Validating user flow issues with Mobile permissions updates
- background & Challenge -
The most currently used Android tablet for our products is the Samsung Galaxy TA3 which is used primarily by drivers to access different Zonar apps such as ZLogs, Coach and EVIR. The increase in laws around data privacy has resulted in more messaging to the drivers on how their data is used and permission requests to gather or use their data. Unfortunately, Android 10 doesn’t allow us to auto grant all the permissions needed for the apps when it is first opened up. Some of the Zonar apps require numerous permissions granted in order to work. The worst-case scenario - if this process was too disruptive to the users, they would opt out and not be able to utilize the full functionality of the app to do their work. It was imperative to test this possible issue before the Mobile Platform went to beta and devise a proactive solution if it was confirmed.
- define -
In early discussions with the Mobile Platform team, my colleague and I noticed that two goals began to emerge for what we wanted to learn:
Research Questions:
· How will the current Android 10 flow impact how users grant permissions?
· How does the volume of permission dialogs impact the user’s acceptance of Zonar apps?
Research Goal: Uncover driver behavior to these new changes in the technology.
It was important to learn how drivers would react to these new changes since any negative reactions could compromise the team’s upcoming beta and future use of the apps themselves.
- research participants -
We were looking for a specific group of users for this experiment:
- Current TA2 users
- Tier level: 4A (customers who have fleets smaller than fifty vehicles).
- Vertical: Pupil & Vocational (PAX) and Freight (FRT)
- Role: Drivers who use our mobile apps such as: Coach, EVIR, ZAccess and ZLogs.
- Number: The goal was to obtain 10 - 20 participants for the entire study.
- Incentives: Originally, the amount was $35 per participant. After a lack of sign ups, we increased the incentive to $65 per participant. The tool Tremendous was used to distribute the incentives after the participants finished the study.
One of the biggest challenges we faced was participant sourcing – the earlier participant leads with CS and PM had not panned out for enticing any drivers to join the study. We then pivoted to expand the participant criteria for GTC users who had prior driving experience. Using Pendo, I also created a guide that advertised the study with an increased incentive; we were able to successfully sign up five participants.
- methodology -
My colleague and I first met with the MoPlat team to get context around the problem, the current flow and alignment on the goals. One of the constraints was time, we had a couple of months to run and finish the research before MoPlat’s beta started, so I devised a plan where we all worked in tandem. My colleague and I would work on crafting the study plan and materials needed for the experiment while the Customer Service Manager and PM volunteered to handle the participant sourcing side for candidates. Their plan was to reach out to customers who had provided excellent product feedback in the past or were interested in participating. I created a Slack channel so we could all keep in contact and it was a place for questions and updates as well. I also created a Confluence page that housed all of the project and research information as well.
For the research side, there were a couple of methods we contemplated using. One method was to conduct user interviews with the drivers. This was discounted because we felt it didn’t provide enough detail and it needed to be paired with a more situational method to get true reactions. Contextual inquiry was another method we considered to see how drivers would react the current flow. This was also rejected due to logistical challenges for remote set up during the pandemic. (Physical on-site research has been scaled back due to safety concerns about COVID- 19). I created a study plan using the methods below:
Part I:
- Moderated Usability Testing: This method was utilized because I wanted to learn and where in the current permissions flow a user would experience friction or get stuck. I wrote out the script which depicted a storyline scenario where a driver would experience the permission prompts with four different tasks that are a part of a daily workflow. My colleague then created a prototype in Maze. We ended up doing five 30-minute moderated sessions via Microsoft Teams. After the usability test, we gave the participants a Survey Monkey link to finish up with part two.
Part II:
- Desirability Study: Since the team was concerned about how the users felt about the current process, I thought this was the perfect method for discovering a user’s attitudes and emotional responses to the flow. For this study, I created it in Survey Monkey and showed a grid list of 65 emotions for users to select which one resonated with them about the process.
- Survey: Since this is a complicated flow, we wanted to get a sense of the usability of the whole process so a SUS questionnaire was added to the Survey Monkey survey.
- analysis -
For this project, my colleague and I split up the work for this area too. She gathered the quantitative usability testing metrics from Maze and Survey Monkey. I worked more on the qualitative side by doing affinity mapping and pulling insights from the findings. I like doing affinity mapping because it gives you a zoomed-out view of the qualitative data and it’s easier to detect patterns.
Microsoft Teams creates transcripts of the interviews. I gathered up all of the transcripts and started tagging different areas (issues, motivations, trends, goals, tasks, quotes, emotions, facts, needs) with different colors. Once I completed tagging, I grabbed all of the tags and copied them into a Miro board for affinity mapping. I first divided out a section for each task to see if any patterns were forming in the tags. I then began to see overarching themes develop and notated them at the top. From the themes I try to pull out the deeper context for insights and write out recommendations for solutions. After I completed my side of the analysis, I synced with my colleague to discuss our work. To showcase the research, we created a power point deck to share the findings with the Mobile Platform team in the Slack channel.
- Learnings -
Overall results
We set out to validate a couple of ideas:
Validate: When presented with a dialog prompting users to grant permissions, users do grant them successfully without locking up or calling customer support?
Correct, this exercise yielded high task completion rates and low error rates.
Validate: The total volume of permission dialogs is not disruptive enough to reduce user acceptance of Zonar apps.
Correct. 60% of respondents disagreed that it would be disruptive enough to reduce user acceptance. The 40% who agreed had an immediate, concrete plan for training.
Emergent patterns:
o Good: Most participants were aware of their employee’s strengths and weaknesses when it came to technology.
o Bad: Confusion on how to deny permissions caused some uncertainty on how to move forward.
sentiment results
insights
Strategic Insight: 75% of the participants we spoke with already have developed ways to help and educate their less tech savvy employees. We can focus on reaching out to the “broadcasters” in the future to help with onboarding future features or apps.
Stakeholder Insight: Mobile Platform is okay to advance to beta.
Product Insight:
o 80% of the participants (who fall in the younger driver demographic) felt that the process was intuitive and straightforward for themselves, although 40% of participants had concerns that the technology may be challenging for older drivers.
o The older demographic tends to be more resistant to change. However, the more familiar and comfortable they become with a new change, the more readily they adopt it as a normal part of their work.
o For the technically adept, the permissions process was seen as a normal software maintenance pattern but technical wording on the prompts could confuse less tech savvy users.
impact
Insight into the driver’s acceptance of the permissions modals provided direction to allow Mobile Platform to advance to beta.
- next steps & reflections -
Next Steps
The following action items will be worked on prior to beta:
Reflections
o An additional challenge we had during participant sourcing was a customer who was enthusiastic about participating, but we when it came time for scheduling they would disappear. Every time the Customer Service Manager reached out to them, they would confirm they were still interested but then fail to connect with us. We ended up stretching out the testing window another week waiting for them due to the earlier difficulties for sourcing. However, extending the testing timeframe time took away for analysis and both my colleague and I were juggling multiple projects. Next time, I plan on having more defined testing period and a longer time buffer for analysis.