User Testing Process
Planning
• Recruit users within the DMV area
• Ask if they would be interested to interact with our prototype
• Meet with the users remotely or in person
• FaceTime
• School
• Home
• Test Materials
• Figma App
• Mobile/Smartphone
Deliver Task
• Explain what will happen during the session
• Tap on the blue buttons
• Go home, swipe, chat, or browse map
• Start recording the prototype on the screen.
User Performs Task
• Observe the user’s actions and behaviors closely while recording the touchpoints on the screen.
Metrics
• Standard Usability Scale Questions
• I think that I would like to use this app frequently.
• I found the app unnecessarily complex.
• I thought the app was easy to use.
• I think that I would need the support of a technical person to be able to use this app.
• I found the various features in this app were well integrated.
• I thought there was too much inconsistency in this app.
• I would imagine that most people would learn to use this app very quickly.
• I found the app very awkward to use.
• I felt very confident using the app.
• I needed to learn a lot of things before I could get going with this app.
Post-Test Questionnaire
• Can you tell us about any frustrations or issues you encountered while using the app?
• Do you have any suggestions for how we could improve the prototype's design or functionality?
• Would you recommend this app to others? Why or why not?
Access to our user testing videos: https://drive.google.com/drive/folders/11Ueo8cxy9jbKy1HUkIIlJjAi7V2TTigx?usp=sharing
Analysis
Based on the given data, we can calculate the following descriptive statistics:
• Sample size: 16
• Gender: 11 deaf women (68.8%) and 5 deaf men (31.2%)
• Primary Language: 16 American Sign Language (100%)
• Mean Age: 26.9 (σ = 4.08, p = 0.3549)
In our recent user testing of the low-fidelity prototype, we monitored SUS to 16 participants and found that the average score was 69.8. This indicates that our prototype performed better than average on the usability scale, with a majority of users likely to recommend it to their friends and family. We used the the System Usability Scale (SUS).
The System Usability Scale (SUS) is widely used to measure the perceived usability of a product or system. It is a quick, effective, and reliable tool for assessing the aspects of effectiveness, efficiency, and overall ease of use. Sauro (2011) conducted a study of 500 participants and found that the average SUS score is 68. Scores above 68 are considered above average, while scores below 68 are considered below average.
In our recent user testing of the prototype, we administered SUS to 16 participants and found that the average score was 69.8. This indicates that our prototype performed better than average on the usability scale, with a majority of users likely to recommend it to their friends and family. We use the following response format for SUS:
We asked participants (n = 16) about their usability experience with this app, so we can gain a better understanding into what aspects of the app work better for them and what areas may need improvement.
The majority of participants (n = 16; mean score of 3.588) rated the app as "strongly disagree" to "strongly agree," according to the data. An aggregate rating of 3.588 suggests that users have a slightly positive opinion of the app, with a score of 3 being seen as neutral.
The majority of the participants (n = 16) regarded the app to be easily accessible and without experiencing any significant difficulties, as indicated by the mean score of 4.000.
The participant's response implies that they thought the app lacked consistency, which could have a bad effect on the user experience. The majority of participants (n = 16) disagreed somewhat with the consistency of the app, according to the mean score of 2.412, but the opinion was not extremely unfavorable.
We asked participants (n = 16) to answer our question about whether they would recommend this app to others. 81.8% percent of participants said "yes," while only 18.2% were unsure.
Additionally, prior research has demonstrated a strong correlation between SUS scores and users' propensity to recommend a good or service. The prototype received a score of 69.8, which shows that the majority of users found it to be usable and effective and that they would be more inclined to recommend it to others.
Participant Quotes
“I had to play around a little bit to understand that there was one different feature in the ‘new love’ vs the ‘new network’ and ‘ new friend’.” The ‘new love’ had a like- dislike feature, while the other two had an opportunity to add friends.”
They would like to see a navigation menu bar where they can switch forth and back between dating, friendship, or networking. This concept is similar to Bumble.
“The messaging UI could be tidied up a bit, especially with the video. It should be displayed in full screen immediately when users open it. Maybe users can choose either landscape or portrait to view a video. I think it would be a great idea to implement an instant video recording feature in the messaging UI!”
This would be included in our high profile prototype.
“The concept is amazing. I like the idea but need more information and other features”
They will better understand the available features when they view the prototype in high profile.
“The app is very confusing for me and I don't know how to use some of the features. I keep seeing blue light as I click other buttons.”
The blue light appears when the icon touched doesn't lead anywhere. If they touch the right icon, blue lights won’t appear.
“This app is nice and a good start. I think some designs need to improve in better ways and keep somethings in simple way”
This is a low profile prototype. It can’t perfectly show our designs. We will be sure to keep it simple when creating a high profile prototype.
Future Directions
Despite our average SUS score, users experienced some issues such as missing information or broken links, indicating a need for further refinement of our prototype in terms of user interface design, navigation, and layout. For example, they noticed the speaking icon, which should be replaced with a signing icon.
