uef-findings-round15.md

Round 15 UEF Pattern Testing Usability Findings

Results overview from Round 15 of the user feedback sessions

Background

The UEF team conducted usability testing to evaluate specific UEF patterns in the context of a non-linear Pro Template application prototype on desktop devices.

The following patterns were evaluated in Round 15 Testing:

• Password (Enter) – Show/Hide
• Text Area (Rich Text)
• Badges Dismissible
• Drop Down Button (Split)
• PRO Template Collapsible Content Area
• Sub-Navigation (Nested Groups)
• Table (Column Selector)
• Table (Column Reorder)
• Table (Search)
• Back to Top (New Design)
• PRO Template Drop Down Tab

Methodology

The UXG UEF team conducted usability testing with six participants on November 14-15, 2018. The evaluations took place in the UXG labs at SSA. All evaluations were conducted on desktop computers.

Rapid Iterative Testing Evaluation (RITE) was the method of choice during round 15. The team conducted three rounds of evaluations with two participants at a time (six total participants). After each round, the facilitators and the UXG UEF team met for a half hour debrief. During each debrief the team discussed the issues found and potential changes to the patterns that would be shown to the next set of participants.

During each evaluation, participants walked through a series of scenarios that allowed them to look at each of the patterns being tested. The prototype design was a fictitious internal application to review an event and send information to coworkers. In order for the prototype to be iterative, it was built with HTML and CSS using the UXG Genesys Design Tool.

After the evaluations were complete, the team analyzed the results including:

• Facilitator Guide metrics on ease of use;
• Participant issues or comments regarding specific UEF patterns or screen details;
• User satisfaction scores on the overall experience as indicated in the post-test questionnaire.

Participant Information

Six participants were involved in the evaluations. Their demographic information was as follows:

• Participants were between the ages of 39 and 65 with an average age of 55
  • The participant who was 39 may have been an outlier and skewed the average
• Five out of six participants have a degree of higher education (4-year or graduate)
  • Three participants have a grad degree
  • Two participants have a 4-year degree
  • One participant has some college
• Five out of six participants access the internet on a Desktop, Tablet and Smartphone
• All participants have a mySocialSecurity account online
• Five out of six participants use a mobile device for emails, browsing, news, social media and banking
• All participants would use a desktop or laptop to access SSA.gov
  • Two participants would use a tablet
  • One participant would use a smartphone

Lessons Learned

The UXG team discussed lessons learned from this round of testing. Those included the following:

• The team liked the testing methodology used
• This methodology might work better with less patterns
• The debrief meetings after each sessions worked well
• The team thought it was easy to update prototype in this manner
• Using HTML made it easy to make the prototype updates
  • Axure would take more time to make the updates and generate
  • If Axure is used, allot additional time for updates
• This method was easier for facilitators because they don't have to come back for an additional debrief
• Doing the debrief right away made it easy to remember the findings
• Breaking up the wall of sticky notes by the different patterns made it easy to debrief
• Color coding the sticky notes was helpful
• Taking pictures after each debrief session was helpful
• Finding a solution where the team does not have to refresh the browser and clear the cache to get the prototype updates will make it easier (this will only work with HTML prototypes)

Metrics

Metrics for this usability test were established by the UEF Workgroup as follows. Each has a target of 80%.

• Completion Rate – Percentage of participants who successfully completed the application without assistance
• Ease of Use – Percentage of participants who indicated the application was “easy” or “very easy” to use, as measured by Questions #3, #5, and #8 of the post-test survey
• User Satisfaction – Percentage of participants who indicated they were “satisfied” or “very satisfied,” as measured by questions #4 and #7 of the post-test survey

What We Learned

Metrics for task completion, ease of use and user satisfaction, as measured by the post-test questionnaire, were as follows:

Metric	Target	Actual
Completion Rate	>=80%	100%
Ease of Use	>=80%	84%
User Satisfaction	>=80%	81%

Post-Test Questionnaire

The following table lists the Post-Test Questionnaire responses. Scale of 1-5, with 1 = lowest and 5=highest Overall

Questions	n=6
1. How well did the website match your expectations?	3.8
2. How well did the website support the task you were asked to perform?	4.1
3. How difficult or easy was the website to use?	4.3
4. Are you satisfied with the content?	3.8
5. How difficult or easy was it to move through sections of the website?	4.3
6. How easy were the words on the website to understand?	3.8
7. How satisfied are you with the speed at which you can complete tasks?	4.3
8. How difficult or easy was it to find information you needed?	4
9. How long would it take you to learn to use this website?	5
10. How confident did you feel using this application?	4.6
11. Average User Satisfaction Score	3.8

Qualitative Assessment

This section discusses the usability issues, as well as observations and participant comments. The patterns tested within this evaluation group the findings.

Password (Enter) – Show/Hide

1. Four out of six participants had no issues with this pattern.
2. One participant thought the icon was a security feature and thought it would turn green if entered correctly.
3. One participant thought the icon was a way to lookup previously saved passwords.

Participant comments:

“Thought the eye was a security feature. It turns green if your password is correct and red if incorrect. Similar to Verizon passphrase.”

“Not what I expected. This is cool. Very creative. Never would have thought of this.”

“If you had a password in the before [this puts it in], this could be where you actually see the password. Expect to show the password.”

“Look up your password (first response)”

“Pretty standard. Great way to verify. See it on a lot of sites.”

“Excited to see this.”

Text Area (Rich Text)

1. There were no major issues with the functionality of the pattern.
2. Most participants compared the text area to Word and other work software. They were comfortable, but wanted more features.
3. All users looked for the save feature in the editor (this may have been caused by the task scenario).
4. Features participants suggested: 4.1. Attachments 4.2. Print options 4.3. Auto numbering (word count)

Participant comments:

“Editor was difficult at first, slowly got it.”

“Not working well, not blaming the editor yet. I didn’t get the basics.”

“Where are the save buttons?”

“Maybe a font selection?”

“Pretty common. Can spell check.”

“Somewhat bare. No color there. Don’t see a way to emphasize with color. Normal view would have a menu bar.”

“Does the job, but want more features.”

Badges (Dismissible)

1. There were no major issues with this pattern.

Participant comments:

2.1 “Straightforward”

2.2 “Very Easy”

Drop Down Button (Split)

1. Most participants were unable to locate the Save option within the drop down menu.
2. The cognitive connection between the Send and Save actions were too far making it difficult for participants to discover the Save by clicking the arrow.

Participant comments:

“Easy to use. Didn’t have to do it like outlook, don’t have to navigate to it.”

“Not used to that.”

“Not sure what the arrow is.” Clicked it “Ah, didn’t expect that. Never saw this type of save as. For me it’s in a different spot.”

“Distracting. Not where it is positioned. Kind of awkward.” Wanted the save in the task bar.

“Didn’t expect to see this here.”

Sub-Navigation (Nested Groups)

1. There were no major issues with this pattern.

Participant comments:

“Color coding really helps.”

“Thought the navigation was pretty good, but wouldn’t know without a label.”

“Straight forward. As expected.”

“Very easy”

“Easy to read.”

“Easy to navigate. For the average user, it might be challenging. Kinda like Outlook, its habit. Navigation is easy, but people that don’t know Outlook might find it hard.”

“Simple. Liked the layout a lot.”

Table (Column Selector)

1. Most participants had no issues with this pattern.
2. Two out of six participants expected more options in the drop down due to the use of the gear icon.
3. Those that work with programs such as MS Excel seemed to understand the feature better.
4. Most users liked the icon in conjunction with the text.

Participant comments:

“I see ‘columns,’ I bet there are other options there. Something to do with columns adding or removing.”

“As expected. Fast!”

“The gear means there are other features to add/manipulate data.”

“This is very narrow. Can only add/remove. Gear means other features to add for review. Means configure. Would want to see add another column based on my needs.”

“Pretty easy. Similar to excel.”

“It says columns. Good icon for that. Self-explanatory.”

“Initially thought it was an excel type drag and hide on right click. Settings, but it didn’t stand out and catch the eye.”

Table (Column Reordering)

1. There were no major issues with this pattern.
2. Most participants liked the interaction and compared it to Excel.
3. One participant wanted to use the feature to resize the columns.
4. Most participants had no issues with the use of the icon. 4.1 Participants read the text along with the icon. 4.2 Many stated they felt the icon without the text would not be recognizable and suggested a tooltip. 4.3 One participant thought it was a fun icon that “helps you lighten the mood so you don’t feel as bad when you don’t get it.”

Participant comments:

“Feels good!”

“Really cool. Makes you think. Get outside our comfort zone. Makes it easier.”

“Accustomed to that, nothing showed me to do it.”

“Would like it to resize the columns.”

“Easy and Fast.”

“I like it. Don’t do it often. Great feature.”

“The finger is saying push L/R because there are 2 arrows. If words weren’t there. I would experiment if I only saw icons.” “Just like it said, to drag column.”

“Good. Easy to do. Like using Excel, you can manipulate fields and says it at the top.”

“It didn’t look like it will [reorder]. Column separation was not there. Move pointer over it, and the [cursor] didn’t change.”

Table (Search)

1. There were no major issues with this pattern.
2. Participants clicked into the text field first and liked the filtering of search results as they typed.
3. Some participants clicked on the magnifying glass expecting to see results that way.

Participant comments:

“That’s nice. Absolutely, very useful.”

“Great! Loved it!”

“Nice and clean.”

“Usually type on the left and the magnifying glass is on the right. This is the other way. SharePoint does it that way.”

“Didn’t have to click on the icon to search. It started searching immediately.”

“That was easy!” Clicked in text box first. Liked when the results came up without clicking search.

“Easy. ‘X’ to clear the keyboard.” Liked the “x”

Back to Top

1. None of the participants noticed the pattern without prompting. When prompted, they understood and liked the feature.
2. Two participants wanted a “go to bottom” icon at the top.

Participant comments:

“Very good, didn’t have to scroll.”

“What if it started all the way at the top? Would I have noticed it instead of showing at the bottom?”

“No, I didn’t notice the icon.”

“Don’t think I need it based on how fast I can scroll. I would need this on a very long scroll page.”

“Good! Top of page, easy rather than scrolling.”

“Use this a lot on shopping sites.”

The design of Back to Top shown to participants at the time of testing is not fully represented in the screenshot above.

PRO Template Collapsible Panels

1. Most participants needed prompting to locate the icons to collapse the panels.
2. One participant thought the arrows meant to go backwards and forwards in the navigation. When prompted, they preferred to see an “x” instead of the arrows.

Participant comments:

“Wouldn’t have gotten that. Arrows usually means next, like next page. Didn’t think it was to get rid of notifications.”

“So used to seeing ‘>’ as next button. My brain is used to seeing an ‘x’ not an arrow. It was confusing for me.”

“Thought it was good. I [wasn’t expecting to be able to collapse the page, but] I figured it out.”

“Worked perfectly. As expected.”

“Nice.”

“Yes! I thought clicking it would present info or hide.”

“Took a moment to find. Maybe where it says ‘Team’, have drop down that says ‘hide’”. “Average user may not know.”

“Now that I know where it is, it’s fine.”

“Really easy. Used to see this type of action to hide things. Like Eclipse, AOL, ALM.”

PRO Template Drop Down Tab

1. Most participants were willing to click the down arrow for more options since they had already used the Drop Down (Split) button on a previous page.
2. Some participants selected the “Reports” portion of the button without prompting.
3. Most participants did not realize there were split actions for the button.
4. Some participants had issues locating the “Create a Report” option. (This could be due to an issue with the scenario because they were confused by creating and sharing reports).

Participant comments:

“Pretty straight forward. It was really cool. You’re paying attention.”

“Took a guess what that was. Didn’t know what’s there until I clicked it.

“Oh! Okay, didn’t notice the button had two actions. So used to using only the drop down button, I didn’t see it.”

Recommendations and Next Steps

Based on this round of testing, the following patterns were found to be problematic for enough participants to necessitate retesting or design refinements:

• Password (Enter)
• Text Area (Rich Text)
• Drop Down Button (Split)
• Table (Column Selector)
• Back to Top (New Design)
• PRO Template Collapsible Panels
• PRO Template Drop Down Tab

Pattern recommendations based on the findings are below.

Pattern	Recommendation	Rationale
Password (Enter)	Keep design as tested but re-test with a different testing scenario.	Two participants had issues understanding the icon.
Text Area (Rich Text)	Keep design as tested but re-test with a different testing scenario.	Due to the nature of the scenario, all participants looked for a save feature within the editor.
Drop Down Button (Split)	1. Re-test with a different scenario context. 2. Use similar actions in the drop menu to the left button label.	Participants had major issues discovering the save action without prompting.
Table (Column Selector)	Re-test with a different icon.	Two participants expected more options because of the similarity with a “Settings” icon.
Table (Column Reorder)	Continue with the current design.	There were no major issues with this pattern.
Table (Search)	Continue with the current design.	There were no major issues with this pattern.
Back to Top	Keep design as tested, but re-test with a larger user base.	All participants needed prompting to discover and use the pattern. After using the pattern, there were no issues.
PRO Template Collapsible Panels	Retest with different icons.	Several participants needed prompting to discover the collapsible interactions. Suggestions were made to use double arrows or other icons.
PRO Template Drop Down Tab	1. Re-test with a different scenario context. 2. Use similar actions in the drop menu to the left button label.	Similar to Drop Down Button (Split) pattern, this pattern was very problematic. Users did not feel the need to select the “Report” section of the button.