“Zebra batch” component usability research findings Q1 2024
In March through April 2024, we conducted usability testing with eight participants with various disabilities (including visual impairments, motor impairments, ADHD, and anxiety and depression). Our research assessed the usability and accessibility of several U.S. Web Design System (USWDS) components. Participants used a mix of assistive technology, including screen readers and screen magnification software, and we tested on desktop and mobile devices in a variety of browsers.
We found that most components performed well overall for all participant groups. We identified the most opportunities for improvements with the memorable date and time picker components, as well as with error handling.
For the memorable date component, we heard that participants value the ability to avoid and fix errors easily, so participants preferred the drop-down option over the text input for the memorable date “month” field. However, we also noticed evidence that multiple input types (e.g., dropdowns and text inputs) could be problematic. We found unhelpful and verbose instruction announcements for both the memorable date and time picker components. Finally, we observed issues with error handling that have wider implications for the guidance USWDS offers web teams.
The USWDS team regularly tests its components for accessibility and usability with people with disabilities. In this round, we tested what we've named the “Zebra” batch of components:
- Banner
- Site alert
- Memorable date
- Time picker
- Radio button
We also tested disabled states and error handling in our prototype, which is a fictional form for reserving a special activity at a fictional national park.
Assess how these components perform for people with disabilities (visual impairment, motor impairment, cognitive disabilities) so we can make improvements to enhance accessibility and usability.
Can users with disabilities (visual, motor, or cognitive) who may use various assistive technologies (screen reader, screen magnifier, voice command, alternative navigation, etc.) interact with these components effectively and easily?
In particular, are components perceivable, operable, and understandable to users? What, if any, problems might users of assistive technology or those who are not neurotypical face when trying to navigate and interact with the prototype elements?
See more details in research questions specific to each component.
Eight people with various disabilities:
- Five with visual impairments
- Three blind
- Two low vision/legally blind (1 with nystagmus)
- One with motor impairment
- One with ADHD
- One with anxiety and depression
Participants live in New York, Connecticut, Atlanta, North Carolina, Colorado.
- Four participants using only screen reader:
- VoiceOver on iPad, Safari (participant had some vision)
- NVDA, PC laptop with Windows 11, Chrome
- JAWS, PC, Windows 11, Chrome
- JAWS, PC laptop, Edge
- One using only screen magnification:
- Zoom built in for Mac
- Three did not use any assistive technology during the session
Five participants on laptops
- Four using PCs
- One using a Mac
Three participants on mobile devices
- One using an iPad
- One using an Android phone
- One using an iPhone
Most visually impaired participants were born blind or legally blind. One participant’s onset of visual problems started 15 years ago.
Proficiency in using screen readers was high for most participants.
- One intermediate participant with two years of experience
- One proficient participant with 13 years of experience
- Two experts participant with over 20 years of experience
- When items are not clearly marked or labeled. All four screen reader users said this.
- Popups and advertising that you can’t skip or can’t figure out how to exit (two users).
- No alt text for images.
- Inaccessible links within maps.
- Inaccessible PDFs that aren’t coded in HTML.
- Tabs without a descriptive, unique label: “I need their contact information. And there is a link called contact. I create [an] open tab. And when I switch to that tab and I check my title with my keyboard shortcut, it's not saying “contact” and…I don't know where I am.”
- Poor contrast makes content hard to perceive.
- Important things are out of their field of view (zoom magnification), so they miss form fields. Then, they get an error message and have to find and fix the problem.
- When grayed out areas they can see aren’t focusable, and they aren’t sure why (might be referring to disabled states).
- Checkboxes often are hidden or not announced properly by the screen reader.
Poor memory and succeptibility to distraction are their biggest challenges, so sites and form designs that ignore that are frustrating.
- Forms that do not save progress automatically. This was the most critical pain point. Sometimes they’re distracted from the task at hand and need to be able to come back to it.
- Forms that force them to recall things they entered on previous pages cause problems.
- **Multi-step forms on multiple pages stop them from referencing information they entered above.**They prefer to have everything on one page so they can easily refer back to previous information entered.
- Forms that do not help them avoid or easily correct errors. They often make errors on forms due to impulsivity and rushing through.
- Sites or forms that force them to enter information in a specific timeframe.
- Forms that don’t tell them the information they’ll need up front.
- Sites or forms with links that take them away from that page. They prefer tooltips so they don’t lose their place.
They weren’t sure if their anxiety or depression affects their experiences or interactions with websites, but they mentioned some challenges:
- Forms or sites that do not help them avoid or correct errors easily were their top frustration. They often skim instead of carefully reading. They remembered recently being stressed when filling out an unemployment benefits form, which led to them making mistakes. The form should make those mistakes obvious right away.
- Emails that do not clearly indicate what they need to do. They said there’s a “blindness to all the garbage we receive in our email” and that they don’t read emails carefully. If something is important, make it obvious.
Overall, first impressions of the prototype were positive. Participants said it seems clean, simple, and well organized, and they praised the heading structure.
| Component | Major friction experienced | Minor friction experienced | Good performance | Needs more research |
|---|---|---|---|---|
| Banner | ✔️ | |||
| Information site alert | ✔️ | |||
| Emergency site alert 1 | ✔️ | |||
| Emergency site alert 2 | ✔️ | |||
| Memorable date 1 | ✔️ | |||
| Memorable date 2 | ✔️ | |||
| Time picker | ✔️ | |||
| Default radio | ✔️ | ✔️ | ||
| Tiled radio | ✔️ | ✔️ | ||
| Error handling | ✔️ | ✔️ | ||
| Disabled state | ✔️ |
| Component | Major friction experienced | Minor friction experienced | Good performance | Needs more research |
|---|---|---|---|---|
| Banner | ✔️ | |||
| Information site alert | ✔️ | ✔️ | ||
| Emergency site alert 1 | ✔️ | |||
| Emergency site alert 2 | ✔️ | |||
| Memorable date 1 | ✔️ | |||
| Memorable date 2 | ✔️ | |||
| Time picker | ✔️ | |||
| Default radio | ✔️ | |||
| Tiled radio | ✔️ | |||
| Error handling | ✔️ | ✔️ | ||
| Disabled state | ✔️ |
Most participants wanted help with entering or choosing correct data and wanted to avoid making mistakes, so they preferred the dropdown in memorable date version 1 (six participants).
That said, some note that having a mix of inputs is problematic (three participants).
- Two low vision participants commented that having a mix of inputs (dropdown and text input) could confuse some users or that they hadn’t encountered that approach before.
- One blind participant used the E command in forms mode to start entering the date because they assumed it would be an edit field, not a dropdown.
- The VA has also found multiple inputs problematic.
Participant behaviors using the dropdown: Visually impaired users tended to type in the field before selecting from the dropdown, while the four sighted users clicked and chose from the dropdown.
- Only two people (both visually impaired) preferred typing over using the dropdown, and one of them (legally blind) said it was good to offer people the option to either type or choose from the dropdown.
- Of the four people who typed in the combo box (either naturally or by being prompted), about half typed in the name and half typed in the number to search for the month.
The helper text announced on memorable date v2 performed much better for screen reader users than the helper text announced on v1. The v2 text was relevant and clear for each field, helping blind screen reader users feel more confident about what format to type in. In v1, hint messaging is overly descriptive and not helpful for screen reader users. It announces the same hint for each field and doesn’t describe the digits or format needed.
- Keep offering the dropdown in memorable date, but make some improvements:
- Only show names of months, but code the component so that a user can search by name or number for month. (#5911), (#5908)
- Improve the hint text to read out field-specific hint labels and digit requirements. (#5902)
- Offer a variant that has three text input fields.
Major pain points for blind screen reader users:
-
The instructions announced when they first interact with the time picker are overly verbose. Participants commented that it's so much information that it's confusing. This even hindered one blind JAWS user from selecting a time. They didn’t realize they had to hit enter to select a time, and the long instructions didn’t help: “It causes confusion with the information and…I have to listen [to] all of those carefully.”
-
To clear the time selection, screen readers announce “clear select contents,” which is unclear. It’d be more helpful if they announced “clear the time” or something similar.
Requiring users to manually expand the time picker instead of auto expanding works fine and is expected behavior for the two participants we asked.
Time picker performed well for most sighted participants.
- Adjust the instructions read by screen readers to be shorter, clearer, and more helpful. (#5905)
- Change the “clear contents” announcement to “clear the time selection” or something similar. (#5905)
Participants had the best experience and corrected errors quickly when they were taken directly to the input that needed correcting and were told what was wrong. Native HTML error handling performed well overall for JAWS and NVDA and in Chrome and Edge browsers. Participants were directed to the input that needed correcting. Screen readers alerted them to what was wrong so they knew how to fix it.
Major pain points: *** Memorable data error handling performed poorly on VoiceOver on mobile.** Sometimes it does not announce what is wrong, and sometimes it only announces “fill out this field” on VoiceOver. VoiceOver users were not taken to the input that needed to be corrected. (See this clip of a session on iPhone with VoiceOver.)🔒
- Memorable data error handling went poorly for Zoom magnification on Mac. The “fill out this field” error only appeared at the input out of their field of view. Zoom magnification prevents automatic scrolling to the field. They had to hunt it down and fix it.
- People didn’t want to be interrupted with error messages when they were in the middle of fixing the error. One NVDA user said it was overzealous in validation — announcing it was invalid after each character typed, which was “annoying.”
- Participants didn’t like that memorable date lets you enter and submit impossible data (like 33 in the ‘Day’ input), which hurts their trust in the site. They expect it to not accept obviously incorrect data, or for it to guide you to fix errors like that and not let you submit them.
Minor friction: The focus indicator was blue on the input after error — it may match expectations better for it to be red with an icon.
- Performed similarly to memorable date. (See this clip of an error announcement while a user tested the time picker using VoiceOver on an iPhone.) 🔒
- Since the error was close to the “submit” button, the zoom magnification user had an easier time locating and fixing the issue.
We should offer robust error handling guidance as well as component error states and variants.
- Conduct a literature review to identify error handling best practices to inform our guidance.
- General error handling guidance should include:
- In addition to inline validation, error messages must appear near the “submit” button and provide helpful information so users can find and fix form errors. This change would help all users, but especially those who use zoom magnification as well as people filling out long forms.
- Give users a chance to correct an error before prematurely announcing their input as incorrect.
- We should offer component styling, previews, and/or variants that show proper error handling best practices for relevant form components.
- For example, the focus indicator should turn red to draw attention to fields that need to be corrected.
Overall, the alerts performed well for all participants.
There’s evidence validating that the emergency site alert should NOT be dismissible.
- One blind person kept exiting the alert message before realizing what it was because they thought it was a popup.
- A low vision participant said she might miss things at the very top of the page since she usually doesn’t go there.
- The person with ADHD said they don’t think you should be able to dismiss it because they impulsively close stuff a lot. They also made a point about considering people with working memory issues who might dismiss an alert and then forget that they saw it.
Blind screen reader users
Informational alert performed well, with minor friction:
- Site alerts (especially emergency alerts) may need to be emphasized with a heading.
-
- One person said it's possible to arrow down and miss the alert, so consider emphasizing it with a heading, star, or some kind of tag.
-
- One person kept exiting the alert message because they thought it was a dialogue box.
-
- Another said they expect an alert to have a heading.
Those with typical AND some sight
- Informational alert performed well overall with minor friction:
-
- One person using zoom magnification said they might miss it since they usually don't go to the very top of the page.
- The positioning of emergency alert 1 (with the banner displayed at the top of the page and the alert displaying under it) seems like the better option overall.
-
- Two people thought v1 stands out more. One person said v2 site alert (where it appears above the banner) looks more like a decoration and is less noticeable.
-
- Another person said v2 makes the banner stand out more, but since we’re more interested in the emergency alert standing out, v1 may be the better option.
-
- One person had no preference between emergency alert v1 or v2.
- Add guidance that we recommend not allowing users to dismiss emergency alerts. We currently do not address dismissibility at all, so we may also have to add guidance for informational alerts.
- For emergency alert, consider adding guidance that recommends: *** To assign a heading to site alerts, especially emergency site alerts.** *** Not allowing users to dismiss emergency site alerts.** *** That the emergency site alert should appear just below the site banner.**
Disabled tiled radio button overall performed well for all participants.
- Screen reader users were all able to perceive that the option was unavailable and why it was unavailable.
- One blind person said she liked that her screen reader announces it rather than keeping it hidden.
- The strikethrough icon that accompanies the mouse was a nice visual aid for sighted users, including a user with low vision.
- All “typically” sighted users easily understood that disabled content was unavailable.
Minor friction:
- Pain point for one NVDA user: They expected that if they tried to select the unavailable option, it would clear all selections. Instead, it kept their previous selection (coyote option) but there wasn’t enough feedback to tell them so. It wasn’t clear that they hadn’t selected “Ride an ATV” and that the coyote option was selected.
- Instead, the screen reader should announce, “You need to select an available option” and clear all selections.
- One legally blind user said the grayed out text was a little hard to read but that bolding the font might help.
- One “typical sighted” person was not sure why an option was disabled and didn’t read the helper text.
Disabled submit button overall performed well (tested with five of eight participants)
- No issues for most sighted users
- Minor friction:
-
- The NVDA user said it’s clear why an option was disabled but suggested putting the disabled button in the tab index so they could still tab to it.
-
- One JAWS user couldn’t tab to the disabled button in forms mode, and therefore they couldn’t perceive the button. Once they exited forms mode, they could find the button and understand why it was disabled.
Move forward with publishing disabled states guidance.
Overall, radio buttons performed well with minor friction.
- One blind screen reader user was annoyed that you can never uncheck all the radio buttons.
Three users said they preferred the tiled radio buttons:
- The touch target seems bigger, like tapping a button (low vision zoom user).
- The tiled buttons look more organized, especially when there are a lot of options (two participants). They look less cluttered, and less formal than the default radio buttons (ADHD participant).
- Investigate use cases for default versus tiled radio buttons and update guidance accordingly.
-
- Explain that tiled radio buttons may be a good option if you want the touch target to appear bigger.
-
- Consider tiled radio buttons for longer lists of options, so they appear more organized.
Banner performed well for three out of four visually impaired participants. They easily found and interacted with it, and they did not find anything confusing. Friction point: One blind participant was navigating by regions and by using the down arrow. Their screen reader would only read “Here’s how you know” and not the contents around it, which was confusing.
Performed well for all three “typically” sighted participants. They easily found and interacted with the banner. The participant with ADHD really liked that the banner did not take them away from the page when they expanded and collapsed it.
Consider solutions to make the banner announcement more useful for screen reader users in forms mode. For example, make more text part of the interactive button because forms mode only takes users to interactive elements.
Prioritize improvements for: Memorable date
- Improve combo box search functionality to return results for both month name or month number.
- Remove the numbers from the drop-down options, and only display month names.Still allow users to search using month name OR number.
- Improve the relevance and usefulness of screen reader hint announcements for each input field. Time picker
- Improve the screen reader instructions announcement to be shorter and more helpful.
- Make the “clear contents” screen reader announcement clearer by making it more specific (e.g., “clear time”) Error handling:
- Conduct a literature review to learn more about error handling best practices.
- Add robust error handling guidance; consider error handling as a new pattern.
- Provide error states styling for component variants and provide previews for relevant form components. Disabled states:
- Continue with publishing disabled states guidance.
Nice-to-have improvements: Banner
- Consider solutions to make the banner announcement more useful for screen reader users in forms mode. For example, include more of the text in the button dropdown. Site alert
- Add guidance about whether site alerts should be dismissible and in what circumstances.
- Further investigate the order that informational site alerts are announced by screen readers and add relevant guidance about if and how to announce alerts.
- Add guidance advocating for the banner to appear at the top of the page, above site alerts. Radio buttons
- Investigate use cases for default versus tiled radio buttons and update guidance accordingly.
| Issue | Recommendation | Github issue |
|---|---|---|
| Memorable date: Dropdown options containing month name and number can be confusing for screen reader users. | Remove the numbers from the options and only display the month names. Still allow users to search using month name OR number. | #5911 |
| Memorable date: Dropdown search functionality doesn't meet some users' expectations. | Update combo box search functionality to allow users to search by either the month name or month number. | #5910 |
| Memorable date: Hint messaging announced to screen readers is unclear and overly descriptive. | Change the hint message announced to screen readers to be relevant to each input and clear about digit requirements. | #5902 |
| Time picker: Verbose and unclear screen reader announcements make selecting and clearing a time difficult for some users. | Adjust the instructions read by screen readers to be shorter, clearer, and more helpful. Change the “clear contents” announcement to “clear the time selection” or something similar. | #5905 |
| Error handling: USWDS should offer more robust and comprehensive guidance for error handling. | Conduct a literature review to identify error handling best practices that will inform our guidance. | #2663 Related issue: #2476 |
| Error handling: USWDS should offer component styling, previews, and/or variants that show proper error handling best practices for relevant form components. | Create a storybook preview for various components using the usa-form-group--error. | #5839 |
| Site alert: Some users could miss site alerts for a variety of reasons (see details above). | Add guidance that helps ensure that site alerts are noticeable to users. | #2664 |
| Radio button: Some users preferred tiled over default radio buttons. | Investigate use cases for default versus tiled radio buttons and update guidance accordingly. | #2665 |
| Banner: Some screen reader setups may not announce the full banner. | Consider solutions that would guarantee the entire banner is announced by screen readers in a multitude of tech setups and in JAWS forms mode. | #5926 |