Strategies in Accessibility Testing

Developing and applying a good testing strategy is essential for delivering accessibility.

User profiles

Start by creating good user profiles. This will serve as a user-centered guide and roadmap for designers and testers. This will help identify the experiences of the entire range of users who may have diverse needs. And by testing with these profiles in mind, testers can more accurately identify any barriers to access for these users.

Taking responsibility for the user experience

We always start from the bedrock understanding that we deliver the user interface and are responsible for the success of that user experience. That might seem obvious, but it is easy to find accessibility advice that starts by focusing on users' disabilities. Why is this difference significant? The first perspective assumes that users, all our users, are simply users and that it is on us to design and deliver an interface that meets their needs. The other focuses on how the user can or cannot use our interface: as if the interface is given, and it is a user's condition that prevents them from using it as we imagined. It's a subtle but essential difference in responsibility.

When resources are limited and deliverables have to be prioritized, it can be easy to start talking about "them" and "they" who cannot use our interface. Continually refocus the conversation on "us" and "we," who have not provided the most suitable interface for all our users.

It's not the user's fault your interface failed

The social model of disability asserts that people with disabilities are not inherently disabled, but instead, they are disabled by the way society is structured and by the barriers that prevent them from fully participating in society. These barriers can take many forms, such as physical, attitudinal, and systemic. The social model of disability views disability as a social construct rather than a medical condition and focuses on changes that society can make to eliminate the barriers which prevent specific types of people from fully participating in that society.

The implication of this change in perspective is profound: if we deliver good user interfaces, we remove any disability people may experience when accessing our content.

"I don't consider myself 'disabled.' I'm lucky enough to have technology that allows me to access everything I need."
Molly Watt, a Deafblind UK member, Accessibility and Usability Consultant ^[1]

What about people with...?

For all of these reasons, we discourage discussing medical conditions when discussing user profiles for accessibility. Unless we intend to deliver medical treatment to users, it's none of our concern what diagnosis they may have, what undiagnosed condition they may have, and what condition they may believe they have or may be experiencing temporarily. As designers, we are experts in meeting user wants and needs, regardless of the reasons for those needs, so this is where we start.

Accessibility and users' needs

Since our day-to-day work is focused on understanding and meeting users' needs, it makes sense to think about the types of needs disabled users may experience.

Needs text

Users who access the web using assistive technologies such as screen readers or Braille displays rely on text content. Providing alternative text for images, graphics, and other visual elements on a webpage helps to remove barriers for these users.

In addition to assistive technologies, alternative text is also helpful for users with limited connectivity. In those cases, images or other visual elements may not load properly, and alternative text can help these users understand what content is missing.

Needs flexibility

Assume and embrace the idea that users may override your interface to suit their needs as they access the content. Providing a robust and flexible interface means it can be accessed in a variety of different ways. If the content is designed to be viewed or listened to, for example, ensure that it can also be read. If an interaction is designed to time out in 5 seconds, ensure there is a way that a user can take longer if they need to. If the navigation uses mouse events and animations, ensure that users can still use it without those things.

As another example, consider the use of color to communicate information, and ask how this content could be accessed without color. This specific example will benefit a range of users:

1. Users who have difficulty distinguishing between specific colors, such as red and green. Web content that relies only on color to convey information may be inaccessible to these users.
2. Users who use high-contrast color combinations to read and navigate web content.
3. Users with certain visual or cognitive impairments and have difficulty distinguishing between colors or experience visual processing difficulties, making it difficult to interpret complex color schemes.

Needs it to work with assistive technology

Disabled users may employ a wide range of devices and software to access the web. Ensuring that web pages do not exclude those users is crucial. Consider, for example, the following list of common assistive technology:

1. Screen readers read the text on a web page aloud, allowing users who cannot see or who benefit from hearing text as they read it, to access the content.
2. Braille displays use raised pins to display text in Braille, allowing users who primarily consume information as Braille to read web content.
3. Speech recognition software allows users with difficulty typing or using a mouse to navigate the web using voice commands.
4. Magnification software enlarges the text and images on a web page, making it easier for low-vision users to read and interact with the content.
5. Alternative input devices, such as switches, joysticks, and touch screens, can be used by users with physical disabilities to more easily navigate and interact with web content.

Needs clarity

While all users benefit from explicit, easy-to-understand content, some groups have specific needs or preferences. Individuals with reading-related, neuro-cognitive difficulties or non-native speakers of the language may find complex information, lengthy passages, or overly abstract concepts hard to follow. Clear, concise, and straightforward language can help these users understand the content of a web page.

Using plain language, avoiding idioms, jargon, or slang, and providing definitions for technical terms can help make content more accessible to these users.

When designing patterns and interactions, follow user expectations, or as Steve Krug put it: "Don't make me think." ^[2] Innovation and novelty have their place in good design, but some cognitive and neurodivergent conditions mean those users will find unexpected interfaces confusing.

"People don't want interfaces to surprise them; they want things to be intuitive and behave as expected." ^[3]

The problem with pigeonholes

We tend to organize our work using categories, but people are not categories. Asking whether an interface is suitable for a blind user or a deaf user, for example, ignores the existence of an estimated 450,000 people in the UK who are deafblind,^[4] having both a hearing and a sight impairment.

Social, cultural, and economic conditions are also influenced by and contribute to users' experience of disability. For example, data show that 27% of people with disabilities live in poverty, while only 21% of people without disabilities do. ^[5] A person with disabilities who is also managing poverty may be primarily limited by a lack of access to resources: their financial status can make it impossible to get a formal diagnosis which may require them to find, organize, travel to, and pay for medical testing. In turn, without a formal diagnosis, they may be unable to access benefits and services that enable them to work.

Needs are not pillars; they combine, contribute, exacerbate, and overlap with one another. This intersectionality ^[6] of conditions can mean we should consider combinations of needs as a regular matter of practice. Users who need text alternatives may be more likely to need plain language, for example.

Perfect accessibility and other illusions about testing

Ideally, we would aim to test software and find that it achieves perfect accessibility ^[7] and does not exclude anyone ever; unfortunately, the practical reality is that this isn't always realistic. Even with infinite time, knowledge, and resources, different needs will require different approaches, and what works well for one group of users may not work at all for another group.

Artfully balancing conflicting needs is actually a well-understood goal in the field of design, and it is no different when designing for the needs of disabled users. Consider the possible conflicts we may have to contend with:

• A high-contrast color scheme that benefits users with visual impairments may be more difficult for users with some types of cognitive or neurological disabilities, such as dyslexia or Irlen syndrome, ^[8] to manage.
• Choosing a font that improves readability for a particular group may have the opposite effect on other groups.
• Optimising an interface for one combination of assistive technology can make it less accessible to other combinations.

When facing conflicting needs, it is important to take a considered, holistic, and practical approach to accessibility. This may involve a frank assessment of what is possible, prioritizing features and releases to avoid getting ahead of your resources, and ongoing testing and refinement to ensure the website is accessible to the broadest possible audience.

Considered exclusion

If we accept that at least some users may experience some amount of exclusion some of the time, the red line for us then becomes the decision that this exclusion will be delivered through a process that is honest, considered, and deliberate, rather than the result of whatever haphazard shortcuts happened to be taken when delivery deadlines started to loom.

Agree on a plan for rolling out accessibility in a realistic way. This should include the "three amigos" ^[9] of stakeholders representing business, development, and testing. Ideally, at least one of the decision makers will be asked to serve as an "accessibility advocate," having additional training and responsibility to constantly ask, "how will this affect our mission to deliver effective accessibility?"

1. Identify the most critical accessibility issues: It's vital to understand and acknowledge the most common and essential accessibility issues that affect the website's disabled user base. For example, keyboard navigation, screen reader compatibility, and color contrast at a minimum. Having this baseline of needs met will mean it is much easier to address more complex needs later on.
2. Prioritize accessibility issues: Once the most critical accessibility issues have been identified, prioritize their delivery based on their severity and impact on the user experience. This will help you focus on the most pressing problems first and allocate limited resources most effectively.
3. Refer to accessibility guidelines and standards: Established accessibility guidelines and standards, such as the Web Content Accessibility Guidelines (WCAG), are a good way to establish a baseline and a starting point.
4. Make good use of automated accessibility testing tools: Automated accessibility testing tools can never address all (or even most) accessibility testing needs. However, they can help identify many common accessibility issues quickly and efficiently (the lowest-hanging fruit). These tools can help you prioritize and address the most critical issues first.
5. Seek feedback from actually disabled users: Soliciting feedback from users with disabilities can provide valuable insights into the most critical accessibility issues and help ensure that the website is accessible to the broadest possible audience. Be careful, however, to avoid drawing general conclusions from small datasets. Statistically speaking, feedback from a handful of users can only prove what that handful of users thinks. Understand the different uses of qualitative and quantitative data.
6. Document your efforts: Finally, document your accessibility goals, decisions, actions, and the resulting progress. This documentation can be used to demonstrate your commitment to accessibility and to identify areas where further improvements may be needed in the future.

An example testing table

Checkpoints for Testers

Considering the range of needs that users have when accessing a webpage, we generally want to test each of these against the goals of content that should be: perceivable, operable, understandable, and robust.

Check that the web page's core content, purpose, or activity under test is documented and understood: this will be the foundation for any functional testing. Then we must ensure that it is accessible in perceivable, operable, understandable, and robust ways for each of the user-needs groups.

• Ensure it can be perceived by users who need to access the content using assistive technology that interprets the text. Test that all core content is accessible using a screen reader.
• Ensure it can be perceived by users who need flexibility in design or layout. Test using a greyscale filter, test while using magnification and zooming, font replacement, and text enlargement.
• Ensure that it can be operated by users who need flexibility in their access. Test using keyboard-only, and test using voice recognition software.
• Ensure that it can be understood by users who need clarity. Test your content using a tool that calculates a Flesch-Kincaid score, ^[10] measuring your text's reading ease and grade level. Compare to the guidelines published by the Plain English Campaign. ^[11] Abbreviations and
• Ensure that it can be understood by users who rely on assistive technology. Ensure that the language of each webpage is detectable via code
• Ensure that it can be accessed in robust ways. Ensure that all web standards are followed and that web pages use valid markup. Test using a variety of different hardware, different screen sizes and resolutions, input devices, assistive technology, and software. Test with slow and intermittent connections and with older software and devices.

Notes and references

1. Technology and Me by Molly Watt ↩︎back to reference 1

2. Mr. Krug's general premise is that it discourages a user whenever they have to stop to think about something (even for a moment) while trying to accomplish a goal. Ideally, we want a website to be as easy to use as possible; it should feel intuitive and self-explanatory. ↩︎back to reference 2

3. Predictable: Make Web pages appear and operate in predictable ways, mdn web docs ↩︎back to reference 3

4. Deafblindness statistics in the UK, Sense ↩︎back to reference 4

5. Inequalities in poverty, The Health Foundation ↩︎back to reference 5

6. Intersectionality, Wikipedia ↩︎back to reference 6

7. The title of this section is paraphrased from the book, Perfect Software And Other Illusions about Testing By Gerald M. Weinberg, 2008 ↩︎back to reference 7

8. Irlen Syndrome is a light-related visual processing condition. ↩︎back to reference 8

9. Three Amigos Agile Alliance ↩︎back to reference 9

10. Flesch-Kincaid readability tests Wikipedia ↩︎back to reference 10

11. How to write in plain English, Plain English Campaign ↩︎back to reference 11