Your usability tests can’t see the users who already left

What I found

I was the lead product designer on Amazon’s Employee Document Management platform for more than two years. EDM handles 4.8 billion documents a year in 67 countries. Candidates use it to sign offer letters, alumni use it to request records, and pretty much everyone in between uses it for something. When I joined, the team had already done solid research: heuristic evaluations, usability tests, SUS scores, and journey maps from pre-hire through alumni.

But in my own interviews, I kept hearing something none of that research had picked up. Most of the employees with disabilities I talked to weren’t using the document portal at all. They downloaded PDFs to their computers and opened them in their own assistive tools, because our default viewer didn’t work with screen readers. The people who did stick with the portal had memorized where the menus were and what order to click things in.

For those employees, the experience we’d designed basically didn’t exist. And it had been that way for a long time without anyone noticing.

Why the research didn’t catch it

The existing research wasn’t bad. It measured how well the product worked for the people using it, which is exactly what it was set up to do. The trouble is that people who’ve given up on a product aren’t in that group. If your methods start from the product, you’re not going to find them.

It gets worse, because a workaround can look fine in the data. If someone downloads a document and reads it in their own screen reader, that still logs as “document accessed.” In our metrics, the task looked done.

Standard testing won’t find the people who stopped using your product. You have to go looking for them.

Pay attention to workarounds

The main thing I’d ask other teams to do is log workarounds as real findings. When someone exports, downloads, prints, or asks a coworker to do a task for them, there’s usually a reason, and often it’s that the product didn’t work for them. It’s easy to write that off as personal preference. A lot of the time, it’s someone adapting.

In interviews, that just means asking a couple more questions. Along with “how do you do this in the tool?”, ask “what do you do instead?” and “when do you skip the tool completely?”

The fix was pretty simple

Once we understood what was going on, the solution wasn’t complicated. I worked with engineering to evaluate the Mozilla PDF reader and build it in, which gave us full screen reader support without much operational overhead. Documents that needed signatures were harder. For those, I had to get legal and finance aligned on a separate approach.

After launch, I went back to the same participants. The reaction was stronger than I expected. People were genuinely moved that someone had thought about them at all. We also saw something we hadn’t planned for: application abandonment dropped after the viewer redesign. And the platform met accessibility compliance for the first time.

“We can’t research every population”

I hear this a lot, and it’s a fair point. Research budgets are limited. But you don’t need a separate accessibility study. You need to include people with disabilities in the studies you’re already running. That costs a lot less than finding the problem years later, which is what happened to us.

It also paid off later. When we added an AI layer to EDM’s employee portal, with plain-language summaries of legal agreements and a mobile briefing card for new hires, the accessibility work was the foundation for it. Fixing the product for the people we’d missed ended up making it better for everyone else, too.

Where to start

• Add participants with disabilities to the recruiting criteria for your next study. Don’t save them for a follow-up.
• Ask “what do you do instead?” in every interview, and write down the workarounds you hear.
• Look for workarounds in your data too: downloads, exports, and people dropping off at the same step over and over.
• When you fix something, go back to the same participants. It builds trust, and it’s the best way to find out if the fix worked.