Your Accessibility Score Is Lying to Your Face

Why do we keep treating accessibility like it's an exam you pass? We've spent years building tools that measure what they can measure automatically, and meanwhile we assume that's good enough. Something is fundamentally broken in how this industry thinks about this.

Three weeks ago I ran Lighthouse and axe-core against juanchi.dev. 98/100. Green. Beautiful. I felt good about myself for exactly four days.

Then I asked Martín — a friend who uses NVDA every single day to navigate the web — to give me real feedback on the site. He sent me an 8-minute audio file. I didn't make it to minute three without wanting to slam the laptop shut.

Real web accessibility: what Lighthouse can't see

Lighthouse and axe-core are brilliant tools. I'm not coming after them. The problem is they detect what's automatically verifiable: color contrast, present alt attributes, form labels, heading order. That covers roughly 30–40% of real accessibility problems.

The other 60% requires a human.

That's not just my opinion. That's what WebAIM says in their manual versus automated auditing studies. Automatic tools can't tell you whether your aria-label makes sense in context, whether your keyboard navigation flow is confusing, or whether your dynamic status announcements are landing at the right moment.

What Martín found on juanchi.dev in under 10 minutes:

1. The navigation menu was announcing its state wrong. I had aria-expanded set correctly, but the button label never changed. NVDA was reading "open menu" even when the menu was already open.
2. The skip links were there… but they didn't actually work. The "skip to content" link existed, it passed automatic validation, but after the skip the focus landed on the container instead of the first interactive element. Result: pressing Tab immediately after the skip threw you right back to the header.
3. CSS animations were toggleable via prefers-reduced-motion — that part I had right — but there was a project carousel that auto-rotated content every 5 seconds, and NVDA was reading that as constant noise.
4. Inline SVG icons had aria-hidden="true" like you're supposed to — but in one specific case, the icon was the only visual indicator of an error state. For a screen reader user: completely invisible.

None of those four problems showed up in the axe report. The score stayed at 98/100 with all of them present.

The code that was failing me (and how I fixed it)

Let's start with the menu. This was my original component:

// ❌ Broken version — still passes Lighthouse
function NavMenu() {
  const [isOpen, setIsOpen] = useState(false);

  return (
    <nav>
      <button
        aria-expanded={isOpen}
        aria-controls="nav-list"
        onClick={() => setIsOpen(!isOpen)}
      >
        {/* Icon changes visually, but the label doesn't */}
        <MenuIcon />
      </button>

      <ul id="nav-list" hidden={!isOpen}>
        {/* items */}
      </ul>
    </nav>
  );
}

The aria-expanded was there. axe checked it, it passed. But NVDA was reading "button" when you focused on it — no context at all. The label was the SVG icon with aria-hidden. Technically correct by the automated rules. In practice: useless.

// ✅ Version that actually works
function NavMenu() {
  const [isOpen, setIsOpen] = useState(false);

  return (
    <nav aria-label="Main navigation">
      <button
        aria-expanded={isOpen}
        aria-controls="nav-list"
        // Label changes with state — the reader announces it
        aria-label={isOpen ? "Close navigation menu" : "Open navigation menu"}
        onClick={() => setIsOpen(!isOpen)}
      >
        {/* Icon is purely decorative now */}
        <MenuIcon aria-hidden="true" />
      </button>

      <ul
        id="nav-list"
        hidden={!isOpen}
        // Role helps contextualize the list when announced
        role="list"
      >
        {/* items */}
      </ul>
    </nav>
  );
}

The skip link was more interesting. The problem wasn't the link itself — it was the target:

// ❌ Focus goes to the div, which isn't usefully focusable
<a href="#main-content" className="skip-link">
  Skip to main content
</a>

<div id="main-content">
  <h1>Title</h1>
  <p>First paragraph...</p>
</div>

When Tab reaches the skip link, you activate it, focus goes to #main-content. But a div doesn't retain focus in a way that makes the next Tab continue inside the div — that's browser-dependent. In some cases, the next Tab jumped back to the top of the document.

// ✅ tabIndex="-1" allows programmatic focus
// without adding it to the natural tab order
<a href="#main-content" className="skip-link">
  Skip to main content
</a>

<main
  id="main-content"
  tabIndex={-1} // Key: allows programmatic focus
  // No outline on focus because the user didn't arrive here via Tab directly
  className="focus:outline-none"
>
  <h1>Title</h1>
  <p>First paragraph...</p>
</main>

The carousel was the easiest fix but the most important one conceptually:

// ❌ Autoplay with no control — a nightmare for screen readers
function ProjectCarousel({ projects }) {
  const [current, setCurrent] = useState(0);

  useEffect(() => {
    // Changes every 5 seconds, interrupting reading
    const timer = setInterval(() => {
      setCurrent(prev => (prev + 1) % projects.length);
    }, 5000);
    return () => clearInterval(timer);
  }, []);

  return <div>{projects[current]}</div>;
}

// ✅ Respects user preferences and offers control
function ProjectCarousel({ projects }) {
  const [current, setCurrent] = useState(0);
  const [isPaused, setIsPaused] = useState(false);
  
  // Detect if the user prefers reduced motion
  const prefersReducedMotion = useMediaQuery(
    "(prefers-reduced-motion: reduce)"
  );

  useEffect(() => {
    // No autoplay if the user asked for it or if paused
    if (prefersReducedMotion || isPaused) return;

    const timer = setInterval(() => {
      setCurrent(prev => (prev + 1) % projects.length);
    }, 5000);
    return () => clearInterval(timer);
  }, [isPaused, prefersReducedMotion]);

  return (
    // aria-live="polite" announces changes without interrupting
    <div
      aria-live="polite"
      aria-label={`Project ${current + 1} of ${projects.length}`}
    >
      {/* Visible pause control — not just for AT */}
      <button
        onClick={() => setIsPaused(!isPaused)}
        aria-label={isPaused ? "Resume rotation" : "Pause rotation"}
      >
        {isPaused ? "▶" : "⏸"}
      </button>
      
      {projects[current]}
    </div>
  );
}

The most common errors that scores don't catch

After talking with Martín and doing the manual audit, I mapped the patterns that show up constantly in projects with perfect scores.

aria-label that makes no sense outside of visual context. You have a button with aria-label="See more". Visually it's obvious from context what "more" means. For a screen reader user listing all the buttons on the page: there are five buttons that say "See more" and none of them are distinguishable. Fix: aria-label="See more React projects", aria-label="See more about this client".

Focus management in modals and dialogs. You open a modal. Focus stays on the button that opened it. The keyboard user has to tab through the entire document to reach the modal content. Or worse: they can tab outside the modal while it's open. This is one of the hardest ARIA patterns to implement correctly and no automated tool validates it completely.

Dynamic status notifications with bad timing. You use aria-live to announce that a form was submitted. But the announcement fires while NVDA is still reading the text of the submit button. The user hears two things simultaneously and understands neither.

Decorative images with empty alt... correct. But inline SVGs without a defined role. That was my exact case. <img alt=""> passes validation. An inline <svg> that's purely decorative needs an explicit aria-hidden="true", and if it carries text or communicates information, it needs a title or aria-label. axe doesn't always catch the wrong case.

I ran into something similar when I built the HAProxy extension for VS Code — the configuration panel had tooltips that were only visible on hover, with no keyboard equivalent. Passed all automated validation. Completely inaccessible.

How to build an accessibility strategy that isn't theater

I'm not telling you to throw away your scores. They work as a first line of defense. What I'm saying is they're the floor, not the ceiling.

My current setup after this whole episode:

Level 1 — Automated (in CI): axe-core via jest-axe on critical components, Lighthouse on main pages. If anything drops below 95, the build fails.

Level 2 — Manual on a regular cadence: Once per sprint, I navigate all new features using only the keyboard. No mouse, no trackpad. Tab, Shift+Tab, Enter, Space, arrow keys. If I can't complete a flow with just the keyboard in a reasonable amount of time, that's a bug.

Level 3 — Real screen reader: I spin up NVDA in a Windows VM (it's free, VoiceOver on Mac works too but there are meaningful differences) and navigate the site. At minimum once per major release.

Level 4 — Real users: Hard to scale, but it's the only level with no blind spots. Martín gave me more actionable feedback in 8 minutes than three hours of automated analysis.

It's the same principle I apply to vibe coding versus stress coding — automated tools give you speed, but when it actually matters you need real human validation. An LLM can generate accessible components with the right ARIA patterns, but it can't tell you whether the experience as a whole makes sense.

If you're curious about the infrastructure side of how I automate these validations, in the post about how Linux executes a binary I talk about why understanding the layers beneath your favorite abstraction changes how you debug — same principle here: Lighthouse is an abstraction layer over WCAG, and WCAG is an abstraction layer over real human experience.

FAQ: real web accessibility

What percentage of accessibility problems does Lighthouse catch automatically? Studies from WebAIM and Deque (the folks who make axe) agree that automated tools catch between 30% and 40% of real problems. The rest requires human evaluation. The percentage varies depending on the type of site and the complexity of the interactions.

What's the difference between axe and Lighthouse for accessibility testing? Lighthouse uses axe-core under the hood for its accessibility audits, so on many points they're measuring the same things. The practical difference: axe-core integrated into your tests (via jest-axe or cypress-axe) gives you feedback during development and can test individual component states. Lighthouse analyzes the fully rendered page and gives you a global score. Use both: axe in unit/integration tests, Lighthouse as a whole-page check.

Is WCAG 2.1 AA enough or do I need to target AAA? For most commercial web projects, WCAG 2.1 AA is the right target. AAA includes criteria that are sometimes impossible to meet without sacrificing functionality (for example, the AAA contrast requirement for normal text is 7:1, which severely limits your color palette). Aim for AA as a legal and best-practice minimum. Implement AAA criteria where it's reasonable without extra friction.

Does Next.js have specific accessibility advantages? Yeah, a few concrete ones. The <Link> component handles announcing page changes to screen readers better than a vanilla SPA. The App Directory router with Server Components reduces the JavaScript needed on the client, which can improve performance on low-cost devices — a real factor for users with assistive technology. And next/image with its required alt attribute prevents one of the most common mistakes. But none of those advantages save you from errors in your own components.

How do I test accessibility with NVDA if I only have a Mac? Three options. First: VoiceOver on Mac (Cmd+F5) — not identical to NVDA but covers most cases. Second: a Windows VM with free NVDA, which is what I do for important validations. Third: BrowserStack, which includes real screen reader testing in some plans. For critical projects, the Windows VM is worth the setup time.

Are SEO and accessibility related? More than most people realize. Google's crawlers are effectively non-visual users — they process the DOM in a way that's similar to how a screen reader does. Alt text on images, semantic heading hierarchy, descriptive links instead of "click here", meaningful HTML structure: all of that benefits both SEO and accessibility. They're not the same thing, but the best practices overlap heavily. When I did the AI codebase analysis of juanchi.dev, one of the patterns that came up was exactly that overlap between semantic structure and indexing performance.

What I learned and didn't expect

The 98/100 score wasn't a lie. It was incomplete. There's an important difference.

Lighthouse and axe measure what they can measure. They're honest within their limits. The problem is we use them as if they were complete when they're partial. That's our mistake, not the tools'.

What hit me hardest about Martín's audio wasn't finding the bugs — that was expected and fixable. It was realizing that I had published the site feeling good about its accessibility. I had checked the box. And that box didn't represent anyone's real experience.

In tech we love reducing complex things to numbers. Uptime at 99.9%. Performance score 100/100. Accessibility 98/100. Numbers are useful as proxies. But when a proxy becomes the goal itself, we lose sight of what the number was originally supposed to represent.

I still need to run the same accessibility review on client projects. I already know what I'm going to find, and I already know it's going to be work. But now I also know the work is worth it — and that the green score in CI doesn't excuse me from doing it.

If you have a high-scoring project that you've never tested with a real screen reader: I'm throwing down the challenge. Open NVDA or VoiceOver, close your eyes, and navigate your own product. Ten minutes. What you find will change how you think about this forever.