[Bug]: error-context.md Page snapshot includes content excluded by ariaSnapshot()

[joshden]

Version

1.62.1

Steps to reproduce

1. Create a Playwright test with the following content:

import { test } from '@playwright/test';

test('failure Page snapshot vs aria-hidden', async ({ page }) => {
  await page.setContent(`
    <h2>BEFORE HIDDEN ROOT</h2>
    <p>Visible content before the hidden subtree.</p>

    <div id="root" aria-hidden="true">
      <h1>ARIA HIDDEN HEADING</h1>
      <p>ARIA HIDDEN CONTENT</p>
    </div>

    <h2>AFTER HIDDEN ROOT</h2>
    <p>Visible content after the hidden subtree.</p>
  `);

  const ariaSnapshot = await page.locator('body').ariaSnapshot();

  console.log(ariaSnapshot);

  throw new Error('Intentional failure — inspect error-context.md');
});

2. Run the test:

npx playwright test aria-hidden.spec.ts

3. Inspect the generated error-context.md under test-results/.../error-context.md.

Expected behavior

The Page snapshot in error-context.md should reflect the same accessibility semantics as locator.ariaSnapshot().

Specifically:

• Content inside aria-hidden="true" should be excluded (or clearly marked as excluded)
• The snapshot should not present a different accessibility tree than ariaSnapshot()

Given the DOM:

<div id="root" aria-hidden="true">

the subtree:

• ARIA HIDDEN HEADING
• ARIA HIDDEN CONTENT

should not appear in the accessibility-based representation unless explicitly annotated as hidden.

Actual behavior

ariaSnapshot() output (correct)

- heading "BEFORE HIDDEN ROOT" [level=2]
- paragraph: Visible content before the hidden subtree.
- heading "AFTER HIDDEN ROOT" [level=2]
- paragraph: Visible content after the hidden subtree.

The aria-hidden="true" subtree is correctly excluded.

---

error-context.md Page snapshot (inconsistent)

- generic [active] [ref=e1]:
  - heading "BEFORE HIDDEN ROOT" [level=2] [ref=e2]
  - paragraph [ref=e3]: Visible content before the hidden subtree.
  - generic [ref=e4]:
    - heading [level=1] [ref=e5]: ARIA HIDDEN HEADING
    - paragraph [ref=e6]: ARIA HIDDEN CONTENT
  - heading "AFTER HIDDEN ROOT" [level=2] [ref=e7]
  - paragraph [ref=e8]: Visible content after the hidden subtree.

This snapshot includes nodes that are excluded from ariaSnapshot() and from role-based queries.

Additional context

The issue is not that Playwright provides multiple representations of the page: that is expected.

The problem is that error-context.md presents a single “Page snapshot” label while actually embedding a hybrid representation that includes:

• accessibility-visible content
• accessibility-hidden content (aria-hidden="true")

This becomes misleading in debugging scenarios involving:

• getByRole() failures
• accessibility tree mismatches
• modal/dialog cleanup issues (e.g. stale aria-hidden on app root)

In such cases, the failure artifact can suggest that content is present and accessible when it is actually excluded from the accessibility tree used by Playwright locators.

Suggested improvement

Either:

Option A: annotate accessibility-hidden nodes

- generic [accessibility-hidden]:
  - heading "ARIA HIDDEN HEADING"
  - paragraph "ARIA HIDDEN CONTENT"

Option B: split snapshots explicitly

# Accessibility snapshot
...matches ariaSnapshot / role semantics...

# Visual / AI page snapshot
...current representation...

This would preserve the usefulness of the visual snapshot while preventing confusion in accessibility-related failures.

Why this matters

In real-world debugging, this led to incorrect conclusions when:

• a modal left aria-hidden="true" on the application root
• role-based locators fail correctly
• but the failure snapshot still showed the “missing” content

This created a contradiction between:

• Playwright’s locator model (getByRole, ariaSnapshot)
• the failure diagnostic artifact (error-context.md Page snapshot)

which misled both me and Claude Code.

Environment

System:
    OS: Linux 6.6 Ubuntu 24.04.2 LTS 24.04.2 LTS (Noble Numbat)
    CPU: (12) x64 13th Gen Intel(R) Core(TM) i7-1355U
    Memory: 11.64 GB / 15.47 GB
    Container: Yes
  Binaries:
    Node: 24.16.0 - /home/josh/.nvm/versions/node/v24.16.0/bin/node
    npm: 11.13.0 - /home/josh/.nvm/versions/node/v24.16.0/bin/npm
  IDEs:
    VSCode: 1.132.0 - /mnt/c/Program Files/Microsoft VS Code/bin/code
    Claude Code: 2.1.228 - /home/josh/.local/bin/claude
    Codex: 0.47.0 - /usr/bin/codex
  Languages:
    Bash: 5.2.21 - /usr/bin/bash
  npmPackages:
    @playwright/test: ^1.62.1 => 1.62.1