Accessibility Tree vs Screenshot for Browser Agents
An accessibility tree gives a browser agent semantic roles, names and hierarchy. A screenshot gives pixels, visual layout and appearance. Each omits information the other can carry, so the right choice depends on the task and the page.
For the category overview, see the full structured-perception category.
What an accessibility tree provides
Accessibility data can expose headings, links, buttons, fields, labels and document structure in a compact form. It is useful for locating controls by meaning and for pages whose semantics are implemented correctly.
It can be incomplete or misleading when the application uses custom widgets, canvas content, hidden duplicates or incorrect accessibility attributes. It also does not fully describe visual occlusion, z-order or pixel-level state.
What a screenshot provides
A screenshot shows the rendered result. It captures charts, canvases, visual alerts, relative placement and content that lacks useful semantics.
The model must infer controls and text from pixels. Small text, dense pages and repeated captures can consume substantial context. A screenshot also does not directly establish whether an apparent element is interactive or how it maps to browser structure.
Add structured browser perception
Structured perception can combine rendered content, interaction targets and layout relationships in a form an agent can query. E2LLM's SiFR snapshots give the model a structured view of the connected page, while focused exploration retrieves more detail around a target.
This is useful when an accessibility tree misses rendered elements or when a screenshot contains the answer but leaves the action target ambiguous. The agent can observe the structured state, choose an explicit browser action and capture again to verify the result.
Choose by failure mode
Use the accessibility tree when semantic navigation is strong and compact structure is the priority. Use screenshots when visual appearance is itself the evidence. Use structured perception when the task needs both page meaning and reliable action context.
Test representative pages rather than a simple demo. Include a dialog, a custom control, a dynamic update and an element that is visually present but difficult to locate semantically.
A practical comparison
- Give each approach the same browser task.
- Measure whether it identifies the correct target.
- Record context used before the first correct action.
- Change the page state and test recovery.
- Verify the final result independently.
The best input is the one that lets the agent complete and verify the real task consistently.
Example: diagnose a hidden control
Imagine a settings page where a save button appears visually but the agent cannot find it in the accessibility tree. A screenshot confirms that the button is rendered, but it does not establish whether an overlay blocks it or which browser node should receive the click.
A structured capture can expose the button's relationship to the dialog, its interaction state and nearby labels. The agent can then act on a grounded target and recapture the page to verify the saved state.
Common questions
Is the accessibility tree the same as the DOM?
No. Browsers derive accessibility information from DOM structure, attributes, styles and platform accessibility rules. Nodes may be renamed, combined or omitted. The result is an accessibility representation rather than a copy of the DOM.
Can screenshots handle every visual application?
They can expose pixels from canvases and visual controls, but reliable interaction may still require coordinate grounding, state tracking and post-action verification. Test dense and dynamic screens rather than assuming visual presence guarantees action precision.
Should an agent receive both on every step?
That can add unnecessary context. Start with the representation best suited to the workflow, retrieve more detail when the state is ambiguous and measure whether the extra input improves verified completion.
For a text-only workflow, work without screenshot-heavy observation. When semantics omit the target, diagnose elements missing from the accessibility tree.