Browser Automation Without Screenshots or a Vision Model
Browser automation can run without screenshots when the agent receives a structured representation of the rendered page. A text-capable model can use roles, visible content, relationships and actionable references to navigate and operate many web interfaces.
For the category overview, see the structured perception approach.
Replace pixels with structured page state
A screenshot gives a model pixels. Text-only browser automation needs another representation that preserves the information required for action: which text is visible, which controls are interactive, how elements relate and what changed after an action.
E2LLM captures pages into SiFR, a structured browser representation designed for model use. The agent can capture the current page, query relevant details, inspect a target and act through explicit browser tools.
This approach works well for document pages, forms, dashboards and navigation whose meaning is available in rendered structure. Visual design review, charts without accessible data, canvas applications and image interpretation may still need screenshots or another visual input.
Use a model with tool calling
A browser agent without a vision model still needs to select tools and produce valid arguments. The host presents E2LLM's MCP tools to the text-capable model and returns structured results to the loop.
Verify the complete path with list_tabs and sifr_capture. Ask the model to locate a visible control, perform a reversible action and capture the result. This establishes that the text-only model can understand the representation and operate the browser.
Build for on-prem deployment
Text-only LLM automation on prem can combine a locally deployed model or approved model endpoint with browser tooling that fits the required network boundary. Treat the components separately:
- The model host controls inference placement.
- The MCP host manages tool connections and the agent loop.
- The browser connector exposes authorized page state and actions.
- The workflow system owns schedules, approvals, records and alerts.
Review the actual data path of every component. A model running on premises does not make a remote tool service local. Choose deployment and service boundaries that satisfy the organization's requirements.
Keep observations bounded
Structured state can still be large. Capture once for the current page, query focused sections and recapture after navigation or material changes. Remove superseded browser payloads from the active model context while retaining task goals, verified facts and authorization boundaries.
This makes browser automation without screenshots useful for both context efficiency and deployment flexibility. The goal is a representation proportionate to the decision, with enough evidence for a correct action.
Questions about text-first deployment
Can a browser agent work without image input?
Use a tool-calling text model with structured browser state and explicit actions.
What does an on-premises text-first stack require?
Keep inference in the approved environment and verify the separate MCP and browser data paths.
Which page representation supports text-only automation?
Provide rendered roles, content, relationships and actionable references in a model-readable structure.
How should an on-premises workflow handle screenshots?
Deploy the model and workflow within the required boundary, then choose browser tooling whose data path meets the same requirement.
Example: operate an internal admin form
An on-prem agent opens a connected internal portal, captures the form and receives labeled fields plus actionable references. It fills approved values, reports the resulting form state and waits for a person to authorize submission.
After approval, the agent submits once and captures the confirmation. No screenshot is required because the workflow depends on field semantics and state changes. If a visual CAPTCHA or canvas-only control appears, the agent routes that exception instead of pretending the structured input covers it.
Common questions
Can every browser task run without vision?
No. Tasks whose evidence is inherently visual require pixels or another visual data source. Use text-first structured perception for interfaces where rendered semantics and interaction state establish the task.
Is text-only automation the same as raw HTML automation?
Raw HTML includes hidden content and application machinery while omitting some rendered relationships. A structured rendered representation is shaped for the agent's navigation and action decisions.
Why use this approach on premises?
It can reduce dependence on remote vision inference and fit environments that already operate local text models. The complete deployment still needs an explicit browser-tool data path and appropriate controls.
To reduce repeated state further, measure and lower browser-agent token use. For visual exceptions, decide when screenshots still add value.