Browser agent

In short

A browser agent is an AI system that carries out a task in a web browser. It reads the page as a screenshot or as structured elements, then clicks, types or scrolls, and uses the updated page to decide what to do next.

Also called: Web agent, Web-browsing agent, AI browser agent
Topic: AI agents
Level: Beginner
First described: Shi, Karpathy, Fan, Hernandez and Liang introduced World of Bits in 2017, a platform where an agent completes tasks on websites by reading the page's pixels and document structure and sending mouse and keyboard actions⁴.

WebArena models a web agent as a system that is given a goal in ordinary language, looks at the current page, and chooses an action. The next page, and the history of earlier actions and observations, become the evidence for the step after that¹. A page can be shown as a screenshot, as HTML, or as an accessibility tree, which keeps the structured elements of the page in a shorter form. The actions include click, type, scroll and going to a URL¹. BrowserGym offers the same kinds of input, the document object model, the accessibility tree and a screenshot, and actions such as clicking an element or clicking at a coordinate².

That is one slice of a wider set of interface agents. BrowserGym's introduction treats a web browser as the environment, and notes that other work covers mobile screens and desktop screens as well². The tasks are meant to be longer than a search box. WebArena told its annotators that a goal should not be something one or two actions can finish¹.

The published results are modest. In WebArena, the best GPT-4 agent had an end-to-end success rate of 14.41%, against human performance of 78.24%¹. BrowserGym's authors say robust web agents remain difficult because real websites are complex and current models are limited². OpenAI's Operator system card adds a different failure. Instructions on a third-party page can pull the model away from the user's goal. The card's safeguard is to ask the user before an action that changes the world, such as a purchase or an email³.

Operator's examples of browser tasks are ordering groceries, booking a reservation and buying tickets, under the user's direction³. BrowserGym's reason for studying this style of agent is compatibility. Acting through the page can avoid building a separate API for every site².

Example

WebArena's published intents include finding the best-rated ergonomic chair and comparing walking and driving times between two places. The tasks are written so that one or two actions are not enough¹.

Why it matters

A browser agent can work on sites that have no API built for it, which is the compatibility BrowserGym describes². Because it acts on a live page, Operator's system card asks the user to confirm a purchase or an email, and warns that the page itself can contain misleading instructions³.

Questions people ask

What can a browser agent do?

In WebArena it can click, type, scroll, change tabs and open a URL, using a screenshot, the page's HTML or an accessibility tree¹. OpenAI's Operator system card says the model should ask before completing a purchase or sending an email³.

Is a browser agent the same as a search engine?

No. WebArena's goals are written so that one or two actions are not enough, and its information-seeking tasks still require moving through pages¹. Operator's system card describes browser tasks such as a grocery order or a reservation, carried out under the user's direction³.

Can browser agents be trusted with a live website?

Treat the results as unreliable. WebArena's best GPT-4 agent succeeded on 14.41% of tasks, against human performance of 78.24%¹. Operator's system card says a page can contain instructions that mislead the model, and that the user should confirm a purchase or an email³.

Related terms

Tools that use it

Sources

Back to the glossary