SLICC · benchmark results · updated 8 October 2026

Browser Agent Benchmark: Claude vs. GPT

AI models are getting good at using a browser, but their scores and costs are far apart. On BU Bench V2.1, Browser Use's set of 200 long, multi-step web tasks, GPT-6.1 Sol at high effort scores 60.0 out of 100 for $2.22 per task. Claude Opus 5.5 at max effort scores 53.5 for $5.21. Across the 12 configurations that ran at least 188 of the 200 tasks, scores range from 27.0 to 60.2, and cost from $0.10 to $5.21 per task.

We add new models as they land. Since 29 September this page has added Claude Sonnet 5.5, GPT-6.1 Sol at four effort levels and, on 8 October, runs with a small decision model in the harness. Runs of Claude Haiku 5.5 are under way. The updates list what changed each time.

A small decision model brings the cost down. Some decisions never need to reach the large model: which button on the page to click, or which line of it holds the price. With SLICC's new intent skill, the large model states each browser step in one sentence and a small model on a GPU carries it out. GPT-6.1 Sol at low effort then costs $0.21 per task instead of $0.80, 74% less, and scores 40 instead of 45 on the same 196 tasks. The GPU's cost is not included. How it works and what it costs in score.

Every run on this page ran inside SLICC, an open-source agent that works in your own browser with the model you choose. Run it on your own tasks.

Ranking and score vs cost

Score is the share of each task's rubric that the judge marked as met, averaged over judged runs and scaled to 100. It is not the share of tasks solved. Cost is the average model spend per task at list prices. Runs is the number of judged runs behind a score. The line joins the configurations that no other configuration beats on both score and cost.

Twelve configurations ran 188 to 200 of the tasks. Five ran only a 20-task screening round and are drawn hatched and hollow: treat them as indicative. The two rows with intent ran 3 to 8 days after the runs they are compared with, on a newer SLICC version, and their time per task is not comparable; see the intent section.

A small model for the small decisions

Many of a browser agent's steps are small: click the search box, type a name, read a total. Without intent, the large model reads a snapshot of the page and picks every control itself. The intent skill splits the work. The large model still decides what to do next and says it in one sentence, such as "choose the Medium pizza size". A small vision model, kev-4b-vision (5.4 GB), finds the control or the text that sentence means and acts on it, and the step returns a few lines instead of a page snapshot. When the small model is not sure which control is meant, it does nothing and returns the likely candidates, so the large model can be more specific. Asked to read something, it returns the closest matching text.

On the same tasks, with intent against without:

• GPT-6.1 Sol at low effort: 40 against 45 (-10.3%), 74% cheaper. n = 196.
• Claude Sonnet 5.5 at low effort: 30 against 35 (-13.5%), 41% cheaper. n = 185.
• GPT-6.1 Sol at low effort with intent against Claude Opus 5.5 at low effort: 40.5 against 37.0 (+9.5%), 50% cheaper. n = 190.
• GPT-6.1 Sol at low effort with intent against Claude Sonnet 5.5 at default effort: 41 against 51 (-19.6%), 83% cheaper. n = 183.

The cost saving holds up. The score loss depends on the model, and it is less certain, because the intent runs ran on SLICC 6.238 from 4 to 6 October, 3 to 8 days after the runs they are compared with, and scores drift between days. On 40 of the tasks, three configurations ran with and without intent on the same day, and no difference was clear: each 95% interval includes zero. Adjusted for the drift measured on those 40 tasks, Claude Sonnet 5.5 at low effort loses about 1 point with intent (95% interval -7 to +6 points), which is within noise. GPT-6.1 Sol at low effort loses about 11 points (-20 to -2), more than the 5 points of the raw comparison.

Two more limits. The cost per task covers the large model only, not the GPU that runs the small one. And runs with intent finish sooner, but by less than the table suggests: runs without intent end with a 2-minute settle wait that the intent runs skip. Without that wait, Claude Sonnet 5.5 at low effort is 18% faster with intent and GPT-6.1 Sol at low effort 33% faster.

The intent skill is open source, in ai-ecoverse/skills.

Head-to-head on the same tasks

Twelve switches worth knowing about, each measured on the tasks both configurations ran. The arrow points from the configuration you would leave to the one you would pick; all cards share the same axes. Every paired comparison is listed further down.

How it was run

The task set

BU Bench V2.1 is a set of 200 long, multi-step web tasks, each scored against a weighted rubric. SLICC ran release v2.1.1. The runner decrypts the tasks in memory only, so no task text is stored in SLICC's repository.

The harness

Each task ran on a fresh SLICC instance: SLICC in Chrome on a cloud VM, with a wiped browser profile. The runner drives it from outside with the slicc command-line tool, so the agent does the task the way it would for a user. For every task the runner opens a new session with the agent's memories erased, sets the model and thinking effort, and sends the task text with a fixed instruction: do not ask clarifying questions, and end with a final answer line. It takes screenshots while the agent works and waits for the agent to finish, then records the cost and exports the transcript.

All models ran on AWS Bedrock, and every run used SLICC's built-in skills. The two intent rows also used the intent skill, at commit 9d10326 of ai-ecoverse/skills#473 before it merged, with kev-4b-vision as the small model on a GPU. Cost per task is SLICC's recorded spend for the agent and all its sub-agents, at list prices from pi-mono's published model list, or from models.dev where that list has no price. Time per task is how long the prompt took. From the full runs on, the runner also waits until every sub-agent is idle, plus a 2-minute quiet period, and that wait is part of the time. The screening round and the intent runs ran without it, so times are not fully comparable between rows.

Default effort is the model's default, adaptive thinking. Claude Opus 5.5 on Bedrock cannot turn thinking off, and low effort is the lowest level that works. Max effort is thinking level xhigh combined with effort max. For GPT-6.1 Sol, low, high and max effort are SLICC's thinking levels low, high and max.

Limits per run

• One hour per run, as the benchmark specifies.
• A cost cap of $10 per run. The screening batch with Claude Fable 5.1, GPT-6 Astra and Kimi K3 ran with a $30 cap.
• A run that reaches a limit is stopped and judged on the work it had done.
• The limits were not always enforced. Until a SLICC fix on 27 September, stopping a run did not stop the agent, so some runs in the screening round kept working and spending without being measured. Their recorded costs are probably too low. Some recorded runs go past the limits, up to 4,274 seconds and, for one GPT-6 Astra run, $44.879.

The judge

The judge is GPT-5.6 Luna on Bedrock. When a Luna verdict is still invalid after repair turns, GPT-5.6 Sol judges the run instead. This page does not break down how many runs each judge scored. The judge follows Browser Use's findings method: it marks each rubric item as met, violated or not assessable, and code computes the score as the met weight over the total weight. A run scores 1 when every item is met and 0 when none is. The judge prompt is read from Browser Use's repository at the pinned release, and the agent's own reasoning is never sent to the judge. Both judge models are from OpenAI, the vendor of the GPT-6 models under test.

Caveats

• The table has no confidence intervals. Most configurations ran each task once. Where Claude Opus 5.5 ran the same task three to five times, its best and worst scores on that task were about 31 points apart on average.
• Errored runs are left out of the scores. 86 runs ended in an error and 17 more have no verdict from the judge. None of them count toward a score; the 17 without a verdict still count toward time and cost. Most errors come from the harness, for example an agent that kept working after the stop signal, a transcript export that timed out or a dropped connection. The errors are not spread evenly. 14 of the 191 runs of Claude Opus 5.5 at max effort errored, 12 of them because the agent ran past its limits or kept working after the stop. Counting those errors as a score of 0 moves its mean from 0.535 to 0.495. 45 of the 200 runs of GPT-6.1 Sol at max effort errored, most of them on long tasks.
• Nine of the 200 tasks have runs from GPT-6.1 Sol only: bu2-135, bu2-136, bu2-155, bu2-156, bu2-175, bu2-176, bu2-182, bu2-195 and bu2-196. They were part of the earlier full-run plan, and the cause of the gap is not confirmed. Claude Sonnet 5.5 at both efforts and GPT-6 Luna also have no run for 3 more tasks each, and Claude Opus 5.5 at max effort for 1.
• Results pool several SLICC versions. Releases shipped while the benchmark ran, so the runs span eight versions, from 6.195.0 to 6.220.1, and the intent runs ran on 6.238.2 to 6.238.6, a few days later. The screening round ran before the full runs and before several harness fixes landed, and the default-effort Claude Opus 5.5 row pools runs from both periods.
• Until a SLICC fix on 28 September, any GPT-6 turn that attached a screenshot the way SLICC sent it failed on Bedrock. GPT-6 Sol and GPT-6 Astra ran only before the fix, and so did part of the GPT-6 Luna runs. The affected runs have not been counted, and the GPT-6 scores may be lower than they would be on the fixed build. GPT-6.1 Sol ran after the fix, so the comparison between the two Sol versions also measures the fix.
• On 30 September, Bedrock returned server errors for hours while the GPT-6.1 Sol runs at high and max effort were going. A SLICC runner fix that day records a run whose last turn died on a provider error as an error instead of judging it. The affected tasks were run again on 30 September and 1 October, and runs that still died count as errors. The earlier runs on this page used the runner from before the fix, which judged those runs anyway. A check of their records found that 0 to 3% of runs per configuration died this way. For GPT-6 Luna the share was 5%.
• The judge gets less evidence than in Browser Use's own runner. It receives at most 10 screenshots, where Browser Use sends up to 50. Deliverable files such as a CSV or JSON output are not passed to it as files; it sees their contents only where they appear in the transcript. SLICC also leaves the judge's reasoning effort unset, where Browser Use's runner uses xhigh.
• These are SLICC's own runs, with SLICC's harness, browser setup and judge configuration. Browser Use's published V2 charts use an earlier 60-task set, so their scores are not comparable with the scores on this page.

All head-to-head comparisons

Each comparison pairs two configurations task by task and counts a pair only when both runs were judged; n is the number of pairs. Each covers its own set of tasks, so a score here can differ from the table above. Every comparison with n below 50 involves a configuration from the screening round. The comparisons with intent are in their own section.

Against the older version

• Claude Sonnet 5.5 against Claude Sonnet 5: 43 against 37 (+16.2%), 77% cheaper, 63% faster. n = 14.
• GPT-6.1 Sol against GPT-6 Sol: 38 against 37 (+1.8%), 53% cheaper, 24% faster. n = 15.

Against the model at the other provider

• Claude Fable 5.1 against GPT-6 Astra: 44 against 43 (+2.3%), 38% cheaper, 240% slower. n = 19.
• Claude Opus 5.5 against GPT-6 Sol: 55 against 33 (+69.2%), 61% cheaper, 27% slower. n = 31.
• Claude Opus 5.5 against GPT-6.1 Sol: 52 against 53 (-2.7%), the same cost to within 1%, 69% slower. n = 188.
• Claude Sonnet 5.5 against GPT-6.1 Sol: 51 against 53 (-4.5%), 19% cheaper, 67% slower. n = 182.
• GPT-6.1 Sol at high effort against Claude Opus 5.5 at max effort: 60 against 55 (+8.7%), 61% cheaper, 45% faster. n = 168.

One tier up at the same provider

• Claude Opus 5.5 against Claude Sonnet 5.5: 52 against 51 (+2.1%), 19% more expensive, 1% slower. n = 182.
• Claude Fable 5.1 against Claude Opus 5.5: 46 against 58 (-20.1%), 349% more expensive, 29% slower. n = 18.
• Claude Opus 5.5 against Claude Sonnet 5: 56 against 37 (+48.7%), 77% cheaper, 52% faster. n = 14.
• GPT-6 Astra against GPT-6 Sol: 40 against 36 (+9.9%), 142% more expensive, 48% faster. n = 16.

Thinking effort, against the same model at its default

• Claude Opus 5.5 at low effort: 37 against 52 (-29.1%), 72% cheaper, 61% faster. n = 188.
• Claude Opus 5.5 at max effort: 54 against 52 (+2.8%), 292% more expensive, 93% slower. n = 174.
• Claude Sonnet 5.5 at low effort: 34 against 51 (-32.2%), 86% cheaper, 76% faster. n = 180.
• GPT-6.1 Sol at low effort: 45 against 53 (-15.2%), 46% cheaper, 13% faster. n = 195.
• GPT-6.1 Sol at high effort: 60 against 53 (+12.4%), 64% more expensive, 85% slower. n = 192.
• GPT-6.1 Sol at max effort: 60 against 52 (+15.9%), 246% more expensive, 150% slower. n = 154.

Attribution and data

The tasks, rubrics and judging method are from BU Bench V2.1 by Browser Use: github.com/browser-use/benchmark, release v2.1.1, commit af6c7f7, dataset SHA-256 0c014b056192e261b06f04ed6da6dcf37f2135150cf58c87ea839bd4e150e5c3. Browser Use did not run, review or endorse these results. At Browser Use's request, this page reproduces no task text, rubric or judge prompt.

The scores are SLICC's own measurements, taken from SLICC's benchmark dashboard of 29 September 2026 and, for GPT-6.1 Sol, from the reports of its benchmark runs of 30 September and 1 October 2026. The intent rows and comparisons come from the run report of 8 October 2026, recomputed from the intent runs of 4 to 6 October and the earlier runs they are compared with. The BU Bench V2.1 run records behind them are not public at the time of writing. The public Hugging Face dataset ai-ecoverse/slicc-bench holds earlier runs on BU Bench V1 only and is not the source of these numbers. The benchmark runner is open source, in the packages/bench folder of the SLICC repository.