Same model, same pass rate, twice the cost: measure what your agent harness sends on every call

A coding agent is two choices: the model, and the harness around it, which is the program that decides what tools exist, what the system prompt says, and what gets sent back to the model on each step. Price lists cover the first choice. A measurement published on 16 September by UC Berkeley's Sky Lab puts a number on the second.

HarnessTax, by Melissa Z. Pan, Shuo Yang, Negar Arabzadeh, Ion Stoica and Matei Zaharia of UC Berkeley with Wei-Lin Chiang of Arena, ran seven models through three harnesses: Claude Code, Codex CLI and Pi, a minimal MIT-licensed agent with about 106,000 GitHub stars. Each of the 21 model and harness pairs got the same 30 randomly sampled tasks from SWE-bench Lite and from Terminal-Bench 2.0, three attempts per task, the harness's own configuration at its high effort setting, a cap of 100 turns, and costs computed from one direct-API price list dated 1 September 2026.

Claude Fable 5 solved 97.8% of SWE-bench Lite attempts in Claude Code, 96.7% in Codex and 96.7% in Pi. An attempt cost $1.33 in Claude Code and $0.67 in Pi, and took 15.3 turns in the first and 15.4 in the second. Across the models they shared, Claude Code cost about 2.0 times Pi and 1.6 times Codex on SWE-bench Lite and about 1.5 times Pi on Terminal-Bench 2.0, while the average effect of the harness on success rate stayed within ±2% on the first benchmark and about ±5% on the second. The authors' summary is that the same model can reach similar success rates "at up to 5x costs."

The same number of steps at twice the price means each step cost twice as much, and the size of a step is something you can read off your own runs.

Turns come from the task, the payload comes from the harness

The input cost of an attempt is the number of model calls multiplied by what each call carries. In the study the first factor barely moved between harnesses, although each harness counts a turn its own way. The second factor starts with a fixed block that goes out on every call before any work: the harness's instructions and a definition for every tool it offers.

The chart data published with the study gives that block for all 630 SWE-bench Lite attempts per harness, measured at the first model call. The token column includes the task prompt; the character columns do not.

Harness (version tested)	Tools offered	Instruction characters	Tool-definition characters	First-call input tokens
Pi 0.85.1	4	2,547	2,873	1,972
Codex CLI 0.146.0	7.4 on average (3 to 9, by model)	23,502	18,114	11,308
Claude Code 2.1.224	23	13,465	76,995	27,011

Codex has the longer instructions of the two vendor harnesses. Claude Code's tool definitions are more than four times the size of Codex's and make up 85% of its instruction and tool characters. And the block is resent with every call, so under prompt caching it is billed as a cache read on most calls and as a cache write whenever the cache is cold. The study flags the same dependency: the extra context "can raise costs, though total spending also depends on caching, generated tokens, and later calls."

Read the number from your own setup

claude -p with --output-format json reports the token usage of the run. Ask for a one-word answer and nearly all of the context is what was loaded before your prompt:

claude -p "Reply with the single word: ok" --output-format json \
  | jq '.usage | .input_tokens + .cache_creation_input_tokens + .cache_read_input_tokens'

Run it from the repository you work in. A -p session loads the same context an interactive session does, including CLAUDE.md, MCP servers, skills and plugins. On the machine this post was checked on, Claude Code 2.1.274 on Opus 5 returned 30,271 tokens from an empty directory.

To separate harness from configuration, remove the configuration with three flags, then remove the built-in tools:

# no MCP servers, no skills or commands, project settings only
claude -p "Reply with the single word: ok" --output-format json \
  --strict-mcp-config --disable-slash-commands --setting-sources project \
  | jq '.usage | .input_tokens + .cache_creation_input_tokens + .cache_read_input_tokens'

# add --tools "Bash,Read,Edit,Write" for four tools, or --tools "" for none

Run	First-call context (tokens)
Full configuration	30,271
Configuration removed	18,809
Configuration removed, four tools	5,295
Configuration removed, no tools	2,441

On this machine 11,462 tokens came from configuration and 16,368 from the default built-in tool definitions, which is 87% of the run with configuration removed. That lines up with the 85% character share in the study, reached by a different route. Configuration has its own levers, from an oversized instruction file to MCP servers and unfiltered tool output, covered in Cut the tokens Claude Code spends before it reads your prompt. The built-in tool definitions are the part those levers leave alone.

For a task with several steps, read each call separately. The stream-json format emits assistant messages as events; this filter keeps the main conversation, drops repeated message IDs, and prints the context each call carried:

claude -p "your task here" --output-format stream-json --verbose \
  | jq -r 'select(.type == "assistant" and .parent_tool_use_id == null)
      | [.message.id, (.message.usage | .input_tokens + .cache_creation_input_tokens + .cache_read_input_tokens)]
      | @tsv' \
  | awk '!seen[$1]++ {print $2}'

Two cautions from testing it. The per-event output_tokens values are not final and did not add up to the run's total, so take output from the closing result line. And use the stream rather than the transcript files under ~/.claude/projects, whose format Anthropic documents as internal and liable to change on any release.

On Codex, codex exec --json closes each turn with a turn.completed event whose usage object carries input_tokens and cached_input_tokens for the whole turn, which gives the total for a run but not the size of each call.

One fix, six attempts, two tool lists

To see the payload in a bill, one small task was run six times on Claude Code 2.1.274 with Opus 5 and configuration removed as above: a three-line Python function with a percentage bug, one failing test, and the instruction to fix it and run pytest. Three attempts used the default tools and three used --tools "Bash,Read,Edit,Write".

	Default tools	Four tools
Attempts passing	3 of 3	3 of 3
Model calls per attempt	3	3
First-call context (tokens)	19,051 to 19,071	5,567 to 5,587
Context across the attempt	58,026 to 58,411	17,572 to 17,718
Cost per attempt	$0.060 to $0.064	$0.078, then $0.035 and $0.037

The work itself was small. Reading the file, making the edit and running the tests grew the context by about 750 tokens between the first call and the last; the rest of each call was the block the harness sent three times. The first four-tool attempt was the most expensive of the six because its prompt prefix was new, so it wrote 6,109 tokens to the cache at the one-hour write price, against about 1,700 for the two attempts after it. With the cache warm, the four-tool attempts cost about 42% less than the default ones. Over the 15 turns Fable 5 averaged on SWE-bench Lite, 13,500 fewer tokens per call is about 200,000 fewer input tokens per attempt.

The dollar figures are Claude Code's own, which it computes from token counts at list price. On a Pro or Max plan that figure does not bill you, and the same tokens draw down your usage windows instead. The cache lifetime also differs by account: an hour on a subscription, five minutes by default on an API key, which changes how often the cold-cache price returns. The prompt cache post covers when that happens.

One small task shows the mechanism. It does not show that four tools are enough for your work.

Scripted runs pay for tools they never call

A claude -p job in CI, a cron script, or a batch over many files usually does not open worktrees, schedule tasks or send push notifications, and it pays for the definitions of those tools on every call. Claude Code documents three ways to shrink what such a job sends:

• --bare skips hooks, skills, plugins, MCP servers, auto memory and CLAUDE.md discovery, and gives Claude the Bash, file read and file edit tools. Anthropic's headless docs call it the recommended mode for scripted and SDK calls and say it will become the default for -p in a future release. It never reads OAuth credentials or the keychain, so it needs ANTHROPIC_API_KEY or an apiKeyHelper in place of a subscription login. It was not measured for this post because the test machine signs in with a subscription.
• --tools "Bash,Read,Edit,Write" restricts the built-in tools and works with a subscription login. It does not touch MCP tools; --disallowedTools "mcp__*" removes those.
• --exclude-dynamic-system-prompt-sections moves the working directory, environment details and git-repo flag out of the system prompt and into the first user message, so the same job running on different machines can reuse one cached prefix.

Interactive work is a different case. The default set includes subagents, web search and fetch, and scheduled tasks, and an exploratory session may need exactly those. The study's authors do not argue for a minimal harness everywhere: "Richer harness features may still benefit other models, workloads, or interaction settings."

Test the pairing on your own tasks

Two limits come with the study. It covers two public benchmarks, and in the authors' words, "These findings can be limited to the two open-source benchmarks we test, which the models may have encountered during training." It is also a self-published write-up without peer review, with disclosed support: Arena sponsored the API access and the Laude Institute provided Anthropic API credits.

What it does support is testing the pairing rather than assuming it. Across the six Anthropic and OpenAI models and both benchmarks, a harness other than the vendor's own produced the highest observed success rate in nine of twelve comparisons. Sonnet 4.6 solved 68.9% of SWE-bench Lite attempts in Codex against 66.7% in Claude Code at a similar cost. GPT-5.6 Sol reached 83.3% on Terminal-Bench 2.0 in Pi against 78.9% in Codex, at $0.42 per attempt against $0.76.

The version of that test worth running is small. Take ten tasks your team runs repeatedly that end in a pass or fail check, usually a test command. Run each three times under each configuration you are weighing: default tools against a trimmed list, or one harness against another. Add up the cost of the runs (total_cost_usd in Claude Code's JSON output), count the passes, and divide the first by the second. The study reported cost per attempt beside success rate; cost per solved task folds the two into one figure for your own code, and it tells you whether the larger payload is buying passes or only tokens.

The first number to collect is the one-liner above, run in the directory where your most frequent scripted job starts. Every call that job makes with the same flags carries at least that much.