Pi, agent harnesses, and how to benchmark them

What a coding-agent harness is, why Mario Zechner built the minimal Pi harness, and why every coding benchmark score is really a model + harness pair.

6 min readlearning

TL;DR: A coding agent is a model inside a harness. Mario Zechner built Pi, a deliberately tiny harness with four tools, because bigger harnesses kept quietly changing what the model sees. And when you try to rank harnesses, you find that every benchmark score is really a model + harness pair. You can't grade the kitchen without also tasting the chef.

1. What even is a harness?

A model on its own is a brilliant chef locked in an empty room. It can tell you how to make pasta, but it can't boil water. The harness is the kitchen we build around it:

the loopsend the request, see what the model wants to do, do it, send back the result, and repeat
the toolsthe hands: read a file, write a file, run a command
the contextwhat the model sees on every turn: system prompt, tool descriptions, history

Claude Code, Codex, OpenCode, Amp, and Cursor's agent are all harnesses.

2. What was going wrong

01The kitchen keeps movingMario found that as Claude Code added features, it started behaving unpredictably. The knives kept changing drawers.
02Hidden context surgeryIn OpenCode he found context pruning that "lobotomizes" the model, and LSP errors injected after every line of an edit.
03Token bloatPlaywright MCP costs about 13.7k tokens of tool descriptions. A short README the agent reads through bash costs about 200–400.
04One size fits nobodyDifferent jobs need different harnesses. The same hammer isn't right for every construction job.
So what's the root cause?

The harness makes lots of decisions for you, invisibly, and those decisions change what the model sees. More features means more hidden text, and you can't tell why the model got confused.

3. So how would you fix it?

If the hidden stuff is the disease, what's the cure?
Have less stuff, and make every addition visible and chosen by you.

But won't a stripped-down agent be dumb?
No. Frontier models have been heavily RL-trained on coding, so they already know what a coding agent is. The chef already knows how to cook and doesn't need a 50-page manual.

So what's the smallest set of tools that still does everything?
Read, write, edit… and bash. Bash can run any program, so every other capability can be a tiny CLI script.

Enter Pi

A minimal terminal coding harness. It has a small, stable core, and every extra capability is something you explicitly add.

4. The four tools

Click each one:

What it leaves out, and how you get it back

Left outThe Pi way
sub-agentsspawn another pi session through bash and read its output
plan modeask Pi to build it, or install a Pi package
big MCP serverssmall CLI tools plus a README the agent reads when needed
huge system promptskills: only a one-line description sits in context, and the full instructions load on demand

Pi can also modify itself, so you can grow a custom harness for a specific job. Mario sees that as a preview of self-modifying software.

layers you can reuse separately
pi-ai          // one API across LLM providers, with token + cost tracking
pi-agent-core  // the loop: run tools, validate, stream events
pi-tui         // terminal UI with differential rendering

// or embed it as an SDK
const { session } = await createAgentSession({ ... });
await session.prompt("What files are in this directory?");

5. So which harness is best?

Easy: give each one the same tasks and count wins?
But which model is running inside each one? A harness never works alone. Terminal-Bench says it outright: it grades the model and the harness together.

So how do benchmarks untangle the chef from the kitchen?
Hold one of them fixed:

6. How much does the kitchen matter?

Sometimes a lot:

same model, same harness, one setting changed

Shortening old tool results as the context fills up. Complete SWE-bench Verified solutions under a tight context window:

full history
43 / 169
shortened history
72 / 169
Harness-Bench: same models, different harnesses
NanoBot (best)
76.2
OpenClaw (worst)
52.4

That's a 23.8-point gap between harnesses.

…and sometimes barely at all. In one bug-fixing study, Claude Code and GitHub Copilot running the same model showed no significant difference, but changing the model did matter.

7. The benchmarks worth knowing

Terminal-Bench 4.0, official leaderboard (as of late Sep 2026)

GPT-6 Astra · Codex
58.18%
Fable 5 · Claude Code
44.55%
Gemini 3.8 Flash
19.09%

Notice a pattern? The top entries run in the model maker's own harness. Labs train their models while using their own harness, so the chef is used to that kitchen.

76.4%Meta-Harness on TB 2.0: a harness found by automated search
2–5turns it saves by snapshotting the environment up front
87.6 → 19.7one model's score when TB 4.0 re-ranked everyone
Before trusting any leaderboard
  • Every score is a model + harness pair. Read the harness first.
  • Scores from different versions (2.0 / 2.1 / 4.0) aren't comparable.
  • Vendor-run comparisons tend to put the vendor's own agent on top.
  • Benchmark-tuned models can crash on a harder refresh ("benchmaxxing").

8. So what should I actually do?

  1. Pick the model first.
  2. Start with its native harness (Claude Code for Claude, Codex for GPT).
  3. Reach for a minimal harness like Pi when you want control, no hidden behavior, or model-agnostic setups. I found no clean head-to-head score for Pi against the big harnesses, so don't assume it wins or loses.
  4. The benchmark that matters most is your own repo: take five tasks you've already done and run them through two harnesses.

Sources

Pi

Benchmarks and leaderboards

Research on harness effects

Written up from a Claude session, Sep 26, 2026.

✦