Agent performance depends on the harness: the system that manages context, state, tools, verification, and completion.
This response builds on PY’s video, Rethinking AI Agents: The Rise of Harness Engineering. The research behind the idea points to a practical conclusion: teams can improve agent performance by improving the system around the model.
[ 01 / THE HARNESS ]
Behavior is a system property.
Two agents using the same model can perform differently because their harnesses assemble different context, expose different tools, preserve different state, and apply different completion checks.
The harness owns the conditions around each model call. It defines the task contract, permission scope, token and time budgets, required artifacts, retry policy, and evidence for completion.
Select the information required for the current decision.
Preserve plans, artifacts, progress, and failures outside the context window.
Limit actions through explicit permissions and defined interfaces.
Connect completion to tests, parsers, policies, and acceptance criteria.
[ 02 / THE EVIDENCE ]
Structure has a cost.
The Natural-Language Agent Harnesses paper evaluated multiple harness configurations with the same underlying goal. On SWE-bench Verified, a full configuration scored 74.4 while using 16.3 million prompt tokens and 642.6 tool calls. A lighter configuration scored 75.2 with 1.2 million prompt tokens and 51.1 tool calls.
The same paper reported a more productive intervention on OSWorld. Migrating an existing harness to an explicit, artifact-backed representation raised performance from 30.4 to 47.2, reduced runtime from 361.5 to 140.8 minutes, and cut model calls from roughly 1,200 to 34.
75.2 SCORE · 1.2M TOKENS · 51.1 TOOL CALLS
30.4 → 47.2 PERFORMANCE
361.5 → 140.8 MINUTES
EXPLICIT STRUCTURE · LESS WASTED WORK
Meta-Harness extends the idea by optimizing harness code from execution traces. It improved online text classification by 7.7 points with four times fewer context tokens and found a harness that improved math reasoning by 4.7 points across five held-out models.
[ 03 / THE DESIGN RULE ]
Earn the complexity.
Harness components should improve the acceptance condition, reduce cost, or make failures easier to diagnose. Extra planners, reviewers, workers, and repair loops need evidence from real traces.
Anthropic’s guidance reaches a similar conclusion: start with simple, composable patterns and add complexity when the task requires it.
- Write a contract.Name the artifact, budget, permissions, and completion condition.
- Externalize state.Store plans, progress, and outputs in durable, inspectable artifacts.
- Verify completion.Use deterministic checks wherever the outcome can be measured.
- Read the traces.Add, remove, or revise stages based on observed failures and cost.
[ 04 / THE FIRST IMPLEMENTATION ]
A bounded runtime.
Start with one agent, a task file, a restricted workspace, a small tool set, durable progress, and a deterministic verifier.
That baseline produces the traces needed for the next design decision. Worker agents, specialized reviewers, and automated harness optimization can follow when the evidence supports them.
Harness engineering makes agent behavior measurable, bounded, and improvable.