Runta and Baseten published FrontierHarness Eval v1.1, evaluating 22 coding-agent harnesses across 5 models on 62 tasks (27,280 runs and 83B tokens). Holding the underlying model fixed, the most expensive harness cost 1.8x to 5.0x more per task than the cheapest, and the best-value harness varied by model—showing that harness choice is an independent axis of evaluation cost and accuracy.
Agent Harness
Also known as: Harness · AI Agent Harness · Agent Runtime · Harness Engineering
Plain English
An agent harness is the software that lets an AI model work through a task. It gives the model tools, passes their results back, and decides when to ask for approval or stop.
Technical Overview
The harness assembles context, calls the model, runs allowed tool calls, and feeds the results into the next turn. It can keep session state across turns, enforce permissions through a sandbox or approval rules, and expose tools through MCP. A coding harness such as Claude Code, Codex, or Muse Code can use that loop to read files, make edits, and run tests. Tool access, context handling, and stopping rules can affect how the same model performs in an evaluation.
Chronological Entries (6)
OpenAI added the ability to replace the tool list in an existing Agents API session. Changes to function tools and MCP servers take effect on the next model call, so an application can change the tools available as a task progresses.
OpenAI released the Agents API in public beta with a managed Codex harness. The service maintains the session and compacts earlier context as it fills, while the developer can choose where tools execute. A task can therefore continue across several context windows without the application implementing its own compaction loop.
OpenAI documented how applications can embed the Codex harness through its app-server protocol. A client can start a turn, stream execution events, and respond to permission requests while providing its own interface and tools.
Anthropic reported an experiment that used an initializer to prepare a repository and later coding sessions to make incremental changes. Each session left progress notes and Git commits, giving the next session a way to recover the state of the task after the earlier conversation left the context window.
Anthropic described an agent that calls tools, reads the results, and uses that feedback to choose its next action. A harness can add checkpoints for human input and stop when the task is complete or an iteration limit is reached.