A capability decomposition

with $H_0$ a no-tools, single-turn baseline. This quantity is intentionally agnostic to whether a gain comes from better prompts, better tools, or better stopping decisions.

The Harness Architecture

Figure 1 depicts the reference design. A request enters a context compiler, which selects instructions and relevant memory. The model proposes a step; a router validates and executes tools; an observer normalizes results; and a verifier decides whether to continue, repair, or stop.

Figure 1. A reference harness for agentic inference. The model is one component in a runtime that controls information flow and feedback.

Context compilation

The context compiler maps a growing trajectory to a bounded working set. It may summarize old observations, retrieve repository-local documentation, or inject a task-specific rubric. Retrieval-augmented generation provides a useful precedent for this separation between parametric knowledge and external context (Lewis et al. 2020). In practice, compilation is also where token budgets become an engineering decision rather than a model property.

Tools and permissions

Tools expose actions with typed schemas: read a file, run a test, issue a database query, or navigate a page. Tool descriptions act as an interface contract, while permissions define the admissible action set. Tool-use research has shown that language models can learn when and how to call APIs (Schick et al. 2023); a harness adds the runtime checks that make those calls observable and interruptible.

State and memory

State is the difference between remembering a plan and merely repeating it. We distinguish ephemeral scratch state, durable task memory, and an append-only trace for audit. The trace is valuable even when it is never shown to the model: it lets the verifier attribute a failure to a bad action, a stale observation, or an invalid assumption.

Verification and recovery

A verifier turns progress into a signal. It can run unit tests, compare a page against a target, or ask a second model to critique an intermediate result. Reflexion-style methods use verbal feedback to improve subsequent attempts (Shinn et al. 2023); harnesses generalize the idea to any executable check. Recovery policies then choose among retrying the same tool, revising the plan, shrinking scope, or asking a human.

Experimental Setup

Tasks

We construct three task families that stress different parts of the loop: repository edits patterned after SWE-bench Verified (Jimenez et al. 2024), browser workflows patterned after WebArena (Zhou et al. 2023), and structured API tasks patterned after tool-use evaluations. Each instance has a binary success criterion and a token-and-tool cost. To keep the comparison focused, every condition uses the same frozen 70B instruction-tuned model and temperature 0.2.

Conditions

We enable components cumulatively: single-turn prompting (Base), structured planning (Plan), typed tools (Tools), working memory (Memory), executable verification (Verify), and recovery with a bounded retry policy (Full). The planner and verifier receive the same model weights as the actor. No additional fine-tuning is used.

Success rate (%) as harness components are enabled. Values are from a simulated controlled study designed to isolate systems effects.

Metrics

We report task success, normalized cost (model tokens plus tool latency), and recovery rate: the fraction of initially failing trajectories repaired within the retry budget. Confidence intervals are omitted because the values are illustrative; the protocol is specified so that a practitioner can reproduce the comparison with a real model.

Results

Table 1 shows a monotonic improvement as the loop gains access to actions and feedback. Typed tools produce the largest single jump on API tasks, while verification and recovery matter most for repository edits, where a plausible patch is not the same as a passing patch. The full harness yields a $3.2\times$ relative improvement over the single-turn baseline at constant parameter count.

Ablation of the full harness on the macro average. Removing one component exposes a characteristic failure mode.

The cost of being capable

Capability is not free. The Full condition uses 1.37$\times$ the tokens of Base and makes 2.8 tool calls per successful task on average. Yet cost per successful task falls by 41% because fewer attempts terminate in an unrecoverable state. This is a systems result: a longer trajectory can be cheaper than a short wrong answer.

Scaling the harness

Figure 2 summarizes a qualitative scaling law observed in the simulation. Gains saturate when the model cannot interpret tool outputs, then resume when the harness supplies structured observations and targeted checks. The curve therefore depends on both sides of the interface; an elaborate harness around an incapable actor mostly produces elaborate traces.

Figure 2. Illustrative harness scaling curve. Depth counts enabled feedback interfaces, not model layers.

Discussion

Why the harness multiplies capability

The model supplies general-purpose heuristics; the harness supplies opportunities to apply them repeatedly. Three mechanisms explain most of the gain:

1. Information gain. Retrieval and observation expose facts that were not in the initial prompt.

2. Actionability. Tools make intermediate decisions testable in the environment.

3. Error correction. Verifiers convert silent mistakes into observations that can change the next step.

These mechanisms resemble the decomposition and reflection strategies reported in recent agent work (Yao et al. 2023; Wang et al. 2024). The harness is therefore a policy over policies: it decides which context the actor sees, which actions are legal, and when a trajectory is good enough.

A control-theoretic view

where $A(b_t)$ is filtered by permissions and budgets. The actor approximates $Q$ with language. This view clarifies why stopping is a learned systems decision rather than a formatting choice.

Failure modes

A harness can amplify the wrong behavior. Prompt injection can poison retrieved context; a permissive router can turn a hallucination into a destructive action; and an overconfident verifier can certify a broken artifact. Long traces also create an observability paradox: more logs make diagnosis possible but can crowd out the evidence needed for the next decision. Robust systems should therefore log everything, show selectively, and fail closed when checks disagree.

Related Work

Our framing connects several lines of research. Transformers established the dominant neural sequence backbone (Vaswani et al. 2017); scaling studies showed that capability can emerge with model size and data (Brown et al. 2020; Wei et al. 2022). ReAct interleaves reasoning and acting (Yao et al. 2023), Toolformer learns API invocation (Schick et al. 2023), and Reflexion adds verbal feedback across attempts (Shinn et al. 2023). Retrieval-augmented generation separates memory from parameters (Lewis et al. 2020). Recent coding-agent benchmarks and open runtimes make the systems layer measurable (Jimenez et al. 2024; Wang et al. 2024). We contribute a common vocabulary for these pieces and an explicit metric for their joint effect.

Limitations and Responsible Use

The numerical study is intentionally simulated and should not be read as a claim about any particular commercial system. Real evaluations must report task sampling, variance across seeds, tool failures, and human escalation. Harnesses also expand the attack surface: credentials, filesystem access, and browser sessions require least-privilege policies and auditable traces. We recommend publishing harness specifications alongside model checkpoints so that comparisons measure complete systems.

Conclusion

A language model becomes an agent when a runtime gives it state, tools, feedback, and a reason to continue. The resulting capability is a property of the pair $(\mathcal{M},H)$, not of the weights alone. This perspective suggests a practical research program: benchmark harness components separately, standardize interfaces, and measure the cost of reliable completion. The punchline remains the title. In agentic systems, the harness is not a footnote; it is where the loop lives.

Acknowledgments

The author thanks the engineers who keep the loop from spinning forever, the evaluators who press the red button, and the users who report that the agent was “almost right.”

Reference Harness Pseudocode

The following pseudocode captures the runtime used for the illustrative study.

state $\leftarrow$ initialize(task)
for step $\leftarrow 1,\ldots,K$ do
context $\leftarrow$ compile(task, state, budget)
action $\leftarrow$ model(context)
if policy.reject(action) then
state.add(error(“permission”)); continue
observation $\leftarrow$ router.execute(action)
state.add(observation)
if verifier.pass(state) then return success
if verifier.fail(state) and recovery.exhausted() then return failure
recovery.update(state)
end for
return failure

A Harness Checklist

For reproducible agent experiments, we recommend reporting: the context compiler and truncation rule; tool schemas, permissions, and timeouts; state serialization and memory retention; verifier implementation and false-positive rate; retry and stopping policy; token, wall-clock, and external-action budgets; and a trace of representative failures. These details are often more predictive of observed behavior than a one-line model name.

References

• Brown, T. B., et al. (2020). Language Models Are Few-Shot Learners .
• Jimenez, C. E., et al. (2024). SWE-bench: Can Language Models Resolve Real-World GitHub Issues? .
• Lewis, P., et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks .
• Schick, T., et al. (2023). Toolformer: Language Models Can Teach Themselves to Use Tools .
• Shinn, N., et al. (2023). Reflexion: Language Agents with Verbal Reinforcement Learning .
• Vaswani, A., et al. (2017). Attention Is All You Need .
• Wang, X., et al. (2024). OpenHands: An Open Platform for AI Software Developers as Generalist Agents .
• Wei, J., et al. (2022). Emergent Abilities of Large Language Models .
• Yao, S., et al. (2023). ReAct: Synergizing Reasoning and Acting in Language Models .
• Zhou, S., et al. (2023). WebArena: A Realistic Web Environment for Building Autonomous Agents .