What Harness Distillation Moves Into Model Weights

Every few months something in the LLM stack gets declared dead, and this season it is the agent harness. The trigger was real research, not a hot take: a team built a specialized harness around a small model, trained the harness's behavior into the model's weights, deleted the harness, and watched the distilled model outperform the original model-plus-harness combination. If a wrapper can be baked into weights and come out stronger, the scaffolding looks doomed.

The conclusion is the wrong shape. Harness distillation is a per-responsibility migration decision dressed up as a verdict on whether agent scaffolding survives. A harness is not one thing; it is a bundle of separable duties, and they migrate at very different rates. Routing, output formatting, tool-call protocol handling, and retry policy are already partially folding into weights, and native tool use plus computer-use models are the visible evidence. Permissions, durable state persistence, verification, and audit trails are control surfaces, and control surfaces demand deterministic guarantees that a probabilistic component cannot provide. Those belong in code no matter how capable the model gets.

The rest of this piece maps that split duty by duty, then traces what each migration does to three things you own: your eval set, your token economics, and your blast radius. It ends with a five-question test you can run against any responsibility in your own stack on a whiteboard.

What the harness distillation debate gets wrong

The experiment behind the current hype cycle is worth understanding precisely, because its details contain the argument against its own headline. Researchers started with a small open model and an automatically "evolved" harness: custom tools, middleware rules blocking risky actions, skills files, and memory, each added to patch an observed failure mode. They then ran training tasks with the small model inside a minimal loop while a stronger reviewer model checked every proposed step against the evolved harness, applying only the smallest correction needed to keep the step within the small model's reach. Fine-tuning on those corrected runs, then removing the harness entirely, produced a model whose reported average task success rose by roughly 21 points, a larger gain than the harness itself provided (the harness distillation paper).

Two details from the same work undercut the "harnesses are dead" reading. The distilled model still ran inside a minimal loop with tool access; the method narrows what a harness must provide rather than eliminating it. And in the chemistry domain, where the harness supplied deep domain knowledge and a validation tool, the distilled model trailed the harnessed version. Procedural behavior, the kind that looks like "inspect before editing, verify before finishing," trains in well. Domain knowledge and context management transfer poorly.

So the productive question is not whether scaffolding survives. It is which responsibilities move, at what cost, and which ones never should. That requires naming the responsibilities first.

What an agent harness is actually responsible for

Anthropic's engineering guidance on building effective agents makes the starting point plain: begin with simple, composable patterns and reach for autonomous agent loops only when the task genuinely demands it (Anthropic's agents guide). Whatever pattern you choose, the harness ends up holding a recognizable set of duties. Naming them precisely gives us the vocabulary for every decision that follows.

Read down the right-hand column and a pattern appears. The first five duties fail soft: a bad route wastes tokens, a malformed call gets retried, a bad format gets rejected by a validator downstream. The last four fail hard: an unauthorized action, a lost transaction, an unverified answer, a missing audit record. That soft-versus-hard split predicts where each duty belongs, and the next two sections walk each side.

Responsibilities already migrating into model weights

Tool-call protocol is furthest along. In the first tool-use era, you wrote the JSON parsing, the argument validation, and the re-prompt loop that recovered from malformed calls. That duty is now a first-class API surface: models accept tool definitions, emit structured calls natively, and handle results without scaffold-level few-shot engineering (Claude's native tool use). The strongest confirmation that the duty has moved is what the field now benchmarks. The Berkeley Function Calling Leaderboard ranks models, not harness code, on invoking real functions across domains (Berkeley function calling results). Nobody ran that benchmark against wrapper parsers, because the responsibility now lives where the benchmark measures it.

Output formatting is close behind. The wrapper's format-fixing loop, the "return only valid JSON" incantation plus a repair path, is being replaced by schema-constrained decoding, where the API guarantees output that validates against a declared schema (OpenAI's structured outputs). When the guarantee is structural, the scaffold's remaining role shrinks to choosing the schema.

Routing and retry policy are partially there. Models increasingly select among provided tools without an external router prompt, and providers now ship computer-use models that act on graphical interfaces directly, evidence that even perception-plus-action loops are folding in. Self-correction inside a turn substitutes for some retry middleware.

Calibration matters here. None of these migrations are total. Function-calling benchmarks exist precisely because protocol failures persist, and the distillation research itself showed context management transfers poorly. But the direction is one-way, and the burden of proof has flipped: the working default is now that the model handles protocol, and your harness needs a specific reason to override it.

The responsibilities that must stay in code

The other side of the taxonomy fails hard, and each duty there fails for the same structural reason: it is a control surface, and control surfaces need guarantees rather than probabilities.

Permissions. The harness decides what may run at all: which files, which queries, which side effects. OWASP's guidance on LLM application risks lists excessive agency among its top concerns, warning against granting models functionality, permissions, or autonomy beyond what the task requires (OWASP on excessive agency). A permission check enforced by the model is a suggestion. Asking a model to "be careful with the database" adds a paragraph to a probability distribution; it enforces nothing. The check must live in code that cannot be talked out of its decision.

State persistence. Durable writes need transactional semantics. A commit either survived the crash or it did not; an idempotency key either deduplicates or it does not. No amount of fine-tuning gives a forward pass those properties. The model can decide what to persist; code and a database must guarantee that it persisted.

Verification. A model checking its own work is a useful signal but never a control; the defendant cannot cross-examine itself. Any verification that gates a release, a payment, or a destructive action needs an external check, deterministic or independently grounded, that the same weights cannot satisfy by being confidently wrong.

Audit trails. An audit trail must be reconstructable by a third party who does not trust the system. If the record of what happened lives in learned behavior, there is no record. Logs, input-output pairs, and decision records belong in append-only storage outside the model.

The pattern across all four: these are exactly the duties where "the model usually gets it right" is a disqualifying sentence. Capability growth does not bend this, because the gap is categorical. A probabilistic component cannot offer a guarantee, and more parameters do not close a categorical gap.

What each migration does to evals, tokens, and blast radius

Moving a duty into weights is not free even when it works. Three things you own change shape, and teams that skip this accounting are the ones that get burned.

Evals migrate with the duty

Every responsibility moved into weights changes the eval set that covers it. While routing lives in your code, you test it with integration tests: this input hits this router, this fallback fires. The moment routing lives in weights, those tests measure nothing. Coverage must move to model-level evals, regression suites of inputs mapped to expected routes, run against the actual model. The industry drift is already visible in benchmarks like SWE-bench, which scores a model inside a deliberately minimal scaffold rather than scoring scaffolds themselves (SWE-bench Verified results). The rule: distillation without a corresponding eval migration is unmeasured. You have not moved the duty; you have deleted your ability to see it.

Token economics shift rather than shrink

Protocol chatter and formatting scaffolds moving into weights genuinely cut per-turn overhead: fewer router prompts, fewer format-repair round trips, fewer "call the tool exactly like this" paragraphs burning tokens on every request. But distillation shifts cost from inference-time tokens to training-time compute, whether you pay that in fine-tuning runs or in whatever margin your provider builds into pricing. The net win depends on traffic volume. A formatting duty hit ten million times a day amortizes training cost quickly; the same duty at ten thousand calls a day may never pay the distillation back. Do that arithmetic per duty, with your actual call volumes, before migrating anything.

Blast radius goes from diffable to silent

When a harness duty lives in code, a behavior change arrives as a versioned diff: reviewable, testable, revertible. When the same duty lives in weights, behavior changes arrive with model updates, silently. Your provider ships a new checkpoint, your distilled retry policy now escalates earlier, your routing drifts toward a costlier tool, and nothing in your repository changed. The failure mode for code is a bad commit you can revert. The failure mode for weights is drift you only detect if the evals from the previous subsection are still running. That dependency, that drift is visible only through evals, is the strongest argument for the migration order below.

A keep-in-code rubric for your own stack

Five questions, answerable on a whiteboard, for any duty in any harness. The result is a keep-in-code rubric that works for any agent scaffolding responsibility:

1. Does it need a guarantee? If the duty must hold every time, not usually, it needs code. Probabilities do not get promoted to guarantees by scale.
2. Does it need external audit? If a third party must reconstruct what happened without trusting the system, the record and its enforcement live outside the model.
3. Must it survive a model swap? If switching providers tomorrow should not change the behavior, keep it in code. Weights are where behavior goes to become model-specific.
4. Does it enforce a security boundary? Permission checks and scope limits face adversaries. Code does not negotiate.
5. Does it need transactional durability? Commits, checkpoints, and idempotency are database properties. No forward pass has them.

Any yes means the duty stays in code, regardless of how capable the next model is. All no, and the duty is a fair candidate for weights, with one precondition: its evals exist first.

A sensible migration order

Given all of the above, a defensible sequence for a team with a live agent in production:

1. Output formatting and tool-call protocol first. These are furthest along provider-side, cheapest to migrate because structured outputs and native tool use already sit in the APIs you call, and easiest to observe failing.
2. Retry policy next, selectively. Where failures are cheap and observable, let the model self-correct. Keep the escalation path in code, because escalation usually touches the control plane.
3. Routing last, and only after eval investment. Routing is where distillation pays best at volume and bites hardest silently, since a drifted route wastes money invisibly. Migrate it only when a route-regression eval set exists and runs on every model update.
4. Control surfaces never. Permissions, state persistence, verification, and audit trails fail hard, need guarantees, and fail categorically inside a probabilistic component. The rubric above exists to keep them off this list permanently.

The common failure mode deserves its own sentence, because it is the one teams actually hit: migrating a duty before its evals exist. The distillation looks clean in a demo, the old integration tests get deleted along with the code they tested, and the first silent model update ships a behavior change nobody can see or attribute. The harness worth keeping is one whose every remaining component has a test that fails loudly when the model beneath it drifts.

The scaffolding was never going to die all at once. It is being unbuilt duty by duty, and the durable engineering decision is knowing which duties you are willing to lose to weights, and which ones you never hand over.