Fork your agent's harness, keep the winner

Everything around a coding agent that you can change is just code and text: the CLAUDE.md, the subagents, the skills, the tools, the memory policy. So when your agent loop stalls, don't guess one fix. At AGNTCon + MCPCon Europe in Amsterdam, Fausto Albers showed his team's move: a meta agent writes several versions of the harness, runs them all on the same task against an eval that sits outside the loop, and keeps the one that wins on whatever you count as cost. The losers get stored too.

He opens with the irony. Attention is all you need, and now attention is the scarce resource. Since 2022 the work he can hand to AI has grown enormously, and his attention didn't follow that curve. His agents make thousands of decisions toward goals he gives them, and he isn't there for most of them. So the question of the talk: how do you build a system that knows what you find important, and what you count as a cost?

A steel beam, drawn in 2D

Fausto runs WonderWhy, which works closely with the applied AI lab for industrial digital twins at the Amsterdam University of Applied Sciences, and he co-founded AI Builders Club in Amsterdam. He has a background in sociology and used to own cocktail bars and restaurants. That's how he got into tech: a startup using machine learning to predict the flavor preferences of hospitality customers.

His case came from a steel company. Its CNC machines know what to do once they get directions compiled from a valid 3D model. But a lot of its clients never get a 3D CAD model. They get a 2D PDF drawing of the beam, a lossy copy of the 3D model, and a trained engineer works out the rest by hand.

Only months ago, turning that drawing into a 3D model wasn't possible, and his team got stuck. So they interviewed the engineers. Each one had the PDF on the left and a CAD program on the right, and rebuilt the beam step by step: making mistakes, correcting them, building a spatial model of the part in their head. That's exactly what they let the agents do, and it works. They delivered it in September. It still costs some money to run, and they're training an open-source model on the Dutch supercomputer so it'll cost almost nothing.

Then the line that sets up the rest of the talk: his team didn't write a single line of that application. They designed the system that built it.

The harness is the part you can edit

His short history of the last few years: prompt engineering, then context engineering, and now it's about designing the harness and running the loop.

He's precise about what "harness" means here. Two things he can't change: the foundation model, and the outer harness, meaning the coding environment, Claude Code or Codex or whatever you run. Everything inside that is his to change, and it's just code and text:

• the instructions, CLAUDE.md and AGENTS.md
• the subagents, their roles and how they coordinate
• the context
• the memory policy ("there's no learning without memory")
• the skills and functions

That's a lot of knobs. Set them up differently and the same task comes out different. And in practice, he says, agents already build these harnesses too.

Keep the judge outside the loop

Around the harness runs the loop: act, check the result, go again. The check can't live inside the harness. The more capable the models get, the better they get at reward hacking, so the eval has to sit outside the system. It can be an agent or plain deterministic code. What matters is that it isn't the thing being graded.

The feedback lands in files on disk, and those files are the input to the next iteration. So every turn of the loop can start from a fresh context window instead of an ever longer conversation. That matters for cost too: my TokenOps note is about how a long agent conversation's input cost grows with the square of its turns.

Watch the cost per accepted change

The number he finds most useful is the cost per accepted change. Each iteration costs something, and the eval is your proxy for progress toward the goal. When the cost keeps rising and the progress stalls, you've hit diminishing returns. The loop still looks very busy. Your credit card is burning.

That's the moment to stop and ask what the harness is missing, because it's just code and text you can change.

And cost doesn't only mean euros. If a client has a six-figure budget and time is the problem, spending a lot of tokens can be the sensible call.

Three harnesses, one task

Here's the part I'd steal. When a loop stalls, the usual move is to think about what could be different, write one new version of the harness and try it. Fausto's team hands that to a meta agent instead: one agent, or a bigger system, that reads every signal the coding harness produces.

The meta agent doesn't write one new harness. It writes three, or many. One gets different instructions, one changes the order of work, one adds or removes a tool. Each version runs the same task, the eval scores it, and you compare outcome and cost against the original.

Every version goes into a database as a snapshot of the coding environment. The one that does best, by whatever you find important, becomes the new harness. The others stay in the database with their evals and traces, because they might come in handy: what fixed a problem in one project might fix one in the next.

In the Q&A someone asked about prompt optimizers like GEPA. He's used them in MLflow on another project, and DSPy, and calls all of it a form of reinforcement learning, of evolutionary algorithms. For more, he pointed to Meta's Hyperagents paper, which builds on the Darwin Gödel Machine, and to JIT-Agent, which trains an open-source model to generate harnesses. He uses that one as well.

The symbol nobody put in the harness

The example that made it concrete came from the steel project. One harness gave the pipeline, version A, a way to crop the technical drawings and zoom in on details. What it didn't know was that some very small symbols on those drawings are very important. That knowledge wasn't in the harness.

The evals stalled. That's a signal, but it doesn't tell you what's wrong or what to change. They explored many directions and landed on a database of those symbols. The next harness gave the pipeline a way to look them up, and that became the policy from then on.

Cost is a state of mind

Keeping every version raises the next problem. With a thousand harnesses in the database, how do you find the one that fits a new situation? His answer leans on how people remember. We don't recall by piling up detail, and the systems we build work very differently from human recall. He thinks we can learn from that.

His example is the same situation, handed to two agents, human or AI. One knows money isn't the problem and there's enough time. It can afford to be curious. Curiosity is expensive, learning is expensive, and it can pay off big. The other knows the project has to ship. There's no room to explore anymore: things just need to be checked.

He's careful here. He doesn't say an AI has emotions. He says states of mind like curiosity, concern or urgency can stand in for what we count as cost. Today a coding agent with a goal is barely aware of what's important to you. It just tries to complete the goal against some eval.

Where my own harness sits

He ended on four things to take home: let useful methods outlive the task, think hard about what costs mean because it's a lot more than tokens, let competing perspectives shape the next move, and learn.

My own setup is the inside of his diagram. A CLAUDE.md per folder, 26 skills of my own, a hook that refuses to open a pull request without an issue number, and a memory folder of about 200 one-fact files. When I correct an agent, it saves the lesson with a why, and the next session reads it. So part of my harness is already written by agents.

But every change is one variant, kept because I said so. No snapshots, no two versions racing on the same task. Reviews judge each code change; nothing judges the harness itself. That's the single-hunch habit Fausto's team moved past.

The number I'm missing is his cost per accepted change. Meterlex already prices every session I run, but nothing joins that to which changes got merged.

The talk is 18 minutes. The harness split is at 6:29, the variants at 10:01 and the symbol story at 11:59. If you've raced two versions of a CLAUDE.md on the same task, I'd love to hear which one won, and why.

Plain text: fork-your-agents-harness.md · Atom feed · All notes