hardeep.cv/Study/HarnessLesson 02
#agents#harnesses#talk-notes

The harness, not the prompt, makes an agent reliable

A harness is the code around the model: tools, context, guardrails, the loop, and a verify step. In this talk, an old model learns to upvote a story on Hacker News, and the prompt never changes.

Tejas KumarAI Developer Advocate, IBM · talk notes
October 23, 20267 min read20 min talk
Watch the talk

What you’ll learn

  1. 01What a harness isSix parts, and only one of them is the model.
  2. 02Why you need oneYou rent the model. Reliability must come from your code.
  3. 03The demoOne prompt, one old model, and a harness that grows.
  4. 04Where a fix goesSort five failures into the part that fixes them.
starts asks runs tool trims caps final answer + trace reports Taskone prompt Modelrented black box Tool registrybrowser tools Agent looprepeats until done Contexttrims history Guardrails15 steps · 50 msgs Verifychecks the trace Honest resultpassed: true / false
  1. The model reads messages and picks the next step. You rent it, and you cannot see inside it.
  2. The tool registry lists what the agent can do. In the demo these are browser tools, such as go to a page, click, and read text.
  3. The agent loop sends the history to the model, runs the tool it picks, adds the result, and asks again.
  4. Context management trims or compacts the history, so a long run does not overflow the context window.
  5. Guardrails are hard limits on the loop. The demo stops at 15 iterations or 50 messages.
  6. Verify is plain code that runs after the loop. It checks what happened, not what the model says happened.
Fig. 1An agent harness is everything around the model. The talk lists six parts: tool registry, model, context management, guardrails, agent loop, and verify.

Why a harness

You rent the model, so reliability must come from your code

pays per token or $20/month answers vary Your harnessyou write it, test it, fix it Rented modela black box behind an API ● you control this side ▲ non-deterministic ▲ routing and context can change
Fig. 2Most of us are not “token billionaires”. We pay rent for a model we cannot inspect, and the provider can change it at any time.

Think of a climbing harness. It ties you to the mountain, which does not move, so you do not fall. A dog harness keeps the dog from running off and, in his words, from bankrupting you with tokens. An agent harness ties a model that changes to code that does not.

Two meanings

“Harness” means two things, and this talk is about the agent kind

Machine learning

Eval harness

inputs model score test suite + runner

It measures a model. Nothing runs in production.

AI engineering

Agent harness

harness agent tools wraps a live agent

It controls an agent while it runs. This is the kind in Fig. 1.

Fig. 3An eval harness is a test runner for a model. An agent harness is the run-time environment around an agent.

The demo

Same prompt, same old model, and only the harness changed

He builds a browser agent live. He picks an older model on purpose, and he does not change the prompt at any point. Each run adds parts to the harness.

Prompt, every run“Upvote a story on Hacker News.”
Model, every rungpt-3.5-turbo-0613, an older model with less skill than current ones.
Lies

Bare agent

▲ upvote loginstuck “I have upvoted …” ✗ no vote · claims done

It hits the login page, gives up, and reports success.

Bounded

+ guardrails

▲ upvote loginstuck ≤ 15 steps · ≤ 50 msgs ✗ still blocked at login

The run can no longer loop forever. The login page still stops it.

Honest fail

+ verify

▲ upvote loginstuck verify: stuck at login passed: false

The harness reports the failure instead of the agent’s claim.

Upvoted

+ login handler

▲ upvote loginharness verify: vote in trace passed: true

The harness logs in with code. The vote lands, and verify agrees.

Fig. 4The same prompt and model in every card. Each card keeps the parts of the card before it and adds more.

The last run

The harness logs in, so the model never has to

click ▲ upvote lands on login page fills form, submits no model call note: you are logged in upvote lands Agentgpt-3.5 Harnessyour code Hacker Newsin a browser Verifypassed: true
  1. The agent clicks upvote. Hacker News wants a signed-in user, so the browser lands on the login page.
  2. The harness checks the URL after each step. It sees the login page, fills in the form, and submits it. The model is not called.
  3. The harness adds a system note to the history that says the agent is now logged in.
  4. The agent continues with the same prompt, and this time the upvote lands.
  5. Verify reads the trace. It finds the click on the vote link and no login page left unresolved, so it reports passed: true.
Fig. 5A login is the same every time, so code handles it. The model only does the part that needs judgment.

Verify

Verify reads the trace, not the agent’s last message

The bare agent said it upvoted the story. It did not. A model’s report on its own work is not evidence. The trace is. It holds every tool call and every page the browser visited.

verify in words (a summary, not the demo code)
Fig. 6Two plain checks on the trace turn a confident lie into an honest failure, and later into a pass you can trust.

Takeaway

Fix behavior in the harness, not in the prompt

A login page, a loop that does not end, a history that grows too long, and a claim with no proof each have a part of the harness that fixes them in code. Code does the same thing every run. A longer prompt does not.

The demo used an old model on purpose. With a good harness, a smaller and cheaper model can do the job. He gave IBM’s open-source OpenRAG as an example of the same idea at company scale: a security harness around retrieval over private data such as PDFs, invoices, and call transcripts.

Agents2025 Harnesses2026 Generated harnesses2027 · his hope ▲ now
Fig. 7His timeline. The last block is a wish, not a forecast. He hopes agents will build their own harness on the fly before they start a task.

What to remember

Ground the model in code you control

  1. A harness is everything around the model: tool registry, context management, guardrails, the agent loop, and verify.
  2. Treat the model as a black box you rent. Do not rest reliability on the prompt alone.
  3. Put hard limits on every run, for example 15 iterations and 50 messages.
  4. Verify in code. Check the trace or the real state, never the agent’s own report.
  5. When the agent fails at a step that is the same every time, such as a login, handle that step in the harness and leave the prompt alone.

Practice

Which part of the harness fixes it?

Pick a part for each failure. “Prompt” is on the list too.

The agent goes back and forth between two pages and burns tokens for an hour.

Guardrails

A cap on iterations or messages ends the run. A prompt that says “do not loop” is a request, not a limit.

The agent says “Done, the form is sent.” Nothing reached the server.

Verify

Check the trace or the real state after the run. The agent’s last message is not proof.

After many tool calls, the agent forgets the task it was given.

Context management

Keep the system prompt and the task, and trim or compact the old tool output in between.

Every run stops at the same cookie banner.

Agent loop

This works like the login handler. The loop sees the banner, closes it with code, and tells the agent. No model call.

The task needs a PDF, and the agent has no way to open files.

Tool registry

Add a tool that reads files. The model cannot do what no tool lets it do.

Related lesson

Evals check the agent before deploy. The harness guards it at run time.

  1. Lesson 01Evals are fuzzy unit tests, and the judge can lieThe same speaker on evals: test cases, LLM judges, and how to check a judge against humans.

Sources

hardeep.cv

© 2026 Hardeep Singh