The late July 2026 smol.ai AINews newsletter highlighted a growing pain point across the field: AI agent benchmarks are hard to trust. Scores swing on the same model when the harness changes. For teams shipping real mobile automation, that means the leaderboard is a starting point, not a decision.
This is called harness sensitivity. The harness is the system-level scaffold around the model: the system prompt, allowed tools, output format, retry logic, environment access, and verification steps. Swap any of those and the same model can look brilliant in one benchmark and mediocre in another. Recent 2026 papers such as Harness-Bench and HarnessX put numbers on the effect and confirmed what agent operators already suspected.
The 30-Second Version
What Harness Sensitivity Actually Means
A benchmark score like "72% on Terminal-Bench" is a single point in a very large configuration space. The paper reports a model, but the number was produced by a harness. Change the retry budget from three to one, swap the JSON schema for free text, or drop one tool from the allowlist, and the same weights can score ten points lower or higher.
- The prompt does heavy lifting. System prompt length, tool descriptions, and few-shot examples routinely move scores by 5 to 15 points.
- Retry logic hides failures. A harness that retries a failed tool call three times reports the average of four attempts, not the model's true one-shot capability.
- Verification is a separate skill. Some harnesses ship with a strong verifier that catches obvious mistakes. Others do not. The verifier score gets attributed to the model.
- Environment access matters. A sandbox with warm caches, seeded credentials, and a stable internet connection is not the environment your mobile agent runs in.
"Agent performance should be interpreted as a property of a model embedded in an execution system, not as a property of the base model alone."
Why This Hits Mobile Automation Harder
A benchmark harness runs on a clean cloud instance with generous timeouts. A mobile agent runs on a phone, over a spotty connection, calling an n8n workflow that talks to a third-party API that rate-limits every ten minutes. The gap between the two is where scores go to die.
| Layer | Benchmark harness | Mobile automation reality |
|---|---|---|
| Latency budget | Multi-minute per step, no user waiting | Sub-second UI feedback expected or the user closes the app |
| Tool reliability | Sandboxed, deterministic, warm caches | Third-party APIs with rate limits, timeouts, and stale data |
| Retry policy | Often three or more retries with backoff | One retry, then fall back to a human handoff |
| Output contract | Structured JSON with a validator | Sometimes JSON, sometimes a deep link, sometimes a native intent |
| Session context | Fresh, seeded, single-user | Long-running, multi-session, with user data mixed in |
A model that hits 80% on a public agent benchmark can drop to 55% inside a mobile automation stack, and the drop is not a model regression. It is the harness gap. Recognising that is the first step to picking a planner that survives real traffic.
How to Read an Agent Benchmark Without Getting Fooled
Public leaderboards are still useful. They just need to be read with the harness in mind. The following checklist is the one that saves the most time when a new model release lands.
- 1Find the harness description. If the paper or blog post does not describe the system prompt, tool set, retry policy, and verifier, treat the score as a directional signal, not a target.
- 2Check the retry budget. A Pass@1 score with no retries is a very different number from Pass@5 with a rescue prompt. Normalise them before comparing models.
- 3Look at the tool list. If the benchmark grants access to a browser, a code interpreter, and a shell, and your mobile stack only has three MCP tools, the transfer is not one to one.
- 4Rerun a small slice in your harness. Ten to twenty representative tasks from your production flow, run through the exact same scaffolding you ship, tell you more than a full public benchmark.
- 5Track cost per successful outcome. A cheaper model that retries five times can beat a frontier model on the leaderboard and lose on your monthly bill.
Designing a Harness You Can Trust
If the harness explains most of the variance, then the harness deserves the same care as the model choice. A good agent harness is boring, explicit, and testable. It reports what happened, not just what worked.
Explicit tool contracts
Every tool call has a schema, every response is validated, and every schema failure is logged with the raw output. This is where the multi-agent hallucination playbook and this guide meet: structured contracts are the single largest reliability lever in both.
One retry, then hand off
Benchmark harnesses can afford five retries because a paper deadline is not a user. A production mobile agent should retry once, then hand off to a fallback model or a human. Otherwise the reported success rate hides a silent latency and cost blowout.
Separate the executor and the advisor
A single model asked to plan, execute, and grade its own work will grade itself generously. The pattern from the executor and advisor pattern for agent orchestration keeps the grader on a different model, and often a more expensive one. That split is what makes harness numbers honest.
Deterministic replay
Log the prompt, tool responses, model output, and final decision for every run. When a regression shows up, replay it against the new model without re-running the whole flow. Without replay, every benchmark run is a fresh guess.
A Common Trap
A Practical Evaluation Loop for Mobile Agent Teams
The goal is not to reproduce a public leaderboard. The goal is to have a repeatable, cheap loop that answers one question: does this new model make our real users' flows finish faster, cheaper, or with fewer escalations?
- 1Freeze a task set. Ten to fifty real tasks from production, with expected outcomes and known edge cases. This is your private benchmark.
- 2Freeze the harness. Same system prompt, same tools, same retry policy, same verifier. Only the model changes between runs.
- 3Run three times. Once is a data point. Three runs surface variance and let you spot a lucky score.
- 4Report four numbers. Pass@1, cost per successful outcome, median latency, and human-escalation rate. A single "accuracy" number hides too much.
- 5Ship behind a routing flag. The new model gets 5% of traffic for a week. Compare the same four numbers against the incumbent. Then decide.
Where This Fits in the Halmob Stack
At Halmob, every agent-backed product has the same three layers: a mobile app that has to feel responsive, an n8n automation layer that does the operational work, and a planner that keeps the two in sync. Harness sensitivity shows up at every seam. A model that plans well in isolation can fall apart the moment it has to call three n8n webhooks in a row and reconcile the results.
The routing rules described in the model routing layer for mobile AI agents and n8n flows assume that scores are grounded in your harness, not a public leaderboard. Pairing that with the executor and advisor split, the structured contracts from the hallucination playbook, and the multi-model orchestration approach from Sakana Fugu multi-model orchestration gives a stack where model swaps are cheap and safe.
The Bottom Line
Agent benchmarks measure a model inside a specific harness on a specific task set in a specific environment. Change any of the three and the number moves. For mobile automation teams, the only score that matters is the one your own harness produces on your own tasks. Build that evaluation loop first, then use public benchmarks the way they were meant to be used: as a shortlist, not a verdict.
For source material, start with the smol.ai AINews newsletter for the late July 2026 discussion on benchmark fairness, and the growing 2026 literature on harness-aware evaluation. For teams that want a harness, routing layer, and evaluation loop wired into a real mobile and automation product, Halmob ships that end to end.