Writing / Tag / 32 of 68
Reliability
Every essay tagged Reliability, newest first.
Reliability for an agent is not one number. A pass rate on a dashboard averages the slow answers, the blown budgets and the confidently wrong responses into the same green figure, and that is usually where the trouble starts. The essays under this tag pull that number apart and look at what actually fails: a flaky step that flips on the same input, a retry that carries poisoned context forward, a crash at step 40 that throws away everything before it.
The common thread is that model nondeterminism is a given, so the system around the model has to absorb it. That means checkpoints, idempotent writes, clean retries, abstention when the agent should not guess, and release gates tied to measurements that predict what customers will experience.
Start with pass^k and per-step consistency, which explains why a five-step task built from 90% steps succeeds about half the time. Then read durable execution with checkpoints and replay and SLOs and error budgets for agents, which together cover how to keep a run alive and how to know when you should not ship.
October 2026
You Turned Off Thinking and Your Agent Stopped Calling Tools
Moving a step to a cheaper, faster model usually means turning the thinking knob down.Your Agent Decides at 0.8 Confidence. You Never Measured 0.8.
A model returns 0.9 and your pipeline treats it as ninety percent sure, so it auto approves, routes, or flags on that number without anyone checking what the number means.Your Long MCP Tool Call Dies at the Client Timeout
A tool that takes four minutes to run cannot answer inside a client request that gives up after sixty seconds.Your Agent Failed at Step 7. The Bug Was at Step 3.
When a multi-step agent produces a confidently wrong answer, the step that threw the error is almost never the step that caused it.Your Agent Retries Are Making the Failure Worse
When an agent step fails and you retry it in the same context, the failed attempt stays in the window and the model conditions its next try on its own mistake.Your Agent's One Reliability Number Is Hiding the Failures That Matter
A single pass rate on a dashboard tells you an agent is "99% reliable" and tells you nothing about which 1% is on fire.When Every AI Provider Goes Down at the Same Time
Early this month three of the big model providers degraded within a few hours of each other, and a lot of AI features went dark with them.
September 2026
Your Agent Got the Right Answer the Wrong Way
Your eval checks the final answer and goes green.Your Agent Would Rather Guess Than Admit It Doesn't Know
Most agents never say "I don't know." They were trained on benchmarks that reward a confident guess over an honest abstention, so in production they invent a policy, a number, or a citation and say it with a straight face.One Flaky Step Is Sinking Your Agent. Vote It Out.
You measured your agent and found one step that flips between right and wrong on the same input.Your Agent Scores 90% on Evals and Still Fails Customers
A 90% pass@1 eval score feels like a passing grade, and then production hands you a stream of complaints anyway.The MCP Tool You Approved Last Week Is Not the One Running Today
You approved an MCP server once, and your agent has trusted its tools ever since.MCP Went Stateless. Your Server Still Holds Sessions.
The 2026-07-28 MCP spec dropped the initialize handshake and the session id, which means any request can now land on any replica behind a plain load balancer.Your Model Advertises 1M Tokens. It Starts Forgetting Around 600K.
A million-token context window landed in half the models this week, and the reflex is to stop retrieving and just paste everything in.A New Model Dropped This Week. Don't Just Bump the String.
Four frontier models shipped in the last 72 hours and your instinct is to change one env var and redeploy.
August 2026
Your Model Started Showing Its Work. Now You Have to Handle It.
Reasoning traces are the feature everyone turned on this month and nobody planned for.Your Agent Called the Same Tool Seventy Times and Billed You for It
A ReAct-style agent calls the same search tool, gets the same unhelpful result, decides another identical call will help, and does it again.Your Agent Passes Every Eval and Still Fumbles Real Conversations
Your eval suite is green.A Customer Hit a Bug Your Agent Will Never Reproduce
A user sends you a screenshot of your agent doing something wrong.Your LLM Judge Prefers the Longer Answer
You wired an LLM up as the grader for your evals, and now every release looks green.Your Agent Retried and Charged the Card Twice
A tool call times out, your agent retries it, and the customer gets billed twice because the first request went through before the response came back.Your Agent Kept Working After the User Left
A user closes the tab and your agent keeps going: three tool calls in flight, two subagents still reasoning, tokens still burning for an answer nobody will read.Let Your Agent Ask Before It Does the Irreversible Thing
Most of what an agent does is safe to let run.Your Agent Just Leaked One Customer's Data to Another
Input defenses stop bad instructions from getting in.One Customer Burned Your Month of LLM Budget by 2pm
LLM spend is per request and wildly variable, so a single runaway agent or one heavy tenant can externalize its cost straight onto your margin.The Agent Is Not Confused. Its Context Is Stale.
In a long session an agent keeps every tool result it ever saw, including the three older versions of a file that has changed twice since.
July 2026
Your Agent Fans Out Faster Than Anything Downstream Can Take
An agent that spawns subtasks and fans out tool calls builds an unbounded work queue, hammers your rate limits, and burns the token budget before the run finishes.Your Agent Dies at Step 40 and Starts Over From Zero
A long agent run that crashes halfway restarts from scratch, re-spending every token it already paid for and re-firing side effects you never wanted twice.Your LLM Returns Almost Valid JSON, and Almost Is Breaking You
Once an LLM feeds a real workflow, "mostly valid" JSON is a failed handoff, not a formatting quirk.Your Agent Keeps Retrying a Tool Call That Will Never Work
Most agent failures in production are not bad reasoning, they are bad tool calls: wrong arguments, expired tokens, timeouts.Your Agent Is One Prompt Doing Six Jobs
A single LLM handling routing, retrieval, reasoning, and execution is easy to prototype and brittle in production.LLM Evals in CI: Catch Regressions Before Your Users Do
Most teams find out a prompt change broke something from a support ticket, not from CI.