Writing / Tag / 53 of 71
Production
Every essay tagged Production, newest first.
Most of what I write ends up tagged production, because the gap I keep seeing is between an agent that works in a demo and one that survives a month of real traffic. The essays here are about that second thing: retries that make a failure worse, tool calls that charge a card twice, a model upgrade nobody evaluated, a provider outage that became your outage.
The pattern across all of them is the same. An agent is a distributed system that happens to call a model, so the old operational questions still apply. What happens on retry? What happens when a dependency is down? Who can prove what it did and why? Most of the answers are boring engineering, and that is the point.
If you are new here, start with idempotency keys for agent tool calls, which is the cleanest example of a production bug that no prompt change can fix. Then read the piece on pass^k and consistency to see why a 90% eval score does not mean what it seems to, and the one on provider outages for the resilience layer every model call should sit behind.
October 2026
Your Background Agent Wakes Up More Than You Think
A scheduled agent fires every few minutes, a webhook fires on every push, and both land on the same agent with nobody watching.Your GenAI Dashboards Go Blank After an OpenTelemetry Upgrade
The GenAI semantic conventions are still in Development, and they rename and remove attributes between versions.Your Agent Traces Are Shipping Customer Prompts to Your Observability Vendor
You turned on GenAI content capture to debug one tool call, and now every prompt and reply is sitting in a third party's searchable storage.Your Agent Decides at 0.8 Confidence. You Never Measured 0.8.
A model returns 0.9 and your pipeline treats it as ninety percent sure, so it auto approves, routes, or flags on that number without anyone checking what the number means.Your Token Budget Is Wrong the Day You Switch Models
You moved to a newer, cheaper model and the bill went up.Stop Hand-Tuning Prompts. Let Your Eval Set Do It.
Most teams still tune prompts by hand, one guess at a time, with no memory of what they already tried and no proof the new version is better.Your Agent Writes Code and Runs It Where Your Secrets Live
The feature that makes a code-writing agent useful, running code it generated a second ago, is also the widest hole in your stack.Your Agent Logs Everything and Can Prove Nothing
A trace with forty-seven agent steps shows an auditor that something happened.Your Agent Failed at Step 7. The Bug Was at Step 3.
When a multi-step agent produces a confidently wrong answer, the step that threw the error is almost never the step that caused it.Your Agent Retries Are Making the Failure Worse
When an agent step fails and you retry it in the same context, the failed attempt stays in the window and the model conditions its next try on its own mistake.Your Evals Are Green and the Product Is Getting Worse
A frozen eval suite stops describing your product within weeks.Your Agent's One Reliability Number Is Hiding the Failures That Matter
A single pass rate on a dashboard tells you an agent is "99% reliable" and tells you nothing about which 1% is on fire.When Every AI Provider Goes Down at the Same Time
Early this month three of the big model providers degraded within a few hours of each other, and a lot of AI features went dark with them.
September 2026
Your Agent Got the Right Answer the Wrong Way
Your eval checks the final answer and goes green.Your Agent Would Rather Guess Than Admit It Doesn't Know
Most agents never say "I don't know." They were trained on benchmarks that reward a confident guess over an honest abstention, so in production they invent a policy, a number, or a citation and say it with a straight face.Nobody Can Tell You How Many Agents You Are Running
A year of shipping agents leaves most teams with a fleet nobody can inventory: no owner, no declared budget, no record of which tools each one can reach.Your Agent's Memory Is Full and Most of It Is Junk
You gave your agent persistent memory and it worked.Your Agent Fetches the Same Row Ten Times a Session
A long-running agent calls the same read tool over and over inside a single session, paying full latency and quota for answers it already had a few turns ago.Your Agent Is Idle Most of the Time It's Working
A tool-using agent spends a surprising share of its wall clock doing nothing, just waiting for a network round trip while the model has already stalled.Your Subagents Are Hiding the Evidence
Spawning a subagent to keep the parent's context clean is the right instinct, and it is also where a lot of long-running agents quietly go wrong.Your Multi-Agent System Already Has a Blackboard
Wire a few agents together with direct handoffs and it works.Your Voice Agent's Dead Air Is an Architecture Problem
A voice agent that goes silent for two seconds after the caller stops talking feels broken, and no faster model fixes it, because the floor is retrieval plus generation plus speech.Your Easy Queries Are Paying for Thinking They Never Use
You turned on extended thinking because it lifted your quality numbers, and a quarter later the invoice had doubled.One Flaky Step Is Sinking Your Agent. Vote It Out.
You measured your agent and found one step that flips between right and wrong on the same input.Your Agent Scores 90% on Evals and Still Fails Customers
A 90% pass@1 eval score feels like a passing grade, and then production hands you a stream of complaints anyway.You Upgraded Your Embedding Model and Silently Broke Retrieval
Swapping the embedding model behind your RAG looks like a one-line config change.The MCP Tool You Approved Last Week Is Not the One Running Today
You approved an MCP server once, and your agent has trusted its tools ever since.Your Agent Fails the Same Way Every Week and Learns Nothing
Your agent trips over the same edge case every Monday, you patch the prompt by hand, and next Monday it trips again.Your AI Agent Shares One API Key. That Is the Problem.
A recent survey found 93% of AI agent projects still authenticate with an unscoped API key, and most agents end up with more access than they need.Content Filters Watch What Your Agent Says. Nothing Watches What It Does.
Your output guardrails inspect the words an agent produces.A New Model Dropped This Week. Don't Just Bump the String.
Four frontier models shipped in the last 72 hours and your instinct is to change one env var and redeploy.Your Agent Started Calling Other Agents. Who Verified Theirs?
A2A hit v1.0 this year and teams are wiring their agents to other teams' and vendors' agents.
August 2026
Your Model Started Showing Its Work. Now You Have to Handle It.
Reasoning traces are the feature everyone turned on this month and nobody planned for.As of This Month, EU Law Says Your AI Output Has to Mark Itself
Article 50 of the EU AI Act took effect on August 2, and it says any image, audio, video, or text your system generates has to carry a machine-readable mark that it was made by AI.Your Agent Called the Same Tool Seventy Times and Billed You for It
A ReAct-style agent calls the same search tool, gets the same unhelpful result, decides another identical call will help, and does it again.You Installed a Skill and Your Agent Will Run Whatever Is Inside
You pulled a skill off a registry, your agent read its instructions, and it is now ready to execute whatever code shipped in the folder.Your Agent Passes Every Eval and Still Fumbles Real Conversations
Your eval suite is green.A Customer Hit a Bug Your Agent Will Never Reproduce
A user sends you a screenshot of your agent doing something wrong.Your RAG Cites a Source That Does Not Say That
Your retrieval is good.Your Agent Makes the User Watch a Spinner for 20 Seconds
Your agent does real work before it answers: it searches, calls three tools, reasons about the results, then writes a reply.Your LLM Judge Prefers the Longer Answer
You wired an LLM up as the grader for your evals, and now every release looks green.Your Agent Retried and Charged the Card Twice
A tool call times out, your agent retries it, and the customer gets billed twice because the first request went through before the response came back.Your Agent Has 200 Tools and Picks the Wrong One
Connect enough MCP servers and your agent carries hundreds of tool definitions into every turn.Your Agent Kept Working After the User Left
A user closes the tab and your agent keeps going: three tool calls in flight, two subagents still reasoning, tokens still burning for an answer nobody will read.Let Your Agent Ask Before It Does the Irreversible Thing
Most of what an agent does is safe to let run.Your Agent Just Leaked One Customer's Data to Another
Input defenses stop bad instructions from getting in.One Customer Burned Your Month of LLM Budget by 2pm
LLM spend is per request and wildly variable, so a single runaway agent or one heavy tenant can externalize its cost straight onto your margin.
July 2026
Your Agent Fans Out Faster Than Anything Downstream Can Take
An agent that spawns subtasks and fans out tool calls builds an unbounded work queue, hammers your rate limits, and burns the token budget before the run finishes.Your Agent Dies at Step 40 and Starts Over From Zero
A long agent run that crashes halfway restarts from scratch, re-spending every token it already paid for and re-firing side effects you never wanted twice.Your Agent Answers the Same Question Fifty Times a Day
Most of what your agent gets asked, it has already answered.Your LLM Returns Almost Valid JSON, and Almost Is Breaking You
Once an LLM feeds a real workflow, "mostly valid" JSON is a failed handoff, not a formatting quirk.Your Agent Will Do Whatever the Web Page Tells It To
The moment your agent reads untrusted content and can also call tools, a hidden instruction in a web page or email can hijack it.Your Agent Keeps Retrying a Tool Call That Will Never Work
Most agent failures in production are not bad reasoning, they are bad tool calls: wrong arguments, expired tokens, timeouts.