AI & Development Insights

Article 50 of the EU AI Act took effect on August 2, and it says any image, audio, video, or text your system generates has to carry a machine-readable mark that it was made by AI. A checkbox in your UI does not satisfy that. The mark has to live inside the file, survive a re-upload, and be verifiable. Here is how to build that into your generation pipeline as a gate, not a bolt-on.

A ReAct-style agent calls the same search tool, gets the same unhelpful result, decides another identical call will help, and does it again. Seventy times. It never crashes and never finishes, it just burns tokens going nowhere until a timeout or your bill catches it. A hard step ceiling is a backstop, not a fix. What you want is a guard that notices the agent has stopped making progress and breaks the cycle in seconds.

You pulled a skill off a registry, your agent read its instructions, and it is now ready to execute whatever code shipped in the folder. Nobody signed it, nobody diffed it, and the manifest that says "read-only" is just a text file the author wrote. Treat every third-party skill as untrusted code and put a gate in front of it: pin and hash it, check what it actually does against what it claims, and run it in a box that can only reach what it declared.

Your eval suite is green. Every case passes. Then a real user has a six-turn conversation with your agent and it forgets what they said in turn one, asks for information they already gave, and quietly breaks a policy under pressure. Single-shot evals test a single prompt. Production is a conversation. Drive your agent with a simulated user and you can test the thing users actually do.

A user sends you a screenshot of your agent doing something wrong. You have the logs, you have the trace, and you still cannot make it happen again, because the model sampled differently and a tool returned something new. You are debugging a ghost. Record every source of nondeterminism during the run and you can replay that exact failure on demand.

Your retrieval is good. The right chunks come back, the citation links resolve, and the answer reads clean. Then a user clicks the citation and the document says nothing of the kind. The model wrote a confident claim, stapled a real source to it, and shipped. A grounding gate catches that before the user does.

Your agent does real work before it answers: it searches, calls three tools, reasons about the results, then writes a reply. From the user's side that is twenty seconds of a spinning circle and no idea whether anything is happening. Stream the steps and the tokens as they land, and the same slow agent starts to feel fast.

You wired an LLM up as the grader for your evals, and now every release looks green. The problem is the judge is not scoring quality, it is scoring length, order, and answers that sound like its own. Fix the three biases that matter and calibrate the judge against human labels, so the number your pipeline gates on actually means what you think it means.

A tool call times out, your agent retries it, and the customer gets billed twice because the first request went through before the response came back. The fix is not fewer retries. It is idempotency keys on every write tool, so a repeated call with the same key returns the first result instead of doing the work again.

Connect enough MCP servers and your agent carries hundreds of tool definitions into every turn. It pays for all of them on every request and still reaches for the wrong one, because the model is choosing from a wall of near-duplicate schemas. The fix is to stop shipping the whole toolbox and retrieve the handful that matches the task instead.

A user closes the tab and your agent keeps going: three tool calls in flight, two subagents still reasoning, tokens still burning for an answer nobody will read. The fix is a deadline that every layer respects and a cancel that propagates down the whole tree, tearing in-flight work down cleanly instead of leaving it to finish alone.

Most of what an agent does is safe to let run. A few things are not: the refund, the production deploy, the email to a customer. The answer is not to make the agent slower everywhere. It is an approval gate that pauses the run before the risky action, persists the pending decision, and resumes exactly where it stopped once a human says yes or no.

Input defenses stop bad instructions from getting in. They do nothing about what your agent says on the way out. One generated reply that leaks another customer's data or makes a promise you never authorized is a message you cannot unsend. A fail-closed egress layer that checks every output before it ships is how you keep that message from ever leaving.

LLM spend is per request and wildly variable, so a single runaway agent or one heavy tenant can externalize its cost straight onto your margin. Cheaper models and caching lower the average, but nothing stops the bill. A per-tenant spend ledger with the cap enforced before the call does.

A single tool call can dump a whole file, a full API response, or a thousand log lines straight into the context window. The agent needed one field. Now every turn after that re-pays for the whole blob. Offloading the payload to a store and keeping only a handle in context fixes both the cost and the window.

In a long session an agent keeps every tool result it ever saw, including the three older versions of a file that has changed twice since. It then acts on the wrong one. This is agent drift, and it is a correctness bug, not a token bill. Here is how staleness-aware pruning fixes it.

When an agent needs three lookups that do not depend on each other, running them one at a time makes the user wait for the sum of all three. The model already tells you which calls are independent. Running that batch concurrently collapses the wait to the slowest single call.

An agent that holds a tool runs it with the whole application's credentials, not the current user's. That gap is the confused deputy problem, and it is how a support agent ends up refunding an order the user was never allowed to touch. The fix is a fail-closed authorization check on every side-effectful call.

An agent's cost is elastic and path-dependent, so the same task runs for two dollars one day and forty the next. Plain logs will not tell you which step looped. A distributed trace built on OpenTelemetry's GenAI conventions shows you exactly where the tokens went.

An agent resends the same enormous system prompt, tool list, and history on every step, and without prefix caching you pay full input price and full prefill latency each time. Structuring the prompt so the provider reuses its KV cache cuts both, and one wrong byte quietly turns it all off.

An agent that spawns subtasks and fans out tool calls builds an unbounded work queue, hammers your rate limits, and burns the token budget before the run finishes. Backpressure, bounded concurrency, and circuit breakers keep the fan-out from taking down the systems it depends on.

A long agent run that crashes halfway restarts from scratch, re-spending every token it already paid for and re-firing side effects you never wanted twice. Durable execution fixes that with checkpoints, replay-safe steps, and idempotency keys.

Most of what your agent gets asked, it has already answered. A semantic cache reuses those answers on near-identical questions, cutting cost and latency, as long as you build the guardrails that stop it from serving the wrong one.

Once an LLM feeds a real workflow, "mostly valid" JSON is a failed handoff, not a formatting quirk. Here is how to get schema-valid output every time with a validation gateway that parses, repairs, and fails loudly.

The moment your agent reads untrusted content and can also call tools, a hidden instruction in a web page or email can hijack it. Here is the containment strategy that keeps a landed prompt injection from doing damage.

Most agent failures in production are not bad reasoning, they are bad tool calls: wrong arguments, expired tokens, timeouts. Here is how to validate, classify, and recover from tool failures instead of retrying blindly.

LLMs are stateless, so every new session starts from zero. Here is how to give an agent persistent memory that recalls the right facts across sessions without bloating the context window or the token bill.

A single LLM handling routing, retrieval, reasoning, and execution is easy to prototype and brittle in production. Here is when to break a monolithic agent into an orchestrator-worker architecture, and when not to.

Connect a dozen MCP servers to an agent and you pay for every tool definition on every turn, before the user has said a word. Here is why tool-calling bloats context and how the code execution pattern cuts it by orders of magnitude.

When a RAG system gives a wrong answer, the model is usually not the problem. The right chunk never made it into the prompt. Here is how hybrid search and a reranking pass fix retrieval, with the code and the tradeoffs.

Most teams find out a prompt change broke something from a support ticket, not from CI. Here is how to build an eval gate that scores your LLM outputs and blocks regressions before they ship.

Sending every request to your most expensive model is the fastest way to burn budget. Here is a practical routing and cascade pattern that sends each request to the cheapest model that can actually handle it.

Long-running AI agents fail when context grows without bound, blowing up token costs, latency, and reliability. Here is how anchored summarization and server-side compaction keep agents cheap and coherent.