Latency

An agent is slow for structural reasons, not model reasons. It calls one tool, waits, then calls the next. It sits idle during every network round trip. It resends the whole prompt and pays full prefill time on each step. It makes the user stare at a spinner for twenty seconds while real work happens out of sight. Each of those is fixable without touching the model, and the essays under this tag take them one at a time.

Some of the fixes are about actual latency: parallel tool execution, speculative execution, prefix caching, a semantic cache. One is about perceived latency, streaming steps and tokens as they land so the same slow agent feels fast. The voice agent essay covers the hardest version, where dead air is measured in hundreds of milliseconds.

Start with parallel tool execution, the simplest change with the largest effect. Then streaming agent progress for the user-facing side, and speculative tool execution once you are ready to borrow branch prediction from CPU design.