Your Agent Makes the User Watch a Spinner for 20 Seconds

You built an agent that actually works. A user asks it something, and it does the sensible thing: it searches your knowledge base, calls a couple of tools, reasons about what came back, and writes a clear answer. The answer is good. The problem is that all of that takes eighteen seconds, and for those eighteen seconds the user is looking at a spinner.

They do not know the agent is searching. They do not know it found three results and is reading them. They do not know it is halfway through writing the reply. They know that they typed something, a circle started spinning, and nothing has happened since. Around the ten second mark they start to wonder if it broke. Around fifteen they reload the page, which kills the request and starts the whole thing over. Some of them just leave.

This is not a model quality problem or an infrastructure problem. Your agent is doing exactly what you want. It is a perceived latency problem, and it is one of the few reliability issues you can fix once and have it pay off on every single request. The work does not get faster. The wait gets legible, and a legible wait is one people will sit through.

Actual latency and perceived latency are different problems

It is worth being precise about what you are optimizing, because the two get conflated constantly.

Actual latency is wall clock time from request to final answer. You attack it by making the work itself smaller or more parallel: running independent tool calls at the same time instead of one after another, caching prompt prefixes, routing simple turns to a cheaper model. Those are real wins and you should pursue them. I have written about the parallel side of that in Your Agent Calls One Tool, Waits, Then Calls the Next.

Perceived latency is how long the wait feels to the person on the other end. It is governed almost entirely by one thing: whether they can see progress. A twenty second wait with a status line that updates every two seconds feels responsive. A five second wait with a blank spinner feels broken. Humans tolerate slow far better than they tolerate silent.

The reason this matters so much for agents specifically is that agents are structurally slow in a way a single model call is not. One agent answer is several model turns plus several tool calls, and the tools reach out to databases, search services, and third party APIs you do not control. You can tune the thing hard and still land in the five to thirty second range for a genuinely useful task. That whole range is the danger zone for a silent UI. So while you keep chipping away at actual latency, streaming progress is what makes the current latency survivable.

The version that makes users leave

Here is the shape of the handler most agents ship with first. It is correct. It also produces the spinner of silence.

// app/api/agent/route.ts
export async function POST(req: Request) {
  const { message } = await req.json();

  // Runs the full loop: model turn, tools, model turn, tools, final answer.
  // Returns nothing until the very end.
  const answer = await runAgentAndWait(message); // ~18s of silence

  return Response.json({ answer });
}

Every second of work happens behind that single await. The client gets one response, at the end, all at once. There is no hook for the UI to show anything in between because the server has committed to saying nothing until it is done.

The fix is to stop returning one lump at the end and start emitting a series of small events as the work happens. Restructure the agent loop as a generator that yields progress, then pipe those yields to the browser over a stream.

Turn the agent loop into an event stream

The core move is to make the agent loop yield typed events instead of building up a result and returning it once. Every meaningful step becomes an event the front end can render: a status change, a tool starting, a tool finishing, a token of the answer.

import Anthropic from "@anthropic-ai/sdk";

const client = new Anthropic();

export type AgentEvent =
  | { type: "status"; text: string }
  | { type: "tool_start"; name: string; input: unknown }
  | { type: "tool_end"; name: string; ok: boolean }
  | { type: "token"; text: string }
  | { type: "done" }
  | { type: "error"; message: string };

async function* runAgent(message: string): AsyncGenerator<AgentEvent> {
  const messages: Anthropic.MessageParam[] = [
    { role: "user", content: message },
  ];

  // Cap the tool-use rounds so a misbehaving loop can never run forever.
  for (let round = 0; round < 8; round++) {
    const stream = client.messages.stream({
      model: "claude-opus-5",
      max_tokens: 4096,
      tools,
      messages,
    });

    // Forward answer tokens the moment the model produces them.
    for await (const chunk of stream) {
      if (
        chunk.type === "content_block_delta" &&
        chunk.delta.type === "text_delta"
      ) {
        yield { type: "token", text: chunk.delta.text };
      }
    }

    const reply = await stream.finalMessage();
    messages.push({ role: "assistant", content: reply.content });

    // No tool calls means the model is done answering.
    if (reply.stop_reason !== "tool_use") {
      yield { type: "done" };
      return;
    }

    // Run every tool the model asked for, announcing each step.
    const results: Anthropic.ToolResultBlockParam[] = [];
    for (const block of reply.content) {
      if (block.type !== "tool_use") continue;

      yield { type: "status", text: labelFor(block.name) };
      yield { type: "tool_start", name: block.name, input: block.input };

      try {
        const output = await runTool(block.name, block.input);
        results.push({
          type: "tool_result",
          tool_use_id: block.id,
          content: output,
        });
        yield { type: "tool_end", name: block.name, ok: true };
      } catch (err) {
        results.push({
          type: "tool_result",
          tool_use_id: block.id,
          content: `Error: ${(err as Error).message}`,
          is_error: true,
        });
        yield { type: "tool_end", name: block.name, ok: false };
      }
    }

    messages.push({ role: "user", content: results });
  }

  yield { type: "error", message: "Agent exceeded its step budget." };
}

A few things worth calling out. The model turn is itself streamed with client.messages.stream, so answer tokens go out as they are generated rather than waiting for the full reply. The status event carries a short human readable label you control (labelFor maps search_docs to something like "Searching the docs"), which is what the user actually sees. The tool lifecycle events (tool_start, tool_end) are the honest signal that work is happening between model turns, which is where most of the silence lived before. And the round cap plus the error branch mean the stream always terminates cleanly, which matters more than it looks like it does, because a stream that never closes is its own kind of hung UI.

Notice what is not in the event union: raw reasoning. You stream the status line and the final answer, not the model's chain of thought. That keeps tool names and internal structure out of the response and avoids showing the user scratch work that sometimes contradicts the final answer.

Push the events to the browser over SSE

Now wrap that generator in a route that speaks Server Sent Events. SSE is the right tool here because the shape of the problem is one directional: the client sends one request, the server pushes a sequence of events back. You do not need the bidirectional machinery of WebSockets for that, and SSE is far less to operate.

// app/api/agent/route.ts
export const runtime = "nodejs";

export async function POST(req: Request) {
  const { message } = await req.json();
  const encoder = new TextEncoder();

  const body = new ReadableStream({
    async start(controller) {
      const send = (event: AgentEvent) =>
        controller.enqueue(
          encoder.encode(`data: ${JSON.stringify(event)}\n\n`),
        );

      try {
        for await (const event of runAgent(message)) {
          send(event);
        }
      } catch (err) {
        send({ type: "error", message: (err as Error).message });
      } finally {
        controller.close();
      }
    },
  });

  return new Response(body, {
    headers: {
      "Content-Type": "text/event-stream",
      // no-transform stops proxies from buffering the stream into one chunk.
      "Cache-Control": "no-cache, no-transform",
      Connection: "keep-alive",
      // Tell nginx specifically not to buffer this response.
      "X-Accel-Buffering": "no",
    },
  });
}

The two headers people forget are the ones that matter most in production. Without no-transform and X-Accel-Buffering: no, a proxy or load balancer in front of your app will happily buffer the entire stream and hand it to the client in one piece at the end, which reproduces the exact silence you were trying to kill, except now it is harder to debug because it works fine on localhost. If your stream feels perfect in development and dead in staging, this is the first place to look.

Read the stream on the client

The last piece is consuming the stream in the browser. A common instinct is to reach for EventSource, but EventSource only does GET requests and cannot send a JSON body, and you usually want to POST the user's message. So read the response body directly with fetch and parse the SSE frames yourself. It is less code than it sounds.

async function streamAgent(
  message: string,
  onEvent: (e: AgentEvent) => void,
  signal?: AbortSignal,
) {
  const res = await fetch("/api/agent", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ message }),
    signal, // lets the caller cancel the whole thing
  });

  const reader = res.body!.getReader();
  const decoder = new TextDecoder();
  let buffer = "";

  while (true) {
    const { value, done } = await reader.read();
    if (done) break;

    buffer += decoder.decode(value, { stream: true });

    // SSE frames are separated by a blank line. Keep the trailing
    // partial frame in the buffer until the rest of it arrives.
    const frames = buffer.split("\n\n");
    buffer = frames.pop() ?? "";

    for (const frame of frames) {
      const line = frame.split("\n").find((l) => l.startsWith("data: "));
      if (line) onEvent(JSON.parse(line.slice(6)) as AgentEvent);
    }
  }
}

The buffering logic is the part to get right. Network chunks do not line up with SSE frame boundaries, so a single read() might hand you two and a half events. Splitting on the blank line separator and holding the trailing partial frame back until more bytes arrive is what keeps you from trying to JSON.parse half an event.

On the React side, the consumer becomes small. Keep a status string and an answer string in state, and update them as events arrive: a status event replaces the status line, a token event appends to the answer, tool_start and tool_end drive a little step indicator, and done clears the status. The user now sees "Searching the docs", then "Checking inventory", then the answer typing itself out word by word. Same eighteen seconds of work. Completely different experience.

The parts people get wrong

Buffering proxies. Covered above, but it is the number one reason streaming works locally and dies in production. Send the anti buffering headers and verify against your real proxy, not just the dev server.

Errors mid stream. Once you have sent a 200 and started streaming, you cannot change the status code. An error that happens on round four of the agent loop has to be delivered as an event inside the stream, which is why the error variant exists in the union and why the route catches and sends it rather than throwing. Your client needs to render that event as a failure state, not wait forever for a done that is never coming.

Abandoned requests. When the user closes the tab, the connection drops but your generator keeps going, calling tools and spending tokens on an answer nobody will read. Wire the request's abort signal through the loop so a disconnect actually stops the work. This is the same discipline I covered in Your Agent Kept Working After the User Left, and streaming makes it more urgent, because a streaming UI invites people to bail the moment they have seen enough.

Truthful progress. The status labels you stream have to match what is actually happening. If you show "Searching the docs" while the agent is really calling a pricing API, you have built a progress bar that lies, and users notice. Drive the labels off the real tool lifecycle events, which is exactly what the generator does. Those same lifecycle events are worth emitting into your traces too, so your observability shows the same story your UI does. I go into that in Your Agent Cost $2 Yesterday and $40 Today and You Cannot See Why.

The takeaway

Your agent is not too slow. It is too quiet. The work it does is legitimate and most of it cannot be removed, but the wait does not have to feel like a wait. Restructure the loop to emit events, push them over SSE, and render them as they arrive, and a slow agent starts behaving like a fast one, without you touching the actual latency at all. It is one of the highest leverage changes you can make, because you build it once and every request benefits.

If your agent works but users keep bouncing before it answers, that gap between correct and usable is usually a streaming problem, not a model problem. Book a consultation call and we can look at where your agent goes silent and get progress flowing back to the people waiting on it.