Viral Ruparel Book a call

Writing / Tag / 4 of 77

Prompt Caching

Every essay tagged Prompt Caching, newest first.

October 2026

3 essays / No. 64 to 73

No. 73 / 21 Oct / 11 min
Your Load Balancer Deletes the Prefix Cache on Every Agent Turn
You self-host inference, you turned on prefix caching, and the cost barely moved.
No. 66 / 12 Oct / 10 min
Your Token Budget Is Wrong the Day You Switch Models
You moved to a newer, cheaper model and the bill went up.
No. 64 / 09 Oct / 11 min
Your Model Router Saves Pennies and Burns Your Prompt Cache
You added a model router to cut costs and the bill went up.

July 2026

1 essay / No. 14 to 14

No. 14 / 29 Jul / 11 min
You Pay Full Price for the Same 8,000-Token Prompt on Every Agent Step
An agent resends the same enormous system prompt, tool list, and history on every step, and without prefix caching you pay full input price and full prefill latency each time.