Latest

Fri, Sep 18, 2026Friday, September 18, 20261 entry

GuideMake it fast and big

Measure Agent Latency: Time to First Token, Decode and the p95 Tail

Measured on small open models you can run on Colab: prefill that grows with the prompt, decode at a fixed cost per token, a prefix cache 22 times faster, and a queue that explodes near capacity.12 min readlatencyinference-cost

Thu, Sep 10, 2026Thursday, September 10, 20261 entry

GuideMake it fast and big

Design a Prompt-Caching Experiment That Measures the Right Thing

Derive when prompt caching pays from Anthropic's write premium and read discount, then run a cold, warm and changed-prefix protocol that keeps cost, context and latency apart.10 min readprompt-cachinginference-cost

That is the first entry in the log.

To browse rather than follow: topics, all articles and all notes. To learn in order: courses and the book.