README.md

5.6 The two phases of inference and prefix caching

This code answers two questions. First, where does a single request spend its time: are reading the input (prefill) and generating the output token by token (decode) the same kind of work? Second, is prefix caching actually taking effect with the way you assemble prompts: over the same long context, how much difference is there between appending only at the end on each turn (append-only) and inserting a current timestamp at the start on each turn (prepend-timestamp)?

Two scripts run the same three experiments:

• api_timing.py: Calls AI Builder Space using streaming responses to measure Time to First Token (TTFT). What happens inside the server has to be inferred from latency.
• local_cache.py: Calls a llama-server instance running on your own machine. The server reports directly how many input tokens were recomputed (prompt_n), how many cached tokens were reused (cache_n), and how many milliseconds prefill and decode each took, giving you the clearest view of the mechanics.

What you need

• api_timing.py: Uses the cloud API only, so any computer works.
• local_cache.py: An Apple Silicon Mac, an NVIDIA GPU (with a CUDA build of llama.cpp), or CPU-only (much slower). The model is a 4-bit GGUF quantization of Qwen3.5-2B (~1.2 GB), so 8 GB of RAM is enough to load it. Experiment 3 uses a context of roughly 50,000 tokens, so leave about 2 GB of extra VRAM or system memory.

Setup and running

Cloud API version:

cd 5_6_prefill_cache
uv venv && source .venv/bin/activate      # Without uv: python3 -m venv .venv
uv pip install -r requirements.txt
export AI_BUILDER_TOKEN=sk_your_key
python api_timing.py                      # Default: deepseek, 3 experiments, ~3 min, ~$0.40
python api_timing.py --model grok-4-fast --only cache

The model must support streaming: deepseek and grok-4-fast work, whereas kimi-k3 and the Gemini models do not support streaming on AI Builder Space.

Local version (uses only the Python standard library; no packages to install):

# 1. Install llama.cpp. Mac: brew install llama.cpp; NVIDIA: build with CUDA per the llama.cpp repository instructions
# 2. Start the server in another terminal; the first run downloads the model automatically (--no-mmproj skips the unused vision weights)
llama-server -hf unsloth/Qwen3.5-2B-GGUF:Q4_K_M --no-mmproj -c 65536 -np 1 --port 8080
# 3. Run the experiments
python3 local_cache.py                    # If not on port 8080, add --url http://127.0.0.1:PORT

What you should see

Experiment 1 (varying input length): Time to First Token (TTFT) grows as the input gets longer. In the local run, you can see prefill throughput directly at several thousand tokens per second.

Experiment 2 (varying output length): Total time grows almost linearly with output length, while time per output token stays nearly flat. Decode runs at one to two hundred tokens per second, more than an order of magnitude slower than prefill. That reflects the two distinct computation regimes: prefill processes the entire input prompt in parallel in one shot, whereas decode can only produce one token per step sequentially.

Experiment 3 (prefix caching): With the append-only pattern, TTFT drops sharply starting on Turn 2. With the prepend-timestamp pattern, every turn is just as slow as Turn 1, because changing the start of the prompt invalidates the entire cached prefix behind it.

Our reference outputs (recorded on 2026-10-23) are in sample_output/:

Across all 6 local turns, total recomputed input tokens come to 51,222 for append-only versus 308,026 for prepend-timestamp, a 6x difference. Local decode stays steady at ~183 tokens/s (5.5 ms per token), while local prefill runs at ~1,703 tokens/s on a 1,041-token input and 3,635 to 5,293 tokens/s once GPU compute saturates (~4,900 tokens/s at 16,865 input tokens). On the cloud API with deepseek, median TTFT rises from 1.23 s at 2,185 input tokens (1.30 s at 9,200; 1.63 s at 77,591) to 3.63 s at 292,995 input tokens, and decode takes ~2.7 to 3.0 ms per output token across 200 to 800 output tokens.

Files in sample_output/:

• local_qwen3.5-2b-q4km_m3ultra_20261023.json and local_m3ultra_console.txt: Local Mac Studio (M3 Ultra) run across all three experiments, executed with English prompts over random ASCII filler text. The .json file stores raw llama-server timing records with English keys; local_m3ultra_console.txt is the console log from that run and uses Chinese table headings and unit labels.
• api_deepseek_20261023.json and api_deepseek_console.txt (all three experiments on deepseek), plus api_grok-4-fast_cache_20261023.json and api_grok-4-fast_cache_console.txt (Experiment 3 on grok-4-fast): Cloud API runs over random ASCII filler text paired with short Chinese prompt questions (api_timing.py keeps those exact prompt strings). The .json files store raw timings under English keys, while the two *_console.txt logs use Chinese headings and unit labels.

Cloud API timings include network and server-queueing jitter and can vary by half a second on the same turn, so focus on medians and trends rather than individual numbers.

One limitation: On AI Builder Space, usage returns only prompt_tokens, completion_tokens, and total_tokens. It does not report cached token counts or bill cache hits separately from cache misses, so api_timing.py can only observe prefix caching through latency, and cache hits on this gateway save wait time rather than money. When you call native provider endpoints directly (for example, DeepSeek's official API, which returns cache-hit token counts in usage), cache hits are billed at a steep discount, so the same difference between append-only and prepending a timestamp also shows up directly on your bill.

Testing with your own data

Both scripts accept --doc your_file.txt to replace the long document in Experiment 3 with your own material, such as the system prompt, tool definitions, and project docs your agent includes on every turn. If you want to check whether your own agent is invalidating its prefix cache, the most direct check is to save the full prompt strings it actually sends on each turn and inspect prompt_n (recomputed tokens) and cache_n (cached tokens) turn by turn with local_cache.py. If prompt_n does not drop to a few dozen tokens from Turn 2 onward, something in the prefix is changing between turns, most commonly a live timestamp, a random ID, or a tool list re-ordered on every turn.