# 2.1 A chat "remembers" only because the whole history is re-sent every turn (English edition)

English twin of `experiments/2_1_stateless/`. Same design, English corpus (`experiments_en/corpus/memos.json`), English questions. Run on 2026-10-14.

## How to run

From the textbook repo root:

```bash
op run --env-file experiments/aib.env -- /Users/grapeot/co/knowledge_working/.venv/bin/python experiments_en/2_1_stateless/run.py gemini-3.8-flash
op run --env-file experiments/aib.env -- /Users/grapeot/co/knowledge_working/.venv/bin/python experiments_en/2_1_stateless/run.py deepseek-v4.1-flash
# DSH screenshots (needs DSH web running on 127.0.0.1:3080; see D below)
/Users/grapeot/co/knowledge_working/.venv/bin/python experiments_en/2_1_stateless/dsh_demo.py
/Users/grapeot/co/knowledge_working/.venv/bin/python experiments_en/2_1_stateless/dsh_figure_en.py
```

The API is AI Builder Space (OpenAI-compatible), temperature 0, system prompt "You are a concise assistant. Answer in English." The bundle is the scripts (not ASR output) of W01, W07, W10, W12, W16 and W17.

## Design

- **A: an 8-turn conversation.** Turn 1 pastes a 328-word bundle: a 33-word preamble ("Here are transcripts of a few of my voice memos from this week. The names may be wrong. On my team, Saoirse handles deployment and Jaxon is on the backend. Please remember that.") plus the six memos (295 words). Turns 2–8 are short questions: "How many people are mentioned?", "Who handles deployment?", "Why did the bill go up?", "What is Aoife working on?", "Who's coming next month?", "What needs to be prepared?", "What's the name of the colleague who handles deployment?". Each turn records the words typed and the input tokens the API billed.
- **B: the turn-8 question with no history.** Same system prompt, no earlier turns.
- **C: a bare "hi".** Not re-run as an experiment: it is language-neutral. The numbers come from `experiments/2_1_stateless/baseline_hi.json` (gemini-3.8-flash 2, gpt-5 7, grok-4-fast 185, deepseek-v4.1-flash 415). A probe on 2026-10-14 re-confirmed gemini 2 and deepseek 415 (see "Probes" below).

## Results

**A, gemini-3.8-flash** (`results_gemini-3.8-flash.json`):

| Turn | Words typed | Billed input tokens | Answer (start) |
|---|---|---|---|
| 1 | 328 | 418 | "Noted: **Saoirse** handles deployment, and **Jaxon** is on backend." |
| 2 | 5 | 692 | "There are **5** people mentioned by name (or **6** including yourself, the speaker)" |
| 3 | 3 | 754 | "**Saoirse** handles deployment." |
| 4 | 6 | 770 | "The Koyeb bill went up because someone changed a service setting to **"always-on,"**" |
| 5 | 5 | 817 | "Aoife is working on an **automated transcription script**" |
| 6 | 4 | 871 | "**Kaveh** is coming to Seattle next month to meet with Jaxon and Saoirse." |
| 7 | 5 | 899 | "For Kaveh's visit: A **10-minute demo** (put together by Saoirse) ..." |
| 8 | 9 | 962 | "**Saoirse**" |

Turn 8: 9 words typed, 962 input tokens billed, answer correct (Saoirse).

**B, gemini-3.8-flash:** the same question with no history billed 24 input tokens. Answer: "I don't have access to your team or company information. Could you provide more context or check your internal team directory?"

**A, deepseek-v4.1-flash** (`results_deepseek-v4.1-flash.json`):

| Turn | Words typed | Billed input tokens |
|---|---|---|
| 1 | 328 | 8,216 (see note) |
| 2 | 5 | 1,253 |
| 3 | 3 | 1,385 |
| 4 | 6 | 1,404 |
| 5 | 5 | 1,451 |
| 6 | 4 | 1,520 |
| 7 | 5 | 1,559 |
| 8 | 9 | 1,630 |

Turn 8 answer: "**Saoirse** handles deployment." No history (B): 434 input tokens, answer: "I don't have any information about your colleagues or team. I can't look up internal personnel details—I only have access to web search, which won't cover your organization's private information."

**Note on deepseek turn 1.** On the AI Builder Space gateway, deepseek-v4.1-flash is not a plain passthrough: the gateway's chat orchestrator offers the model a server-side web-search tool on every call and sums the prompt tokens of every tool round into the billed `prompt_tokens` (source: `ai_builder_space/backend/app/services/chat_orchestrator.py`, `tools = list(CHAT_TOOL_DEFINITIONS)`, `total_prompt += ...`). Evidence that this is what happened on English turn 1:
- turn 1 alone, repeated three more times, billed 8,127, 13,866 and 8,174 input tokens, with 5–25 s latency (a plain call takes 2–5 s);
- the English W01 memo alone with "Please remember this." billed 8,220 tokens (gemini: 99);
- the Chinese turn 1 re-run the same day still billed 751 (unchanged from the zh run), so the Chinese bundle does not trigger the search and the English one does;
- the model itself says "I only have access to web search" (B above), and its reply to "hi" begins "Hi! How can I help you today? I can search the web for up-to…". So the 415-token "hi" overhead is, at least in part, that tool's definition.
- The search results are not kept in the conversation history (turn 2 is back to 1,253), which is why the series drops after turn 1.
We did not inspect the raw gateway response (the tool-execution summary), so "the gateway ran a web search" is an inference from the gateway source and the measurements above, not a logged observation.

From turn 2 on, the deepseek series grows the same way as gemini: 1,253 → 1,630 while each turn types 3–9 words.

## Probes (2026-10-14, not part of the series)

| Call | deepseek-v4.1-flash | gemini-3.8-flash |
|---|---|---|
| "hi" (no system prompt) | 415 | 2 |
| Chinese turn 1 (zh run's exact message) | 751 | 361 |
| English W01 + "Please remember this." | 8,220 | 99 |
| "Who handles deployment on my team?" | 430 | 19 |
| English turn 1, three repeats | 8,127 / 13,866 / 8,174 | – |

## D: context usage inside an agent tool (DSH screenshots)

`dsh_demo.py` drives the DeepSeek Harness web UI with Playwright (model DeepSeek-V41-Flash via Ollama Cloud, reasoning effort High, "Full access" and "Standard mode" as the UI defaulted; workspace = the public deepseek_harness checkout). The sidebar is collapsed before the first message, so no session history appears in any screenshot; the demo session was archived at the end. Six turns: the 343-word bundle (same as A plus "and just reply "OK". Don't read or write any files, and don't call any tools."), then "How many people are mentioned?", "Who handles deployment?", "Why did the bill go up?", "Who's coming next month?", "What's the name of the colleague who handles deployment?". Screenshots: `assets/ch02_01_en/dsh_turn{1..6}.png`; numbers: `dsh_results.json`; chapter figure: `assets/ch02_01_en/dsh_usage.png` (cropped from turn 6 by `dsh_figure_en.py`, with the per-turn usage, the session total and cache hit, and the context meter boxed).

| Turn | Words typed | This turn's usage | Session total | Cache hit |
|---|---|---|---|---|
| 1 | 343 | 18.4K | 18.4K | 66% |
| 2 | 5 | 18.7K | 37.1K | 83% |
| 3 | 3 | 18.8K | 55.9K | 88% |
| 4 | 6 | 19K | 74.9K | 91% |
| 5 | 4 | 19.1K | 94K | 93% |
| 6 | 9 | 19.2K | 113K | 94% |

- Turn 1: 343 words typed, 18.4K tokens used. The rest is the harness's own system prompt and tool descriptions.
- Each later turn types 3–9 words and costs another 18.7K–19.2K tokens (about 19K), because the whole window is sent again.
- The context meter reads "2% of context used" throughout (1M window).
- Turn 6 answered "**Saoirse**." (plus a caveat that the transcripts may have names wrong).
- Difference from zh: the cache hit rate starts at 66%, not 0%. The harness's long fixed prefix was already in the provider's cache from earlier use, so the first turn already hit it. It still rises every turn (66% → 94%).

## Files

- `run.py`, `results_gemini-3.8-flash.json`, `results_deepseek-v4.1-flash.json`: A and B.
- `dsh_demo.py`, `dsh_figure_en.py`, `dsh_results.json`: D.
- `results.json`: summary of everything above.
- `CONTRACT.md`: the facts the English chapter may use.
- Chart data: `experiments_en/chart_data/ch02_01_tokens.json` (gemini series; typed = words).