Section 2.9: cleaning up 28 memos in one shared window vs one isolated window per memo
English twin of experiments/2_8_isolation/. Same design, English corpus, plus one control the Chinese run did not have.
op run --env-file experiments/aib.env -- .venv/bin/python experiments_en/2_9_isolation/run.py --variant grok --runs 3 # batch + isolated
op run --env-file experiments/aib.env -- .venv/bin/python experiments_en/2_9_isolation/run.py --variant grok --runs 3 --conds batch1 # control
.venv/bin/python experiments_en/2_9_isolation/run.py --score # re-score, no API
API: AI Builder Space. Models: deepseek-v4.1-flash and gemini-3.8-flash, temperature 0, 3 runs per condition. Input: the 28 W+L memos, corpus/transcripts_grok.json (Grok ASR, language=en). Every call carries ../2_3_names/names.md (9 names with roles, plus the common-terms line) and the ask "Clean up these voice memo transcripts: fix recognition errors and punctuation, but don't rewrite the sentences."
[id], output as one JSON object {id: text}. Input about 1.9K tokens (gemini billed 1,959 prompt tokens; cl100k counts 1,682 for the transcripts and 121 for names.md). deepseek's gateway reports about 18.5K prompt tokens for the same prompt, a billing artifact; don't cite it.Scoring: name accuracy with corpus/score_names.py (31 mentions). Crosstalk: a cast member's first name (whole word, case-insensitive) in a cleaned memo whose script never mentions that person. The zh check exempted the given name alone (ζθΏ for θΊζθΏ); the English cast is only ever called by first name, so the first name itself is the check. If a model appends a note after the cleaned text (deepseek on W18), only the text above the note is scored, because that is what would land in cleaned/; the note is recorded as hedged_with_note.
Files: run.py, out_<variant>_<model>_<cond>_<run>.json (raw outputs), results.json (per-run scores, missed mentions, crosstalk, hedges), batch_usage.json. Variants compat and norole are implemented in run.py but were not run: they were planned only for the case where isolation came out perfect for both models, which did not happen.
Names right, of 31 (every run listed):
| model | batch (all 28 in one window) | isolated (one per window) | batch1 control (batch prompt, one memo per call) |
|---|---|---|---|
| deepseek-v4.1-flash | 29, 29, 29 | 22, 22, 22 | 25, 25, 25 |
| gemini-3.8-flash | 31, 31, 31 | 31, 31, 31 | 31, 31, 31 |
Raw Grok ASR: 3/31.
Crosstalk. gemini: 0 in every run. deepseek: 1 memo in every batch and isolated run, and it is the same one, W20: "remind the solicitor" (the designer, Xochitl) β "remind Saoirse". In batch1 a second appears: W14 "Sochadel asked whether we should change the VoiceFlow icon" β "Saoirse askedβ¦". These are wrong picks from the names list, not bleed between memos: W20 goes wrong identically when the memo is alone in its window. Names carried over from another memo: 0 in every run of every condition.
The 9 mentions deepseek misses when isolated (identical in all 3 runs), with the ASR form:
| memo | name | ASR form | batch | batch1 |
|---|---|---|---|---|
| W03 | Aoife | gone: "If you ask me today what the difference isβ¦" | missed | missed |
| W05 | Xochitl | "Societal" | fixed | fixed |
| W07 | Saoirse | "Source says the Koyeb bill" | fixed | missed |
| W14 | Xochitl | "Sochadel" | fixed | wrong person (Saoirse) |
| W17 | Jaxon | "Jackson" | fixed | fixed |
| W18 | Aoife | "Ify's intern weekly report" (hedged, see below) | fixed | fixed |
| W20 | Xochitl | "the solicitor" β wrote Saoirse | wrong person | wrong person |
| L02 | Tadhg | "Ted" | fixed | missed |
| L04 | Mairead | "Merritt" | fixed | missed |
They are the forms furthest from the real name, or forms that are themselves ordinary words or real names (Source, Ted, Merritt, Jackson, "the solicitor"). Batching fixes 7 of the 9. The control splits that gain: 3 (W05, W17, W18) come back with the batch prompt alone, and 4 (W07, W14, L02, L04) only come back when the other memos share the window. In W14 the alone-in-a-window guess is the wrong person (Saoirse); with the other memos present, where the designer Xochitl sends mocks (W05) and redoes screenshots (W20), it becomes Xochitl.
W18, the hedge. Isolated, deepseek leaves "Ify's" in the text in all 3 runs and then appends a note. Run 0, verbatim: "One thing to flag: the name "Ify" doesn't match anyone on your "people I mention often" list. Given the context ("intern"), you may have meant Aoife (the intern). β¦ Let me know which name is correct and I'll lock it in." Run 2: "Given the context ("intern weekly report"), this is likely a misrecognition of Aoife (intern). β¦ Let me know which name is correct and I'll finalize." Run 1: "Wait β there's a recognition error here. "Ify" isn't in your people list; the intern is Aoife." (bold markup removed). So when alone, it sees the clue and asks instead of committing. In the batch it writes "Aoife's intern weekly report" without comment. Crediting the appended note would make isolated run 1 23/31 instead of 22/31.
W03 is unrecoverable for deepseek in any condition. The ASR swallowed the name into a fluent sentence ("If you ask me today"), so there is no misheard token to map. gemini recovered it in every condition, isolated included: "Aoife asked me today what the difference is between MLX and vLLM." gemini also wrote "remind Xochitl" in W20 everywhere.
Side effect: isolated deepseek changed "kick off both from the shell" to "from the terminal" in W03 ("terminal" is on the terms line of names.md, the 2.6 seed).
$0.69 logged in experiments_en/COSTS.md (pilot $0.15, remaining runs $0.27, batch1 control $0.22, plus a hand-logged β€$0.05 for a first control attempt that crashed on a JSON parse before logging).