README.md

Experiment corpus: 38 synthetic voice memos (English edition)

This is the shared input for the English module 2 experiments (2.1–2.9). It holds 38 short voice memos in natural spoken American English, each 40–90 words, recorded as if someone were talking into a phone. Every person in them is fictional. The memos mirror the Chinese corpus one to one: the ids are the same, and each memo does the same job its Chinese twin does. See MAPPING.md.

• W01–W20, work memos. These plant names whose spelling can't be recovered from sound, plus AI jargon and the speaker's verbal signature "I mean".
  • Spelling variants: Jaxon and Kaitlynn.
  • Opaque spellings: Saoirse, Aoife, Xochitl, Mairead and Tadhg.
  • Kept from the Chinese corpus: Priya and Kaveh.
  • In W03, W06, W13 and W20, "shell" means the command line. The author writes that sense as "terminal", and W06 is where he makes that single edit for 2.6.
  • W10 is a memo about names being misheard. W18 has "Aoife's intern weekly report", the cross-memo clue for 2.9.
• L01–L08, life memos. These contain 9 everyday "shell"s: the Shell gas station, eggshell, taco shell, seashell, shell pasta, pistachio shells and a hermit crab's shell. They are the negatives for 2.6. Mairead (cousin) and Tadhg (friend) appear here.
• I01–I10, idea memos. Each carries one sharp, non-consensus claim, used in 2.5. The claims include: prompt engineering is basically dead; multi-agent is really about context isolation; tok/s is misleading.

Across the 28 W+L memos there are 31 name mentions and 22 term mentions, the same per-role and per-term counts as the Chinese corpus. "I mean" appears in 11 of the 28 W+L memos and in 4 of the I memos. The scripts have no um/uh, because the ASR strips them anyway.

How the audio was made

The memos are read in the author's voice. This is a voice clone the author consented to, made with local Qwen3-TTS (Qwen/Qwen3-TTS-12Hz-1.7B-Base, via tts-clone qwen-clone on Apple Silicon). The clone works across languages: the reference recording is the author speaking Chinese. That recording lives in the private/ directory of the tts_clone_skill project and is not distributed. Synthesis is free and local, and took about 20 minutes for 38 memos (tts.log).

Files

• memos.json: the scripts plus about, people (9) and terms (12). It is generated by build_memos.py, which also writes lines.json and prints the per-memo counts. "terminal" is in the terms list (the author's written form, mirroring Apple in the Chinese list) but is never spoken and never scored.
• lines.json: the TTS input, [{"id", "text"}].
• audio/*.wav: 38 files, about 9.5 minutes in total.
• transcripts_*.json: AI Builder Space ASR output, {memo_id: text}, all with language=en:
  • grok: /audio/grok-transcription, no vocabulary.
  • grok_terms: the same, with terms set to the 9 names and the 11 spoken terms.
  • compat: the OpenAI-compatible /audio/transcriptions route (whisper-1).
• score_names.py → name_scores.json: name and term accuracy per transcript set.
• MAPPING.md: Chinese id → English id, what each memo plants, and which section uses it.

Scoring rules

• Matching is case-insensitive and whole-word.
• A possessive counts as a mention ("Aoife's" contains "Aoife").
• Inside multi-word terms, a space may also be a hyphen or nothing.
• Per memo, the correct count is min(mentions in the script, mentions in the transcript).
• Only W+L memos are scored.
• Case-sensitive counts are reported alongside, because case-insensitive matching accepts "Lora" for LoRA and "VLLM" for vLLM.

Reproduce

# 0. (optional) regenerate memos.json and lines.json from build_memos.py
.venv/bin/python experiments_en/corpus/build_memos.py
# 1. synthesize (needs the local tts-clone and the private reference audio)
tts-clone qwen-clone --ref-audio <reference.wav> --ref-text-file <reference.txt> \
  --lines-file experiments_en/corpus/lines.json --out-dir experiments_en/corpus/audio/
# 2. transcribe, three ways (skips any transcripts_*.json that already exists)
op run --env-file experiments/aib.env -- .venv/bin/python experiments_en/corpus/transcribe.py
# 3. score
.venv/bin/python experiments_en/corpus/score_names.py

Results (2026-10-14)

The Chinese baseline was 4/31 names in both raw engines, 19–20/22 terms, and 21/31 names with terms.

Names.

• Jaxon comes out as Jackson (5/5) in both raw engines, and Kaitlynn as Caitlin or Kaitlyn.
• Saoirse becomes Sorsa, "Source", Sarah's, Cyrus, Saras or Therese.
• Aoife becomes Oifi or Ify. Twice, in W03 and W18, it disappears into the sentence ("If you ask me today…", "If his intern weekly report…").
• Kaveh becomes Kevi, Kevish, Kewei, Kavvy or "Coverage".
• Mairead becomes Merritt, Mara, or "My aunt" (compat, L01).
• Tadhg becomes Ted, Taj or Tatch.

Names that survive raw ASR.

• Priya is right 3/4 (grok) and 4/4 (compat). This matches the Chinese corpus, where Priya also survived.
• The designer was first named Niamh, but every engine spelled it right in 2 of 3 memos, so it was replaced with Xochitl (now grok "Societal"/"Sochadel", compat "So Chito"/"So Chido"; W05, W14 and W20 were re-synthesized and re-transcribed).
• compat also got Aoife once (W13) and Kaveh once (W19).

Terms. Raw grok gets fewer terms right than in Chinese: Koyeb→Coyab, vLLM→ValM/VLM, LoRA→"Elora", VoiceFlow→"voice flow". compat gets 20/22. With terms, grok gets every term (two are uppercase VLLM) and 27/31 names. The four misses are W03 Aoife (the phrase is gone), W07 Saoirse ("Source says"), W12 Kaveh (Kavish) and W20 Niamh (Nave).

Things to know about the transcripts.

• "I mean" survives: grok kept 15/15 and compat 14/15 (compat dropped it, with the "So", at the start of W01). "shell" survives 13/13 in every engine. Grok writes the gas station lowercase in L01 ("stop at the shell").
• compat already writes numbers as digits ("10 in the morning", "8 at night", "10-minute", "40 cents", "around 9"). Grok keeps the words. The 2.4 digits-and-a.m./p.m. rule therefore only has something to fix on the grok transcripts, which is what the experiments use.
• Grok sometimes moves sentence boundaries: "let's not rush it next Wednesday. We'll look at…" (W01), and "We're out of both marats coming over" (L01). It also mishears a few ordinary words: "verbal takes" for tics (W11), "M file" for env file (W06), "Naive reg" for RAG and "TD" for TDD (I04 and I05).
• Grok also keeps a few spoken repetitions, such as "at the, the new sizes" (W20) and "the, the trick" (L04).

Scores after the Niamh → Xochitl swap (W+L, 31 name mentions): grok 3/31, compat 6/31, grok with terms 28/31. Terms unchanged (grok 14/22, compat 20/22, grok with terms 22/22).