# Experiment corpus: 38 synthetic voice memos (English edition)

This is the shared input for the English module 2 experiments (2.1–2.9). It holds 38 short voice memos in natural spoken American English, each 40–90 words, recorded as if someone were talking into a phone. Every person in them is fictional. The memos mirror the Chinese corpus one to one: the ids are the same, and each memo does the same job its Chinese twin does. See `MAPPING.md`.

- **W01–W20, work memos.** These plant names whose spelling can't be recovered from sound, plus AI jargon and the speaker's verbal signature "I mean".
    - Spelling variants: Jaxon and Kaitlynn.
    - Opaque spellings: Saoirse, Aoife, Xochitl, Mairead and Tadhg.
    - Kept from the Chinese corpus: Priya and Kaveh.
    - In W03, W06, W13 and W20, "shell" means the command line. The author writes that sense as "terminal", and W06 is where he makes that single edit for 2.6.
    - W10 is a memo *about* names being misheard. W18 has "Aoife's intern weekly report", the cross-memo clue for 2.9.
- **L01–L08, life memos.** These contain 9 everyday "shell"s: the Shell gas station, eggshell, taco shell, seashell, shell pasta, pistachio shells and a hermit crab's shell. They are the negatives for 2.6. Mairead (cousin) and Tadhg (friend) appear here.
- **I01–I10, idea memos.** Each carries one sharp, non-consensus claim, used in 2.5. The claims include: prompt engineering is basically dead; multi-agent is really about context isolation; tok/s is misleading.

Across the 28 W+L memos there are 31 name mentions and 22 term mentions, the same per-role and per-term counts as the Chinese corpus. "I mean" appears in 11 of the 28 W+L memos and in 4 of the I memos. The scripts have no um/uh, because the ASR strips them anyway.

## How the audio was made

The memos are read in the author's voice. This is a voice clone the author consented to, made with local Qwen3-TTS (`Qwen/Qwen3-TTS-12Hz-1.7B-Base`, via `tts-clone qwen-clone` on Apple Silicon). The clone works across languages: the reference recording is the author speaking Chinese. That recording lives in the `private/` directory of the tts_clone_skill project and is not distributed. Synthesis is free and local, and took about 20 minutes for 38 memos (`tts.log`).

## Files

- `memos.json`: the scripts plus `about`, `people` (9) and `terms` (12). It is generated by `build_memos.py`, which also writes `lines.json` and prints the per-memo counts. "terminal" is in the terms list (the author's written form, mirroring Apple in the Chinese list) but is never spoken and never scored.
- `lines.json`: the TTS input, `[{"id", "text"}]`.
- `audio/*.wav`: 38 files, about 9.5 minutes in total.
- `transcripts_*.json`: AI Builder Space ASR output, `{memo_id: text}`, all with `language=en`:
    - `grok`: `/audio/grok-transcription`, no vocabulary.
    - `grok_terms`: the same, with `terms` set to the 9 names and the 11 spoken terms.
    - `compat`: the OpenAI-compatible `/audio/transcriptions` route (whisper-1).
- `score_names.py` → `name_scores.json`: name and term accuracy per transcript set.
- `MAPPING.md`: Chinese id → English id, what each memo plants, and which section uses it.

### Scoring rules

- Matching is case-insensitive and whole-word.
- A possessive counts as a mention ("Aoife's" contains "Aoife").
- Inside multi-word terms, a space may also be a hyphen or nothing.
- Per memo, the correct count is min(mentions in the script, mentions in the transcript).
- Only W+L memos are scored.
- Case-sensitive counts are reported alongside, because case-insensitive matching accepts "Lora" for LoRA and "VLLM" for vLLM.

## Reproduce

```bash
# 0. (optional) regenerate memos.json and lines.json from build_memos.py
.venv/bin/python experiments_en/corpus/build_memos.py
# 1. synthesize (needs the local tts-clone and the private reference audio)
tts-clone qwen-clone --ref-audio <reference.wav> --ref-text-file <reference.txt> \
  --lines-file experiments_en/corpus/lines.json --out-dir experiments_en/corpus/audio/
# 2. transcribe, three ways (skips any transcripts_*.json that already exists)
op run --env-file experiments/aib.env -- .venv/bin/python experiments_en/corpus/transcribe.py
# 3. score
.venv/bin/python experiments_en/corpus/score_names.py
```

## Results (2026-10-14)

| ASR | names right | terms right (case-sensitive) |
|---|---|---|
| Grok, no vocabulary | 5/31 | 14/22 (10) |
| OpenAI-compatible route | 8/31 | 20/22 (17) |
| Grok, names and terms in `terms` | 27/31 | 22/22 (20) |

The Chinese baseline was 4/31 names in both raw engines, 19–20/22 terms, and 21/31 names with `terms`.

**Names.**
- Jaxon comes out as Jackson (5/5) in both raw engines, and Kaitlynn as Caitlin or Kaitlyn.
- Saoirse becomes Sorsa, "Source", Sarah's, Cyrus, Saras or Therese.
- Aoife becomes Oifi or Ify. Twice, in W03 and W18, it disappears into the sentence ("If you ask me today…", "If his intern weekly report…").
- Kaveh becomes Kevi, Kevish, Kewei, Kavvy or "Coverage".
- Mairead becomes Merritt, Mara, or "My aunt" (compat, L01).
- Tadhg becomes Ted, Taj or Tatch.

**Names that survive raw ASR.**
- Priya is right 3/4 (grok) and 4/4 (compat). This matches the Chinese corpus, where Priya also survived.
- **The designer was first named Niamh**, but every engine spelled it right in 2 of 3 memos, so it was replaced with Xochitl (now grok "Societal"/"Sochadel", compat "So Chito"/"So Chido"; W05, W14 and W20 were re-synthesized and re-transcribed).
- compat also got Aoife once (W13) and Kaveh once (W19).

**Terms.** Raw grok gets fewer terms right than in Chinese: Koyeb→Coyab, vLLM→ValM/VLM, LoRA→"Elora", VoiceFlow→"voice flow". compat gets 20/22. With `terms`, grok gets every term (two are uppercase VLLM) and 27/31 names. The four misses are W03 Aoife (the phrase is gone), W07 Saoirse ("Source says"), W12 Kaveh (Kavish) and W20 Niamh (Nave).

**Things to know about the transcripts.**
- "I mean" survives: grok kept 15/15 and compat 14/15 (compat dropped it, with the "So", at the start of W01). "shell" survives 13/13 in every engine. Grok writes the gas station lowercase in L01 ("stop at the shell").
- **compat already writes numbers as digits** ("10 in the morning", "8 at night", "10-minute", "40 cents", "around 9"). Grok keeps the words. The 2.4 digits-and-a.m./p.m. rule therefore only has something to fix on the grok transcripts, which is what the experiments use.
- Grok sometimes moves sentence boundaries: "let's not rush it next Wednesday. We'll look at…" (W01), and "We're out of both marats coming over" (L01). It also mishears a few ordinary words: "verbal takes" for tics (W11), "M file" for env file (W06), "Naive reg" for RAG and "TD" for TDD (I04 and I05).
- Grok also keeps a few spoken repetitions, such as "at the, the new sizes" (W20) and "the, the trick" (L04).


**Scores after the Niamh → Xochitl swap (W+L, 31 name mentions):** grok 3/31, compat 6/31, grok with `terms` 28/31. Terms unchanged (grok 14/22, compat 20/22, grok with `terms` 22/22).