# 5.8 Real-time speech recognition: transcribing while you speak, and where the vocabulary list goes

This folder answers two engineering questions about real-time speech recognition (ASR). First, where latency goes in a streaming pipeline: how long after you start speaking the first words appear, and how long you wait after stopping for the finalized transcript. Second, where a proper-noun vocabulary list belongs: passed directly into the speech recognizer via `terms`, applied afterward by a post-ASR correction LLM, or both, and what each choice costs in accuracy and post-stop wait time.

There are two scripts:

- `run.py`: Runs a controlled benchmark on the sample voice memos from Module 2 (each paired with a verbatim ground-truth script). It streams each recording to the AI Builder Space real-time speech WebSocket at real-time speaking speed and compares five configurations: no vocabulary list; ASR `terms` only; post-ASR LLM correction; both ASR `terms` and post-ASR LLM correction; and, as a reference baseline, uploading the full audio clip for batch transcription after speech ends. For each configuration it counts how many colleague name mentions are spelled right and records both latency metrics.
- `live.py`: A small interactive tool for your own testing. Speak into your microphone and watch incremental draft text print as you talk; press Enter when finished to commit and receive the finalized transcript. You can also pass `--file` to stream a WAV file at real-time speed.

## Hardware requirements

Everything runs over APIs, so any computer works. Using `live.py` with live speech requires a working microphone input device.

## Setup and usage

```bash
cd 5_8_realtime
uv venv && source .venv/bin/activate      # Or use python3 -m venv .venv if uv is not installed
uv pip install -r requirements.txt
export AI_BUILDER_TOKEN=sk_your_key

python run.py --lang en                   # English sample memos
python run.py --lang zh                   # Mandarin Chinese sample memos (~6 min, ~$0.20)
python live.py --language en              # Speak English into your mic; press Enter when done
python live.py                            # Default --language is zh
python live.py --language en --terms names.txt   # Pass a vocabulary list during ASR (one term per line)
```

On first run, `run.py` automatically downloads the 38 sample voice memos from Module 2 (about 13 MB per language) into `audio_en/` or `audio_zh/`, and runs only the 23 memos that mention colleague names. Audio frames are sent at real-time speaking speed (0.1 seconds per chunk) to simulate a live speaker, so streaming each memo takes as long as the recording itself.

The real-time endpoint offers two models: `gpt-live-transcribe` (the default dedicated real-time transcription model) and `--model gpt-realtime` (a general-purpose real-time multimodal model).

## What you should see

Reference outputs from our runs on 2026-10-23 (recorded on a Mac Studio calling AI Builder Space, two passes each in English and Mandarin Chinese) are in `sample_output/`. Across the 23 name-bearing memos, `run.py` scores 31 colleague name mentions and 20 technical term mentions:

| Configuration | Colleague names right (English, 2 runs) | Colleague names right (Mandarin Chinese, 2 runs) | Post-stop wait for final text |
|---|---|---|---|
| No vocabulary list | 3/31, 3/31 | 3/31, 3/31 | ~0.9-1.0 s |
| ASR `terms` only | 26/31, 27/31 | 22/31, 24/31 | ~0.8-1.0 s |
| Post-ASR LLM correction (`deepseek`) | 20/31, 21/31 | 22/31, 23/31 | ~2.5-2.8 s |
| Both: ASR `terms` + post-ASR LLM correction | 27/31, 28/31 | 30/31, 29/31 | ~2.2-2.5 s |
| (Reference) Batch upload after speaking, no list | 3/31, 3/31 | 4/31, 4/31 | ~1.4-3.1 s |

With `gpt-live-transcribe`, the first incremental words appear about 1.0-1.2 seconds after speech starts (1.2 s median in the English runs, 1.0-1.2 s in the Chinese runs), and draft text continues streaming while you speak. Once speech ends, the finalized transcript arrives in under 1 second, faster than waiting to upload the full clip after speaking.

Key observations from the runs:

- **Passing the vocabulary list in ASR `terms` adds essentially zero latency.** Colleague-name accuracy jumps from 3/31 to 26-27/31 in English (and 22-24/31 in Mandarin Chinese), while the wait after speech ends stays at ~0.8-1.0 seconds (`0.84-0.85 s` median in the English runs). Technical terms hit 20/20 across all `terms` runs.
- **Post-ASR LLM correction adds a full LLM call after every utterance**, with a median LLM step of 1.5-1.8 seconds (pushing total post-stop wait to ~2.5-2.8 seconds). In English it scores lower than passing `terms` during ASR (20-21/31 vs. 26-27/31), while in Mandarin Chinese the two single-stage methods score about the same (22-23/31 vs. 22-24/31).
- **The two stages catch different error types, which matters far more in Chinese than in English.** Without `terms`, the acoustic recognizer turns unfamiliar names into ordinary words that look plausible in context. In the English runs (`sample_output/en_gpt-live-transcribe_run1_20261023.json`), `"Aoife asked me today"` (`W03`) becomes `"If he asked me today"`, `"Mairead's coming over"` (`L01`) becomes `"my rant coming over"`, and `"Tadhg says"` (`L02`) becomes `"Touch says"`; in the Chinese runs, `祁骁` turns into everyday words like `取消` or `其消息`. Because the post-ASR LLM only reads text and never hears the audio, it often leaves those ordinary phrases alone, whereas passing `terms` at the ASR stage lets the decoder match the audio directly. Adding post-ASR LLM correction on top of `terms` in English nudges accuracy from 26-27/31 to 27-28/31 (fixing `"If its intern weekly report"` to `"If Aoife's intern weekly report"` in `W18`) at the cost of more than doubling post-stop wait to ~2.2-2.5 seconds, so `terms` alone already captures nearly the full gain. In Mandarin Chinese, homophone ambiguity leaves rare character spellings wrong even when `terms` is enabled (such as `蔺思远`), and the post-ASR LLM fixes them against the vocabulary list, lifting accuracy from 22-24/31 to 29-30/31.
- **Trade-off with `--model gpt-realtime`.** In a Mandarin Chinese comparison pass (`sample_output/zh_gpt-realtime_20261023.json`), passing `terms` to `gpt-realtime` gets 28/31 colleague names right within ~1.0 second after speech ends, outperforming `gpt-live-transcribe` with `terms` alone. However, `gpt-realtime` emits zero text while you are speaking; the first text arrives only after speech finishes and commits (median 12.2 seconds, roughly the full clip duration). Pick between streaming words while speaking and end-of-segment accuracy based on your application.

### Files in `sample_output/` and ground-truth inputs

- **English runs (`--lang en`, input from `memos_en.json`):**
  - `sample_output/en_gpt-live-transcribe_run1_20261023.json` and `sample_output/en_gpt-live-transcribe_run2_20261023.json`: per-memo English transcripts, latency numbers, and missed-name counts for the two English `gpt-live-transcribe` runs described in Section 5.8.
  - `sample_output/en_gpt-live-transcribe_run1_console.txt` and `sample_output/en_gpt-live-transcribe_run2_console.txt`: saved console summaries from those two English runs. These `.txt` logs were captured when the benchmark script printed Chinese status lines and table headers, so their labels are in Chinese while the run data is for `--lang en`.
  - `memos_en.json`: English ground-truth memo scripts, colleague names (`Jaxon`, `Saoirse`, `Kaitlynn`, `Aoife`, `Xochitl`, `Priya`, `Kaveh`, `Mairead`, `Tadhg`), and technical terms (note: its top-level `"about"` description string is in Chinese).
- **Mandarin Chinese runs (`--lang zh`, input from `memos_zh.json`):**
  - `sample_output/zh_gpt-live-transcribe_run1_20261023.json` and `sample_output/zh_gpt-live-transcribe_run2_20261023.json`: per-memo Chinese transcripts, timings, and missed-name counts from the two Chinese `gpt-live-transcribe` runs.
  - `sample_output/zh_gpt-realtime_20261023.json`: per-memo Chinese transcripts and timings from the `--model gpt-realtime` comparison pass.
  - `sample_output/zh_gpt-live-transcribe_run1_console.txt`, `sample_output/zh_gpt-live-transcribe_run2_console.txt`, and `sample_output/zh_gpt-realtime_console.txt`: saved console logs (with Chinese labels) for the three Mandarin Chinese runs.

## Using your own data

`run.py` takes two custom inputs: a directory of `<id>.wav` recordings and a ground-truth JSON file formatted like `memos_en.json`:

```json
{"people": {"Colleague Name": "role description"}, "terms": ["Term1", "Term2"],
 "memos": [{"id": "001", "text": "Verbatim transcript of what was spoken in this recording"}]}
```

Run it with `python run.py --lang en --audio my_audio_dir --memos my_memos.json` (pass `--lang zh` for Chinese recordings; `zh` is the default if `--lang` is omitted). Both the ASR `terms` list and the post-ASR correction vocabulary list are built from `people` and `terms`. Only memos whose ground-truth `text` mentions at least one name in `people` are included in the comparison.

Recordings must be 24 kHz, mono, 16-bit WAV files. Convert phone recordings (such as `.m4a`) first with `ffmpeg`:

```bash
ffmpeg -i recording.m4a -ar 24000 -ac 1 -sample_fmt s16 recording.wav
```

Scoring rule: if a vocabulary item appears `n` times in the ground-truth script and `k` times in the model transcript, it counts as `min(n, k)` hits. English uses case-insensitive whole-word matching, and Chinese uses case- and space-insensitive substring matching, matching the scoring in Module 2.

The microphone capture path in `live.py` was verified on a Mac using a virtual audio device (playing recorded audio into the virtual device and capturing from it), and has not been tested with a live speaker talking into a physical microphone.