README.md

5.8 Real-time speech recognition: transcribing while you speak, and where the vocabulary list goes

This folder answers two engineering questions about real-time speech recognition (ASR). First, where latency goes in a streaming pipeline: how long after you start speaking the first words appear, and how long you wait after stopping for the finalized transcript. Second, where a proper-noun vocabulary list belongs: passed directly into the speech recognizer via terms, applied afterward by a post-ASR correction LLM, or both, and what each choice costs in accuracy and post-stop wait time.

There are two scripts:

• run.py: Runs a controlled benchmark on the sample voice memos from Module 2 (each paired with a verbatim ground-truth script). It streams each recording to the AI Builder Space real-time speech WebSocket at real-time speaking speed and compares five configurations: no vocabulary list; ASR terms only; post-ASR LLM correction; both ASR terms and post-ASR LLM correction; and, as a reference baseline, uploading the full audio clip for batch transcription after speech ends. For each configuration it counts how many colleague name mentions are spelled right and records both latency metrics.
• live.py: A small interactive tool for your own testing. Speak into your microphone and watch incremental draft text print as you talk; press Enter when finished to commit and receive the finalized transcript. You can also pass --file to stream a WAV file at real-time speed.

Hardware requirements

Everything runs over APIs, so any computer works. Using live.py with live speech requires a working microphone input device.

Setup and usage

cd 5_8_realtime
uv venv && source .venv/bin/activate      # Or use python3 -m venv .venv if uv is not installed
uv pip install -r requirements.txt
export AI_BUILDER_TOKEN=sk_your_key

python run.py --lang en                   # English sample memos
python run.py --lang zh                   # Mandarin Chinese sample memos (~6 min, ~$0.20)
python live.py --language en              # Speak English into your mic; press Enter when done
python live.py                            # Default --language is zh
python live.py --language en --terms names.txt   # Pass a vocabulary list during ASR (one term per line)

On first run, run.py automatically downloads the 38 sample voice memos from Module 2 (about 13 MB per language) into audio_en/ or audio_zh/, and runs only the 23 memos that mention colleague names. Audio frames are sent at real-time speaking speed (0.1 seconds per chunk) to simulate a live speaker, so streaming each memo takes as long as the recording itself.

The real-time endpoint offers two models: gpt-live-transcribe (the default dedicated real-time transcription model) and --model gpt-realtime (a general-purpose real-time multimodal model).

What you should see

Reference outputs from our runs on 2026-10-23 (recorded on a Mac Studio calling AI Builder Space, two passes each in English and Mandarin Chinese) are in sample_output/. Across the 23 name-bearing memos, run.py scores 31 colleague name mentions and 20 technical term mentions:

With gpt-live-transcribe, the first incremental words appear about 1.0-1.2 seconds after speech starts (1.2 s median in the English runs, 1.0-1.2 s in the Chinese runs), and draft text continues streaming while you speak. Once speech ends, the finalized transcript arrives in under 1 second, faster than waiting to upload the full clip after speaking.

Key observations from the runs:

• Passing the vocabulary list in ASR terms adds essentially zero latency. Colleague-name accuracy jumps from 3/31 to 26-27/31 in English (and 22-24/31 in Mandarin Chinese), while the wait after speech ends stays at ~0.8-1.0 seconds (0.84-0.85 s median in the English runs). Technical terms hit 20/20 across all terms runs.
• Post-ASR LLM correction adds a full LLM call after every utterance, with a median LLM step of 1.5-1.8 seconds (pushing total post-stop wait to ~2.5-2.8 seconds). In English it scores lower than passing terms during ASR (20-21/31 vs. 26-27/31), while in Mandarin Chinese the two single-stage methods score about the same (22-23/31 vs. 22-24/31).
• The two stages catch different error types, which matters far more in Chinese than in English. Without terms, the acoustic recognizer turns unfamiliar names into ordinary words that look plausible in context. In the English runs (sample_output/en_gpt-live-transcribe_run1_20261023.json), "Aoife asked me today" (W03) becomes "If he asked me today", "Mairead's coming over" (L01) becomes "my rant coming over", and "Tadhg says" (L02) becomes "Touch says"; in the Chinese runs, 祁骁 turns into everyday words like 取消 or 其消息. Because the post-ASR LLM only reads text and never hears the audio, it often leaves those ordinary phrases alone, whereas passing terms at the ASR stage lets the decoder match the audio directly. Adding post-ASR LLM correction on top of terms in English nudges accuracy from 26-27/31 to 27-28/31 (fixing "If its intern weekly report" to "If Aoife's intern weekly report" in W18) at the cost of more than doubling post-stop wait to ~2.2-2.5 seconds, so terms alone already captures nearly the full gain. In Mandarin Chinese, homophone ambiguity leaves rare character spellings wrong even when terms is enabled (such as 蔺思远), and the post-ASR LLM fixes them against the vocabulary list, lifting accuracy from 22-24/31 to 29-30/31.
• Trade-off with --model gpt-realtime. In a Mandarin Chinese comparison pass (sample_output/zh_gpt-realtime_20261023.json), passing terms to gpt-realtime gets 28/31 colleague names right within ~1.0 second after speech ends, outperforming gpt-live-transcribe with terms alone. However, gpt-realtime emits zero text while you are speaking; the first text arrives only after speech finishes and commits (median 12.2 seconds, roughly the full clip duration). Pick between streaming words while speaking and end-of-segment accuracy based on your application.

Files in sample_output/ and ground-truth inputs

• English runs (--lang en, input from memos_en.json):
  • sample_output/en_gpt-live-transcribe_run1_20261023.json and sample_output/en_gpt-live-transcribe_run2_20261023.json: per-memo English transcripts, latency numbers, and missed-name counts for the two English gpt-live-transcribe runs described in Section 5.8.
  • sample_output/en_gpt-live-transcribe_run1_console.txt and sample_output/en_gpt-live-transcribe_run2_console.txt: saved console summaries from those two English runs. These .txt logs were captured when the benchmark script printed Chinese status lines and table headers, so their labels are in Chinese while the run data is for --lang en.
  • memos_en.json: English ground-truth memo scripts, colleague names (Jaxon, Saoirse, Kaitlynn, Aoife, Xochitl, Priya, Kaveh, Mairead, Tadhg), and technical terms (note: its top-level "about" description string is in Chinese).
• Mandarin Chinese runs (--lang zh, input from memos_zh.json):
  • sample_output/zh_gpt-live-transcribe_run1_20261023.json and sample_output/zh_gpt-live-transcribe_run2_20261023.json: per-memo Chinese transcripts, timings, and missed-name counts from the two Chinese gpt-live-transcribe runs.
  • sample_output/zh_gpt-realtime_20261023.json: per-memo Chinese transcripts and timings from the --model gpt-realtime comparison pass.
  • sample_output/zh_gpt-live-transcribe_run1_console.txt, sample_output/zh_gpt-live-transcribe_run2_console.txt, and sample_output/zh_gpt-realtime_console.txt: saved console logs (with Chinese labels) for the three Mandarin Chinese runs.

Using your own data

run.py takes two custom inputs: a directory of <id>.wav recordings and a ground-truth JSON file formatted like memos_en.json:

{"people": {"Colleague Name": "role description"}, "terms": ["Term1", "Term2"],
 "memos": [{"id": "001", "text": "Verbatim transcript of what was spoken in this recording"}]}

Run it with python run.py --lang en --audio my_audio_dir --memos my_memos.json (pass --lang zh for Chinese recordings; zh is the default if --lang is omitted). Both the ASR terms list and the post-ASR correction vocabulary list are built from people and terms. Only memos whose ground-truth text mentions at least one name in people are included in the comparison.

Recordings must be 24 kHz, mono, 16-bit WAV files. Convert phone recordings (such as .m4a) first with ffmpeg:

ffmpeg -i recording.m4a -ar 24000 -ac 1 -sample_fmt s16 recording.wav

Scoring rule: if a vocabulary item appears n times in the ground-truth script and k times in the model transcript, it counts as min(n, k) hits. English uses case-insensitive whole-word matching, and Chinese uses case- and space-insensitive substring matching, matching the scoring in Module 2.

The microphone capture path in live.py was verified on a Mac using a virtual audio device (playing recorded audio into the virtual device and capturing from it), and has not been tested with a live speaker talking into a physical microphone.