# 3.9 Self-tuning loop, holdout set and gates (English runs)

> **About the model name.** The code requests `deepseek-v4-flash`, the name AI Builder Space accepts (see `WORKER` in `lib.py`). When DeepSeek released V4.1 Flash on 2026-09-10, it retired V4 Flash, and requests to the old name `deepseek-v4-flash` are now served by DeepSeek-V4.1-Flash (DeepSeek API Change Log). These runs took place in October 2026, so the text calls the worker **deepseek-v4.1-flash**.

This folder holds the English eval set for Module 3 and the self-tuning experiment built on it. It supplies:

- the dataset, scoring rule and baseline used in 3.2 (building the eval), and the noise measurements, worst-case items and raw ASR mishearings used in 3.4 and 3.5;
- the self-tuning curve, the three gate designs and the final comparison in 3.9, "The self-tuning loop."

All numbers below come from the English runs, finished on 2026-10-20.

## The question

- If a strong model rewrites a cheap model's correction prompt from its failures on a small eval set, how much of the gain carries over to data it has never seen?
- If you put a gate on that loop, how should the gate be set, and can the gate itself block real progress?

## Data and metric

**Items (`items_en.json`).** 150 items, each with a raw ASR transcript (`raw`) and the reference text (`truth`):

- 38 memos (W01–W20, L01–L08, I01–I10; `source: memo`) from the Module 2 English corpus.
- 112 meeting lines (M000–M111; `source: meeting`) from the synthetic meeting in 3.7. Each line's audio was transcribed on its own by Grok on AI Builder Space, with `language=en` and no vocabulary list. The raw results are cached in `meeting_lines_raw_en.json`.

The truth texts contain 225 mentions of names and terms (memos 53, meeting 172; people 142, terms 83). 39 items contain none.

**Names list (`names_en.md`).** 9 people (Jaxon, Saoirse, Kaitlynn, Xochitl, Aoife, Kaveh, Priya, Mairead, Tadhg) and 11 terms (Koyeb, vLLM, KV cache, LoRA, MLX, AI Builder Space, Superlinear, intake, VoiceFlow, Codex, prefill), 20 entries in total. The worker receives this list together with the skill.

**Starting skill (`skill_v0_en.md`, 37 words, 229 characters):** "You will receive a raw speech-to-text transcript. Using the list of names and terms below, correct any names and terms that were transcribed wrong. Do not change anything else. Output only the corrected text, with no explanation."

**Metric (`score_item` and `aggregate` in `lib.py`).** Only the 20 entries in the names list count. For each entry, if the truth has *n* mentions and the output has *k*, then correct = min(*n*, *k*) and wrong insertions = *k* − *n* when *k* > *n*. Score = (correct − wrong insertions) / total expected mentions. Matching is case-insensitive and whole-word; multi-word terms also match with a hyphen or without the space ("KV-cache", "KVcache"), and possessives ("Aoife's") count. A score of 0.78 means about 78 net correct per 100 names and terms that should be there. A model that inserts names freely can score below zero.

## Baseline and noise (used in 3.2 and 3.4)

`baseline.py en` → `baseline_en.json` (one run):

| | correct | wrong insertions | score |
|---|---|---|---|
| Raw ASR | 101/225 | 0 | 0.4489 |
| `skill_v0_en.md` + deepseek-v4.1-flash | 199/225 | 24 | 0.7778 |

All 24 wrong insertions come from meeting lines, and 21 of them from a single item, M036. In 7 of the 150 outputs (M012, M018, M036, M070, M080, M084, M086), the worker treated the transcript as a message to answer: it said there was no transcript to correct and responded to the content instead. On M036 it listed the entire names list while doing so. Without M036, the v0 score would be (199 − 3)/225 = 0.8711.

`noise.py en <skill>` → `noise_en.jsonl` (same skill, all 150 items, 3 runs; temperature 0 throughout):

| skill | run 1 | run 2 | run 3 | items whose pass/fail flips across runs |
|---|---|---|---|---|
| v0 (`skill_v0_en.md`) | 0.7244 | 0.7244 | 0.6533 | 3 |
| round-2 skill (`runs/en_ungated_1/skill_r02.md`, 782 words) | 0.76 | 0.8089 | 0.7556 | 1 |

Together with the baseline run, four v0 runs span 0.6533–0.7778. The round-2 skill is not stable either: one flipping item moves the 150-item total by about 0.05.

## Self-tuning without a gate

**Setup (`autotune.py`).**

- Worker: deepseek-v4.1-flash on AI Builder Space, temperature 0 (`lib.py`).
- Tuner: `gpt-6.1-sol` through a ChatGPT subscription (OAuth), reasoning effort `high`.
- Meta-prompt (`meta_prompt_en.md`): the current skill, the names list, the tune-set score, and every tune-set failure with its raw transcript, the worker's output, the expected text, the missed entries and the wrongly added entries. The tuner returns a complete new skill. The meta-prompt contains no warning against overfitting, because a first version written by a reader usually would not have one either.
- Splits: for each seed (1, 2, 3), the 150 items are shuffled with that seed and split into **tune 24 / holdout 60 / test 66**. The tuner sees only tune-set failures. Holdout plus test make up the **126 unseen items**. The same seed produces the same split in every variant (`runs/en_*_<seed>/split.json`).

Name and term mentions per split:

| seed | tune | holdout | test | unseen (holdout + test) |
|---|---|---|---|---|
| 1 | 39 | 91 | 95 | 186 |
| 2 | 35 | 95 | 95 | 190 |
| 3 | 45 | 87 | 93 | 180 |

Without a gate (`runs/en_ungated_{1,2,3}/`), every proposal is accepted for 8 rounds. Because single runs are noisy, each round's skill was run 3 times on the tune set and 3 times on the 126 unseen items (the first run is in `log.json`; `curve_reeval.py` adds two more, stored in `curve_reeval_en.json`). `curve_summary.py` averages them over the three seeds into `curve_ungated_en.json`:

| round | tune mean | unseen mean | per-seed means min–max | single runs min–max | prompt words (mean) |
|---|---|---|---|---|---|
| 0 | 0.673 | 0.743 | 0.636–0.860 | 0.581–0.868 | 37 |
| 1 | 0.930 | 0.867 | 0.842–0.886 | 0.812–0.889 | 409 |
| 2 | 0.982 | 0.825 | 0.719–0.884 | 0.597–0.911 | 698 |
| 3 | 1.000 | 0.855 | 0.781–0.894 | 0.737–0.900 | 902 |
| 4 | 0.975 | 0.850 | 0.776–0.889 | 0.769–0.889 | 958 |
| 5 | 1.000 | 0.830 | 0.728–0.884 | 0.661–0.889 | 1,110 |
| 6 | 0.998 | 0.834 | 0.746–0.882 | 0.726–0.889 | 1,178 |
| 7 | 1.000 | 0.868 | 0.833–0.886 | 0.801–0.894 | 1,258 |
| 8 | 1.000 | 0.829 | 0.751–0.877 | 0.737–0.879 | 1,237 |

The gain on unseen items happens in round 1 (0.743 → 0.867) and stops there. Rounds 2–8 move between 0.825 and 0.868 while the tune set sits at 0.975–1.0 and the prompt grows from 409 to about 1,240 words (229 → 7,975 characters). Seed 1 shows the overfitting most clearly: unseen 0.636 → 0.842 (round 1) → 0.719 (round 2), then 0.728–0.833, while its tune score is 1.0 from round 3 on.

**Copying from the tune set (`copy_check.py en`).** After removing the 20 vocabulary entries from both texts, a tune item counts as copied if any 8 consecutive words of its raw transcript or its truth appear in the round-8 skill. Seed 1: 2 of 24 tune items (W10, M053); seed 2: 3 (W20, M018, W17); seed 3: 2 (M101, L04). Unseen items, checked the same way as a control: 0 in every seed. An example from `runs/en_ungated_2/skill_r08.md`, where W20 is a tune item: "For example, "remind the solicitor the app store screenshots need to be redone" becomes "remind Xochitl the app store screenshots need to be redone.""

## Gates

Same seeds and splits as above. Each round's proposal takes effect only if it passes the gate; otherwise the current skill stays, and the tuner is told: "Your previous proposal was rejected by the gate: on a holdout set you cannot see, it lowered the score or broke items that used to be right. The failures below are still for the version currently in force; try a different approach." No holdout content is shown. The loop stops after 3 rejections in a row, or after 8 rounds.

An item counts as *right* when it has no missed and no wrongly added entries. *Broken* items were right under the current skill and are wrong under the proposal; *fixed* items are the reverse. The current skill's holdout numbers are the ones measured when it was accepted (or at round 0); they are not re-measured each round.

| Gate | Flag | Rule |
|---|---|---|
| 1, strict | `--gate-strict` | Holdout score does not drop, and no holdout item breaks. |
| 2, single baseline | `--gate` | Holdout score rises by at least 0.01 (`MARGIN`), and broken items do not outnumber fixed ones. |
| 3, averaged baseline | `--gate-avg` | Same as gate 2, but the holdout set is run twice and the two-run means are compared. Broken and fixed items are counted on the first run. |

With 87–95 mentions in a holdout split, one mention is worth 0.0105–0.0115, so a margin of 0.01 means at least one more net correct mention. Every proposal in gates 1 and 2 was judged on a single run, so part of what the gates react to is noise.

Results (`runs/en_{gatedstrict,gated,gatedavg}_{1,2,3}/log.json`):

- **Gate 1, strict:** 12 proposals, 1 accepted (seed 1, round 3; 298 words; broke 0, fixed 3). 11 of the 12 broke 1–3 holdout items that had been right. 6 of the 12 raised the holdout score above the current version, and 5 of those 6 still broke 1–3 items and were rejected. Seeds 2 and 3 stopped at v0 after 3 rejections.
- **Gate 2, single baseline:** 14 proposals, 5 accepted (seed 1 round 1; seed 2 rounds 1 and 2; seed 3 rounds 1 and 2). v0's single holdout runs were low: seed 1 0.6264, seed 2 0.4737, seed 3 0.8621, so the first proposal cleared the bar easily in every seed (seed 2's round 1 broke 3 items but fixed 4). Final skills: 354 / 660 / 682 words.
- **Gate 3, averaged baseline:** 15 proposals, 4 accepted (seed 1 rounds 1 and 4; seed 2 rounds 1 and 2). Seed 3 never moved: v0 scored 0.8736 on both holdout runs, well above its average elsewhere (test mean 0.5591). Its three proposals averaged 0.8563, 0.8794 and 0.8506; 0.8794 was higher but under the 0.8836 bar. Seed 3 stopped at v0 after round 3.

## Final comparison on the test set

`final_eval.py en` → `final_eval_en.json`: each stopping skill is run 3 times on that seed's 66-item test split, which no decision used, and the three scores are averaged.

| version | seed 1 | seed 2 | seed 3 | mean | words (mean) |
|---|---|---|---|---|---|
| v0 | 0.6105 | 0.8421 | 0.5591 | 0.671 | 37 |
| ungated, round 2 | 0.6632 | 0.8632 | 0.8279 | 0.785 | 698 |
| ungated, round 8 | 0.7509 | 0.8632 | 0.8316 | 0.815 | 1,237 |
| gate 1, strict | 0.7228 | 0.8316 | 0.5233 | 0.693 | 124 (298 / 37 / 37) |
| gate 2, single baseline | 0.8491 | 0.8421 | 0.7921 | 0.828 | 565 (354 / 660 / 682) |
| gate 3, averaged baseline | 0.8421 | 0.8316 | 0.4875 | 0.720 | 429 (636 / 613 / 37) |

Single test runs across this table span 0.3011–0.9032; v0's runs alone span 0.3333–0.8526. Gate 2 gave the highest mean with prompts less than half the length of eight ungated rounds. On seeds 1 and 2, gate 3 is close to gate 2; its mean is pulled down by seed 3 staying at v0, which scored 0.8736 on the holdout set and 0.487 on test.

## Files

Data and prompts:

- `items_en.json`: the 150 items (`id`, `source`, `raw`, `truth`, plus `speaker` for meeting lines).
- `meeting_lines_raw_en.json`: cached raw Grok transcripts of the 112 meeting lines.
- `names_en.md`, `skill_v0_en.md`, `meta_prompt_en.md`: the names list, the starting skill and the tuner's meta-prompt.

Code:

- `lib.py`: scoring (`score_item`, `aggregate`) and the worker call.
- `build_items.py`: builds `items_en.json`.
- `baseline.py`, `noise.py`, `autotune.py`, `final_eval.py`, `curve_reeval.py`, `curve_summary.py`, `copy_check.py`: the experiment steps described above.
- `run_en.sh`: runs the full English experiment in order.

Results:

- `baseline_en.json`: per-item outputs and scores for the v0 baseline.
- `noise_en.jsonl`: the two noise measurements.
- `curve_reeval_en.json`, `curve_ungated_en.json`: the ungated curve, per seed and averaged.
- `final_eval_en.json`: the final test-set comparison.
- `runs/en_<variant>_<seed>/` for variants `ungated`, `gatedstrict`, `gated`, `gatedavg` and seeds 1–3: `split.json` (item ids per split), `skill_rNN.md` (the skill proposed in each round; `skill_r00.md` is v0), `results_rNN.json` (per-item outputs and scores for each round), `log.json` (scores per round and, for gated runs, `accepted`, `regressions`, `fixed`, and `holdout_mean` where applicable), and `final_skill.md` (the skill in force when the run stopped).

## How to rerun

Run the commands from this folder. The scripts need Python with `httpx`, and an `AI_BUILDER_TOKEN` for the worker calls on AI Builder Space.

```bash
export AI_BUILDER_TOKEN=...

python baseline.py en                       # -> baseline_en.json
python noise.py en skill_v0_en.md           # appends one line to noise_en.jsonl

# Self-tuning, one seed at a time (seed = 1, 2 or 3)
python autotune.py en 1                     # no gate, 8 rounds  -> runs/en_ungated_1/
python autotune.py en 1 --gate-strict       # gate 1             -> runs/en_gatedstrict_1/
python autotune.py en 1 --gate              # gate 2             -> runs/en_gated_1/
python autotune.py en 1 --gate-avg          # gate 3             -> runs/en_gatedavg_1/

python noise.py en runs/en_ungated_1/skill_r02.md
python final_eval.py en                     # needs the ungated runs for all three seeds
python curve_reeval.py en                   # -> curve_reeval_en.json
python curve_summary.py en                  # -> curve_ungated_en.json
python copy_check.py en                     # copying check on the round-8 ungated skills
```

`run_en.sh` runs the noise, autotune (three seeds in parallel per variant), final and curve steps in order and writes each step's output to `runs_en_<step>.out`. It skips any step that already has a `done_en_<step>` marker file; those markers are not included here, so a fresh copy runs everything. Set `P` at the top of the script to your Python interpreter before running it. It does not run `baseline.py`, `curve_summary.py` or `copy_check.py`.

Notes:

- **The tuner client is local to the author.** `autotune.py` calls `gpt-6.1-sol` through a ChatGPT-subscription OAuth client that lives outside this folder and is added to the import path at the top of the file. To rerun, replace `request_result(prompt, model=..., reasoning_effort="high", timeout=...)` with any OpenAI-compatible client that returns the answer text. A different tuner will produce different skills.
- **The worker can be changed** with the `WORKER_MODEL` environment variable (default `deepseek-v4-flash`).
- **Rerunning overwrites results.** `autotune.py` writes into `runs/en_<variant>_<seed>/`, and `baseline.py`, `final_eval.py` and `curve_reeval.py` overwrite their JSON files. Copy the folder first if you want to keep the published results for comparison. Even at temperature 0 the worker is not deterministic, so expect your numbers to differ from ours by about the noise shown above.
- **Worker calls.** Every evaluation in `autotune.py` runs all 150 items (tune, holdout and test; the per-round test score is logged but no decision reads it), plus a second 60-item holdout run under `--gate-avg`. An ungated seed therefore makes 9 × 150 worker calls. The tuner runs on the subscription and is not billed per token.
- **Rebuilding the items** with `python build_items.py en` is optional, since `items_en.json` is included. It reads the Module 2 English corpus (published at [https://www.ai-builders.com/resources/aibuilder2027/m02/en/experiments/corpus/](https://www.ai-builders.com/resources/aibuilder2027/m02/en/experiments/corpus/)) from `../../experiments_en/corpus/` relative to this folder and reuses the cached transcripts in `meeting_lines_raw_en.json`; it only calls the ASR API for meeting lines missing from that cache, which requires the per-line meeting audio from `3_7_formulation` (`build_meeting.py en` there).