README.md

3.9 Self-tuning loop, holdout set and gates (English runs)

About the model name. The code requests deepseek-v4-flash, the name AI Builder Space accepts (see WORKER in lib.py). When DeepSeek released V4.1 Flash on 2026-09-10, it retired V4 Flash, and requests to the old name deepseek-v4-flash are now served by DeepSeek-V4.1-Flash (DeepSeek API Change Log). These runs took place in October 2026, so the text calls the worker deepseek-v4.1-flash.

This folder holds the English eval set for Module 3 and the self-tuning experiment built on it. It supplies:

• the dataset, scoring rule and baseline used in 3.2 (building the eval), and the noise measurements, worst-case items and raw ASR mishearings used in 3.4 and 3.5;
• the self-tuning curve, the three gate designs and the final comparison in 3.9, "The self-tuning loop."

All numbers below come from the English runs, finished on 2026-10-20.

The question

• If a strong model rewrites a cheap model's correction prompt from its failures on a small eval set, how much of the gain carries over to data it has never seen?
• If you put a gate on that loop, how should the gate be set, and can the gate itself block real progress?

Data and metric

Items (items_en.json). 150 items, each with a raw ASR transcript (raw) and the reference text (truth):

• 38 memos (W01–W20, L01–L08, I01–I10; source: memo) from the Module 2 English corpus.
• 112 meeting lines (M000–M111; source: meeting) from the synthetic meeting in 3.7. Each line's audio was transcribed on its own by Grok on AI Builder Space, with language=en and no vocabulary list. The raw results are cached in meeting_lines_raw_en.json.

The truth texts contain 225 mentions of names and terms (memos 53, meeting 172; people 142, terms 83). 39 items contain none.

Names list (names_en.md). 9 people (Jaxon, Saoirse, Kaitlynn, Xochitl, Aoife, Kaveh, Priya, Mairead, Tadhg) and 11 terms (Koyeb, vLLM, KV cache, LoRA, MLX, AI Builder Space, Superlinear, intake, VoiceFlow, Codex, prefill), 20 entries in total. The worker receives this list together with the skill.

Starting skill (skill_v0_en.md, 37 words, 229 characters): "You will receive a raw speech-to-text transcript. Using the list of names and terms below, correct any names and terms that were transcribed wrong. Do not change anything else. Output only the corrected text, with no explanation."

Metric (score_item and aggregate in lib.py). Only the 20 entries in the names list count. For each entry, if the truth has n mentions and the output has k, then correct = min(n, k) and wrong insertions = k − n when k > n. Score = (correct − wrong insertions) / total expected mentions. Matching is case-insensitive and whole-word; multi-word terms also match with a hyphen or without the space ("KV-cache", "KVcache"), and possessives ("Aoife's") count. A score of 0.78 means about 78 net correct per 100 names and terms that should be there. A model that inserts names freely can score below zero.

Baseline and noise (used in 3.2 and 3.4)

baseline.py en → baseline_en.json (one run):

	correct	wrong insertions	score
Raw ASR	101/225	0	0.4489
skill_v0_en.md + deepseek-v4.1-flash	199/225	24	0.7778

All 24 wrong insertions come from meeting lines, and 21 of them from a single item, M036. In 7 of the 150 outputs (M012, M018, M036, M070, M080, M084, M086), the worker treated the transcript as a message to answer: it said there was no transcript to correct and responded to the content instead. On M036 it listed the entire names list while doing so. Without M036, the v0 score would be (199 − 3)/225 = 0.8711.

noise.py en <skill> → noise_en.jsonl (same skill, all 150 items, 3 runs; temperature 0 throughout):

skill	run 1	run 2	run 3	items whose pass/fail flips across runs
v0 (skill_v0_en.md)	0.7244	0.7244	0.6533	3
round-2 skill (runs/en_ungated_1/skill_r02.md, 782 words)	0.76	0.8089	0.7556	1

Together with the baseline run, four v0 runs span 0.6533–0.7778. The round-2 skill is not stable either: one flipping item moves the 150-item total by about 0.05.

Self-tuning without a gate

Setup (autotune.py).

• Worker: deepseek-v4.1-flash on AI Builder Space, temperature 0 (lib.py).
• Tuner: gpt-6.1-sol through a ChatGPT subscription (OAuth), reasoning effort high.
• Meta-prompt (meta_prompt_en.md): the current skill, the names list, the tune-set score, and every tune-set failure with its raw transcript, the worker's output, the expected text, the missed entries and the wrongly added entries. The tuner returns a complete new skill. The meta-prompt contains no warning against overfitting, because a first version written by a reader usually would not have one either.
• Splits: for each seed (1, 2, 3), the 150 items are shuffled with that seed and split into tune 24 / holdout 60 / test 66. The tuner sees only tune-set failures. Holdout plus test make up the 126 unseen items. The same seed produces the same split in every variant (runs/en_*_<seed>/split.json).

Name and term mentions per split:

seed	tune	holdout	test	unseen (holdout + test)
1	39	91	95	186
2	35	95	95	190
3	45	87	93	180

Without a gate (runs/en_ungated_{1,2,3}/), every proposal is accepted for 8 rounds. Because single runs are noisy, each round's skill was run 3 times on the tune set and 3 times on the 126 unseen items (the first run is in log.json; curve_reeval.py adds two more, stored in curve_reeval_en.json). curve_summary.py averages them over the three seeds into curve_ungated_en.json:

round	tune mean	unseen mean	per-seed means min–max	single runs min–max	prompt words (mean)
0	0.673	0.743	0.636–0.860	0.581–0.868	37
1	0.930	0.867	0.842–0.886	0.812–0.889	409
2	0.982	0.825	0.719–0.884	0.597–0.911	698
3	1.000	0.855	0.781–0.894	0.737–0.900	902
4	0.975	0.850	0.776–0.889	0.769–0.889	958
5	1.000	0.830	0.728–0.884	0.661–0.889	1,110
6	0.998	0.834	0.746–0.882	0.726–0.889	1,178
7	1.000	0.868	0.833–0.886	0.801–0.894	1,258
8	1.000	0.829	0.751–0.877	0.737–0.879	1,237

The gain on unseen items happens in round 1 (0.743 → 0.867) and stops there. Rounds 2–8 move between 0.825 and 0.868 while the tune set sits at 0.975–1.0 and the prompt grows from 409 to about 1,240 words (229 → 7,975 characters). Seed 1 shows the overfitting most clearly: unseen 0.636 → 0.842 (round 1) → 0.719 (round 2), then 0.728–0.833, while its tune score is 1.0 from round 3 on.

Copying from the tune set (copy_check.py en). After removing the 20 vocabulary entries from both texts, a tune item counts as copied if any 8 consecutive words of its raw transcript or its truth appear in the round-8 skill. Seed 1: 2 of 24 tune items (W10, M053); seed 2: 3 (W20, M018, W17); seed 3: 2 (M101, L04). Unseen items, checked the same way as a control: 0 in every seed. An example from runs/en_ungated_2/skill_r08.md, where W20 is a tune item: "For example, "remind the solicitor the app store screenshots need to be redone" becomes "remind Xochitl the app store screenshots need to be redone.""

Gates

Same seeds and splits as above. Each round's proposal takes effect only if it passes the gate; otherwise the current skill stays, and the tuner is told: "Your previous proposal was rejected by the gate: on a holdout set you cannot see, it lowered the score or broke items that used to be right. The failures below are still for the version currently in force; try a different approach." No holdout content is shown. The loop stops after 3 rejections in a row, or after 8 rounds.

An item counts as right when it has no missed and no wrongly added entries. Broken items were right under the current skill and are wrong under the proposal; fixed items are the reverse. The current skill's holdout numbers are the ones measured when it was accepted (or at round 0); they are not re-measured each round.

Gate	Flag	Rule
1, strict	--gate-strict	Holdout score does not drop, and no holdout item breaks.
2, single baseline	--gate	Holdout score rises by at least 0.01 (MARGIN), and broken items do not outnumber fixed ones.
3, averaged baseline	--gate-avg	Same as gate 2, but the holdout set is run twice and the two-run means are compared. Broken and fixed items are counted on the first run.

With 87–95 mentions in a holdout split, one mention is worth 0.0105–0.0115, so a margin of 0.01 means at least one more net correct mention. Every proposal in gates 1 and 2 was judged on a single run, so part of what the gates react to is noise.

Results (runs/en_{gatedstrict,gated,gatedavg}_{1,2,3}/log.json):

• Gate 1, strict: 12 proposals, 1 accepted (seed 1, round 3; 298 words; broke 0, fixed 3). 11 of the 12 broke 1–3 holdout items that had been right. 6 of the 12 raised the holdout score above the current version, and 5 of those 6 still broke 1–3 items and were rejected. Seeds 2 and 3 stopped at v0 after 3 rejections.
• Gate 2, single baseline: 14 proposals, 5 accepted (seed 1 round 1; seed 2 rounds 1 and 2; seed 3 rounds 1 and 2). v0's single holdout runs were low: seed 1 0.6264, seed 2 0.4737, seed 3 0.8621, so the first proposal cleared the bar easily in every seed (seed 2's round 1 broke 3 items but fixed 4). Final skills: 354 / 660 / 682 words.
• Gate 3, averaged baseline: 15 proposals, 4 accepted (seed 1 rounds 1 and 4; seed 2 rounds 1 and 2). Seed 3 never moved: v0 scored 0.8736 on both holdout runs, well above its average elsewhere (test mean 0.5591). Its three proposals averaged 0.8563, 0.8794 and 0.8506; 0.8794 was higher but under the 0.8836 bar. Seed 3 stopped at v0 after round 3.

Final comparison on the test set

final_eval.py en → final_eval_en.json: each stopping skill is run 3 times on that seed's 66-item test split, which no decision used, and the three scores are averaged.

version	seed 1	seed 2	seed 3	mean	words (mean)
v0	0.6105	0.8421	0.5591	0.671	37
ungated, round 2	0.6632	0.8632	0.8279	0.785	698
ungated, round 8	0.7509	0.8632	0.8316	0.815	1,237
gate 1, strict	0.7228	0.8316	0.5233	0.693	124 (298 / 37 / 37)
gate 2, single baseline	0.8491	0.8421	0.7921	0.828	565 (354 / 660 / 682)
gate 3, averaged baseline	0.8421	0.8316	0.4875	0.720	429 (636 / 613 / 37)

Single test runs across this table span 0.3011–0.9032; v0's runs alone span 0.3333–0.8526. Gate 2 gave the highest mean with prompts less than half the length of eight ungated rounds. On seeds 1 and 2, gate 3 is close to gate 2; its mean is pulled down by seed 3 staying at v0, which scored 0.8736 on the holdout set and 0.487 on test.

Files

Data and prompts:

• items_en.json: the 150 items (id, source, raw, truth, plus speaker for meeting lines).
• meeting_lines_raw_en.json: cached raw Grok transcripts of the 112 meeting lines.
• names_en.md, skill_v0_en.md, meta_prompt_en.md: the names list, the starting skill and the tuner's meta-prompt.

Code:

• lib.py: scoring (score_item, aggregate) and the worker call.
• build_items.py: builds items_en.json.
• baseline.py, noise.py, autotune.py, final_eval.py, curve_reeval.py, curve_summary.py, copy_check.py: the experiment steps described above.
• run_en.sh: runs the full English experiment in order.

Results:

• baseline_en.json: per-item outputs and scores for the v0 baseline.
• noise_en.jsonl: the two noise measurements.
• curve_reeval_en.json, curve_ungated_en.json: the ungated curve, per seed and averaged.
• final_eval_en.json: the final test-set comparison.
• runs/en_<variant>_<seed>/ for variants ungated, gatedstrict, gated, gatedavg and seeds 1–3: split.json (item ids per split), skill_rNN.md (the skill proposed in each round; skill_r00.md is v0), results_rNN.json (per-item outputs and scores for each round), log.json (scores per round and, for gated runs, accepted, regressions, fixed, and holdout_mean where applicable), and final_skill.md (the skill in force when the run stopped).

How to rerun

Run the commands from this folder. The scripts need Python with httpx, and an AI_BUILDER_TOKEN for the worker calls on AI Builder Space.

export AI_BUILDER_TOKEN=...

python baseline.py en                       # -> baseline_en.json
python noise.py en skill_v0_en.md           # appends one line to noise_en.jsonl

# Self-tuning, one seed at a time (seed = 1, 2 or 3)
python autotune.py en 1                     # no gate, 8 rounds  -> runs/en_ungated_1/
python autotune.py en 1 --gate-strict       # gate 1             -> runs/en_gatedstrict_1/
python autotune.py en 1 --gate              # gate 2             -> runs/en_gated_1/
python autotune.py en 1 --gate-avg          # gate 3             -> runs/en_gatedavg_1/

python noise.py en runs/en_ungated_1/skill_r02.md
python final_eval.py en                     # needs the ungated runs for all three seeds
python curve_reeval.py en                   # -> curve_reeval_en.json
python curve_summary.py en                  # -> curve_ungated_en.json
python copy_check.py en                     # copying check on the round-8 ungated skills

run_en.sh runs the noise, autotune (three seeds in parallel per variant), final and curve steps in order and writes each step's output to runs_en_<step>.out. It skips any step that already has a done_en_<step> marker file; those markers are not included here, so a fresh copy runs everything. Set P at the top of the script to your Python interpreter before running it. It does not run baseline.py, curve_summary.py or copy_check.py.

Notes:

• The tuner client is local to the author. autotune.py calls gpt-6.1-sol through a ChatGPT-subscription OAuth client that lives outside this folder and is added to the import path at the top of the file. To rerun, replace request_result(prompt, model=..., reasoning_effort="high", timeout=...) with any OpenAI-compatible client that returns the answer text. A different tuner will produce different skills.
• The worker can be changed with the WORKER_MODEL environment variable (default deepseek-v4-flash).
• Rerunning overwrites results. autotune.py writes into runs/en_<variant>_<seed>/, and baseline.py, final_eval.py and curve_reeval.py overwrite their JSON files. Copy the folder first if you want to keep the published results for comparison. Even at temperature 0 the worker is not deterministic, so expect your numbers to differ from ours by about the noise shown above.
• Worker calls. Every evaluation in autotune.py runs all 150 items (tune, holdout and test; the per-round test score is logged but no decision reads it), plus a second 60-item holdout run under --gate-avg. An ungated seed therefore makes 9 × 150 worker calls. The tuner runs on the subscription and is not billed per token.
• Rebuilding the items with python build_items.py en is optional, since items_en.json is included. It reads the Module 2 English corpus (published at https://www.ai-builders.com/resources/aibuilder2027/m02/en/experiments/corpus/) from ../../experiments_en/corpus/ relative to this folder and reuses the cached transcripts in meeting_lines_raw_en.json; it only calls the ASR API for meeting lines missing from that cache, which requires the per-line meeting audio from 3_7_formulation (build_meeting.py en there).