README.md

2.4 Skill: success criteria and boundaries (English edition)

Twin of experiments/2_4_skill/. Does a one-page skill change what a model does when you only say "clean this up"?

How to run

op run --env-file experiments/aib.env -- .venv/bin/python experiments_en/2_4_skill/run.py
.venv/bin/python experiments_en/2_4_skill/run.py --rescore   # re-score the saved outputs, no API calls

The API is AI Builder Space, and the models are deepseek-v4.1-flash and gemini-3.8-flash. The input is ../corpus/transcripts_grok.json (raw Grok ASR, no vocabulary), for the 28 work and life memos W01–W20 and L01–L08. Each memo gets one fresh call at temperature 0. The design is at the top of run.py. Raw outputs are in out_<model>_<condition>.json, and the summary is in results.json. Cost: $0.29 for 112 calls (see ../COSTS.md).

Design

Both conditions get the same names list, names.md, built from the corpus cast and terms. The only variable is the skill, cleanup_skill.md, which has five rules:

• Fix punctuation and sentence breaks, and fix only obvious mishearings.
• Spell names and terms the way names.md spells them.
• Write times as digits with a.m. or p.m., and write other spoken numbers as digits: "10 a.m.", "8 p.m.", "40 cents", "a 10-minute demo".
• Keep the speaker's wording and habits, including "I mean". Don't turn a casual note into formal or corporate prose.
• Don't merge sentences, drop details or summarize. Output only the cleaned text.

The ask: "Clean up this voice memo transcript. Output only the cleaned-up full text." This mirrors the Chinese ask, 帮我整理一下这段语音转写，只输出整理好的全文.

Scoring is by regex. A first-try pass means all three checks hold:

• keep_tic: "I mean" appears at least as many times as in the transcript the model was given. It appears in 11 of the 28 memos.
• numbers: the house style for numbers. The Grok transcripts still spell out five numbers, and each must become digits with the word gone:
  • W02 "ten in the morning" → 10 a.m.
  • W08 "eight at night" → 8 p.m.
  • W12 "ten minute demo" → 10-minute
  • L02 "around nine" → 9
  • L05 "forty cents" → 40 cents

Across all 28 memos, no time may appear in another format ("10 AM", "8:00 PM", "10am").

• no_rewrite: the output's word count is within 85–115% of the transcript's.

The Chinese run had a fourth check, that fillers (嗯/啊/呃) are removed. The English ASR strips um and uh, so that check has nothing to measure. The Chinese CJK-spacing check became the numbers check. The Chinese scorer counted 就是说 against the script, which produced 3 artefactual failures where the ASR had dropped a word. The English scorer counts "I mean" against the transcript.

Before using the numbers rule, we checked that the Grok transcripts still spell out these numbers. The compat (whisper) transcripts already write digits, so they could not be used here.

Diagnostics are recorded but are not part of the pass:

• names right (31 mentions);
• preamble or markdown in the output;
• contractions;
• casual markers kept ("so", "honestly", "like", "kind of", and so on);
• word-level similarity to the transcript.

Results (2026-10-14)

These are the violations, failed memos only.

• deepseek-v4.1-flash, no skill:
  • "I mean" deleted in L03.
  • Numbers missed in W02, W08, W12, L02 and L05. In W08 the model converted the time to its own format, "8 PM"; it left the other four as words.
• gemini-3.8-flash, no skill:
  • "I mean" deleted in W03, W15, W20, L03, L05 and L07.
  • Numbers missed in W02 ("10:00 AM"), W08 ("8:00 PM"), W12 ("ten-minute"), L02 ("nine") and L05 ("forty cents").

What the models do by default.

1. They delete the verbal signature. Gemini dropped "I mean" in 6 of the 11 memos that have it, and deepseek in 1. Both models dropped it in L03, replacing it with a dash.
2. They write numbers in their own style, not yours. With no skill, each model left some numbers as words and converted others into its own format ("10:00 AM", "8 PM"). Neither model produced "10 a.m." without being told.
3. Light register drift, gemini only. In L05, "the Costco one is like forty cents cheaper" became "Costco is about forty cents cheaper". In L07, "The doctor said more vegetables… get the pistachios" became "The doctor said to eat more vegetables… I should get the pistachios… Also, no more snacks".

What they do not do in English: memo-speak and summarizing. No output fell outside the 85–115% length band. No output had a preamble or markdown. Mean word-level similarity to the transcript was 0.95 for gemini and 0.97 for deepseek. The Chinese failure, where a casual note was rewritten into a 周报 (corporate weekly report), does not reproduce.

Names, a diagnostic only, since both conditions have the list:

• Gemini recovered Aoife from "If you ask me today" in W03.
• Deepseek confidently turned W20's "the solicitor" into "Saoirse". The right name is Xochitl.
• Because names.md lists "terminal" as a term, deepseek also changed W03's "from the shell" to "from the terminal", even with no skill. The Chinese names list with "Apple" had the same side effect.

Before/after pairs (quotable)

W01, the Chinese chapter's first example. In English it shows only name fixes and one sentence-break fix, with no register change:

Grok: So this afternoon, I went over the deployment with Jackson and Sorsa. I mean, the service is sitting on Coyab right now, and she says the cold start is way too slow, so she wants to spin up ValM on our own box. I said, let's not rush it next Wednesday. We'll look at the KV cache hit rate with Prea first and then decide.

gemini-3.8-flash, no skill: So this afternoon, I went over the deployment with Jaxon and Saoirse. I mean, the service is sitting on Koyeb right now, and she says the cold start is way too slow, so she wants to spin up vLLM on our own box. I said let's not rush it. Next Wednesday, we'll look at the KV cache hit rate with Priya first and then decide.

L03, the kindergarten snack share. Both models delete "I mean" by default:

Grok: Reminder, the kids' kindergarten is doing a snack share next week. The teacher said cut up fruit or crackers and no peanuts. Hard-boiled eggs are fine too. I mean, just peel them first so there's no shell left on. I need to get it all ready Thursday night.

deepseek-v4.1-flash, no skill: Reminder: the kids' kindergarten is doing a snack share next week. The teacher said cut-up fruit or crackers, and no peanuts. Hard-boiled eggs are fine too — just peel them first so there's no shell left on. I need to get it all ready Thursday night.

gemini-3.8-flash, no skill: Reminder: The kids' kindergarten is doing a snack share next week. The teacher said cut-up fruit or crackers, and no peanuts. Hard-boiled eggs are fine too—just peel them first so there's no shell left on. I need to get it all ready Thursday night.

W02, the number style:

Grok: Note to self, tomorrow at ten in the morning I'm going through the voice flow requirements with Caitlin.

gemini, no skill: Note to self: Tomorrow at 10:00 AM, I'm going through the VoiceFlow requirements with Kaitlynn.

gemini, skill: Note to self: tomorrow at 10 a.m. I'm going through the VoiceFlow requirements with Kaitlynn.