# 2.6 Flow-back: a single edit overfits; batch history draws the boundary (English edition)

Twin of `experiments/2_5_overgeneralize/`, with the homonym **shell** in place of 苹果. `PILOT.md` explains the choice. The author writes the command-line sense of "shell" as "terminal", the word on his screen. Everyday senses stay "shell".

## How to run

```bash
op run --env-file experiments/aib.env -- .venv/bin/python experiments_en/2_6_overgeneralize/run.py
op run --env-file experiments/aib.env -- .venv/bin/python experiments_en/2_6_overgeneralize/run.py --batch-only   # step 3 only
.venv/bin/python experiments_en/2_6_overgeneralize/run.py --rescore   # re-score step 2 from saved outputs, no API calls
```

The API is AI Builder Space, and the models are `deepseek-v4.1-flash` and `gemini-3.8-flash`. The input is `../corpus/transcripts_grok.json`, and the names list is `../2_4_skill/names.md`, which, like the Chinese list with Apple, includes "terminal" among the terms. Everything is in `results.json`:

- `proposals`: step 1;
- `downstream`: step 2, with full outputs;
- `batch_proposals`: step 3;
- `hand_scored`: the hand classification of every rule.

Cost: $0.20 for 86 calls. The pilot (`PILOT.md`, `pilot_homonym.py`) cost a further $0.52.

## Design (same run counts as the Chinese run)

1. **Single-edit distillation.** The model sees only W06 before and after the one edit ("If you run intake from the shell instead of from the editor" → "…from the terminal…"). It is asked to "Turn this edit into one rule for my cleanup rule file, so it is applied automatically whenever I clean up a transcript. Output only the rule, in one sentence." 5 runs per model, temperature 1.
2. **Downstream damage.** Clean up 11 memos at temperature 0, with `names.md` plus one of three rules:
   - none;
   - global: `Rule: in transcripts, always write "shell" as "terminal".`
   - scoped: `Rule: write "shell" as "terminal" only when it means the command line on a computer; in any everyday sense, keep "shell".`

   The memos are the work positives W03, W13 and W20 (3 command-line mentions, which should become "terminal") and the life negatives L01–L08 (9 everyday mentions, which should stay "shell"). The 9 everyday mentions:
   - the Shell gas station in L01 and L05;
   - a beach shell in L02;
   - an eggshell in L03;
   - a taco shell, twice, in L04;
   - shell pasta in L06;
   - a pistachio shell in L07;
   - a hermit crab's shell in L08.

   Each mention is located by the words around it ("stop at the ___", "that ___ pasta", and so on), so a model's commentary is never counted as a conversion.
3. **Batch-history distillation.** All 12 memos in the 38-memo history that contain "shell" go to the model at once. That is 4 work memos (W03, W06, W13, W20), shown before and after the edit to "terminal", and 8 life memos (L01–L08), shown unedited. The model is asked for one rule. 5 runs per model, temperature 1.

Rules are classified by hand:
- **scoped (command line)**: the condition picks out the command-line sense.
- **over-narrow**: there is a condition, but it would miss the other command-line mentions.
- **global**: no condition.

## Results (2026-10-14)

### Step 1, single edit (verbatim, 5 runs each)

deepseek-v4.1-flash: 2 global, 1 over-narrow and 2 scoped.

1. Replace the word "shell" with "terminal" when it refers to running a command from the command line. *(scoped)*
2. Replace "shell" with "terminal" when referring to running intake. *(over-narrow)*
3. Replace "shell" with "terminal". *(global)*
4. Replace "shell" with "terminal". *(global)*
5. Replace "shell" with "terminal" when it refers to running a command. *(scoped)*

gemini-3.8-flash: 1 global and 4 scoped.

1. Replace "the shell" with "the terminal". *(global)*
2. Replace "shell" with "terminal" when referring to the command-line environment.
3. Replace "shell" with "terminal" when referring to the command-line interface.
4. Replace the word "shell" with "terminal" when referring to the command line.
5. Replace "shell" with "terminal" when referring to the command-line interface.

### Step 2, downstream

| rule | model | command-line "shell" → terminal | everyday "shell" wrongly → terminal | everyday "shell" kept |
|---|---|---|---|---|
| none | deepseek | 0/3 | 0/9 | 9/9 |
| none | gemini | 0/3 | 0/9 | 9/9 |
| **global** | deepseek | 3/3 | **9/9** | 0/9 |
| **global** | gemini | 3/3 | **9/9** | 0/9 |
| scoped | deepseek | 3/3 | 0/9 | 9/9 |
| scoped | gemini | 3/3 | 0/9 | 9/9 |

**Resisted: 0.** Under the global rule, neither model left a single everyday mention alone. Deepseek even announced in L01 "per your rule, 'shell' → 'terminal.'" and then wrote "stop at the terminal and fill up". In the pilot, deepseek once refused to apply the rule; that did not happen at temperature 0 on this corpus.

Verbatim absurd outputs under the global rule, from gemini-3.8-flash. deepseek-v4.1-flash makes the same 9 conversions:

- L01: "On the way home, stop at the terminal and fill up, then grab eggs and milk."
- L02: "Bring the kids, they can each find a terminal to take home"
- L03: "Hard-boiled eggs are fine too. I mean, just peel them first so there's no terminal left on."
- L04: "the trick is to warm each terminal in the oven for a couple of minutes so it doesn't crack, and then fill the terminal right before you eat."
- L05: "Gas at the Terminal by the school went up again" (deepseek: "Gas at the terminal by the school")
- L06: "some of that terminal pasta for the kids"
- L07: "get the pistachios that are still in the terminal"
- L08: "why a hermit crab has to move into a new terminal"

Other notes:

- With no rule, deepseek did not follow "output only the cleaned-up text" in 6 of the 11 memos. In W03, W20, L01, L03, L04 and L05 it added commentary or "Changes made" lists. In W03 it refused to clean up at all and asked whether "VLM" meant vLLM. The anchor-based scoring is unaffected.
- Deepseek with the scoped rule prefixed W20 with a note ("…confirms 'shell' here means the command line, so it becomes 'terminal.'").
- Undo is the "none" row: delete the global rule, and wrong conversions return to 0/9.

### Step 3, batch history (verbatim, 5 runs each)

deepseek-v4.1-flash: 5/5 scoped, and all 5 also state what to leave alone.

1. Replace "shell" with "terminal" only when it refers to a command-line interface, leaving all other senses of "shell" unchanged.
2. Replace "shell" with "terminal" only when it refers to a command-line interface for running programs, not when it means a physical shell, gas station, food item, or other non-computing sense.
3. Replace "shell" with "terminal" only when it refers to a command-line interface for running programs, but leave it unchanged when it means a gas station, seashell, eggshell, taco shell, pasta, nut shell, or animal shell.
4. Replace "shell" with "terminal" only when it refers to a command-line environment for running programs or scripts; leave it unchanged when it means a gas station, seashell, eggshell, taco shell, pasta, nut shell, or physical shell.
5. Keep "shell" when it means a physical shell (seashell, egg, taco, pasta, pistachio, crab) or the Shell gas station, but change it to "terminal" when it means the command-line environment.

gemini-3.8-flash: 5/5 scoped, with the positive condition only. The negative side is implied, not stated.

1. Replace "shell" with "terminal" when referring to the command-line interface or computing environment.
2. Replace "shell" with "terminal" when referring to the command-line interface.
3. Same as 1.
4. Same as 1.
5. Replace "shell" with "terminal" when it refers to the command line or computing interface.

### Summary

| how the rule was distilled | deepseek-v4.1-flash scoped to the command line | gemini-3.8-flash scoped to the command line |
|---|---|---|
| one edit (W06), on the spot | 2/5 (2 global, 1 over-narrow) | 4/5 (1 global) |
| 12 retrieved records (4 edited work + 8 untouched life) | 5/5 (5/5 also name the exceptions) | 5/5 |

Variance context: in the pilot, with a different W06 wording, the single-edit cell gave 2/5 global for deepseek and 0/5 for gemini (`PILOT.md`). Across the 10 runs, that is deepseek 4/10 global and gemini 1/10.