# 3.7 Process formulation vs. outcome formulation (English runs)

This folder holds the experiment behind section 3.7 of the English edition, "Long-horizon delegation: describe the destination, not the steps," in particular the part titled "The experiment: same strong model, two ways of asking." It also provides the meeting recordings and the names list used in that section's hands-on exercise.

Everything described here is the English run: 12 trials (two difficulty tiers × two formulations × 3 reps), run in October 2026.

## The question

There are two ways to hand off a task to an AI agent:

- **Process formulation:** do these steps, and you are done.
- **Outcome formulation:** meet these criteria, and you are done; how you get there is up to you.

When a task is complex enough that nobody can list, in advance, steps that guarantee the result, does process formulation hit a ceiling? And if it does, does the ceiling come from how the task is framed, rather than from a weak model or a poorly written plan?

To rule out the last two explanations, both arms use the same strong model, and the plan in the process arm is written by that same model after it has seen every goal the outcome arm is graded on.

## Design

**Models.** The executor is `gpt-6.1-sol`, called through a ChatGPT subscription (OAuth) against the Codex Responses endpoint, reasoning effort `medium` (recorded as `effort` in each `trials/*/log/summary.json`). The planner in the process arm is also `gpt-6.1-sol`, reasoning effort `high`. Both arms get the same single tool (`shell`) and the same starting folder; the only difference is the prompt. Speech recognition is Grok file transcription on AI Builder Space, documented for the agent in `fixture_en/API.md`. The agent calls it itself with the `AI_BUILDER_TOKEN` environment variable.

**Conditions.**

| Arm | Prompt |
|---|---|
| Process | Task + the planner's steps + "Follow the steps below exactly. The task is done once every step is done." (`prompts_en/process.tmpl`) |
| Outcome | Task + acceptance criteria + "The task is done only when all of these hold. How you get there is up to you; before you hand it over, make sure each one actually holds." (`prompts_en/outcome.tmpl`) |

The planner (`prompts_en/planner.tmpl`) sees the task, the same acceptance criteria labeled "Goals (visible only to you)", and a listing of the files in the folder with their sizes and durations. It has not listened to anything. The executor in the process arm receives only the task and the steps. Each process trial gets its own freshly generated plan, saved as `trials/en_*_process_*/plan.md`.

**Two tiers.**

- *Simple:* turn one voice memo, `memos/W01.wav`, into `notes/W01.md`. Four criteria (`prompts_en/criteria_simple.md`), including "The verbal tic "I mean" is removed." This is a task whose steps can be listed in advance.
- *Complex:* turn a synthetic meeting of about 54 minutes, delivered as two audio files, into `minutes/transcript.md` (`prompts_en/task_complex.md`). Five criteria (`prompts_en/criteria_complex.md`, verbatim):
  1. Every stretch of the recording where someone is speaking has matching text in the transcript. Nothing is left out.
  2. No passage appears twice.
  3. Every paragraph starts with an [hh:mm:ss] timestamp giving its position in the whole meeting (counted from the start of the first recording file), accurate to within 10 seconds.
  4. Names and terms from names.md are spelled correctly.
  5. It is verbatim: no summarizing, no rewording, nothing cut.

**The synthetic meeting** (`meeting_en/`). A fictional biweekly sync of a small team building VoiceFlow, a voice dictation product. The script (`script.json`) was written by `gpt-6.1-sol` from `script_prompt.md`. Every line was synthesized with Gemini TTS preset voices (host Charon, Jaxon Puck, Kaitlynn Kore, Xochitl Aoede, Aoife Leda, Saoirse Despina, Kaveh Iapetus); no one's voice was cloned. The meeting runs 3,220.35 seconds in total, including a 300-second break, with 112 voiced lines. `truth.json` holds the exact start and end time of every line.

The meeting contains two problems that do not produce any error message:

- **A quiet phone participant.** Saoirse dials in by phone. Her lines are resampled to 8 kHz, band-limited to 300–3,400 Hz, turned down to 0.06× gain, and mixed with line noise (`config.json`: `phone_gain`, `phone_noise`). She has 8 turns. When the full recording goes through the ASR API, parts of her speech are dropped without any error; when her stretches are clipped and transcribed on their own, the text comes back. In pilot runs, a gain of 0.12× lost only 1 of her 8 turns in English, so the English meeting uses 0.06×.
- **Two overlapping files.** The recording "switches devices" partway through. `meeting_part1.m4a` ends at 1,801.57 s of the meeting, and `meeting_part2.m4a` starts at 1,761.57 s, so the two files share 40 seconds. Appending part 2 to the end of part 1 repeats those 40 seconds and shifts every part-2 timestamp by 40 seconds.

The pilot runs also ruled out two other candidate problems: long stretches of silence did not make the API invent words, and a file of more than 50 minutes went through in a single request.

**Sandbox.** `harness/mini_agent.py` runs every agent command under macOS `sandbox-exec`. The agent can write only to its working folder and temp directories, and it cannot read the `experiments/` directory (where the ground truth lives) or the local credential folders. The value of `AI_BUILDER_TOKEN` is replaced with `<redacted>` in everything the model sees and in the logs.

**Grading.** The agent never sees the ground truth. `grade.py` scores the complex tier against `meeting_en/truth.json`:

- A line counts as *covered* if the normalized transcript contains a fuzzy match for it (rapidfuzz partial ratio of at least 70).
- A line counts as *duplicated* if a second match is still found after the first one is removed.
- Each transcript paragraph is matched to the script line that best matches its opening. The paragraph is *off* if its timestamp falls outside that line's start-to-end interval widened by 10 seconds on each side.
- *All criteria met* (`all_criteria_met` in `results.json`) means all 112 lines covered, no duplicates, no paragraph off, and all 8 of Saoirse's turns covered. Names and terms are counted separately. Verbatim fidelity is checked indirectly: a summarized or reworded line fails to match its script line.

`grade_simple.py` passes a simple-tier note when it exists, "I mean" no longer appears, six names and terms (Jaxon, Saoirse, Koyeb, vLLM, Priya, KV cache) are present, and its similarity to the script with the tic removed is at least 0.85.

## Results

| English, 3 reps per arm | Process | Outcome |
|---|---|---|
| Complex: all criteria met (covered, no duplicates, all timestamps within ±10 s, phone 8/8) | 0/3 | 3/3 |
| Lines covered | 111/112, 112/112, 112/112 | 112/112 ×3 |
| Duplicated lines | 1, 1, 1 | 0, 0, 0 |
| Part-2 paragraphs off by more than 10 s | 169/171, 99/101, 103/105 | 0/59, 0/54, 0/91 |
| Saoirse's 8 phone turns kept | 7/8, 8/8, 8/8 | 8/8 ×3 |
| Agent reported steps it could not do | 2/3 (reps 1, 2) | 0/3 |
| Agent declared done, criteria not met | 1/3 (rep 3) | 0/3 |
| Names and terms, complex (capped at expected) | 475/486 | 482/486 |
| Simple tier (W01) passed | 1/3 | 3/3 |
| Minutes per complex run | 10.0, 8.8, 11.8 | 7.6, 7.6, 9.6 |
| Tool calls per complex run | 28, 40, 53 | 35, 32, 34 |

Reading the three complex process runs one by one, three things stand out.

**All three plans made the same wrong assumption.** Without having heard the audio, each planner assumed part 2 starts where part 1 ends. Plan 3 (`trials/en_complex_process_3/plan.md`, 14 steps, 1,062 words) puts it this way in step 2: "Set their meeting-time offsets to zero and the exact duration of part 1, respectively." Plan 2: "Record the exact start offset of part 2 as the decoded duration of part 1." The executors followed this, so the overlapping passage was transcribed twice and the part-2 timestamps were pushed late by roughly the length of the overlap (169/171, 99/101 and 103/105 part-2 paragraphs off by more than 10 seconds).

**The plans contained many checks, but the checks tested the plan's own assumption.** Plan 3 includes an independent second transcription pass with boundaries shifted by 60 seconds (step 9) and a final validation (step 13) whose fifth bullet reads "Part 2 timestamps include part 1's exact duration." Those checks helped with the quiet phone participant (all 8 of her turns were kept in 2 of 3 runs, 7 of 8 in the third), but they could not reveal that the timeline itself was wrong. Rep 3 ended with: "Saved the transcript to `minutes/transcript.md`, with meeting-relative timestamps and speaker labels. One unrecoverable word is marked `[unintelligible]`. The saved file passed the formatting and timestamp checks." In that run, 103 of 105 part-2 paragraphs were off.

**The plans required something the executor could not do.** Step 10 of plan 3 reads: "Use playback or an audio-capable inspection tool to check the actual sound; matching API outputs alone do not constitute verification." A command-line agent cannot listen to audio. Reps 1 and 2 said so in their final messages ("Saved an **unverified automated draft** to `minutes/transcript.md` … The task is **not fully complete**" and "…**not fully audio-verified**. I could not perform the required direct listening reviews, so the task is not fully complete."). Rep 3 declared the task done. Every English run delivered a transcript; none refused.

The outcome runs chose their own checks against the criteria. Rep 1's final message reports "…the 40-second recording overlap included only once." Rep 3's reads: "Recovered the quiet phone-in speech, removed the recordings' 40-second overlap, and checked timestamps, duplicate passages, and supplied name spellings." The full action logs are in `trials/en_complex_outcome_*/log/transcript.jsonl`.

**Simple tier.** Process reps 1 and 2 failed on one name: both notes read "So this afternoon, I went over the deployment with Jaxon and Xochitl." where the speaker said Saoirse. Both plans told the executor to change a name only when the audio supports it (rep 1, step 8: "Apply a listed spelling only where that name or term is actually spoken; do not substitute merely similar words."), so the executor kept the wrong name, and both runs reported success. One name across three samples is too little to claim a difference between the formulations on simple tasks.

## Files

- `harness/mini_agent.py`: a minimal agent, one model plus one `shell` tool, with the sandbox and redaction described above.
- `run_trial.py`: runs one trial. It builds a fresh working folder, has the planner write a plan (process arm only), runs the agent, copies the deliverable and logs back into `trials/`, and grades the result.
- `grade.py`, `grade_simple.py`: grading for the complex and simple tiers.
- `summarize.py`: collects every trial's grade and agent summary into `results.json`.
- `results.json`: one row per trial. The English trials are the rows with `"lang": "en"`.
- `build_meeting.py`, `gemini_tts.py`: synthesize the meeting from the script and cut it into two overlapping files.
- `prompts_en/`: `task_simple.md`, `task_complex.md`, `criteria_simple.md`, `criteria_complex.md`, and the three prompt templates `planner.tmpl`, `process.tmpl`, `outcome.tmpl`.
- `fixture_en/`: `API.md` and `names.md`, copied into the agent's working folder at the start of every trial.
- `meeting_en/`: `script_prompt.md` (the prompt that produced the script), `script.json`, `config.json` (voices, phone settings, break, overlap), `truth.json` (ground-truth timeline), and `recordings/meeting_part1.m4a`, `recordings/meeting_part2.m4a` (the two files the agent receives).
- `trials/en_<tier>_<arm>_<rep>/`: for each trial, `prompt.md` (what the executor received), `planner_prompt.md` and `plan.md` (process arm), `log/transcript.jsonl` (the full agent transcript), `log/summary.json` (model, effort, tool calls, time, final message), `workspace/` (the deliverable and the agent's other working files, without audio or hidden folders), and `grade.json`.

The per-line TTS `.wav` files and the uncut meeting `.wav` are not included. Rebuild them with `build_meeting.py en` (see below).

## How to rerun

Run the commands from this folder. The Python scripts need `httpx`, `numpy`, `soundfile` and `rapidfuzz`; the audio steps need `ffmpeg`/`ffprobe` and `sox`. The agent sandbox uses `sandbox-exec`, so the harness as written runs on macOS only.

```bash
# AI Builder Space key; the agent uses it to call the speech-to-text API
export AI_BUILDER_TOKEN=...

# One trial: <lang> <tier: simple|complex> <arm: process|outcome> <rep>
python run_trial.py en complex outcome 1
python run_trial.py en complex process 1
python run_trial.py en simple outcome 1
python run_trial.py en simple process 1

# Re-grade a transcript or note directly
python grade.py en trials/en_complex_outcome_1/workspace/minutes/transcript.md
python grade_simple.py en trials/en_simple_outcome_1/workspace/notes/W01.md

# Collect all trials into results.json
python summarize.py
```

Things to adjust before running:

- **The model client is local to the author.** `run_trial.py` (planner) and `harness/mini_agent.py` (executor) call `gpt-6.1-sol` through a ChatGPT-subscription OAuth client that lives outside this folder and is added to the import path at the top of each file. To rerun, swap in any OpenAI-compatible client: the planner needs one text completion (`request_result(prompt, model=..., reasoning_effort=...)` returning the answer text), and the executor's `call_model` needs a Responses-style call that supports a function tool. Results with a different model will differ from ours.
- **The working-folder location.** `run_trial.py` sets `RUNS` to a local directory outside the repository where it builds each trial's working folder; point it somewhere on your machine. If a trial's folder already exists, the script moves it to the trash with the `trash` command first.
- **The simple tier needs the Module 2 English corpus.** `W01.wav` and the memo script (`memos.json`) come from the Module 2 English corpus, which the scripts expect at `../../experiments_en/corpus/` relative to this folder. Download it from [https://www.ai-builders.com/resources/aibuilder2027/m02/en/experiments/corpus/](https://www.ai-builders.com/resources/aibuilder2027/m02/en/experiments/corpus/).

**Rebuilding the meeting.** The two `.m4a` recordings are included, so you do not need to rebuild them to rerun trials. If you want to regenerate the audio:

```bash
export GEMINI_API_KEY=...
python build_meeting.py en
```

This synthesizes every line again with Gemini TTS (the per-line `.wav` cache is not included), mixes the timeline, applies the phone processing, and writes new `recordings/*.m4a` and a new `truth.json`. TTS output varies from call to call, so the new audio will not match the published files line for line; grade new trials against the `truth.json` produced with the audio they used.

**Doing the hands-on exercise instead.** You do not need the harness to try this yourself. Give your own coding agent `meeting_en/recordings/meeting_part1.m4a`, `meeting_en/recordings/meeting_part2.m4a`, `fixture_en/names.md` and `fixture_en/API.md`, plus the five criteria above, as described in the chapter.