README.md

2.1 One request, written once in a chat box and once in a workspace: a "voice memos to text" script (English edition)

English twin of experiments/2_1_workspace/. Same design, English request, English fixture (audio and API.md), English success check. Run on 2026-10-14. Both conditions: 3 trials each, done.

How to run

From the textbook repo root. Both conditions use the Chinese experiment's venv (experiments/2_1_workspace/.venv: openai, httpx, requests), so A's generated code and B's agent see the same Python.

# A: chat box, trial n
op run --env-file experiments/aib.env -- experiments/2_1_workspace/.venv/bin/python experiments_en/2_1_workspace/run_chat.py gemini-3.8-flash 1
# A counterfactual (not counted): replace only the guessed URL with the real one and run again
op run --env-file experiments/aib.env -- experiments/2_1_workspace/.venv/bin/python experiments_en/2_1_workspace/counterfactual_url.py
# B: workspace agent, trial n
op run --env-file experiments/aib.env -- experiments/2_1_workspace/.venv/bin/python experiments_en/2_1_workspace/run_agent.py 1
# check one run directory; validate the check itself
experiments/2_1_workspace/.venv/bin/python experiments_en/2_1_workspace/check.py <run_dir>
experiments/2_1_workspace/.venv/bin/python experiments_en/2_1_workspace/check.py --validate

Design

Both conditions get the same sentence:

Write me a Python script that uses the AI Builder Space speech recognition API to transcribe the voice memos in the memos/ folder to text, and saves each one as a .txt file with the same name in transcripts/. The API key is in the environment variable AI_BUILDER_TOKEN.

• A: chat box. One API call whose only message is that sentence, model gemini-3.8-flash. The code in the reply is saved verbatim as script.py in a fresh folder that has only memos/ (no API.md), the key is in the environment, and it is run once, unmodified.
• B: workspace agent. A fresh scratch folder outside the repo (so the agent can't read common.py or other trials) with memos/ and API.md. The same sentence plus "You are working in a folder: you can read files, run commands, and look at the results." plus a safety rule (read the key only from the environment variable, never write it to a file or print it, work only inside the current folder). One headless turn of Antigravity CLI: agy --print ... --model gemini-3.8-flash-high --mode accept-edits --sandbox --dangerously-skip-permissions --print-timeout 10m --output-format json. The agent may edit and re-run as often as it likes; no human input. Afterwards the folder is copied to trials/B<n>/ws and scanned for the key.
  • One deliberate difference from the zh command: no --new-project (local AGY policy: it leaves a permanent empty project in the Antigravity desktop app). The agent still worked in the scratch folder as its working directory.
• Success (same for both, check.py). The 5 memos (W01, W02, W05, L01, I01) each have transcripts/<same name>.txt, non-empty, at least 20 English words, and word-level similarity to the script in experiments_en/corpus/memos.json of at least 0.5. The last rule excludes files holding an error message or raw JSON. Validation (check_validation.json): the known-good grok and compat transcripts score 0.916–1.0 on all 5; a raw JSON dump scores 0.08 and an error string 0.05.
• fixture/API.md is the English translation of the zh API.md (same facts: base URL, grok endpoint, fields, response shape, curl example, the OpenAI-compatible route). The example text is the real grok output for English W01 from experiments_en/corpus/transcripts_grok.json; the full-response shape is from the zh api_check/api_check.json, with language=en and W01's duration.

Results

A: chat box (gemini-3.8-flash)

Model time 13.6–17.8 s; script run 0.1–0.3 s.

• All three guessed a host that does not exist (both api.ai-builder.run and api.ai-builder.space return NXDOMAIN). Two of three guessed the same host the zh run guessed three times (api.ai-builder.space). The real base URL is https://space.ai-builders.com/backend/v1.
• Everything else was right: Authorization: Bearer <key>, the OpenAI-compatible multipart upload (file, model=whisper-1), iterating memos/, writing <same name>.txt.
• Two of three hedged in the code or notes: A1's comment "Base endpoint for AI Builder Space (adjust base URL if using a custom gateway)"; A1 and A2 read an optional override from an environment variable (AI_BUILDER_BASE_URL, AI_BUILDER_API_URL), and A2's notes say "If your AI Builder Space workspace uses a custom base URL or custom model identifier, you can override the endpoint without changing the code". A3 hard-coded the URL with no caveat.
• Swallowed errors. All three wrap each file's request in try/except, print the error, and move on. All three exit with code 0 and leave transcripts/ empty. A1 and A2 send the errors to stderr, so stdout shows only the progress lines and then "Processing complete." / "Batch transcription completed." A3 prints "FAILED" plus the error for each file on stdout and no closing message, so its failure is visible in the terminal, but its exit code still says success.

Counterfactual (not counted, counterfactual_url.json). Only the guessed host replaced with the real one, nothing else changed: A1, A2 and A3 all succeed, 5/5 each (similarity 0.916–1.0).

B: workspace agent (Antigravity CLI 1.3.5, gemini-3.8-flash-high)

• All three used the endpoint that appears only in API.md: https://space.ai-builders.com/backend/v1/audio/grok-transcription. As in zh, agy's event log doesn't record file reads, so "it read API.md" is inferred from the URL.
• All three created a venv with uv venv, installed requests, wrote transcribe.py, ran it, and reported verifying the output before finishing (B2: "Verified all 5 voice memos (I01, L01, W01, W02, W05) were transcribed successfully and their corresponding .txt files exist in transcripts/."). The log does not show whether the agent opened the files or only listed them.
• In every trial the sandbox blocked one attempt to run the script (operation not permitted: python, or a traceback in B3); the agent recovered on its own within the same turn.
• None of the three agent scripts swallows errors: a failed request raises or calls sys.exit(1), so a failure would show up as a non-zero exit.
• B1 took 251.5 s (zh range: 107–128 s); B2 and B3 were in the zh range or faster. agy reported 157K–308K input tokens per trial (subscription, not billed to AI Builder Space).

Caveats

• n=3 per condition, one model each.
• A's failure comes from a fact it had no way to know (the URL), not from coding ability; B had that fact in API.md. The comparison is built that way on purpose: the point is that material inside the agent's reach gets used, and that an agent that runs its own code catches the failure that A reports as success.
• B also ran on the high-reasoning tier (gemini-3.8-flash-high), A on plain gemini-3.8-flash.
• A ran in a venv with requests preinstalled; without it A would fail earlier with ImportError.
• The similarity threshold exists to exclude JSON and error text; it does not grade transcription quality.

Files

• fixture/memos/ (5 wav, copied from experiments_en/corpus/audio/), fixture/API.md.
• run_chat.py, run_agent.py, check.py, counterfactual_url.py; check_validation.json.
• trials/A<n>/: response.md (raw model reply), run/script.py (verbatim code), run_stdout.txt, run_stderr.txt, result.json, counterfactual_url_fixed/.
• trials/B<n>/: prompt.md, stdout.json (agy JSON incl. the final message), events.log, result.json, ws/ (the agent's folder after the run).
• results.json: summary.