README.md

2.8 (English edition): put every skill in the window, or only an index

This is the English re-run of the Chinese edition's experiments/2_7_skills/ (which feeds sections 2.7 and 2.8). Same design and code structure. What changed: an English skill pool, 30 newly generated English tasks, English prompts, and a different set of models (no GPT anywhere). Run date 2026-10-14. Spend for this experiment: about $34.5 (itemized in ../COSTS.md).

How to reproduce

# 1. Build the English skill pool (no API calls). Needs local checkouts of the source repos; see build_pool.py.
.venv/bin/python experiments_en/2_8_skills/build_pool.py
# 2. Tasks: gemini drafts 2 candidates per skill, 3 models answer with no skill, hand review picks 30
op run --env-file experiments/aib.env -- .venv/bin/python -u experiments_en/2_8_skills/make_tasks.py draft
op run --env-file experiments/aib.env -- .venv/bin/python -u experiments_en/2_8_skills/make_tasks.py baseline
.venv/bin/python experiments_en/2_8_skills/make_tasks.py finalize        # reads selection.json
# 3. Main runs (one line per model)
op run --env-file experiments/aib.env -- .venv/bin/python -u experiments_en/2_8_skills/run.py grok-4-fast none full_1 index index_rep2 full_all
# 4. Grade (primary judge), optional second judge, aggregate
op run --env-file experiments/aib.env -- .venv/bin/python -u experiments_en/2_8_skills/grade.py
op run --env-file experiments/aib.env -- .venv/bin/python -u experiments_en/2_8_skills/grade.py secondary
# 5. Description rewrite and re-run
.venv/bin/python experiments_en/2_8_skills/fix_descriptions.py plan
op run --env-file experiments/aib.env -- .venv/bin/python -u experiments_en/2_8_skills/fix_descriptions.py rewrite
op run --env-file experiments/aib.env -- .venv/bin/python -u experiments_en/2_8_skills/run.py grok-4-fast index_fixed
op run --env-file experiments/aib.env -- .venv/bin/python -u experiments_en/2_8_skills/grade.py && .venv/bin/python experiments_en/2_8_skills/grade.py analyze
# 6. Hand spot-check of the judge
.venv/bin/python experiments_en/2_8_skills/grade.py sample 20   # then label spotcheck_review.json by hand
.venv/bin/python experiments_en/2_8_skills/grade.py agree

Gateway facts that affect how to read the results

All calls go through AI Builder Space, the same OpenAI-compatible API readers use.

• deepseek-v4.1-flash (gateway id deepseek-v4-flash) runs inside the gateway's orchestrator, which drops client-supplied tools. Its index condition therefore uses a plain-text protocol: the model replies with a line READ_SKILL: <name>, and the skill text comes back as the next user message (same as the Chinese run). The orchestrator also runs its own web searches on some prompts. 16 deepseek answers say in their own text that a search ran and returned nothing relevant, and the debug trace flagged only 1 of those 16 (results.json → meta.search_mentions). In full_all, 3 calls show double the prompt tokens (about 682K instead of 336K), which matches an extra search round. We saw no case where a search produced the answer, but this column cannot be fully trusted.
• gemini-3.8-flash also runs in the orchestrator. With client tools attached it returns an empty message, so its index condition uses the same text protocol as deepseek. It is a reasoning model: completion tokens include several hundred reasoning tokens even on trivial prompts. Gateway bug: if the last user message contains certain words (we confirmed now, know, today, latest, current, price, weather, recent), gemini requests fail with HTTP 400 "Requests ending with a model turn are not supported". lib.call works around it by appending one more user turn, Please respond to the message above., and records nudged=true. This applied to 3/30 questions in none/full_1/full_all, and to 29/30 index tasks (the returned skill text almost always contains "now" or "know"). It never affects the first-turn decision about which skill to open, except on those 3 questions.
• grok-4-fast is served as xAI grok-4.3 (non-reasoning). It passes client tools through, so its index condition uses native read_skill tool calls, as in the Chinese run. So grok is the only model on native tool calling here, and protocol and model are confounded for grok.
• Context limits. No model was blocked. All three accepted the whole pool on all 30 tasks: 336,479 (deepseek), 349,907 (gemini) and 318,141 (grok) prompt tokens. The Chinese run's "doesn't even fit" cell came from gpt-5's 272K limit, and nothing in this run reproduces it.
• Cost is the gateway's per-call estimated_cost_usd from the orchestrator trace, with no cache discount. Implied prices per 1M tokens (input / output): gemini 0.75 / 3.75, grok 1.25 / 2.50, deepseek about 0.15 input.

Design

Skill pool (pool.json, pool_text.json). Starts from the Chinese run's 139 skills (experiments/2_7_skills/pool.json):

• context-infrastructure skills (31): taken from the English mirror grapeot/context-infrastructure-en, at the committed HEAD of the local checkout. 30 of them were mostly Chinese in the zh pool.
• writing-skill skills (7): taken from that repo's skills_en/ folder.
• Skills already in English (86): kept byte for byte from the zh pool. One of them, bestpractice_academic_paper_conversion, has no en-mirror copy and was already English.
• Dropped (15): skills that are mostly Chinese (CJK characters > 0.5 × Latin words outside code fences, or a Chinese description) and have no English version. These are bright_horizons_photo_sync, chat_gpt_oauth, eink_diary, grok_oauth, ecg_plot, health_quantification, four m5stack_sticks3 skills, maven, skill_open_router_data_scraper and three voice-lora skills.
• Replacements (15): English skills from other public grapeot repos: send_email, typefully_post, typefully_metrics and workflow_presentation_slides (extra files in context-infrastructure-en), plus apple_calendar, colab, ai_session_export, docx, skill_google_docs, tiff_icc_profile, skill_smtp, superlinear-brand, track17, chinese_visa_photo and gpt-live.

Provenance (source repo, commit, action) is recorded per skill in pool.json. After the swap, no skill is mostly Chinese and no description contains Chinese. The pool is 139 skills, 325,094 tokens (cl100k) and 1,445,885 characters, against 369,217 tokens in zh. The index is 5,316 tokens (cl100k), against 6,682 in zh. The secret scan hits only placeholders (example.com addresses, replace-with-your-... keys).

Tasks (tasks.json).

• Drafting. gemini-3.8-flash read each skill and drafted 2 candidate questions in English. The prompt is the zh DRAFT_SYS translated: a realistic user question, no skill file name, no copying of distinctive sentences, plus a verbatim excerpt and a rubric. Candidates whose excerpt was not verbatim were dropped, leaving 260.
• Baseline filter. All three models answered each candidate with no skill. In 177 candidates all three were wrong (cross-model judge).
• Hand selection. We picked 30 from those 177, with one target skill each: 8 from context-infrastructure-en and 22 from 22 different other repos. The rules mirror the zh selection. We dropped answers that conflict with another skill, private config values, exact-sentence recall, and one candidate where search was used (selection.json). Two questions were edited because they named the skill or CLI file (T16, T24).
• Cue tags (task_cues.json). 7 of the 30 are "no cue" questions (T06, T25 to T30). They don't name the product or tool, and several skills in the pool fit the request. Unlike zh, where the tags were added after seeing the failures, these tags were set at selection time, before any with-skill run.

Conditions (one call per model per task, no history between tasks): none; full_1 (only the target skill); full_all (all 139, fixed seed-27 permutation); index (139-line index, up to 4 opens); index_rep2 (exact repeat, also the control for the rewrite); index_fixed (rewritten descriptions). We skipped full_10 and full_30 to stay in budget; the code supports them.

Grading. The judge prompt is the same as zh. No model grades itself: gemini grades deepseek and grok, and deepseek grades gemini. A second pass with the other judge (deepseek for deepseek/grok answers, gemini for gemini answers) covered all 506 answers.

Results

Correct answers out of 30 (primary judge; results.json → cells)

model	none	only the right skill (full_1)	all 139 (full_all)	index	index re-run
deepseek-v4.1-flash	2	30	30	24	23
gemini-3.8-flash	1	30	30	30	29
grok-4-fast	0	30	29	9	8

The none scores come from the task filter. Each task was failed by all three models in the baseline pass, and the 3 correct answers here are re-run noise (deepseek T07, T19; gemini T10).

Tokens, latency and cost: all skills vs index

model	condition	mean prompt tokens / question	mean latency (s)	$ per 30 questions
deepseek-v4.1-flash	full_1	4,874	6.8	0.061
	full_all	371,712 (median 336,479)	7.9	1.712
	index	32,748	19.5	0.446
gemini-3.8-flash	full_1	2,630	7.5	0.119
	full_all	349,907	9.7	7.944
	index	15,928	11.4	0.439
grok-4-fast	full_1	2,571	4.3	0.103
	full_all	318,141	15.3	11.945
	index	7,925	7.5	0.311

Index tokens, latency and cost add up all rounds of a task. The deepseek full_all mean includes 3 calls that went through an extra gateway search round. Moving from all skills to the index cut cost by 97% for grok, 94% for gemini and 74% for deepseek. Latency fell for grok (15.3 → 7.5 s), rose slightly for gemini (9.7 → 11.4 s) and more than doubled for deepseek (7.9 → 19.5 s), because the index adds at least one round trip.

Index failures: didn't open any / opened the wrong one / opened the right one but answered wrong

model	index wrong	opened nothing	opened only other skills	opened target, still wrong	correct when target opened
deepseek-v4.1-flash	6	5	1 (T26 → apple_calendar)	0	24/24
gemini-3.8-flash	0	0	0	0	30/30
grok-4-fast	21	21	0	0	9/9
re-run, all models	8 + 1 + 22	5 / 0 / 22	2 / 0 / 0	0 / 1 / 0	23/23, 29/30, 8/8

All 27 first-run index failures were routing failures: 26 never opened a skill, and 1 opened the wrong one. When the target was opened, the answer was right 63 times out of 63 (60/61 in the re-run). Opening behaviour differs a lot by model. Gemini opened 1 to 3 skills per task (mean 1.47) and never zero. Deepseek usually opened one (mean 1.07) and none on 5 tasks. Grok opened nothing on 21/30 (22/30 in the re-run), including clearly cued questions that name Docling, OpenReview, Bilibili, Codex, Narwal or Apple Watch. It answered those from its own knowledge.

Run-to-run flips between index and index_rep2: deepseek 1 task (T07), gemini 1 (T28), grok 3 (T01, T10, T12).

Cue vs no-cue questions (index; re-run in brackets)

model	23 cued	7 no-cue
deepseek-v4.1-flash	20 (19)	4 (4)
gemini-3.8-flash	23 (23)	7 (6)
grok-4-fast	9 (8)	0 (0)

The English counterpart of the zh Resend/Outlook example is T26. It asks about the default time zone of calendar invite / calendar update without naming Outlook, and deepseek opened apple_calendar in both runs. The zh-style Resend question itself (T27, received list --after without naming Resend) did not confuse deepseek or gemini in English: both opened the Resend skill.

Description rewrite (index_fixed) vs plain re-run

Procedure as in zh. plan collects, per model, the index tasks that were wrong and never opened the target (deepseek 6, gemini 0, grok 21; 21 distinct target skills). gemini-3.8-flash then rewrote those 21 one-line descriptions from the skill body and the old line only. It never saw the tasks and was told to say when to open the skill, not what is in it. We read all 21 and made one hand edit. For codex_cli, "troubleshooting ... backend compatibility" sat too close to T09's symptom and was softened (original kept in descriptions_fixed.json → hand_edits). The rewritten lines are 184 characters on average against 170 before, and the full index grew from 5,316 to 5,378 tokens. Both versions are in English, so the language of the descriptions did not change.

model	wrong and target not opened	correct after rewrite	correct in plain re-run (control)	passing sample still correct: rewrite	passing sample: re-run
deepseek-v4.1-flash	6	1 (T15)	0	8/10 (broke T07, T20)	9/10
gemini-3.8-flash	0	–	–	10/10	10/10
grok-4-fast	21	2 (T01, T05)	1 (T01)	8/9	7/9

The rewrite recovered 1 of 6 for deepseek against 0 for the control, and 2 of 21 for grok against 1. That is within run-to-run noise. Deepseek still opened nothing on 3 of its 6 (T05, T26, T29) and opened the wrong skill on two (T28 → tavily, T09 → temporal_info_verification). Grok still opened nothing on 18 of 21. The deepseek T20 regression came from opening ai_session_export instead of skill_opencode_data, possibly because of the new "export session history" wording, or possibly noise.

Judge reliability

• Second judge. Primary and secondary judges agree on 501 of 506 answers. The 5 disagreements fall in deepseek none (2), deepseek full_all (1), deepseek index_fixed (1) and gemini index_rep2 (1). Secondary-judge totals per cell are in results.json (correct_secondary_judge). No cell moves by more than 1.
• Hand spot-check. I drew 20 primary judgments with a fixed seed (10 judged correct, 10 judged wrong) and labelled them against the excerpt, spotcheck.json and spotcheck_review.json. Agreement was 20/20. The closest calls were T20 (grok wrote --include-subagents with an invented explanation; judged wrong) and T03 none (deepseek suggested deleting "a stored token file" without the specific file; judged wrong). Both are defensible.

Limitations

• Small sample. 30 tasks per cell, so one task is 3.3 points. Don't read differences of 2 to 3 tasks.
• One call per task per condition. Run-to-run variation is estimated only by the single re-run.
• Different task set from zh. These are newly generated English tasks, not translations. Gemini generated them, and it is also a tested model. The baseline filter (all three models wrong without the skill) and hand review limit the bias, but a generator/model affinity could still help gemini.
• Two judges, one of them tested. Each answer is graded by a model other than its author. Both judges are also tested models.
• Protocol confound. grok uses native tool calls, while deepseek and gemini use the text protocol. Grok's reluctance to open skills could come from the model, the protocol, or both.
• Gateway behaviour.
  • deepseek's orchestrator runs web searches that its trace mostly doesn't report (see above).
  • gemini needed the nudge turn on most index tasks.
  • Neither showed a case where it changed the answer, but readers reproducing this should expect it.
• No dilution test between 1 and 139 skills. full_10 and full_30 were not run. At about 320K to 350K tokens, no model lost more than one task, and we don't know what happens on a bigger pool.
• Cost estimates. Costs are gateway estimates without cache discounts. Treat absolute dollars as orders of magnitude and the ratios as the reliable part.

Files

• build_pool.py → pool.json (per-skill provenance, description, CJK stats, scan hits; dropped list), pool_text.json
• lib.py (HTTP calls with gateway workarounds, English prompts, judge), make_tasks.py → candidates_raw.json, candidates.json, candidates_baseline.json, candidates_summary.json; selection.json → tasks.json, task_cues.json
• run.py → raw/<model>__<condition>.json (answer, opened/requested skills, tokens, latency, cost, web flag, nudge flag)
• grade.py → graded/, results.json (cells, analysis, spotcheck, meta)
• fix_descriptions.py → index_fixed_plan.json, descriptions_fixed.json
• spotcheck.json, spotcheck_review.json; *.log run logs