README.md

5.5 Turning thinking on and off on your own tasks

This script answers one practical question: for the exact same model, how many extra tasks does turning on thinking mode solve on your workload, and how many extra seconds and dollars does that cost?

It runs a batch of tasks with known ground-truth answers twice on the same model and prompt, changing only reasoning_effort: none for thinking off, and high for full thinking. Each mode runs the full task set multiple times (3 runs per mode by default), because model outputs have run-to-run variance and a single pass can mistake sampling luck for a real gain.

What you need

Cloud API calls only, with no local GPU required. Any computer running Python 3.9+ works, along with an AI Builder Space API key.

Setup and usage

cd 5_5_thinking
uv venv && source .venv/bin/activate      # Without uv: python3 -m venv .venv && source .venv/bin/activate
uv pip install -r requirements.txt        # Or: pip install -r requirements.txt
export AI_BUILDER_TOKEN=sk_your_key
python run.py                             # Default: grok-4-fast, 16 sample tasks, 3 runs per mode, about 3 minutes
python run.py --model gpt-5 --repeats 2   # Switch to another model

Supported models: both grok-4-fast and gpt-5 allow turning thinking completely off (none), which makes them well suited for this comparison. grok-4.5 and grok-4.7 only go down to low, and kimi-k3 only supports low, high, and max; deepseek on AI Builder Space runs with thinking off and cannot enable it.

What you should see

The script prints a summary table first: accuracy for each mode (with the correct count from each individual run in parentheses), median latency, average output tokens, and cost per full pass over the task set. Below that, it lists how many runs each task passed with thinking off versus thinking on, flagging items where thinking helped or hurt. Per-item logs (including the last few hundred characters of each model reply) are saved as JSON files under results/. Open that file and inspect any missed items: sometimes the model made a real calculation error, and sometimes it skipped the requested answer format.

Our reference outputs are in sample_output/ (recorded on 2026-10-23 on a Mac Studio calling AI Builder Space):

Note on language in the starter and sample files: there are no separate English-language run files in sample_output/. The built-in starter set tasks.jsonl uses Chinese-language task prompts (including English target strings in count_e, reverse, and arrange). In sample_output/, grok-4-fast_20261023.json and gpt-5_20261023.json come from those Chinese-language runs (English JSON keys and numeric metrics, with model reply snippets mostly in Chinese), and grok-4-fast_console.txt and gpt-5_console.txt are console logs from the same runs captured with Chinese table headers and status labels (思考关 for thinking off, 思考开 for thinking on, and ← 思考有用 where thinking helped).

Thinking mode helped on problems that need an intermediate scratchpad: letter counting (count_e, where both models missed runs with thinking off: 0/2 on gpt-5 and 2/3 on grok-4-fast), calendar date math (date, 2/3 on grok-4-fast), multi-step arithmetic on the electricity bill (power_bill, 1/3 on grok-4-fast), and permutation counting on grok-4-fast (arrange, 2/3 on grok-4-fast and 2/2 on gpt-5). Tasks such as factual lookup (capital), field extraction (extract), and unit conversion (unit) hit 100% with thinking off on both models, so turning on thinking only added latency and cost. Your own numbers will differ from ours, especially with thinking off, where the score can swing by several tasks from pass to pass.

Using your own data

Put 10 to 20 labeled examples from your own workload into a JSONL file, one task per line:

{"id": "invoice_07", "question": "...your input...", "answer": ["ground-truth answer", "another accepted form"]}

Then run python run.py --tasks my_tasks.jsonl. Grading is handled by is_correct in run.py: SYSTEM asks the model to end with 答案： (matching tasks.jsonl), extract_answer takes the text after the last 答案： (or falls back to the last line), and is_correct strips spaces and punctuation before comparing against answer. When the ground-truth answer is numeric, is_correct compares by numerical value (so "130 degrees" counts as 130). If your tasks return longer text that cannot be graded by exact match, or if you want to change the answer prefix for your own prompts, edit is_correct (and SYSTEM / extract_answer if needed) to run your own check, such as verifying required fields.

Token prices are listed in PRICES at the top of run.py (USD per million tokens, based on October 2026 AI Builder Space rates). Update PRICES if rates change or you test a different model. Thinking tokens are billed at the output-token rate and are already included in the output token count.