# 5.5 Turning thinking on and off on your own tasks

This script answers one practical question: for the exact same model, how many extra tasks does turning on thinking mode solve on your workload, and how many extra seconds and dollars does that cost?

It runs a batch of tasks with known ground-truth answers twice on the same model and prompt, changing only `reasoning_effort`: `none` for thinking off, and `high` for full thinking. Each mode runs the full task set multiple times (3 runs per mode by default), because model outputs have run-to-run variance and a single pass can mistake sampling luck for a real gain.

## What you need

Cloud API calls only, with no local GPU required. Any computer running Python 3.9+ works, along with an AI Builder Space API key.

## Setup and usage

```bash
cd 5_5_thinking
uv venv && source .venv/bin/activate      # Without uv: python3 -m venv .venv && source .venv/bin/activate
uv pip install -r requirements.txt        # Or: pip install -r requirements.txt
export AI_BUILDER_TOKEN=sk_your_key
python run.py                             # Default: grok-4-fast, 16 sample tasks, 3 runs per mode, about 3 minutes
python run.py --model gpt-5 --repeats 2   # Switch to another model
```

Supported models: both `grok-4-fast` and `gpt-5` allow turning thinking completely off (`none`), which makes them well suited for this comparison. `grok-4.5` and `grok-4.7` only go down to `low`, and `kimi-k3` only supports `low`, `high`, and `max`; `deepseek` on AI Builder Space runs with thinking off and cannot enable it.

## What you should see

The script prints a summary table first: accuracy for each mode (with the correct count from each individual run in parentheses), median latency, average output tokens, and cost per full pass over the task set. Below that, it lists how many runs each task passed with thinking off versus thinking on, flagging items where thinking helped or hurt. Per-item logs (including the last few hundred characters of each model reply) are saved as JSON files under `results/`. Open that file and inspect any missed items: sometimes the model made a real calculation error, and sometimes it skipped the requested answer format.

Our reference outputs are in `sample_output/` (recorded on 2026-10-23 on a Mac Studio calling AI Builder Space):

| Model | Thinking off | Thinking on | Median latency (Off → On) | Avg output tokens (Off → On) | Cost per 16-task pass (Off → On) |
|---|---|---|---|---|---|
| `grok-4-fast` (3 runs per mode) | 14.3/16 (13, 16, 14) | 16.0/16 (16, 16, 16) | 2.1s → 8.1s | 91 → 672 | $0.0086 → $0.0319 |
| `gpt-5` (2 runs per mode) | 15.0/16 (15, 15) | 16.0/16 (16, 16) | 1.9s → 2.6s | 42 → 193 | $0.0185 → $0.0666 |

Note on language in the starter and sample files: there are no separate English-language run files in `sample_output/`. The built-in starter set `tasks.jsonl` uses Chinese-language task prompts (including English target strings in `count_e`, `reverse`, and `arrange`). In `sample_output/`, `grok-4-fast_20261023.json` and `gpt-5_20261023.json` come from those Chinese-language runs (English JSON keys and numeric metrics, with model reply snippets mostly in Chinese), and `grok-4-fast_console.txt` and `gpt-5_console.txt` are console logs from the same runs captured with Chinese table headers and status labels (`思考关` for thinking off, `思考开` for thinking on, and `← 思考有用` where thinking helped).

Thinking mode helped on problems that need an intermediate scratchpad: letter counting (`count_e`, where both models missed runs with thinking off: 0/2 on `gpt-5` and 2/3 on `grok-4-fast`), calendar date math (`date`, 2/3 on `grok-4-fast`), multi-step arithmetic on the electricity bill (`power_bill`, 1/3 on `grok-4-fast`), and permutation counting on `grok-4-fast` (`arrange`, 2/3 on `grok-4-fast` and 2/2 on `gpt-5`). Tasks such as factual lookup (`capital`), field extraction (`extract`), and unit conversion (`unit`) hit 100% with thinking off on both models, so turning on thinking only added latency and cost. Your own numbers will differ from ours, especially with thinking off, where the score can swing by several tasks from pass to pass.

## Using your own data

Put 10 to 20 labeled examples from your own workload into a JSONL file, one task per line:

```json
{"id": "invoice_07", "question": "...your input...", "answer": ["ground-truth answer", "another accepted form"]}
```

Then run `python run.py --tasks my_tasks.jsonl`. Grading is handled by `is_correct` in `run.py`: `SYSTEM` asks the model to end with `答案：` (matching `tasks.jsonl`), `extract_answer` takes the text after the last `答案：` (or falls back to the last line), and `is_correct` strips spaces and punctuation before comparing against `answer`. When the ground-truth answer is numeric, `is_correct` compares by numerical value (so `"130 degrees"` counts as `130`). If your tasks return longer text that cannot be graded by exact match, or if you want to change the answer prefix for your own prompts, edit `is_correct` (and `SYSTEM` / `extract_answer` if needed) to run your own check, such as verifying required fields.

Token prices are listed in `PRICES` at the top of `run.py` (USD per million tokens, based on October 2026 AI Builder Space rates). Update `PRICES` if rates change or you test a different model. Thinking tokens are billed at the output-token rate and are already included in the output token count.