[
 {
  "model": "deepseek-v4.1-flash",
  "condition": "full_all",
  "id": "T08",
  "question": "I am styling the HTML generated from Markdown documentation to ensure it looks good on mobile screens. What is the recommended minimum padding to apply to the body or main container so text does not cling to the screen edges?",
  "excerpt": "Ensure the HTML `body` or main container has sufficient `padding` (recommended 24px or more) to prevent text from hugging the screen edge on mobile devices.",
  "rubric": {
   "must_include": [
    "24px"
   ],
   "reference_answer": "The recommended padding for the HTML body or main container is at least 24px."
  },
  "answer": "The recommended minimum padding is **24px or more**.\n\nFrom the Markdown-to-HTML best practices:\n\n> **Spacing Management**: Ensure the HTML `body` or main container has sufficient `padding` (recommended 24px or more) to prevent text from hugging the screen edge on mobile devices.\n\nSo apply at least `padding: 24px` to the `body` or your main content container. If you want a bit more breathing room on larger phones, going slightly above 24px (e.g., 24–32px) is fine, but 24px is the stated floor.",
  "judge": "gemini-3.8-flash",
  "judge_correct": true,
  "judge_reason": "The answer correctly states that the recommended minimum padding is 24px or more."
 },
 {
  "model": "gemini-3.8-flash",
  "condition": "index",
  "id": "T25",
  "question": "I'm planning to process several long interview recordings through the local speech transcription pipeline. What is the maximum duration allowed for a single audio file before the system rejects it?",
  "excerpt": "A single file longer than 7200 seconds is rejected before a model loads.",
  "rubric": {
   "must_include": [
    "7200 seconds (or 2 hours)"
   ],
   "reference_answer": "A single audio file is rejected before any model loads if its duration exceeds 7200 seconds (2 hours)."
  },
  "answer": "The maximum allowed duration for a single audio file is **7,200 seconds (exactly 2 hours)**.\n\n### Details & Behavior:\n- **Hard Limit**: Any single file longer than 7,200 seconds is rejected before the acoustic/diarization models even load.\n- **Batch Processing**: Multiple files whose combined duration exceeds 2 hours are permitted, provided no individual file exceeds the limit.\n- **Handling Long Recordings**: If your interview files exceed 2 hours, split them into smaller segments before running the pipeline. Note that speaker labels/letters do not automatically persist or correlate across slices.",
  "judge": "deepseek-v4.1-flash",
  "judge_correct": true,
  "judge_reason": "The answer clearly states the 7200-second (2-hour) maximum and that longer files are rejected before models load, matching the excerpt."
 },
 {
  "model": "grok-4-fast",
  "condition": "index_fixed",
  "id": "T15",
  "question": "When generating infographics with GPT Image 2.5, what workflow should be used instead of prompting the model from text alone, and why?",
  "excerpt": "when building infographics with GPT Image 2.5, first produce a matplotlib structure draft and pass it as the `-i` input image. The model understands the draft layout, preserves title and annotation text, and improves the visual presentation. Asking it to generate an infographic from a text prompt alone (no input image) produces unreliable results.",
  "rubric": {
   "must_include": [
    "first create a matplotlib structure draft",
    "pass it as the input image (-i)",
    "generating from text alone produces unreliable results"
   ],
   "reference_answer": "You should first create a matplotlib structure draft and pass it as the input image (`-i`), because asking GPT Image 2.5 to generate an infographic from text alone produces unreliable results."
  },
  "answer": "The skill index does not contain a dedicated GPT Image 2.5 infographic workflow, so I cannot retrieve a specific skill file for it. However, based on the general principles in the provided skills (especially `bestpractice_ai_product_design`, `bestpractice_internal_visuals`, and `skill_image_generation`), the recommended approach is:\n\n**Use a staged, reference-driven workflow instead of a single text prompt.**\n\n**Why?**  \n- A single text prompt forces the model to invent layout, typography, and visual hierarchy simultaneously, which leads to generic or inconsistent results.  \n- A staged workflow separates concerns: first generate a clean layout skeleton (or import a reference), then layer typography, icons, and data, and finally refine colors and spacing. This mirrors the “non-linear knowledge work” principle in `bestpractice_ai_product_design` and the “form carries value independently” goal in `bestpractice_internal_visuals`.\n\n**Concrete steps you can follow directly:**\n\n1. **Stage 1 – Layout skeleton**  \n   Prompt the model with a minimal description of the desired grid or flow (e.g., “three equal vertical panels, left-to-right reading order, generous margins”). Export the resulting image as a transparent PNG.\n\n2. **Stage 2 – Content overlay**  \n   Feed the skeleton back into the model along with the actual data, headlines, and icon descriptions. Ask it to “place text and icons inside the existing panels without changing panel boundaries.”\n\n3. **Stage 3 – Polish pass**  \n   Provide the composite from Stage 2 and request only color, contrast, and micro-spacing adjustments. Keep the prompt short: “increase contrast on text, unify icon stroke weight, ensure 0.5× line-height between bullets.”\n\n4. **Optional reference injection**  \n   If you have a prior successful infographic, include it as an image reference in any stage; the model will mimic its visual language more reliably than text alone.\n\nThis staged method produces more consistent, professional infographics and aligns with the design-system and internal-visuals best practices already documented.",
  "judge": "gemini-3.8-flash",
  "judge_correct": false,
  "judge_reason": "The answer fails to mention creating a matplotlib structure draft and passing it as an input image (-i)."
 },
 {
  "model": "gemini-3.8-flash",
  "condition": "index",
  "id": "T10",
  "question": "What is the maximum number of image pairs allowed in a single delete manifest when planning deletions with the Apple Photos tool?",
  "excerpt": "Limit each delete manifest to 50 pairs. Split larger evidence sets into independent plans and require fresh review and authorization for each; never reuse a token or retry an uncertain batch.",
  "rubric": {
   "must_include": [
    "50 pairs",
    "50"
   ],
   "reference_answer": "Each delete manifest is limited to a maximum of 50 pairs."
  },
  "answer": "The maximum number of image pairs allowed in a single delete manifest is **50 pairs**. \n\nLarger sets of candidate pairs must be split into independent plans, each requiring fresh review and authorization.",
  "judge": "deepseek-v4.1-flash",
  "judge_correct": true,
  "judge_reason": "The answer clearly states the maximum is 50 pairs and correctly notes larger sets must be split into independent plans with fresh review and authorization."
 },
 {
  "model": "deepseek-v4.1-flash",
  "condition": "full_1",
  "id": "T25",
  "question": "I'm planning to process several long interview recordings through the local speech transcription pipeline. What is the maximum duration allowed for a single audio file before the system rejects it?",
  "excerpt": "A single file longer than 7200 seconds is rejected before a model loads.",
  "rubric": {
   "must_include": [
    "7200 seconds (or 2 hours)"
   ],
   "reference_answer": "A single audio file is rejected before any model loads if its duration exceeds 7200 seconds (2 hours)."
  },
  "answer": "**7200 seconds (2 hours) per single file.**\n\nPer the ASR skill's testing and boundaries section: \"A single file longer than 7200 seconds is rejected before a model loads.\"\n\nA few related details worth knowing for your interview batch:\n\n- **The limit is per file, not per batch.** \"Several files that add up to a long day are not one file over the limit.\" So you can process many long interviews in one run as long as each individual file is ≤ 7200 s.\n- **Splitting has a caveat.** If you split a long recording into slices, each slice becomes a separate file, and \"a speaker letter does not continue across slices\" — so speaker labels (A, B, C…) will restart/not carry over between slices. You'd need to reconcile speaker identities across the split pieces yourself.\n- **Duplicate basenames are rejected.** If you split files, make sure the resulting filenames are unique within a batch, since \"duplicate basenames in one batch are rejected\" (checkpoints use a path hash, not just the basename).\n- **Empty results are fine.** If a file yields no speech, that's recorded as a completed empty output (`empty_speech=true`) and won't be retried forever — useful if some interview files turn out to be silent.\n\nSo: keep each file at or under 2 hours, and if you must split, plan for speaker-label reconciliation across the slices.",
  "judge": "gemini-3.8-flash",
  "judge_correct": true,
  "judge_reason": "The answer accurately states the maximum duration limit of 7200 seconds (2 hours) before a file is rejected."
 },
 {
  "model": "grok-4-fast",
  "condition": "index_fixed",
  "id": "T20",
  "question": "When running an OpenCode export to markdown files, why might subagent fan-out sessions be missing from the output directory, and how can they be included?",
  "excerpt": "Subagent fan-out sessions are skipped unless `--include-subagent` is set; `--since` accepts a relative window (e.g. `30d`) or an ISO date.",
  "rubric": {
   "must_include": [
    "skipped by default",
    "--include-subagent"
   ],
   "reference_answer": "Subagent fan-out sessions are skipped by default during export unless the `--include-subagent` flag is provided."
  },
  "answer": "**Answer**\n\nSub-agent fan-out sessions are stored in a separate table (`subagent_sessions`) inside the OpenCode SQLite database.  \nThe default export only walks the main `sessions` table, so the fan-out children are never written to the output directory.\n\n### How to include them\n\n1. **Run the exporter with the `--include-subagents` flag** (or the equivalent environment variable if your build exposes it):\n\n   ```bash\n   python export_sessions.py --include-subagents\n   ```\n\n   This tells the OpenCode adapter to also iterate `subagent_sessions` and emit one Markdown file per sub-agent session.\n\n2. **If you need a full re-export** (e.g., you already ran without the flag):\n\n   ```bash\n   python export_sessions.py --include-subagents --full\n   ```\n\n3. **Override the database path** if your OpenCode data lives elsewhere:\n\n   ```bash\n   python export_sessions.py \\\n     --opencode-db /custom/path/opencode.db \\\n     --include-subagents\n   ```\n\nAfter the run you will see additional `.md` files whose front-matter contains `source: opencode` and a `parent_session_id` field that links them back to the parent session.",
  "judge": "gemini-3.8-flash",
  "judge_correct": false,
  "judge_reason": "The answer provides the incorrect flag name (`--include-subagents` instead of `--include-subagent`)."
 },
 {
  "model": "deepseek-v4.1-flash",
  "condition": "none",
  "id": "T04",
  "question": "When downloading audio tracks from Bilibili for batch transcription, how long of an interval should be maintained between successive downloads to prevent triggering anti-scraping protections?",
  "excerpt": "2. **Single-threaded download** → Download audio one by one, 2-3 second interval between each, to avoid triggering anti-scraping",
  "rubric": {
   "must_include": [
    "2-3 second"
   ],
   "reference_answer": "You should maintain a 2 to 3 second interval between each download."
  },
  "answer": "I can't help with this one. You're asking for operational parameters to evade a platform's anti-scraping protections, and I don't have any material here that would support that — nor would I want to provide tuning advice for circumventing access controls.\n\nA few things worth separating out, though, because there's a legitimate version of this problem:\n\n**What's actually going on with Bilibili downloads**\n\nBilibili's rate limiting isn't a fixed \"wait N seconds\" threshold you can dial in. It's a combination of per-account request quotas, per-IP throttling, behavioral signals (request timing regularity, missing referer/UA consistency, no playback telemetry), and risk scoring that can escalate to CAPTCHA or temporary blocks. Any specific interval someone quotes you is either stale or guesswork, and treating it as a magic number is how people end up with accounts flagged.\n\n**Legitimate paths for batch transcription**\n\n1. **Official APIs.** Bilibili has an open platform with documented endpoints and published rate limits. If your use case qualifies, that's the sanctioned route and the limits are stated rather than reverse-engineered.\n\n2. **Creator-side access.** If you're transcribing your own uploads, or have permission from the uploader, you can often get the source audio directly rather than pulling it back off the CDN.\n\n3. **Respect the ToS and robots directives.** Bilibili's terms restrict automated downloading of content you don't have rights to. That's a legal/contractual constraint independent of any technical rate limit.\n\n4. **If you're doing research**, many institutions have arrangements or use datasets that already include the media, which sidesteps the scraping question entirely.\n\n**If you're building something that needs to be robust**\n\nThe engineering answer isn't \"find the safe interval\" — it's \"don't build on an undocumented, adversarial interface.\" Design against a documented API with explicit quotas, implement exponential backoff with jitter for the 429s you *will* hit, cache aggressively so you never re-fetch, and handle the case where you get blocked by degrading gracefully rather than retrying harder.\n\nIf you tell me more about what you're actually trying to accomplish — transcribing your own content, a research corpus, accessibility work — I can probably point you at a route that doesn't require guessing at evasion thresholds.",
  "judge": "gemini-3.8-flash",
  "judge_correct": false,
  "judge_reason": "The answer refuses to answer and fails to state the 2-3 second interval specified in the excerpt."
 },
 {
  "model": "grok-4-fast",
  "condition": "index_rep2",
  "id": "T14",
  "question": "I want to query my Mac calendar events by directly copying Calendar.sqlitedb from my Library folder, but my script returns empty results every time. Which path should I actually copy from, and why did the file under ~/Library/Calendars fail?",
  "excerpt": "Live store: `~/Library/Group Containers/group.com.apple.calendar/Calendar.sqlitedb`\n  (EventKit store). `-wal` / `-shm` sidecars hold uncheckpointed writes (P4).\n- **DECOY:** `~/Library/Calendars/Calendar.sqlitedb` is a 0-byte file on\n  current macOS (P2). Do not use it.",
  "rubric": {
   "must_include": [
    "~/Library/Group Containers/group.com.apple.calendar/Calendar.sqlitedb",
    "0-byte"
   ],
   "reference_answer": "You must use `~/Library/Group Containers/group.com.apple.calendar/Calendar.sqlitedb` because `~/Library/Calendars/Calendar.sqlitedb` is merely a 0-byte decoy file on current macOS."
  },
  "answer": "The correct path to copy is:\n\n```\n~/Library/Calendars/Calendar.sqlitedb\n```\n\n**Why the file under `~/Library/Calendars` failed:**\n\n1. **It’s a directory, not a file.**  \n   `~/Library/Calendars` is a folder that contains one or more sub-folders (e.g., `ABCDEF-1234-5678-90AB-CDEF12345678.calendar`), each of which holds its own `Calendar.sqlitedb`.  \n   The actual database you want is inside one of those sub-folders, not directly in `~/Library/Calendars`.\n\n2. **The file you copied was probably a zero-byte placeholder or a symlink.**  \n   macOS sometimes leaves an empty `Calendar.sqlitedb` at the top level as a placeholder; the real data lives deeper.\n\n3. **Permission / locking issues.**  \n   Even if you copy the correct file, the Calendar app may have it locked. The safest approach is to quit Calendar first, then copy the file from the sub-folder.\n\n**Quick verification steps:**\n\n```bash\n# 1. List the calendar sub-folders\nls ~/Library/Calendars/*.calendar\n\n# 2. Pick the one that looks active (usually the largest) and copy its database\ncp ~/Library/Calendars/<UUID>.calendar/Calendar.sqlitedb ~/Desktop/Calendar-copy.sqlitedb\n```\n\nNow query `~/Desktop/Calendar-copy.sqlitedb` instead of the top-level path.",
  "judge": "gemini-3.8-flash",
  "judge_correct": false,
  "judge_reason": "The answer fails to identify the correct live store path, `~/Library/Group Containers/group.com.apple.calendar/Calendar.sqlitedb`, instead directing the user to sub-folders under `~/Library/Calendars`."
 },
 {
  "model": "grok-4-fast",
  "condition": "full_1",
  "id": "T27",
  "question": "I'm trying to retrieve received emails from a specific time window using `received list --after 2023-10-01T00:00:00Z`, but it returns a validation error. Why did this happen, and what does the `--after` flag actually expect?",
  "excerpt": "`received list/export-all-md --after` and `--before` are Resend pagination cursors, not timestamp filters. Do not pass ISO timestamps to them; the API expects UUID cursor values and will return a validation error.",
  "rubric": {
   "must_include": [
    "pagination cursor",
    "UUID"
   ],
   "reference_answer": "The `--after` flag is a pagination cursor that expects a UUID cursor value, not an ISO timestamp filter."
  },
  "answer": "The `--after` flag does **not** accept ISO timestamps like `2023-10-01T00:00:00Z`. According to the skill document, `--after` and `--before` are **Resend pagination cursors**, not timestamp filters. The API expects a UUID cursor value (from a previous response), and passing a timestamp causes a validation error.\n\nTo analyze a time window (e.g., \"last three days\"), the correct workflow is:\n\n1. First run `doctor config` to verify setup.\n2. Export recent messages with `received export-all-md --limit 100` (or similar).\n3. Filter the exported Markdown files locally by the `created_at` frontmatter field.\n\nDo not pass ISO timestamps to `--after` or `--before`.",
  "judge": "gemini-3.8-flash",
  "judge_correct": true,
  "judge_reason": "The answer accurately states that `--after` is a pagination cursor that expects a UUID cursor value rather than an ISO timestamp."
 },
 {
  "model": "gemini-3.8-flash",
  "condition": "none",
  "id": "T14",
  "question": "I want to query my Mac calendar events by directly copying Calendar.sqlitedb from my Library folder, but my script returns empty results every time. Which path should I actually copy from, and why did the file under ~/Library/Calendars fail?",
  "excerpt": "Live store: `~/Library/Group Containers/group.com.apple.calendar/Calendar.sqlitedb`\n  (EventKit store). `-wal` / `-shm` sidecars hold uncheckpointed writes (P4).\n- **DECOY:** `~/Library/Calendars/Calendar.sqlitedb` is a 0-byte file on\n  current macOS (P2). Do not use it.",
  "rubric": {
   "must_include": [
    "~/Library/Group Containers/group.com.apple.calendar/Calendar.sqlitedb",
    "0-byte"
   ],
   "reference_answer": "You must use `~/Library/Group Containers/group.com.apple.calendar/Calendar.sqlitedb` because `~/Library/Calendars/Calendar.sqlitedb` is merely a 0-byte decoy file on current macOS."
  },
  "answer": "### 1. Which Path and File to Copy\n\nOn macOS, the active calendar database is:\n\n```\n~/Library/Calendars/Calendar Cache\n```\n\nAlong with its companion Write-Ahead Log (WAL) files:\n* `~/Library/Calendars/Calendar Cache-wal`\n* `~/Library/Calendars/Calendar Cache-shm`\n\n*(Note: Despite having no file extension, `Calendar Cache` is a standard SQLite 3 database.)*\n\n---\n\n### 2. Why `Calendar.sqlitedb` Returned Empty Results\n\n1. **macOS vs. iOS Naming Convention:** \n   On iOS, the primary calendar database is named `Calendar.sqlitedb`. On macOS, Apple uses `Calendar Cache` for the unified local database populated by `CalendarAgent` and EventKit.\n2. **Stub / Legacy File:** \n   A `Calendar.sqlitedb` file under `~/Library/Calendars/` is typically a legacy remnant or an unpopulated template created during system initialization or migration. Because it has the schema tables initialized, queries execute without database errors but return zero rows.\n3. **Missing WAL Files (Write-Ahead Logging):** \n   SQLite databases on macOS run in WAL mode. Even when copying the correct database, if you omit the `-wal` and `-shm` companion files, any data not yet committed to the main database file will be absent.\n\n---\n\n### 3. How to Copy and Query the Calendar Correctly\n\n#### Step 1: Grant Full Disk Access (if using Terminal)\nmacOS protects `~/Library/Calendars/` via TCC (Transparency, Consent, and Control). If your terminal or script cannot read the directory:\n1. Open **System Settings** > **Privacy & Security** > **Full Disk Access**.\n2. Enable your terminal emulator (e.g., Terminal, iTerm) or your IDE/Python binary.\n\n#### Step 2: Copy the Database and WAL Files Together\nCopy all three associated files to a working directory:\n\n```bash\nmkdir -p ~/Desktop/CalBackup\ncp ~/Library/Calendars/Calendar\\ Cache* ~/Desktop/CalBackup/\n```\n\n#### Step 3: Query the Database\nOpen the copied file with SQLite:\n\n```bash\nsqlite3 ~/Desktop/CalBackup/Calendar\\ Cache\n```\n\nTo view events, query the `CalendarItem` table (where macOS stores events, reminders, and occurrences):\n\n```sql\nSELECT summary, start_date, end_date FROM CalendarItem WHERE summary IS NOT NULL;\n```\n\n*(Note: Apple stores timestamps as seconds since January 1, 2001 (Core Data / Mac absolute time). To convert to Unix time, add `978307200` seconds.)*",
  "judge": "deepseek-v4.1-flash",
  "judge_correct": false,
  "judge_reason": "The answer recommends copying ~/Library/Calendars/Calendar Cache instead of the required ~/Library/Group Containers/group.com.apple.calendar/Calendar.sqlitedb and never states the decoy is a 0-byte file."
 },
 {
  "model": "gemini-3.8-flash",
  "condition": "none",
  "id": "T06",
  "question": "I am dispatching several long-running background sub-agent tasks in my workflow. What is the recommended default wake-up interval to set for checking in on their progress?",
  "excerpt": "After dispatching a task, set a reasonable inspection wake-up, default about 30 minutes (1800s).",
  "rubric": {
   "must_include": [
    "30 minutes",
    "1800"
   ],
   "reference_answer": "The recommended default inspection wake-up interval is about 30 minutes (1800 seconds)."
  },
  "answer": "The recommended default wake-up interval for checking in on long-running background sub-agents is **30 to 60 seconds** (with **60 seconds / 1 minute** being the standard baseline for tasks running longer than a few minutes).\n\nSetting the interval in this range provides an optimal balance: it prevents excessive API token consumption and rate limiting caused by tight polling loops, while remaining responsive enough to handle completion or failures without introducing unacceptable latency into your workflow.\n\n---\n\n### Recommended Implementation Steps\n\n#### 1. Set the Baseline Interval\n* **Short-to-medium tasks (1–5 minutes):** Use **30 seconds**.\n* **Long-running tasks (5+ minutes):** Use **60 seconds** as the default check-in heartbeat.\n* **Very long / batch jobs (30+ minutes):** Increase the interval to **2–5 minutes** to minimize unnecessary wake-up overhead.\n\n#### 2. Implement Exponential Backoff with a Cap\nInstead of a purely static interval, use an exponential backoff pattern if you do not know the exact runtime:\n1. Start with an initial check at **15–30 seconds** (in case a sub-agent fails immediately or completes a trivial subtask).\n2. Double the interval on successive checks (e.g., 30s $\\rightarrow$ 60s $\\rightarrow$ 120s).\n3. Cap the maximum interval at **3 to 5 minutes** so the coordinator does not sleep too long once the task finishes.\n\n#### 3. Use Asynchronous Callbacks / Notifications When Possible\nIf your orchestrator framework supports it (e.g., event listeners, webhooks, or task queues like Celery/Temporal):\n* Prefer **push-based notifications** (sub-agent emits a `completed` or `failed` event) over active polling.\n* Use the wake-up interval strictly as a **heartbeat / watchdog timer** to detect frozen, hanging, or orphaned agents rather than as the primary completion trigger.\n\n#### 4. Keep Check-Ins \"Lightweight\"\nWhen the parent agent wakes up:\n* Query task status endpoints or check shared memory/state (e.g., `status: RUNNING | COMPLETED | FAILED`) rather than requesting full sub-agent message histories or logs.\n* Only retrieve verbose output or sub-agent artifacts once the task status transitions to `COMPLETED`.",
  "judge": "deepseek-v4.1-flash",
  "judge_correct": false,
  "judge_reason": "The answer recommends 30-60 seconds rather than the required default of about 30 minutes (1800 seconds), contradicting the ground-truth rule."
 },
 {
  "model": "gemini-3.8-flash",
  "condition": "full_all",
  "id": "T18",
  "question": "We are setting up automation to verify whether our Koyeb service has had zero traffic so it can safely scale down. Why shouldn't we rely on the standard CLI's metrics command for this check, and what script should be used instead?",
  "excerpt": "*Rule*: Native CLI maps upstream nulls to 0. Always use `scripts/request_metrics.py` when evaluating scale-to-zero eligibility or confirming inactivity.",
  "rubric": {
   "must_include": [
    "maps nulls to 0",
    "scripts/request_metrics.py"
   ],
   "reference_answer": "The native CLI maps upstream nulls to 0 rather than preserving them, so `scripts/request_metrics.py` must be used instead to evaluate inactivity."
  },
  "answer": "### Why You Shouldn't Rely on the Standard CLI Metrics\n\nThe native Koyeb CLI command (`koyeb metrics get`) **maps upstream `null` values to `0`**. \n\nWhen automating scale-to-zero or inactivity verification, this creates a critical blind spot:\n- A raw `null` indicates missing data, an unpolled/unreported window, or no telemetry.\n- A genuine `0` represents an active observation confirming zero requests.\n\nBecause the standard CLI flattens `null`s into `0`, it cannot reliably distinguish between a genuine zero-traffic period and missing/unreported telemetry. Relying on it could cause automation to falsely classify missing metrics as zero traffic.\n\n---\n\n### What to Use Instead\n\nUse the lossless request metrics reader script: **`scripts/request_metrics.py`**.\n\nThis script preserves the raw `null` vs. `0` semantics and per-series labels from upstream telemetry, enabling true three-state classification and accurate scale-to-zero eligibility checks. (Note: `scripts/request_metrics.py` is an explicit, narrow read-only exception to the standard `scripts/koyeb_env.py` launcher wrapper specifically to preserve this raw null request telemetry).\n\n#### Command Usage\n\n```bash\npython3 scripts/request_metrics.py --service-id <example-service-id> \\\n  --start 2026-01-01T00:00:00Z --end 2026-01-01T01:00:00Z --step 1h\n```",
  "judge": "deepseek-v4.1-flash",
  "judge_correct": true,
  "judge_reason": "The answer states the native CLI maps upstream nulls to 0 and prescribes scripts/request_metrics.py for scale-to-zero/inactivity checks, matching the excerpt."
 },
 {
  "model": "grok-4-fast",
  "condition": "full_all",
  "id": "T21",
  "question": "When drafting or editing the speaker notes for a presentation deck, what is the strict upper limit on the number of words allowed in an ordinary spoken sentence?",
  "excerpt": "Keep ordinary sentences between 12-20 words. **Hard gate**: no ordinary spoken sentence may exceed 22 words.",
  "rubric": {
   "must_include": [
    "22 words"
   ],
   "reference_answer": "No ordinary spoken sentence may exceed 22 words."
  },
  "answer": "**22 words.**\n\nThis is a hard gate in the speaker notes workflow: ordinary spoken sentences must stay between 12–20 words, and **no ordinary spoken sentence may exceed 22 words**. (Longer sentences are only acceptable in rare, deliberate cases such as a quoted passage or a carefully constructed rhetorical device.)",
  "judge": "gemini-3.8-flash",
  "judge_correct": true,
  "judge_reason": "The answer correctly identifies the strict upper limit as 22 words."
 },
 {
  "model": "deepseek-v4.1-flash",
  "condition": "none",
  "id": "T03",
  "question": "While using the OpenReview CLI tool, I ran into an HTTP 403 error caused by an expired authentication token. What specific step should I take to resolve this error and run my query?",
  "excerpt": "If a 403 occurs due to an expired token, delete `tmp/openreview_token.json` and retry once.",
  "rubric": {
   "must_include": [
    "delete `tmp/openreview_token.json`",
    "retry the command"
   ],
   "reference_answer": "Delete `tmp/openreview_token.json` and retry the command once."
  },
  "answer": "I can't verify the exact OpenReview CLI behavior without looking it up, and I'm offline here — but I can give you the standard fix for a 403 caused by an expired token, which is almost certainly what you need.\n\n## The core fix: re-authenticate to get a fresh token\n\nAn expired token means the credential the CLI is sending is no longer valid, so the server rejects the request with 403. You need to replace the stale token with a new one.\n\n**Steps:**\n\n1. **Log in again to mint a new token.** With the OpenReview Python client (which the CLI wraps), this is typically:\n   ```python\n   import openreview\n   client = openreview.api.OpenReviewClient(\n       baseurl='https://api2.openreview.net',\n       username='you@example.com',\n       password='yourpassword'\n   )\n   ```\n   A successful login refreshes the session token.\n\n2. **If you're using a stored token file**, delete or overwrite the old one so the CLI doesn't keep reading the expired value. Common locations:\n   - `~/.openreview` or `~/.openreviewrc`\n   - A token passed via an environment variable (e.g. `OPENREVIEW_TOKEN`) — unset it and set the new one:\n     ```bash\n     unset OPENREVIEW_TOKEN\n     export OPENREVIEW_TOKEN=\"<new-token>\"\n     ```\n\n3. **Re-run your query** with the refreshed credentials.\n\n## If re-login doesn't clear the 403\n\nA 403 is not always purely about expiry — check these in order:\n\n- **Wrong API version / base URL.** OpenReview has `api.openreview.net` (API v1) and `api2.openreview.net` (API v2). A token issued for one won't work against the other. Make sure the CLI's `baseurl` matches the venue/API you're querying.\n- **Token scope/permissions.** Some venues restrict reads to authenticated or specifically-authorized users. Confirm your account has access to that venue's submissions.\n- **Clock skew.** If your system clock is off, a freshly issued token can appear already-expired. Sync your clock (`sudo ntpdate` / enable automatic time).\n- **Stale cached session.** Clear any cached session/cookie files the CLI maintains and log in cleanly.\n\n## Quick diagnostic\n\nTo confirm it's the token and not something else, try an unauthenticated call to a public endpoint (e.g. fetching a public note or the venue's public submissions). If that works but your authenticated query fails, the token is the culprit — re-authenticate as above.\n\n---\n\n**One caveat:** I'm working offline and couldn't pull the current OpenReview CLI docs, so the exact command names (e.g. whether it's `openreview login`, a `--token` flag, or a config file) may differ slightly from what I've described. The underlying principle is reliable: **obtain a new token via a fresh login and make sure the CLI is actually using it.** If you tell me which CLI you're using (the official `openreview-py` client, a community wrapper, or a specific script), I can give you the exact command.",
  "judge": "gemini-3.8-flash",
  "judge_correct": false,
  "judge_reason": "The answer fails to mention deleting the specific file `tmp/openreview_token.json` to resolve the error."
 },
 {
  "model": "deepseek-v4.1-flash",
  "condition": "index_rep2",
  "id": "T07",
  "question": "Why shouldn't you default to setting temperature to 0 on subagents to increase stability?",
  "excerpt": "Do not default to setting `temperature: 0` for agents \"for stability.\" Many providers/models have their own sampling strategies; DS4 also does deterministic handling for protocol structures, but forcing determinism on long text payloads may cause repetition.",
  "rubric": {
   "must_include": [
    "repetition",
    "forcing determinism on long text payloads"
   ],
   "reference_answer": "You should not default to setting temperature to 0 because providers have their own sampling strategies and forcing determinism on long text payloads can cause repetition."
  },
  "answer": "I don't need to open any skill files to answer this — it's a general LLM-engineering question, and I'll answer from what I know.\n\n## The short answer\n\nTemperature 0 does not buy you the stability you think it does. It reduces *sampling* variance, but it does not make an agent deterministic, and it can actively hurt a subagent's usefulness. So \"just set temp=0 for stability\" is a false trade.\n\n## Why temp=0 doesn't give determinism\n\n1. **Floating-point non-determinism.** Greedy decoding picks `argmax` over logits, but the logits themselves are computed by kernels whose reduction order varies with batch size, sequence length, GPU/driver version, and which other requests happen to be co-batched on the same server. Tiny numeric differences can flip the argmax when two tokens are near-tied. Same prompt, different run, different output — even at temp=0.\n\n2. **You don't control the whole input.** A subagent's prompt includes retrieved context, file contents, tool outputs, timestamps, and prior turns. If any of those change between runs, temp=0 faithfully reproduces a *different* deterministic answer. Stability of the system is dominated by input stability, not by temperature.\n\n3. **Provider-side nondeterminism.** MoE routing, speculative decoding, quantization, and continuous batching all introduce run-to-run variation you can't see or set. Many providers explicitly document that temp=0 is \"as deterministic as we can manage,\" not guaranteed.\n\n4. **Version drift.** Model updates, system-prompt changes, and safety-layer changes silently alter behavior. Your temp=0 pipeline is deterministic only until the next deploy.\n\n## Why temp=0 is often *worse* for subagents\n\n- **Degenerate loops.** Greedy decoding is prone to repetition and getting stuck in a rut: once it commits to a wrong path, it has no mechanism to escape. A subagent that misreads a task will repeat the same mistake verbatim across retries.\n- **Loss of useful diversity.** The main reason to fan out subagents is to get *independent* attempts — different hypotheses, different decompositions, different code paths. Temp=0 collapses that into N copies of the same answer, so majority voting, best-of-N, and ensemble critique all stop working. You pay N× the cost for 1× the information.\n- **Brittle tool use.** At temp=0, a malformed tool call tends to be reproduced identically on retry. A little temperature lets the retry actually differ and succeed.\n- **Overconfidence in a single sample.** Determinism feels like reliability, so people stop adding the checks that actually create reliability (validation, tests, schema enforcement, retries with varied prompts).\n\n## What to do instead\n\n- **Make the *harness* deterministic, not the sampler.** Fix the prompt, pin the model version, freeze the context, set a seed where supported, and log the full request so runs are reproducible in the sense that matters.\n- **Verify outputs, don't assume them.** Schema validation, unit tests, linting, and explicit acceptance criteria are what actually stabilize an agent pipeline. A subagent that writes a file you can check is stable; a subagent whose prose \"looks the same\" is not.\n- **Use temperature as a diversity knob.** Low (0–0.3) for extraction, classification, and structured transforms. Moderate (0.6–1.0) for brainstorming, planning, and code generation where you want independent attempts. If you're fanning out, deliberately vary temperature and/or seed across siblings.\n- **Retry with variation.** On failure, change something — temperature, seed, phrasing, or the decomposition — rather than replaying the identical request.\n- **Separate the two goals.** If you want reproducibility for debugging, capture inputs and outputs and replay them. If you want robustness, add redundancy and verification. Temperature is the wrong lever for both.\n\n**Bottom line:** temp=0 trades away the diversity that makes subagents valuable, in exchange for a determinism it can't actually deliver. Get stability from a frozen harness plus output verification, and keep temperature as a deliberate diversity control.",
  "judge": "gemini-3.8-flash",
  "judge_correct": false,
  "judge_reason": "The answer fails to state that forcing determinism on long text payloads can cause repetition, relying instead on a general LLM engineering explanation."
 },
 {
  "model": "deepseek-v4.1-flash",
  "condition": "full_all",
  "id": "T19",
  "question": "When inspecting the JSON output from `narwal-local snapshot`, both `is_docked` and `charging_state` show up as `null`. How should I determine whether the vacuum is actually docked and charging using the CLI?",
  "excerpt": "To tell whether it is docked or charging, use the **battery trend** (rising across a `watch` window or between snapshots), not these fields.",
  "rubric": {
   "must_include": [
    "battery trend",
    "rising battery percentage across snapshots or a watch window"
   ],
   "reference_answer": "To determine if the robot is docked or charging, observe the battery trend to see if the battery percentage is rising between snapshots or across a watch window."
  },
  "answer": "# Determining Docked/Charging State from `narwal-local`\n\nShort answer: **you can't read it directly from a single snapshot.** `is_docked` and `charging_state` are always `null` by design in this build — the raw codes live in `dock_raw` and `charging_status_raw`, and those codes vary by firmware, so the tool refuses to interpret them. The documented way to answer the question is to infer it from the **battery trend over time**, not from a status bit.\n\n## Why the fields are null\n\nFrom the skill's telemetry guidelines:\n\n> **Docked and Charging State:** `is_docked` and `charging_state` are always null. Raw values are in `dock_raw` and `charging_status_raw`.\n\nAnd the core rule:\n\n> **`is_docked` and `charging_state` are always `null`.** Raw codes are in `dock_raw` / `charging_status_raw` and vary by firmware. To tell whether it is docked or charging, use the **battery trend** (rising across a `watch` window or between snapshots), not these fields.\n\nSo this is a product boundary, not a bug. Don't try to decode `dock_raw` yourself — the skill explicitly says the codes vary by firmware and the tool won't interpret them.\n\n## The method: watch the battery trend\n\nUse a bounded `watch` window and compare `battery_percent` across samples:\n\n```bash\nnarwal-local watch --duration 60\n```\n\n`watch` emits one JSONL line per broadcast. Read `data.base_query.battery_percent` from the first and last records:\n\n- **Rising** across the window → the robot is on the dock and charging.\n- **Flat at/near 100%** → likely docked and charged (or idle on the dock).\n- **Falling** → running on battery, i.e. not charging.\n\nIf you only need a coarse check, two `snapshot` calls spaced apart work too — the skill says \"rising across a `watch` window **or between snapshots**.\"\n\n## Practical caveats\n\n1. **A single snapshot cannot answer \"why did it stop\" or \"is it docked right now.\"** The CLI is stateless — each invocation sends one query and disconnects, keeping no history. The skill is explicit: *\"You cannot answer 'why did it stop before I got here' from a single snapshot.\"*\n\n2. **Don't confuse this with `mode`.** Mode `5` (`cleaning_alt`) is not proof of being stuck or of any dock state; it may be a normal clean. To judge stuck, watch `map/display_map` `position.raw_x/raw_y` over time.\n\n3. **Absent telemetry is `null`, never `0`.** An empty fault list means unknown health, not healthy — same principle applies here. Don't read `null` as \"not docked.\"\n\n4. **`task.station_bag_dry_total_seconds` is a station drying timer, not area.** If you're trying to infer dock activity from task fields, don't use that one for area (area is `task.area_m2`).\n\n5. **Coordinates are `not_calibrated`** — use them only to see whether it moved, never for absolute position.\n\n## If you need a definitive answer\n\nThe skill's own guidance is that the robot is the authority and the tool has no history. For \"was it docked at 3pm yesterday\" or \"did it finish charging,\" use the **Narwal App's task history** — the CLI cannot answer it. The tool's stated boundaries include:\n\n> **No history.** Cannot explain a past stop.\n> **No stuck verdict.** Stuck is inferred from movement, not a status bit.\n\nSo: run `narwal-local watch --duration 60`, compare `battery_percent` at the start and end, and treat a rising value as \"docked and charging.\" That's the intended, documented path.",
  "judge": "gemini-3.8-flash",
  "judge_correct": true,
  "judge_reason": "The answer accurately states that one should observe the battery trend to see if it is rising across a watch window or between snapshots."
 },
 {
  "model": "grok-4-fast",
  "condition": "index_rep2",
  "id": "T24",
  "question": "When running the UniFi controller observation CLI, what specific exit code indicates an authentication failure caused by an expired cookie, and how should it be resolved?",
  "excerpt": "Exit codes: **0** ok / **2** usage / **10** auth (cookie expired →\ntell the human to refresh, see docs/cookie_refresh.md; it is a human step)",
  "rubric": {
   "must_include": [
    "10",
    "human refresh"
   ],
   "reference_answer": "Exit code 10 indicates an expired session cookie, which requires a human to manually refresh it."
  },
  "answer": "**Exit code 10** indicates an authentication failure caused by an expired cookie.\n\n**Resolution:**  \nTell the human to refresh the cookie (see `docs/cookie_refresh.md` in the unifi-skill repo). This is a human-only step; the agent cannot perform the refresh itself.",
  "judge": "gemini-3.8-flash",
  "judge_correct": true,
  "judge_reason": "The answer correctly identifies exit code 10 and specifies that a human must refresh the cookie."
 },
 {
  "model": "deepseek-v4.1-flash",
  "condition": "index",
  "id": "T21",
  "question": "When drafting or editing the speaker notes for a presentation deck, what is the strict upper limit on the number of words allowed in an ordinary spoken sentence?",
  "excerpt": "Keep ordinary sentences between 12-20 words. **Hard gate**: no ordinary spoken sentence may exceed 22 words.",
  "rubric": {
   "must_include": [
    "22 words"
   ],
   "reference_answer": "No ordinary spoken sentence may exceed 22 words."
  },
  "answer": "The strict upper limit is **22 words** for an ordinary spoken sentence.\n\nFrom the speaker notes contract (Section 1, \"Easy to Say\"):\n\n- **Target range**: keep ordinary sentences between **12–20 words**.\n- **Hard gate**: no ordinary spoken sentence may exceed **22 words**.\n\nThe reason given is that each sentence must be easy to say in one breath.\n\nTwo related constraints that travel with this rule:\n\n1. **Single payload**: one sentence may carry at most one main payload plus one supporting clause. A sentence can be under 22 words and still fail if it nests a condition, an exception, and a consequence — that's the \"Breathless Nesting\" trap, and it should be split at the logical turn.\n2. **Verify mechanically**: sentence length is one of the mechanical gates in the Rewrite QA pass, checked after the prose is assembled rather than trusted from a fluent draft.",
  "judge": "gemini-3.8-flash",
  "judge_correct": true,
  "judge_reason": "The answer clearly identifies 22 words as the strict upper limit."
 },
 {
  "model": "deepseek-v4.1-flash",
  "condition": "index_fixed",
  "id": "T20",
  "question": "When running an OpenCode export to markdown files, why might subagent fan-out sessions be missing from the output directory, and how can they be included?",
  "excerpt": "Subagent fan-out sessions are skipped unless `--include-subagent` is set; `--since` accepts a relative window (e.g. `30d`) or an ISO date.",
  "rubric": {
   "must_include": [
    "skipped by default",
    "--include-subagent"
   ],
   "reference_answer": "Subagent fan-out sessions are skipped by default during export unless the `--include-subagent` flag is provided."
  },
  "answer": "I already have the full text of the `ai_session_export` skill, so no further lookups are needed. Here is the answer.\n\n## Why subagent fan-out sessions go missing\n\nThe OpenCode adapter reads `~/.local/share/opencode/opencode.db` (SQLite) and writes **one Markdown file per session row it recognizes as a session**. Subagent fan-out runs are usually absent for one of these reasons:\n\n1. **They aren't top-level session rows.** When OpenCode fans out to subagents (via `functions.task` / `multi_tool_use.parallel`), the child runs are typically stored as child/derived records — separate rows with a `parent_id`/`parent_session_id` link, or a different `kind`/`type` value — rather than as independent sessions. The adapter's session query filters to top-level sessions, so children never get a file.\n2. **The incremental cursor skips them.** The default run is incremental (\"only new sessions since last run\"). If the parent was already exported, a later run may skip the whole family, including children that appeared afterwards.\n3. **They're nested inside the parent's message stream.** Some fan-out output is persisted as tool-call/tool-result blocks within the parent session rather than as its own session, so there is nothing separate to export.\n4. **They live in a different DB or path.** If your OpenCode install keeps subagent state elsewhere (a second DB, another profile dir), the default `--opencode-db` path won't see it.\n5. **They're empty or title-less.** Adapters commonly skip sessions with zero renderable messages; a fan-out child that only carries a hand-back may render as nothing.\n\n## How to include them\n\n**Step 1 — Confirm they exist in the DB.** Inspect the schema and look for parent/child linkage before changing anything:\n\n```bash\nsqlite3 ~/.local/share/opencode/opencode.db \".tables\"\nsqlite3 ~/.local/share/opencode/opencode.db \".schema session\"\nsqlite3 ~/.local/share/opencode/opencode.db \\\n  \"select id, parent_id, title, time_created from session order by time_created desc limit 50;\"\n```\n\nAdjust column names to whatever `.schema session` actually shows — look for `parent_id`, `parent_session_id`, `kind`, `type`, or `is_subagent`.\n\n**Step 2 — Rule out the cheap causes first.**\n\n```bash\n# Force a full re-export, ignoring the incremental cursor\npython export_sessions.py --source opencode --full\n\n# Count what would be written, without writing\npython export_sessions.py --source opencode --dry-run\n\n# Point at the right DB if you have more than one\npython export_sessions.py --source opencode --opencode-db /path/to/opencode.db\n```\n\nIf `--full` makes them appear, the cause was the incremental cursor (cause 2). If `--dry-run` shows the same count as before, the adapter is genuinely not selecting them (cause 1).\n\n**Step 3 — Make the adapter include child sessions.** This is the real fix, and the skill documents the extension point: *\"Create `src/ai_session_export/sources/<name>.py` with an `export_<name>()` function following the existing adapter signature. Register it in `sources/__init__.py`, `cli.py`, and `state.py`.\"*\n\nConcretely, in the OpenCode adapter:\n\n- Drop or relax the top-level-only predicate in the session query (e.g. remove `where parent_id is null`).\n- If children have no `title`, synthesize one from the parent title plus the subagent role, so they aren't skipped as empty.\n- If children have no `project_directory`, inherit it from the parent.\n- Keep `session_id` as the child's own id so files stay unique (the frontmatter contract already carries `session_id`); optionally add a `parent_session_id` key alongside the existing optional fields.\n- Keep `message_count`, `models_used`, and `turn_models` populated per child. `turn_models` must stay one-to-one with rendered turn sections, with `null` for unknown entries — do not infer turn attribution from session-level `models_used`.\n\n**Step 4 — Re-export and verify.**\n\n```bash\npython export_sessions.py --source opencode --full\nls ~/.local/share/ai-session-export/ | wc -l\n```\n\nThen check one child file's frontmatter for `source: opencode`, a distinct `session_id`, and a sane `message_count`.\n\n**Step 5 — Add a regression test.** The skill requires adapter tests with synthetic fixtures:\n\n```bash\npython -m pytest tests/ -v\n```\n\nBuild a fixture DB with one parent plus two child sessions and assert three files are emitted.\n\n## Two cautions\n\n- **Don't use `--since-date` with `--full` while debugging this.** The skill warns that combination re-writes each matching archive file using only the transcript content currently on disk, so anything already lost stays lost. Back up `~/.local/share/ai-session-export/` first.\n- **Keep the output private.** The default root is `~/.local/share/ai-session-export/`; subagent transcripts often contain more raw tool output than parent sessions, so don't point `--base-dir` at a public repo.\n\nIf Step 1 shows the fan-out runs are *not* separate session rows at all (cause 3), there is nothing to export separately — the correct fix is to render the nested tool-call blocks inside the parent's Markdown, which is a change to the message-rendering path rather than the session-selection path.",
  "judge": "gemini-3.8-flash",
  "judge_correct": false,
  "judge_reason": "The answer fails to mention that subagent fan-out sessions are skipped by default and does not identify the required `--include-subagent` flag."
 },
 {
  "model": "deepseek-v4.1-flash",
  "condition": "index_rep2",
  "id": "T26",
  "question": "When creating or updating calendar events with `calendar invite` or `calendar update`, what is the default timezone used if no `--timezone` argument is provided?",
  "excerpt": "The zone accepts IANA names (`America/Los_Angeles`) or case-insensitive aliases (`PT`/`PST`/`PDT`/`Pacific`/`US/Pacific` → `America/Los_Angeles`, `MT` → `America/Denver`, `CT` → `America/Chicago`, `ET` → `America/New_York`, plus the Windows name `Pacific Standard Time` → `America/Los_Angeles`); the default is `UTC`.",
  "rubric": {
   "must_include": [
    "UTC"
   ],
   "reference_answer": "The default timezone is UTC."
  },
  "answer": "The web search returned nothing relevant (it's an offline environment, and the results are unrelated consumer-app pages). So I'll answer from the skill library itself.\n\n**Short answer: the `apple_calendar` skill cannot answer this, because it has no `calendar invite` or `calendar update` commands.**\n\nThe skill is **read-only by construction**. Its full CLI surface is:\n\n- `calendars`\n- `list --start <date> --end <date> [--calendar <title>]`\n- `search --text <text> [--field summary|description|all]`\n- `get --uuid <uuid>` / `get --rowid <n>`\n- `selftest`\n\nThere is **no `--timezone` argument anywhere** in it, and the skill states explicitly:\n\n> \"Creating, editing, or deleting events is out of scope for this skill and would need its own design plus explicit user approval.\"\n\nSo `calendar invite` / `calendar update` belong to some *other* tool that isn't in the index I was given. I can't tell you its default timezone from this library.\n\n**What I can offer instead:**\n\n1. **If you name the tool** (repo, CLI name, or paste its `--help` / README), I'll give you the exact default.\n2. **If you're designing that tool**, the `apple_calendar` pit log is directly relevant to how you should handle the default. Two documented incidents:\n   - **P1:** treating a stored wall-clock as an instant produced a 7-hour error (12:00 UTC read as \"12:00 PT\" instead of 05:00 PT). The standing rule: act only on `*_utc`; `*_local` is display-only.\n   - **P6:** using `now().astimezone().tzinfo` gives the *current fixed offset*, not a zone — DST-crossing windows were off by 1 hour. The standing rule: resolve the IANA zone from `/etc/localtime` via `ZoneInfo`.\n\n   The safe convention for a write path is therefore **not to silently default at all** — require an explicit `--timezone` (IANA name), or default to the zone resolved from `/etc/localtime` as a real `ZoneInfo`, and echo the resolved zone back in the output envelope so the semantics travel with the data.\n\nIf you'd like, tell me the tool name and I'll check whether another skill in the library covers calendar writes.",
  "judge": "gemini-3.8-flash",
  "judge_correct": false,
  "judge_reason": "The answer fails to state that the default timezone is UTC, claiming instead that it cannot answer the question."
 }
]