{
 "skill": "skill_v0_en.md",
 "scores": [
  0.6711,
  0.6844,
  0.6667
 ],
 "flipping": [
  "M023",
  "M084"
 ],
 "answered_per_run": {
  "0": [
   "M018",
   "M036",
   "M084",
   "M086"
  ],
  "1": [
   "M018"
  ],
  "2": [
   "M012",
   "M018",
   "M080"
  ]
 },
 "detail": {
  "M023": [
   {
    "pass": true,
    "answered": false,
    "misses": {},
    "extras": {},
    "output": "A repeatable loss of audio before processing completes would stop it. So would cross-workspace access to a transcript, obviously, or a billing path that submits the same recording repeatedly. Those checks are passing right now. A slower response under a burst would make me reduce the cohort or add a queue limit, not automatically cancel. Incorrect names are a quality gate, but I need Aoife's measured baseline before I propose a responsible threshold."
   },
   {
    "pass": false,
    "answered": false,
    "misses": {
     "Aoife": 1
    },
    "extras": {},
    "output": "A repeatable loss of audio before processing completes would stop it. So would cross-workspace access to a transcript, obviously, or a billing path that submits the same recording repeatedly. Those checks are passing right now. A slower response under a burst would make me reduce the cohort or add a queue limit, not automatically cancel. Incorrect names are a quality gate, but I need IFA's measured baseline before I propose a responsible threshold."
   },
   {
    "pass": true,
    "answered": false,
    "misses": {},
    "extras": {},
    "output": "A repeatable loss of audio before processing completes would stop it. So would cross-workspace access to a transcript, obviously, or a billing path that submits the same recording repeatedly. Those checks are passing right now. A slower response under a burst would make me reduce the cohort or add a queue limit, not automatically cancel. Incorrect names are a quality gate, but I need Aoife's measured baseline before I propose a responsible threshold."
   }
  ],
  "M084": [
   {
    "pass": false,
    "answered": true,
    "misses": {},
    "extras": {
     "Jaxon": 1,
     "Saoirse": 2,
     "Xochitl": 2,
     "Aoife": 2,
     "Mairead": 1,
     "Tadhg": 2,
     "KV cache": 1,
     "prefill": 1
    },
    "output": "I don't have a transcript to correct — your message is the meeting request itself, not a raw speech-to-text transcript. So there's nothing for me to fix names or terms in.\n\nIf you paste the transcript, I'll return only the corrected text. In the meantime, here's what I'd need from the report to answer your three questions, since I can't see it either:\n\n**1. Define \"name recall\" explicitly.** The three readings I'd expect in the room:\n- **Strict recall** — the exact name appears verbatim in the output.\n- **Fuzzy recall** — the name appears with a minor spelling variant or partial form (e.g., first name only).\n- **Semantic recall** — the right person is identified even if the name is wrong or absent (e.g., \"the backend engineer\" for Jaxon).\n\nThese give very different numbers on the same outputs. The report should state which one it scores, whether matching is case- and diacritic-sensitive (relevant for Saoirse, Aoife, Mairead, Tadhg, Xochitl), and whether a wrong name counts as a miss or as a separate error class.\n\n**2. Baseline vs. hinted.** I'd want, per condition: n, the recall definition used, the point estimate with a confidence interval, and the exact hint format — because \"hinted\" could mean a name list in the prompt, a retrieval step over prior context, or a post-hoc correction pass. Those aren't comparable, and a hint that leaks the answer inflates recall without telling you anything about the system.\n\n**3. Where the average hides a failure.** The usual suspects worth disaggregating:\n- **Rare and non-English names** — averages are dominated by common names; Aoife, Xochitl, Tadhg, and Saoirse are exactly where a mean will look fine while the tail fails.\n- **Per-name vs. per-mention** — one heavily repeated name can carry the average.\n- **Position in context** — recall that collapses past a certain context length, which matters directly for prefill and KV cache behavior.\n- **Confident wrong names** — a miss is visible; a fluent substitution is not, and it's the failure that erodes trust fastest.\n- **Conditional accuracy** — recall among outputs that were otherwise correct, versus recall overall.\n\nSend the transcript and the report numbers, and I'll do the correction pass and then dig into the disaggregation."
   },
   {
    "pass": true,
    "answered": false,
    "misses": {},
    "extras": {},
    "output": "I don't have the transcript or report you're referring to — no speech-to-text text was included in your message, and there's nothing in this conversation for me to correct or analyze. Could you paste the raw transcript (and the report, if it's separate)? Once I have it, I'll return only the corrected text as instructed, and I can also address the name-recall definition, baseline vs. hinted comparison, and where the averages mask failures."
   },
   {
    "pass": false,
    "answered": false,
    "misses": {},
    "extras": {
     "Jaxon": 1,
     "Saoirse": 1,
     "Kaitlynn": 1,
     "Xochitl": 1,
     "Aoife": 1,
     "Kaveh": 1,
     "Priya": 1,
     "Mairead": 1,
     "Tadhg": 1,
     "Koyeb": 1,
     "vLLM": 1,
     "KV cache": 1,
     "LoRA": 1,
     "MLX": 1,
     "AI Builder Space": 1,
     "Superlinear": 1,
     "intake": 1,
     "VoiceFlow": 1,
     "Codex": 1,
     "prefill": 1
    },
    "output": "I don't have the transcript to correct — your message contains the question you want answered, but not the raw speech-to-text text with the names and terms in it. Paste the transcript and I'll return only the corrected version, with Jaxon, Saoirse, Kaitlynn, Xochitl, Aoife, Kaveh, Priya, Mairead, Tadhg, Koyeb, vLLM, KV cache, LoRA, MLX, AI Builder Space, Superlinear, intake, VoiceFlow, Codex, and prefill fixed as needed and nothing else changed."
   }
  ]
 }
}