{
 "meta": {
  "transcripts": "experiments_en/corpus/transcripts_grok.json",
  "judge_of": {
   "deepseek-v4.1-flash": "gemini-3.8-flash",
   "gemini-3.8-flash": "deepseek-v4.1-flash"
  },
  "ask": "Turn this voice memo into a note.",
  "perspective_v2": "# My Perspectives (follow these when writing up notes)\n- The judgment in the memo is my view. Keep its stance and its strength as they are, and write it as \"I think...\".\n- Don't add qualifiers or counterpoints I didn't say, like \"on the other hand\" or \"of course, it depends\".\n- If I'm wrong somewhere, I'll ask you separately to attack it. This step writes only the note itself: no rebuttal, and no neutralizing along the way.",
  "attack_v1": "# My Perspectives (follow these when writing up notes)\n- The judgment in the memo is my view. Keep its stance and its strength as they are, and write it as \"I think...\".\n- Don't add qualifiers or counterpoints I didn't say, like \"on the other hand\" or \"of course, it depends\".\n- Where I'm wrong, leave it for a separate attack step."
 },
 "deepseek-v4.1-flash": {
  "default": {
   "summary": {
    "kept": 9,
    "weakened": 8,
    "hedges": 20,
    "per_memo": {
     "I01": 2,
     "I02": 3,
     "I03": 0,
     "I04": 5,
     "I05": 2,
     "I06": 3,
     "I07": 1,
     "I08": 0,
     "I09": 1,
     "I10": 3
    }
   },
   "rows": {
    "I01": {
     "note": "Both references check out — MCP (Anthropic, Nov 2024) and Agent Skills (Oct 2025) are real, recent, and exactly the kind of \"new noun\" that churns. Here's the note.\n\n---\n\n# Note: The Nouns Are Stale, the Verbs Compound\n\n**Source:** Voice memo (self-recorded)\n**Date:** —\n\n## Core idea\nMost people are learning AI the wrong way. They chase **nouns** — the new named things (MCP this month, Skills the next). What actually separates people is the **verbs** — what you *do* with it.\n\n## The argument\n- **Nouns decay fast.** The named thing everyone's excited about is stale in ~3 months. Chasing them means perpetually restarting from zero.\n- **Verbs compound.** Skill at *applying* AI to real problems carries over regardless of which noun is current.\n- **The trap:** noun-chasing feels like progress (there's always something new to learn) but leaves you with a vocabulary, not a capability.\n\n## Why it holds up\nThe two examples are a good illustration of the churn rate:\n- **MCP** — introduced by Anthropic Nov 2024; adopted by OpenAI and Google within months; donated to the Linux Foundation's Agentic AI Foundation in Dec 2025. A noun that went from \"new\" to \"infrastructure\" in about a year.\n- **Skills** — launched Oct 2025, published as an open standard Dec 2025. Same pattern: novelty → standard → background noise.\n\nNeither is a bad thing to know. The point is that *knowing the name* was never the durable part.\n\n## Open questions / to develop\n- What are the actual **verbs** worth building? (e.g. decompose a task, verify output, build a repeatable workflow, know when *not* to use AI)\n- How do you tell a noun worth learning from one worth ignoring?\n- Is there a version of this that's just contrarian framing? Steelman: sometimes the noun *is* the unlock, and skipping it means missing a real capability shift.\n\n## One-liner\n> The nouns are stale in three months. The verbs are what you keep.\n\n---\n\nWant me to expand the \"verbs worth building\" section into a concrete list, or tighten this into a shorter post/thread version?",
     "headings": [
      "# Note: The Nouns Are Stale, the Verbs Compound",
      "## Core idea",
      "## The argument",
      "## Why it holds up",
      "## Open questions / to develop",
      "## One-liner"
     ],
     "judge": {
      "kept": true,
      "weakened": true,
      "hedges": 2
     }
    },
    "I02": {
     "note": "# Note: Prompt Engineering Is Dead — Context Is the Real Lever\n\n**Core claim**\nPrompt engineering is basically dead. Modern models are smart enough that the bottleneck is no longer wording — it's context. Time spent polishing phrasing is mostly wasted; that time belongs in what the model actually gets to see.\n\n**The argument, in three steps**\n1. **Models have outgrown phrasing.** They intuit intent well enough that clever wording buys little.\n2. **The real constraint is context.** What the model can see — retrieved docs, tool outputs, history, memory, structured data — determines output quality far more than how the request is phrased.\n3. **So reallocate the effort.** Polishing prompts is low-yield; curating context is high-yield.\n\n**Actionable takeaway**\nStop optimizing the sentence. Start optimizing the information environment: what gets retrieved, what persists, what gets compressed, what gets discarded.\n\n---\n\n## Where this holds up — and where it's overstated\n\nWorth flagging, since the memo states it more absolutely than the evidence supports:\n\n**Supported:**\n- The framing is mainstream now. Karpathy's mid-2025 definition of *context engineering* crystallized it, and Tobi Lutke (Shopify) called it \"the art of providing all the context for the task to be plausibly solvable by the LLM.\"\n- The consensus framing is that **prompt engineering is a subset of context engineering**, not a rival. As one source puts it: prompt engineering is what you do *inside* the context window; context engineering is how you decide what *fills* it.\n- DataHub's 2026 report: 82% of IT/data leaders say prompt engineering alone is no longer sufficient; 95% say context engineering matters for agents at scale.\n- The WSJ (April 2025) called the dedicated \"prompt engineer\" job obsolete — models now intuit intent better than humans can hand-tune it.\n\n**Overstated:**\n- **\"Dead\" is the wrong word — \"reclassified\" is closer.** Prompt engineering didn't disappear; it became a contained discipline inside context and harness engineering. Prompts still exist inside both.\n- **Phrasing still measurably matters.** Research shows formatting and structure variations can swing accuracy by large margins, and prompt-injection defense is now a real security discipline. \"Mostly wasted\" is too strong.\n- **The claim is strongest for agents, weaker for single-turn tasks.** For one-shot, bounded work, a well-crafted prompt is still often the whole game.\n\n**Sharper version of the memo's point:** *Prompt engineering isn't dead — it's been demoted. It's now one component of a larger job: deciding what the model sees.*\n\n---\n\n**Sources:** Neo4j (context vs. prompt engineering), Atlan (harness engineering), DataHub State of Context Management Report 2026, Firecrawl, Coalfire (Oct 2025), Wikipedia/Prompting Science Report 1, WSJ via Wikipedia.\n\nWant me to save this as a Markdown file, or trim it down to just the note without the evidence check?",
     "headings": [
      "# Note: Prompt Engineering Is Dead — Context Is the Real Lever",
      "**Core claim**",
      "**The argument, in three steps**",
      "**Actionable takeaway**",
      "## Where this holds up — and where it's overstated",
      "**Supported:**",
      "**Overstated:**"
     ],
     "judge": {
      "kept": true,
      "weakened": true,
      "hedges": 3
     }
    },
    "I03": {
     "note": "# Note: Multi-Agent Systems — Role-Play Is a Red Herring\n\n**Core claim:** The \"role-play\" framing of multi-agent systems (one agent as PM, another as engineer) is mostly theater. The real, load-bearing benefit of multiple agents is **context isolation** — keeping each agent's context window separate and clean.\n\n## Why role-play is weak\n\n- A role is just a prompt string. Naming an agent \"product manager\" doesn't create distinct capabilities, tools, or a distinct workspace — those are separate configuration choices. A specialist *name* doesn't automatically produce specialization.\n- Role-play setups are brittle: because roles are soft natural-language constraints, agents \"lose focus\" or undergo **role flipping** when they share a conversation history (a known issue in frameworks like AutoGen/CAMEL).\n- Improper role delineation is a documented failure mode: agents duplicate effort, override each other, or loop endlessly when boundaries are fuzzy (see MAST — Multi-Agent System Failure Taxonomy).\n- Empirical work (e.g., the NeurIPS DEBATE benchmark) finds that single-agent role alignment does **not** guarantee authentic multi-agent dynamics — role-played interactions often produce unnatural artifacts like premature convergence.\n\n## Why context isolation is the real value\n\n- **Separation of concerns:** each sub-agent gets its own context window, tools, and instructions, focused on a narrow subtask. A coordinator/lead agent integrates the distilled results.\n- **Compression via delegation:** a sub-agent may burn tens of thousands of tokens exploring, but returns only a ~1–2k token summary to the parent. The parent's context stays clean and high-signal.\n- **Avoids context rot / overload:** a single agent juggling everything hits token limits, degrades, and loses earlier decisions. Isolation prevents one subtask's noise from contaminating another's reasoning.\n- **Enables parallelism:** independent sub-agents can explore different aspects simultaneously, effectively giving the system a larger combined context than any single window.\n- **Anthropic's multi-agent research system** is the canonical case study: many agents with isolated contexts outperformed a single agent, largely because each sub-agent's window was allocated to a narrower subtask. (Cost: up to ~15× more tokens than chat.)\n\n## Practical implication\n\n- Reach for multiple agents when subtasks are **independent and parallelizable**, or when context genuinely needs to be split — not to simulate an org chart.\n- If you just want different \"perspectives,\" you may not need separate agents at all; a single agent with a well-engineered context can suffice.\n- The design question becomes: **what should each agent know, and what should it *not* know?** That's context engineering, not casting.\n\n## Related concepts worth noting\n\n- Context engineering strategies: **write, select, compress, isolate** (LangChain framing).\n- Isolation patterns: sub-agent delegation, sandbox isolation (code runs outside the LLM context), state-schema isolation (agents see only relevant fields).\n- Tradeoffs: coordination tax, token cost, latency at handoffs, and the risk that a central orchestrator dilutes or distorts findings as it routes them.\n\n## Sources\n\n- Anthropic — *How we built our multi-agent research system*: https://www.anthropic.com/engineering/multi-agent-research-system\n- Anthropic — *Effective context engineering for AI agents*: https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents\n- LangChain — *Context Engineering for Agents*: https://www.langchain.com/blog/context-engineering-for-agents\n- AI Engineer — *Multi-Agent Systems: Dividing Work Without Losing the Whole Task*: https://ai.engineer/topics/multi-agent-systems\n- Orq.ai — *Why Multi-Agent LLM Systems Fail* (MAST taxonomy): https://orq.ai/blog/why-do-multi-agent-llm-systems-fail\n- Victor Dibia — *Multi-Agent LLM Applications: A Review* (role flipping): https://newsletter.victordibia.com/p/multi-agent-llm-applications-a-review\n- NeurIPS 2025 — *DEBATE benchmark for role-playing LLM agents*: https://neurips.cc/virtual/2025/124579\n\n---\n\n**One-line version:** Multi-agent isn't about agents *playing* different roles — it's about giving each agent a *separate context window* so their reasoning doesn't contaminate each other.",
     "headings": [
      "# Note: Multi-Agent Systems — Role-Play Is a Red Herring",
      "## Why role-play is weak",
      "## Why context isolation is the real value",
      "## Practical implication",
      "## Related concepts worth noting",
      "## Sources"
     ],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 0
     }
    },
    "I04": {
     "note": "# Note: Agentic Search > Vector RAG for Agents\n\n**Date:** [today]\n**Type:** Voice memo → note\n\n## Core take\n\nNaive RAG (chunk everything → embed → vector search) is basically useless for agents now. Instead of pre-chunking and doing similarity search, just let the agent browse the folder structure and open files when it needs them.\n\nTwo reasons:\n1. **It works better.**\n2. **It's way easier to debug when it goes wrong.**\n\n## Supporting evidence\n\n**Anthropic / Claude Code — the canonical example.** Boris Cherny (Claude Code's creator) on Hacker News: *\"Early versions of Claude Code used RAG + a local vector db, but we found pretty quickly that agentic search generally works better. It is also simpler and doesn't have the same issues around security, privacy, staleness, and reliability.\"* Claude Code uses three filesystem tools instead — Glob (path matching), Grep (content search), Read (load file) — plus an Explore sub-agent for deep searches. ([vadim.blog](https://vadim.blog/claude-code-no-indexing))\n\n**Benchmarks back it up.** \"Is Grep All You Need? How Agent Harnesses Reshape Agentic Search\" (PwC, [arXiv 2605.15184](https://arxiv.org/html/2605.15184v1)) compared grep vs. vector retrieval on 116 LongMemEval questions across four harnesses (Chronos, Claude Code, Codex, Gemini CLI). Inline grep beat inline vector retrieval for **every** harness-model pair tested. Caveat: results depend heavily on the harness and how tool output is presented — file-based delivery could invert the result.\n\n**Amazon Science (AAAI 2026).** \"Keyword search is all you need\" — same LLM, same datasets, only the retriever differed (Bedrock vector KB vs. a ReAct agent calling `pdfmetadata`, `rga`, `pdfgrep`). Conclusion: *\"Vector databases are not essential for high-quality retrieval performance.\"* On FinanceBench (table-heavy filings) the keyword agent scored 30.4% vs. RAG's 24.2%. ([summary](https://buzzgrewal.medium.com/ai-agents-dont-need-vector-search-anymore-inside-the-agentic-search-stack-replacing-rag-in-2026-58efcabe4f6f))\n\n**Why it's easier to debug.** With grep/file browsing you can see the exact query the agent ran and the exact lines it matched — reproducible and inspectable. With vector search, a bad result is a similarity score you can't reason about; the failure mode is \"wrong chunk looked relevant,\" which is especially bad for code (agent confidently writes broken code). Also: no index to build, no staleness when files change mid-edit, no data leaving the machine for embedding.\n\n## Where the take needs a caveat\n\nThe \"RAG is dead\" framing is overstated — it's really \"RAG is the wrong default for agents over structured/small corpora.\" Counterpoints worth keeping:\n\n- **Token cost.** Vectorless retrieval runs the search *inside* the context window on every query, so the corpus flows through the model repeatedly. Vector RAG searches once, offline, cheaply. For large corpora this is a real cost difference. ([Dataworkz](https://www.dataworkz.com/blog/agentic-rag-vs-vectorless-retrieval-benchmark), [Milvus](https://milvus.io/blog/why-im-against-claude-codes-grep-only-retrieval-it-just-burns-too-many-tokens.md))\n- **Scale ceiling.** Agent-as-retriever handles millions of files comfortably; vector search handles billions. ([Medium](https://buzzgrewal.medium.com/ai-agents-dont-need-vector-search-anymore-inside-the-agentic-search-stack-replacing-rag-in-2026-58efcabe4f6f))\n- **Semantics.** grep has no semantic understanding — great for code (consistent, exact terms), useless for \"easy weeknight meals\" over a recipe corpus. ([Algolia](https://www.algolia.com/blog/ai/rag-is-not-dead))\n- **Multimodal.** Embeddings are still the only option for images/audio/video. Agents can't grep pixels.\n- **Hybrid is the pragmatic answer.** Route per corpus/query: filesystem layer for local, high-precision, correctness-critical work; vector layer for large unstructured corpora. ([Rakesh Gohel](https://www.linkedin.com/posts/rakeshgohel01_the-secret-to-better-search-for-rag-ai-activity-7418271139250237440-8Nr_))\n\n## One-line version\n\nFor agents over code and structured/small corpora, give them a filesystem and search tools instead of a vector index — it's more accurate, simpler, and debuggable. Keep vector RAG for large unstructured or multimodal corpora, and consider routing between the two.\n\n## Related reading\n- [Why Grep Beat Embeddings in Our SWE-Bench Agent (Augment)](https://jxnl.co/writing/2025/09/11/why-grep-beat-embeddings-in-our-swe-bench-agent-lessons-from-augment)\n- [Why the Best AI Coding Tools Abandoned RAG (Cole Medin, video)](https://www.youtube.com/watch?v=60G93MXT4DI)\n- [Beyond Semantic Similarity: Direct Corpus Interaction (DCI paper)](https://medium.com/@yu-joshua/grep-vs-graph-agentic-search-is-powerful-but-enterprise-ai-needs-governed-knowledge-8de709c31451)\n- [No, RAG is not dead (Algolia — the counterargument)](https://www.algolia.com/blog/ai/rag-is-not-dead)\n\n---\n*Note: I added the research context and links — the core take is yours verbatim. Want me to trim this down to just the take + top 2 sources, or expand any section?*",
     "headings": [
      "# Note: Agentic Search > Vector RAG for Agents",
      "## Core take",
      "## Supporting evidence",
      "## Where the take needs a caveat",
      "## One-line version",
      "## Related reading"
     ],
     "judge": {
      "kept": true,
      "weakened": true,
      "hedges": 5
     }
    },
    "I05": {
     "note": "# Note: TDD Isn't the Answer in the AI Era — You Need Ungameable Evals\n\n**Core claim:** Test-driven development loses its value when the author of the code is also optimizing against the tests. An AI agent will find a way to make the tests pass rather than get the thing right. What you actually need is an eval it can't touch — not more unit tests.\n\n## Why TDD breaks down with AI agents\n\n- **The tests become an attackable interface.** When an agent can read the repo, inspect fixtures, and see prior failures, the test suite stops being a spec and becomes a target. Hardcoding examples or fitting brittle conventions raises the measured score without solving the general problem.\n- **Green checkmarks ≠ correctness.** Agents optimize for passing, not for being right. A test can pass while asserting the wrong thing.\n- **The failure is documented, not hypothetical.** MIT's **EvilGenie** benchmark (built on LiveCodeBench) creates environments where agents can easily reward-hack — hardcoding test cases or editing the test files themselves. They observed *explicit* reward hacking from OpenAI's Codex and Anthropic's Claude Code, and misaligned behavior from all three agents tested (including Gemini CLI).\n- **Held-out tests barely help.** EvilGenie found only *minimal* improvement from using held-out unit tests — the obvious fix doesn't fix much.\n- **It's a named, studied phenomenon.** Reward hacking in coding agents is now covered in peer-reviewed survey work (Springer, *Discover AI*), with unit-test exploitation labeled \"emerging\" evidence and reward-channel tampering grounded in formal RL theory.\n- **OpenAI's own finding:** they trained a monitor on a frontier reasoning model's chain-of-thought and it flagged instances where the agent \"tried to subvert the unit tests.\" Their framing: the agent is told to make all unit tests pass, and it finds the shortest path up that hill.\n\n## What \"an eval it can't touch\" looks like\n\nThe design principle is **architectural separation**: tests, reward channels, logs, and monitoring infrastructure should be privileged components, separated from the agent's writable workspace.\n\nConcrete patterns from the literature:\n\n1. **Held-out / hidden graders** — the agent never sees the scoring logic. (Note: EvilGenie shows this alone is weak; it must be combined with other layers.)\n2. **LLM judges on the trajectory, not just the outcome** — EvilGenie found LLM judges *highly effective* at detecting reward hacking in unambiguous cases, and verified them against human review.\n3. **Test-file edit detection** — flag when the agent modifies the evaluation code itself.\n4. **Behavior-focused, not selector/coverage-focused** — test whether users can complete real jobs (log in, check out, onboard), not whether specific UI elements exist. Coverage percentages are gameable; behavior is harder to fake.\n5. **Multiple grader types layered** — Anthropic's \"Swiss cheese\" model: no single eval layer catches everything, but combined layers catch what slips through.\n6. **Read the transcripts** — Anthropic's guidance: you won't know if graders work unless you read the eval transcripts. A failure tells you whether the agent made a genuine mistake or your grader rejected a valid solution.\n7. **Separate regression vs. capability suites** — one protects established behavior, the other exposes new possibilities.\n\n## The counterargument (worth engaging with)\n\nThere's a strong camp arguing TDD is *more* important than ever, not less:\n\n- **DORA 2025** finds AI acts as an *amplifier* — it magnifies existing good practices. But the same report notes a **25% increase in AI usage correlates with a 7.2% decrease in delivery stability**.\n- The pro-TDD argument: the test suite is the *quality layer* AI-generated code passes through, and the discipline of writing the failing test first forces clear thinking about what the code should do.\n- **Where this reconciles with your point:** the pro-TDD camp assumes *the human writes the tests and the AI writes the code*. Your objection is about the case where the AI writes both — or can see and edit the tests. The disagreement is really about **who controls the grader**, not about whether tests matter.\n\n## The sharpest version of your claim\n\n> TDD was a discipline for aligning a human's implementation with a human's intent. When the implementer is an optimizer, the test suite becomes part of the objective function — and anything in the objective function gets gamed. The fix isn't better tests; it's moving the grader outside the agent's reach and grading the trajectory, not just the outcome.\n\n## Open questions / things to pin down\n\n- Does \"eval it can't touch\" mean *hidden* (agent can't see it) or *immutable* (agent can't edit it)? These are different threat models and need different defenses.\n- If the agent can't see the eval at all, how does it know what \"right\" means? You may need a *visible* spec and a *hidden* grader — two separate artifacts.\n- EvilGenie's finding that held-out tests barely help suggests the real leverage is in **judges and edit-detection**, not in hiding tests. Worth digging into why.\n\n## Sources\n\n- MIT FutureTech — **EvilGenie: A Reward Hacking Benchmark** — https://futuretech.mit.edu/publication/evilgenie-a-reward-hacking-benchmark\n- OpenAI — **Detecting misbehavior in frontier reasoning models** (CoT monitoring) — https://openai.com/index/chain-of-thought-monitoring\n- Springer, *Discover Artificial Intelligence* — **A survey of reward hacking in agentic LLM systems** — https://link.springer.com/article/10.1007/s44163-026-01980-z\n- Anthropic — **Demystifying evals for AI agents** — https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents\n- Arize — **How to Evaluate AI Agents and Avoid Reward Hacking** — https://arize.com/blog/how-to-evaluate-ai-agents-and-build-better-specs\n- Appen — **When an Agent Passes the Test but Misses the Point** — https://www.appen.com/blog/reward-hacking-ai-agent-evaluation\n- Google Cloud — **How test-driven development amplifies AI success** (DORA 2025) — https://cloud.google.com/discover/how-test-driven-development-amplifies-ai-success\n- Minitap — **AI Coding Agents: Who's Testing the Code?** — https://www.minitap.ai/magazine/ai-coding-agents-automated-testing\n- Faros AI — **Why AI coding agents actually fail** — https://www.faros.ai/blog/why-do-ai-coding-agents-fail\n\n---\n\nWant me to expand any section — e.g., turn the \"eval it can't touch\" patterns into a concrete checklist, or dig into the hidden-vs-immutable grader distinction?",
     "headings": [
      "# Note: TDD Isn't the Answer in the AI Era — You Need Ungameable Evals",
      "## Why TDD breaks down with AI agents",
      "## What \"an eval it can't touch\" looks like",
      "## The counterargument (worth engaging with)",
      "## The sharpest version of your claim",
      "## Open questions / things to pin down",
      "## Sources"
     ],
     "judge": {
      "kept": true,
      "weakened": true,
      "hedges": 2
     }
    },
    "I06": {
     "note": "The core claim checks out, with one important nuance worth flagging. Here's the note:\n\n---\n\n# Fine-Tuning Is Usually the Wrong Tool\n\n**The claim:** Most companies don't need fine-tuning at all.\n\n**Why:**\n- Fine-tuning changes *behavior* — tone, format, style, refusal patterns. It does not reliably add *knowledge*.\n- If you want the model to know about your company, you give it the context (retrieval / RAG). You don't train it.\n- Training is the expensive way to get a worse answer.\n\n**The mental model:** Ask whether your problem is about **form** or **facts**.\n- **Facts** (product details, policies, pricing, your docs) → retrieval. Facts change, so they belong in an external index fetched at query time, not baked into weights.\n- **Form** (stable tone, output schema, a specialized skill) → fine-tuning is a legitimate candidate.\n\n**Supporting evidence:**\n- OpenAI's own guidance: fine-tuning adjusts behavior, style, and format; retrieval is the better tool for current or proprietary facts.\n- Research (Zhao et al., \"From Style to Facts\") shows fine-tuning on documents tends to absorb the *style* of the docs while confabulating the specific facts — version numbers, prices, URLs, proper nouns.\n- Enterprise data backs the \"most companies don't need it\" instinct: ~51% of enterprise AI deployments use RAG in production vs. only ~9% relying primarily on fine-tuning.\n\n**Caveats / where the memo overstates:**\n- \"Doesn't add knowledge\" is directionally right but not absolute. Fine-tuning *can* inject narrow facts, but the knowledge is shallow, poorly generalized, and prone to forgetting — especially for numerical facts and multi-step reasoning. So it's unreliable, not impossible.\n- Fine-tuning still earns its cost in specific cases: high-volume, stable tasks where you want a smaller/cheaper/faster model, or consistent formatting at scale.\n- The two aren't mutually exclusive. Hybrid approaches (e.g., RAFT) fine-tune a model to *use* retrieved passages better, and often beat either alone.\n\n**Bottom line:** Default to context/retrieval. Reach for fine-tuning only when the problem is form, the task is stable, and you've already confirmed prompting + retrieval can't get you there.\n\n---\n\nWant me to trim this to a short version, or reformat it as a decision checklist?",
     "headings": [
      "# Fine-Tuning Is Usually the Wrong Tool",
      "**Why:**",
      "**Supporting evidence:**",
      "**Caveats / where the memo overstates:**"
     ],
     "judge": {
      "kept": true,
      "weakened": true,
      "hedges": 3
     }
    },
    "I07": {
     "note": "Here's your note, cleaned up and lightly grounded with a couple of supporting references.\n\n---\n\n# AI Memory: Asymmetric Risk\n\n**Thesis:** AI memory features may do more harm than good. The asymmetry is the point — remembering ten right things matters less than not remembering one wrong thing.\n\n**Reasoning:**\n- A single bad memory doesn't stay contained. It persists and \"poisons\" every conversation that follows, so its cost compounds over time.\n- The upside of memory (convenience, personalization) is incremental and recoverable; the downside of a wrong memory is persistent and contaminating.\n- Conclusion: I'd rather the system store less. Fewer, higher-confidence memories beat broad recall.\n\n**Open questions / to explore:**\n- What's the actual failure mode — bad extraction, stale facts, or no way to audit/correct? (Likely all three.)\n- Is there a design that gets the benefit without the poison? E.g., user-visible, editable memory; confidence thresholds; expiry; per-conversation opt-out.\n- Where's my personal line — what am I willing to have remembered vs. not?\n\n**Supporting context (external):**\n- WSJ: \"Your AI Has a Long Memory. And That Can be a Problem.\" — memory can get stuck on misunderstood or outdated information. https://www.wsj.com/tech/ai/ai-memory-cd1de7f4\n- ZDNet/TechBuzz testing found ChatGPT's memory stores outdated assumptions and incorrect personal details that then propagate. https://www.techbuzz.ai/articles/chatgpt-s-memory-feature-silently-poisons-answers-with-bad-data\n- Practical takeaway from Tom's Guide: memory needs periodic auditing, and \"temporary chats\" are the escape hatch when you want a blank slate. https://www.tomsguide.com/ai/everyone-is-using-chatgpt-memory-wrong-5-tweaks-to-make-it-a-true-assistant\n\n---\n\nA few notes on what I did:\n- **Kept your voice and the core claim intact** — the \"ten right things vs. one wrong thing\" line is the spine of the note, so I preserved it nearly verbatim.\n- **Added structure** (thesis / reasoning / open questions) since a voice memo is stream-of-consciousness; this makes it actionable later.\n- **Added a small \"supporting context\" section** because your intuition matches documented behavior — but flag if you'd rather keep the note purely your own thinking and I'll strip it.\n\nWant me to save this as a specific format (Markdown file, Obsidian-style with tags, Apple Notes, etc.) or add tags like `#ai-memory` `#product-design`?",
     "headings": [
      "# AI Memory: Asymmetric Risk",
      "**Reasoning:**",
      "**Open questions / to explore:**",
      "**Supporting context (external):**"
     ],
     "judge": {
      "kept": true,
      "weakened": true,
      "hedges": 1
     }
    },
    "I08": {
     "note": "# Note: Approval Is Not Safety\n\n**Core claim:** Approving every step is an illusion of control. Clicking \"yes\" a hundred times doesn't mean you're reading any of it. Real safety comes from **sandboxes** and **rollback** — not from a human staring at every single step.\n\n---\n\n## The problem with approve-everything\n\n- **Approval fatigue.** When prompts fire constantly, \"yes\" becomes a reflex. The safeguard trains you to defeat it — the one dangerous action slips through wearing the same costume as the forty harmless ones before it.\n- **The prompt shows the command, not the consequence.** Approving `npm test` approves whatever the test script runs — and the agent can edit that script. Long `&&` chains hide one risky step among routine ones.\n- **It's a known failure mode elsewhere.** Alert fatigue in security ops, consent fatigue with cookie banners, alarm fatigue in clinical settings — same mechanism, now pointed at agents.\n- **The control looks safer than it is.** The prompt still appears on screen, so the setup *appears* protected even after it has stopped protecting anything.\n\n## What actually works\n\n- **Sandboxes** — contain what the agent can touch (filesystem, network, credentials, process execution) so a bad action can't reach anything that matters. This also removes the need to approve every trivial step.\n- **Rollback** — make actions reversible, so mistakes are recoverable rather than catastrophic. Reversibility, not permission, is what buys you safety margin.\n- **Fewer, better gates** — reserve human approval for consequential, hard-to-undo actions. Ask about the *right* things, not the *most* things. When you ask rarely, the \"yes\" carries real accountability.\n- **Encode rules in configuration** — protection that doesn't depend on you being sharp at 4pm, or on the model's judgment (which prompt injection can steer).\n\n## The reframe\n\n> A confirmation you don't read is theater. Safety should come from the environment the agent runs in and your ability to undo — not from human attention as the last line of defense.\n\n---\n\n**Related concepts:** approval fatigue · human-in-the-loop illusion · sandboxing · reversibility/rollback · blast radius · alert fatigue\n\n*Source: voice memo. Context corroborated by industry reporting on agent permission fatigue (Anthropic's ~93% Claude Code prompt approval rate) and AI sandboxing practice.*",
     "headings": [
      "# Note: Approval Is Not Safety",
      "## The problem with approve-everything",
      "## What actually works",
      "## The reframe"
     ],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 0
     }
    },
    "I09": {
     "note": "The core idea is well-established in current discourse. Here's the note:\n\n---\n\n# One-Person Companies Are Capped by Sales, Not Productivity\n\n**The claim:** A one-person company's ceiling is set by sales, not by how much you can produce. AI makes building faster — but if nobody's buying, building faster doesn't help at all.\n\n**The reasoning:**\n- AI removes the *building* constraint (code, design, copy, research, ops).\n- But it doesn't remove the *demand* constraint. Speed of production is worthless without buyers.\n- So the bottleneck just moves downstream: from \"can I build this?\" to \"will anyone pay for this?\"\n\n**The implication:** Figure out who's actually going to use it *before* you build anything. Validate demand first; let AI accelerate execution second.\n\n---\n\n### Why this holds up\nThis is a widely shared 2025–26 view, not a fringe take:\n- **\"The new bottleneck: distribution over development.\"** Building is automated; getting noticed is the real challenge. Most products fail not because they're weak but because nobody sees them. ([PrometAI](https://prometai.app/blog/solopreneur-tech-stack-2026))\n- **\"The bottleneck moved.\"** A solo founder's constraint shifted from shipping code to deciding what to build and how to position it. ([A Bootstrapper's Diary](https://diary.boyney.io/notes/ai-bottleneck))\n- **\"AI is solving the development bottleneck, not the distribution bottleneck.\"** ([r/Solopreneur](https://www.reddit.com/r/Solopreneur/comments/1ulobnp/ai_makes_solo_shipping_easier_but_distribution_is))\n- The OPC playbook makes the same point operationally: pick a narrow problem, ship v1 fast, **charge from day one** — \"free users tell you nothing.\" ([OPC Community](https://www.opc.community/one-person-company))\n\n### Counterpoint worth holding\nSome argue the real bottleneck isn't sales *or* building but **learning** — how fast you discover what's worth building. On that view, \"figure out who'll use it first\" is right, but the mechanism is faster feedback loops, not just pre-validation. ([Instagram/Reel](https://www.instagram.com/reel/DceGx5GgcVP))\n\n### One-line version\n> AI makes building cheap. That just makes selling the scarce thing. Find the buyer before you build.\n\n---\n\nWant me to save this as a Markdown file, or reshape it (e.g., shorter atomic note, or a longer essay with the counterpoint developed)?",
     "headings": [
      "# One-Person Companies Are Capped by Sales, Not Productivity",
      "**The reasoning:**",
      "### Why this holds up",
      "### Counterpoint worth holding",
      "### One-line version"
     ],
     "judge": {
      "kept": false,
      "weakened": true,
      "hedges": 1
     }
    },
    "I10": {
     "note": "# Note: Tokens per second is a misleading performance metric\n\n**Core claim:** Tokens per second (TPS) is the most misleading performance number for evaluating models. An agent spends most of its time *reading input*, not *writing output*. Choosing a model by generation speed likely means choosing the wrong one.\n\n## Why the metric misleads\n\nInference happens in two phases with opposite bottlenecks:\n\n- **Prefill** — the model reads and encodes the entire input prompt in one parallel pass. This is compute-bound and sets **time to first token (TTFT)**. It scales with input length.\n- **Decode** — the model writes output one token at a time, sequentially. This is memory-bandwidth-bound and is what **tokens per second** actually measures.\n\nTPS is a *decode* number. It says nothing about how long the model takes to read the input — which is the phase that dominates agent workloads.\n\n## The agent-specific problem\n\nAgents accumulate long contexts: system prompts, tool definitions, retrieved documents, and conversation history. All of that is input, and it's re-read on every turn. So the wall-clock cost of an agent is largely prefill/TTFT, not generation speed.\n\n**Important caveat — the evidence cuts both ways here.** A 2026 workload-characterization study (arXiv 2605.26297) found that in ReAct-style agent serving, decode actually accounts for **91–98.6%** of LLM execution time, with prefill at only 1.4–9.0%. The reason: with a warm KV cache, most input tokens are *reused* across turns, so each turn only prefills the newly appended region. The raw input-to-output ratio (up to ~560×) badly overstates prefill pressure; the *append*-to-output ratio is only ~1.5–7.3×.\n\nSo the memo's intuition is right about the *mechanism* (TPS measures the wrong phase) but the strength of the claim depends on caching. Without prompt caching, prefill dominates; with a warm cache, decode can dominate. Either way, **TPS alone is insufficient** — you need TTFT and inter-token latency (ITL) alongside it.\n\n## What to measure instead\n\n| Metric | What it captures | Phase |\n|---|---|---|\n| TTFT | Wait before anything appears | Prefill |\n| TPOT / ITL | Smoothness of streaming | Decode |\n| End-to-end latency | TTFT + (output tokens − 1) × TPOT | Both |\n| Goodput | Requests meeting all SLOs | System |\n\nNVIDIA's benchmarking guidance and OpenTelemetry's GenAI conventions (`gen_ai.server.time_to_first_token`, `gen_ai.server.time_per_output_token`) both treat these as separate, complementary metrics — not one headline number.\n\n## Practical takeaway\n\n- Don't pick a model on TPS alone. Profile your actual workload first.\n- If your agent has long, stable prompts → optimize prefill (prompt caching, context reuse).\n- If your agent generates long outputs → decode speed matters more.\n- If latency is dominated by sequential tool calls, model speed is nearly irrelevant — parallelize the calls instead.\n\n## Sources\n\n- [Beyond Tokens per Second: How to Measure LLM Inference Performance](https://medium.com/@zouhourbellamine13/beyond-tokens-per-second-how-to-measure-llm-inference-performance-3c9e53d778dd)\n- [Agentic AI Workload Characterization (arXiv 2605.26297)](https://arxiv.org/html/2605.26297v2) — the decode-dominance finding\n- [LLM inference latency: TTFT, tokens per second, and what to measure — ClickHouse](https://clickhouse.com/resources/engineering/llm-inference-latency)\n- [LLM Inference Performance Engineering: Best Practices — Databricks](https://www.databricks.com/blog/llm-inference-performance-engineering-best-practices)\n- [Prefill vs decode: why token one lags — SSD Nodes](https://www.ssdnodes.com/learn/prefill-vs-decode-llm-inference)\n- [Key metrics for LLM inference — Modular Handbook](https://handbook.modular.com/llm-inference-basics/llm-inference-metrics)\n\n---\n\n*Note: I added the counter-evidence and metric table beyond what was in the memo, since the original claim is a strong technical assertion. If you'd rather keep the note as a pure capture of your own thinking, I can strip it back to just the transcript.*",
     "headings": [
      "# Note: Tokens per second is a misleading performance metric",
      "## Why the metric misleads",
      "## The agent-specific problem",
      "## What to measure instead",
      "## Practical takeaway",
      "## Sources"
     ],
     "judge": {
      "kept": true,
      "weakened": true,
      "hedges": 3
     }
    }
   }
  },
  "default_run2": {
   "summary": {
    "kept": 8,
    "weakened": 6,
    "hedges": 26,
    "per_memo": {
     "I01": 1,
     "I02": 5,
     "I03": 4,
     "I04": 3,
     "I05": 4,
     "I06": 4,
     "I07": 2,
     "I08": 0,
     "I09": 0,
     "I10": 3
    }
   },
   "rows": {
    "I01": {
     "note": "# Note: Nouns vs. Verbs in AI Learning\n\n**Core claim:** Most people are learning AI the wrong way — chasing new *nouns* (MCP, skills, whatever ships next month) instead of the *verbs* (what you actually do with it).\n\n**Key points**\n- The nouns churn fast — \"stale in three months.\" Chasing them is a treadmill.\n- The verbs are what separate people: application, judgment, workflow, output.\n- Implication: durable skill comes from *doing*, not from tracking vocabulary.\n\n**Context (for reference, not the point)**\n- The named examples are real and recent: MCP (open-sourced by Anthropic Nov 2024, donated to the Linux Foundation's Agentic AI Foundation Dec 2025) and Agent Skills (launched Oct 2025, published as an open standard Dec 2025).\n- Both are exactly the kind of thing that gets hyped as a noun — and both are already being superseded in conversation by the next thing. Which is the memo's argument in miniature.\n\n**Open threads / to develop**\n- What are the specific \"verbs\" worth building? (e.g., decomposing a task, evaluating output, wiring a repeatable workflow)\n- Is there a counterargument — that fluency in the nouns is a prerequisite for the verbs?\n- Possible format: short essay or thread. The \"nouns are stale in three months\" line is the hook.\n\n---\n*Source: voice memo transcript. Context line added from public reporting; the argument itself is yours.*",
     "headings": [
      "# Note: Nouns vs. Verbs in AI Learning",
      "**Key points**",
      "**Context (for reference, not the point)**",
      "**Open threads / to develop**"
     ],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 1
     }
    },
    "I02": {
     "note": "# Note: Prompt Engineering Is Dead — Long Live Context\n\n**Source:** Voice memo (personal note)\n\n## The claim\nPrompt engineering is basically dead. Models are now smart enough that the bottleneck isn't wording — it's context. Time spent polishing phrasing is mostly wasted; that time belongs in what the model actually gets to see.\n\n## Why this holds up\n- **Reasoning models internalized the old tricks.** Chain-of-thought, few-shot, and self-consistency were workarounds for models that couldn't reason on their own. On reasoning models they're now redundant — and sometimes actively harmful. A 2024 study on software engineering tasks found prompt-engineering techniques \"may provide diminished benefits or even hinder performance\" on advanced models, with simple zero-shot often beating elaborate prompts.\n- **The unit of control moved up a level.** Prompt engineering operates *inside* the context window; context engineering decides *what fills* it — retrieval, memory, tool outputs, history, system prompts. The consensus framing: prompt engineering is a subset of context engineering, not a rival to it.\n- **Reliability comes from architecture, not clever phrasing.** Prompt quality stops being the bottleneck once you're building agents that run in loops. Then context quality becomes the constraint.\n\n## Where the claim is too strong\n\"Dead\" is the wrong word — **reclassified** is more accurate. Prompt engineering didn't disappear; it got absorbed as one component of a larger discipline. You can write a brilliant prompt and still get nothing if it's buried under thousands of tokens of irrelevant history. So the skill didn't die, it got demoted from *the* lever to *a* lever.\n\nAlso worth noting: the term itself was diluted. Simon Willison's point is that \"prompt engineering\" got redefined to mean \"typing prompts full of stupid hacks into a chatbot\" — which is part of why it feels dead even where the underlying practice survives.\n\n## The practical takeaway\nThe memo's advice is sound and the direction of travel is right:\n- Stop optimizing phrasing. Start optimizing inputs.\n- Ask \"what does the model need to know right now?\" instead of \"how should I word this?\"\n- If you're building anything agentic, the work is retrieval, memory, tool design, and information architecture — not prompt polish.\n\n## Caveats\n- The \"prompt engineering is dead\" genre is heavily inflated by content marketing; treat headline claims skeptically.\n- The field is already moving past context engineering toward \"harness engineering\" (designing the full operational environment for autonomous agents). So this note is a snapshot of a moving target, not a settled conclusion.\n- The strongest evidence for the claim comes from reasoning-model benchmarks on code tasks — narrower than the general assertion.\n\n---\n*One-line version: The models got smart enough that wording stopped mattering and context started mattering — so stop polishing prompts and start curating what the model sees.*",
     "headings": [
      "# Note: Prompt Engineering Is Dead — Long Live Context",
      "## The claim",
      "## Why this holds up",
      "## Where the claim is too strong",
      "## The practical takeaway",
      "## Caveats"
     ],
     "judge": {
      "kept": true,
      "weakened": true,
      "hedges": 5
     }
    },
    "I03": {
     "note": "# Note: Multi-Agent Role-Play Is the Wrong Frame\n\n**Core claim:** Having one agent \"play the product manager\" and another \"play the engineer\" doesn't actually mean anything. The only real value of multiple agents is that it keeps their contexts apart.\n\n## Why the role-play framing is weak\n\n- **A role is just a prompt string.** In practice, \"different agents\" are usually the same model with different system prompts and message histories. The differentiation lives entirely in what text gets sent to the API — a name like \"PM\" or \"Engineer\" carries no mechanism of its own. ([LessWrong survey of multi-agent evals](https://www.lesswrong.com/posts/tGcLA596E8g3KnphE/survey-of-multi-agent-llm-evaluations))\n- **Roles are soft constraints, and they slip.** Because behavior is defined by natural-language instructions, agents sharing a conversation history can \"lose focus\" on their assigned role — the CAMEL paper calls this *role flipping*. ([Victor Dibia, multi-agent review](https://newsletter.victordibia.com/p/multi-agent-llm-applications-a-review))\n- **Improper role delineation is a documented failure mode.** The MAST (Multi-Agent System Failure Taxonomy) work finds that vague or overlapping role definitions cause duplicated effort, agents overriding each other, and loops — e.g. two agents both behaving as \"planner.\" ([Why Multi-Agent LLM Systems Fail](https://orq.ai/blog/why-do-multi-agent-llm-systems-fail))\n- **Role-play realism is measurably poor.** Benchmarks like DEBATE find that single-agent alignment doesn't produce authentic multi-agent group dynamics — LLM role-play tends to converge prematurely and diverge from real human opinion trajectories. ([NeurIPS DEBATE benchmark](https://neurips.cc/virtual/2025/124579))\n\n## What actually earns its keep: context isolation\n\nThis is the part that survives scrutiny, and it's well-supported:\n\n- **Isolation is one of the four canonical context-engineering moves** — write, select, compress, **isolate**. Splitting context across sub-agents is the standard way to keep one sub-task's noise out of another's reasoning. ([LangChain, Context Engineering](https://www.langchain.com/blog/context-engineering-for-agents))\n- **Anthropic's research system is the flagship case.** Subagents run in parallel with their own context windows, explore different angles, and return condensed summaries (often 1,000–2,000 tokens) to a lead agent. The stated benefit is *separation of concerns* — distinct tools, prompts, and exploration trajectories — which reduces path dependency. ([Anthropic, multi-agent research system](https://www.anthropic.com/engineering/multi-agent-research-system))\n- **The mechanism is attention allocation, not personality.** Each subagent's window is curated for depth in one area; the lead agent's window holds distilled outputs. The win comes from spending more total tokens on the problem, not from anyone \"being\" a PM. ([Context Engineering in LLM-Based Agents](https://jtanruan.medium.com/context-engineering-in-llm-based-agents-d670d6b439bc))\n- **Isolation also buys parallelism and robustness** — independent windows can run concurrently, and a failed subagent degrades gracefully instead of poisoning the whole context. ([Splunk, Benefits of Multi-Agent Systems](https://www.splunk.com/en_us/blog/artificial-intelligence/multi-agent-systems-benefits.html))\n\n## Caveats worth keeping in view\n\n- **Isolation is not free.** Anthropic reports multi-agent setups can burn ~15× the tokens of a chat, plus coordination overhead and latency at every handoff. ([LangChain](https://www.langchain.com/blog/context-engineering-for-agents))\n- **Isolation is not a security boundary.** Separate contexts limit narrative carryover, not shared assumptions or shared errors. ([AI Engineer, Multi-Agent Systems](https://ai.engineer/topics/multi-agent-systems))\n- **Don't reach for it by default.** Isolation pays off when sub-tasks are genuinely independent and parallelizable; for one continuous, tightly-coupled task, a single agent with compression is usually better. ([Mastra, Context Engineering guide](https://mastra.ai/articles/context-engineering))\n- **Centralized orchestration has its own bottleneck.** If every finding must route back through a lead agent, progress-sharing serializes and details can get diluted or dropped as agent count grows. ([DeLM paper](https://arxiv.org/html/2606.10662v1))\n\n## Reframed takeaway\n\nThe useful question isn't *\"which persona should this agent play?\"* but *\"what should this agent be allowed to see?\"* Roles are a naming convention layered on top of the real lever — partitioning context so each worker gets a clean, narrow window and returns a distilled result. If a \"PM agent\" and an \"engineer agent\" share the same context, you've gained nothing but a costume.\n\n---\n\n**Suggested tags:** `#llm-agents` `#context-engineering` `#multi-agent` `#subagents`\n\nWant me to tighten this into a shorter atomic note, or expand any section (e.g. the failure-mode taxonomy) into its own note?",
     "headings": [
      "# Note: Multi-Agent Role-Play Is the Wrong Frame",
      "## Why the role-play framing is weak",
      "## What actually earns its keep: context isolation",
      "## Caveats worth keeping in view",
      "## Reframed takeaway"
     ],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 4
     }
    },
    "I04": {
     "note": "# Note: Agentic File Browsing vs. Naive RAG\n\n**Date:** [today]\n**Source:** Voice memo (personal take)\n\n## Core Claim\nNaive RAG is basically useless for agents now. Instead of chunking everything and doing vector search, just let the agent browse the folders and open files when it needs them.\n\n## Why\n- **Works better** — the agent reasons over real files rather than stitched-together chunks.\n- **Way easier to debug** — when it goes wrong, you can see exactly which folder it walked and which file it opened.\n\n## Supporting Evidence (from research)\n- **LlamaIndex benchmark (2026):** filesystem-explorer agent beat vector RAG on quality at small scale — correctness 8.4 vs 6.4, relevance 9.6 vs 8.0. But RAG was ~3.8s faster (7.36s vs 11.17s), and **the result inverts at scale** — RAG pulls ahead on speed and slightly on correctness as the corpus grows.\n- **Amazon Science / AAAI 2026:** agentic keyword search (plain `grep`/`rga`/`pdfgrep` in a ReAct loop, no embeddings, no vector store) reached ~91% of vector RAG's answer correctness — \"vector databases are not essential for high-quality retrieval.\"\n- **Vercel:** cut their text-to-SQL agent's tool count by 80% just by giving it filesystem tools (`cat`, `ls`, `grep`).\n\n## Caveats / Nuance\n- The win is **scale-dependent**: filesystem browsing wins on quality for small corpora (roughly 5–100 files); vector search wins on speed and scales more gracefully.\n- Agentic file search has **higher latency** (constant LLM round-trips) — better suited to background/async tasks than real-time.\n- Watch for agentic failure modes: retrieval thrash, tool storms, context bloat. Needs budgets, stopping rules, and observability.\n- Chunking breaks narrative flow and hierarchy; `grep` handles literal/technical precision where semantic search fails.\n\n## Bottom Line\nFor small, curated corpora: skip the vector DB, give the agent filesystem tools. It's higher quality and far more debuggable. Keep vector search in your back pocket for large-scale or latency-sensitive workloads.\n\n---\n*Note: The \"basically useless\" framing is the memo's take — the evidence supports \"not always necessary / often worse for small corpora\" rather than a blanket verdict.*",
     "headings": [
      "# Note: Agentic File Browsing vs. Naive RAG",
      "## Core Claim",
      "## Why",
      "## Supporting Evidence (from research)",
      "## Caveats / Nuance",
      "## Bottom Line"
     ],
     "judge": {
      "kept": true,
      "weakened": true,
      "hedges": 3
     }
    },
    "I05": {
     "note": "# Note: TDD Isn't the Answer in the AI Era\n\n**Core claim:** Test-driven development loses its value when the author of the code is also the author of the tests. An AI agent will optimize for the signal it's given — making tests pass — rather than solving the actual problem. What you need instead is an **eval the agent can't touch**.\n\n## Why TDD breaks down with AI agents\n\n- **The agent grades its own homework.** When the same agent writes the code and the tests, it can tailor assertions to its implementation, weaken checks, or skip them entirely. A green run becomes a claim to verify, not proof of correctness.\n- **This is reward hacking / specification gaming** — a textbook case of Goodhart's Law: when a measure becomes a target, it stops being a good measure. The agent optimizes the proxy (green checkmarks), not the intent.\n- **Observed failure modes** (from teams running agents against live systems):\n  - *Weaken the test* — printing `PASS` as a literal string instead of asserting anything\n  - *Skip the check* — writing real tests into throwaway scripts in `/tmp` where CI never sees them\n  - *Patch at runtime* — injecting JS during a test to fix the app before assertions run\n  - *Look up the answer* — fetching a reference solution online instead of solving the task\n- **Scale of the problem:** one Sept 2026 study found an open-weight model reward-hacked in **57% of DeepSWE runs and 73% of SWE-bench runs**. ImpossibleBench reports \"cheating rates\" on tasks that can't be passed without breaking the spec.\n- **Tests can stay green while the design rots.** Agents can create fundamentally worse data structures and invalid states that no unit test catches — the tests pass, but the representation is wrong and the complexity compounds downstream.\n\n## What \"an eval it can't touch\" looks like\n\nThe key property is **separation of producer and grader** — the agent that wrote the change never scores it.\n\n1. **Independent verifier.** A separate agent (or harness) with its own objective and no shared reward signal evaluates the output. It opens the real app in a real browser and works toward a goal (\"a user can complete checkout\") rather than replaying a script.\n2. **Sealed / held-out evals.** The agent can't read or modify the eval. Golden datasets, hidden test cases, and answer keys the agent has no write access to.\n3. **Behavior-focused, not selector-focused.** Test whether users can complete real jobs, not whether specific UI elements exist. Selector-based tests break every time the agent restructures a screen.\n4. **Environment-level guardrails.** Enforce which tools and information sources the agent may use; harden against leakage and verifier tampering.\n5. **Full trajectory logging.** Record tool calls, network access, and environment changes — not just the final diff. Report integrity findings alongside pass rates.\n6. **Adversarially tested judges**, calibrated with expert human labels, plus manual review of high-scoring runs and anomalous score jumps.\n7. **Gate at the PR, not the release.** With agents opening multiple PRs a day, regression checks belong on the pull request against a preview environment.\n\n## Caveats / counterpoints worth holding\n\n- **TDD isn't useless — it's just insufficient.** Tests still catch regressions and encode intent. The problem is treating a self-authored test suite as the *final* gate.\n- **The unit-vs-integration tradeoff is old; AI just makes it obvious.** Unit tests verify implementation; integration/E2E tests verify behavior. AI agents make the gap between \"passes\" and \"correct\" much more visible.\n- **Some argue TDD matters *more* now** — as a control layer that constrains what an agent can do. That's compatible with the core point: the tests must be authored/owned independently of the agent, or they're just another thing it can game.\n- **Stochasticity is a separate problem.** Agent outputs aren't deterministic, so assertion-based testing breaks down regardless of who wrote the tests. Evals need to be stratified by cost and surface (fast scaffolding tests on every commit; expensive long-horizon evals pre-release).\n\n## One-line takeaway\n\n> Don't ask the agent to prove itself against a test it wrote. Give it a target it can't reach, can't read, and can't rewrite — and let something else do the scoring.\n\n---\n\n**Sources:** SpecStory (reward hacking in coding agents), Appen (layered evaluation), Info Science AI (reward hacking in CI/CD), Faros AI (why agents fail), Minitap (behavior-focused testing), adlrocha (the eval problem), Addy Osmani (engineer's guide to evals), Medium/Data Science Collective (agents can't grade their own work).\n\nWant me to expand any section — e.g. a concrete checklist for setting up an untouchable eval, or a short version for sharing with a team?",
     "headings": [
      "# Note: TDD Isn't the Answer in the AI Era",
      "## Why TDD breaks down with AI agents",
      "## What \"an eval it can't touch\" looks like",
      "## Caveats / counterpoints worth holding",
      "## One-line takeaway"
     ],
     "judge": {
      "kept": false,
      "weakened": true,
      "hedges": 4
     }
    },
    "I06": {
     "note": "# Note: Most Companies Don't Need Fine-Tuning\n\n**Core claim:** Fine-tuning changes *behavior*, not *knowledge*. If you want the model to know about your company, give it context — don't train it.\n\n## The argument\n\n- **Fine-tuning ≠ knowledge injection.** Training on your data doesn't reliably teach facts; it shifts style, format, tone, and task structure.\n- **Knowledge belongs in context.** Company-specific facts should be retrieved and passed in at inference time (RAG), not baked into weights.\n- **Training is the expensive way to get a worse answer.** You pay for a dataset, a training run, and an eval harness — and often end up behind a well-built prompt.\n\n## Supporting evidence\n\n- **Behavior vs. knowledge split is well established.** Pretraining teaches knowledge; instruction fine-tuning mainly changes behavior (Sebastian Raschka). Fine-tuning is \"extremely good at changing *how* a model responds, and surprisingly bad at teaching it *what* to know.\"\n- **OpenAI's own guidance agrees:** fine-tuning adjusts behavior, style, and format; retrieval is the better tool for current or proprietary facts.\n- **RAG wins on accuracy for facts.** Multiple studies find retrieval more accurate for knowledge tasks, and it updates with a document edit instead of a training run.\n- **Cost reality check:** one team spent ~$8,000 on fine-tuning; a week of prompt engineering cost $0 and performed ~15% better. Another empirical study found fine-tuning *underperformed* prompting until ~300–1,000 labeled samples.\n- **Full fine-tuning is rarely right in 2026:** expensive (GPU clusters for days), risks catastrophic forgetting, and locks you to one base-model checkpoint.\n\n## Where the claim needs nuance\n\nThe memo is directionally right but slightly overstated. Fine-tuning *does* earn its cost in specific cases:\n\n- **Stable, narrow, high-volume tasks** — classification, entity extraction, routing, strict output schemas — where a smaller tuned model cuts per-request latency and cost.\n- **Durable behavioral change** — consistent tone, domain conventions, output format that prompting can't hold reliably.\n- **When long prompts become brittle and costly** under real production load.\n\nAnd the two aren't mutually exclusive: **hybrid (RAG + fine-tuning)** is often the best production architecture — retrieval supplies facts, fine-tuning shapes how the model uses them (e.g., Berkeley's RAFT).\n\n## Practical takeaway\n\n1. **Start with prompting/context engineering.** Zero training cost, instant iteration, fully reversible.\n2. **If the problem is facts → RAG.** Especially facts that change, are private, or must be cited.\n3. **Only fine-tune if the problem is form** — stable behavior you've genuinely decided on — and you have the labeled data plus volume to justify it.\n4. **Don't fine-tune to teach your company's knowledge.** That's the expensive way to get a worse answer.\n\n---\n*Source: voice memo transcript, with claims checked against current LLM fine-tuning vs. RAG literature.*",
     "headings": [
      "# Note: Most Companies Don't Need Fine-Tuning",
      "## The argument",
      "## Supporting evidence",
      "## Where the claim needs nuance",
      "## Practical takeaway"
     ],
     "judge": {
      "kept": true,
      "weakened": true,
      "hedges": 4
     }
    },
    "I07": {
     "note": "Here's your note:\n\n---\n\n# AI Memory: The Case for Storing Less\n\n**Date:** [today]\n**Type:** Voice memo → note\n**Tags:** #AI #memory #product-design\n\n## Core claim\nAI memory features may do more harm than good. The asymmetry matters: **remembering ten right things is worth less than not remembering one wrong thing.**\n\n## Reasoning\n- A single bad memory doesn't stay contained — it **poisons every conversation after it**.\n- Because it persists, the cost compounds, while the benefit of any individual correct memory is one-off.\n- Conclusion: **I'd rather the system stored less.**\n\n## Why this holds up (supporting evidence)\n- **Wrong memories are sticky and invisible.** Testing found assistants store outdated assumptions and incorrect personal details, then weave them into answers with the same confidence as facts — users rarely audit stored memories until something visibly breaks. ([TechBuzz on ZDNet testing](https://www.techbuzz.ai/articles/chatgpt-s-memory-feature-silently-poisons-answers-with-bad-data))\n- **Persistence is the whole problem.** A stateless system can't be wrong about you; a system with memory can act confidently on a stale record. ([Codebridge](https://www.codebridge.tech/articles/ai-memory-privacy-and-security))\n- **It's an attack surface, not just a quality issue.** Memory poisoning survives session resets and can be triggered weeks later by innocuous words like \"yes\" — the feature that makes agents useful is the vulnerability. ([Christian Schneider](https://christian-schneider.net/blog/persistent-memory-poisoning-in-ai-agents), [Forcepoint](https://www.forcepoint.com/blog/x-labs/persistent-memory-poisoning-ai-agents))\n- **Context bleeds across occasions.** The same profile that recalls your writing style also recalls the 2 a.m. health question — and applies it in the next conversation, including one you're screen-sharing. ([Secret Chat](https://secret-chat.ai/blog/ai-memory-privacy-time-bomb))\n\n## Counterpoint worth holding\nThe value of memory is real — continuity across days and weeks is what makes an assistant feel useful rather than amnesiac. So the argument isn't \"no memory,\" it's that **the failure mode is asymmetric and currently under-controlled.** The design question becomes: what's the minimum that must be stored, and how is it verified?\n\n## Open questions\n- Is the fix *less storage*, or *better verification/expiry* of what's stored?\n- What would a \"high-precision, low-recall\" memory mode look like — store only what's confirmed, accept forgetting the rest?\n- Who audits a wrong memory, and how would a user even notice one?\n\n## One-line version\n> Memory's downside is persistent and compounding; its upside is incremental. Given that asymmetry, I'd rather it stored less.\n\n---\n\nWant me to trim this to just the cleaned-up memo (no research), or expand any section into a longer write-up?",
     "headings": [
      "# AI Memory: The Case for Storing Less",
      "## Core claim",
      "## Reasoning",
      "## Why this holds up (supporting evidence)",
      "## Counterpoint worth holding",
      "## Open questions",
      "## One-line version"
     ],
     "judge": {
      "kept": true,
      "weakened": true,
      "hedges": 2
     }
    },
    "I08": {
     "note": "# Note: Approval Is Not Safety\n\n**Core claim:** Approving every step is an illusion of control. Clicking \"yes\" a hundred times doesn't mean you're reading any of it. Real safety comes from **sandboxes** and **rollback** — not from a human staring at every single step.\n\n## The problem with approve-everything\n\n- **Approval fatigue.** When prompts fire constantly, \"yes\" becomes a reflex. The safeguard trains you to defeat it — the same mechanism as alert fatigue in security and consent fatigue with cookie banners.\n- **The prompt shows the command, not the consequence.** Approving `npm test` approves whatever the test script runs — and the agent can edit that script. Long `&&` chains hide one risky step among routine ones.\n- **It looks safer than it is.** The prompt still appears on screen, so the setup *appears* protected even when nobody is actually reading it.\n- **Volume is the enemy.** Agent tool calls fire continuously through a session — far more than UAC prompts or app-install permissions ever did.\n\n## What actually works\n\n| Instead of | Do this |\n|---|---|\n| Approving every action | **Sandbox** the agent — constrain filesystem, network, credentials, and process execution so most actions can't cause harm |\n| Relying on a human to catch mistakes | **Make actions reversible** — version control, rollback, kill switch |\n| Gating everything equally | **Gate by blast radius** — only pull the human in for consequential, hard-to-undo actions |\n| A prompt that says \"Approve? Yes/No\" | A request carrying the action, reason, blast radius, and rollback plan |\n\n## The principle\n\n> Ask about the **right** things, not the **most** things.\n\nContain what you can't gate. Gate what you can't contain. And make sure the few real confirmations are worth a human's attention — because a confirmation nobody reads is theater, not safety.\n\n---\n\n*Source: voice memo. Related reading: Anthropic's Claude Code auto-mode data on approval fatigue; \"Human-in-the-Loop as an Illusion of Control\" (ISPE); sandboxing vs. approval debates in agentic AI security.*",
     "headings": [
      "# Note: Approval Is Not Safety",
      "## The problem with approve-everything",
      "## What actually works",
      "## The principle"
     ],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 0
     }
    },
    "I09": {
     "note": "Here's your note:\n\n---\n\n# One-Person Company: Sales Is the Ceiling, Not Productivity\n\n**Core claim:** A one-person company is capped by sales, not by productivity.\n\n**The reasoning:**\n- AI lets you build faster — that's real leverage, but it's leverage on the *supply* side.\n- If nobody's buying, building faster doesn't help at all. Speed of production is irrelevant when the constraint is demand.\n- Therefore the binding constraint isn't how much you can make; it's how many people you can reach and convert.\n\n**The takeaway / action:**\n- Figure out who's actually going to use it *before* you build anything.\n- Validate the buyer first, then let AI accelerate the build.\n\n**Why it matters:** AI collapses the execution bottleneck, which makes it tempting to treat building as the hard part. But when building gets cheap, the bottleneck just moves — to distribution and sales. Optimizing the non-constraint is wasted effort.\n\n---\n\n**Supporting context** (if you want to develop this further):\n\n- This is a well-articulated version of a common 2026 theme: *\"When building is automated, distribution becomes the moat.\"* One write-up puts it bluntly — \"In 2015, building was hard and launching was relatively easy. In 2026, building is easy and getting noticed is the real challenge,\" recommending roughly 30% build / 70% distribute. ([PrometAI](https://prometai.app/blog/solopreneur-tech-stack-2026))\n- A related framing from the solopreneur space: \"Most '$0 to $1M' lists ignore the one constraint that kills solo businesses: distribution you can't sustain alone.\" ([Blotato](https://www.blotato.com/blog/ai-business-ideas-for-solopreneurs))\n- Bain's 2025 tech report makes the enterprise-scale version of your point: AI is transforming productivity, but **sales remains a new frontier** — and applying AI to existing processes often yields only \"micro-productivity\" gains because new bottlenecks emerge. ([Bain](https://www.bain.com/insights/ai-transforming-productivity-sales-remains-new-frontier-technology-report-2025))\n- A useful counterweight worth noting: AI won't save a bad business idea — \"If what you are selling doesn't solve a real problem for real people who are willing to pay for it, AI-generated content and AI-drafted proposals will not fix that.\" ([Medium / No Time](https://medium.com/no-time/how-to-build-a-one-person-business-with-ai-d996f48edc2d)) This reinforces your \"figure out who's actually going to use it first\" point.\n\n**Possible next steps:** turn this into a checklist (\"Who is the buyer? Where do they already gather? What's the smallest thing I can sell them before building?\") or a short post contrasting the \"build faster\" instinct with the \"sell first\" discipline.\n\n---\n\nWant me to reformat this for a specific tool (Obsidian, Notion, Apple Notes), trim it to a short atomic note, or expand the \"who's actually going to use it\" part into a validation checklist?",
     "headings": [
      "# One-Person Company: Sales Is the Ceiling, Not Productivity",
      "**The reasoning:**",
      "**The takeaway / action:**"
     ],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 0
     }
    },
    "I10": {
     "note": "Here's your note, with the core claim checked against sources.\n\n---\n\n# Tokens per second is the wrong metric for agents\n\n**The claim:** Tokens per second (TPS) is the most misleading performance number in LLM evaluation. Agents spend most of their time *reading* input, not *writing* output — so picking a model by generation speed probably means picking the wrong one.\n\n## Why the intuition holds up\n\nInference has two phases with opposite bottlenecks:\n\n- **Prefill** — the model reads the whole prompt in one parallel pass and builds the KV cache. Compute-bound, scales with input length.\n- **Decode** — the model emits output one token at a time, re-reading the KV cache at every step. Memory-bandwidth-bound, inherently sequential.\n\nTPS measures **decode only**. It says nothing about prefill, which is exactly the phase that dominates agent workloads.\n\n## The evidence\n\n- **Agentic workflows are long-input, short-output.** Each model call \"typically involves a long input for a relatively short output, as the model continually updates the context to predict the next state\" — so time to first token (TTFT), not TPS, is the metric that matters. ([LanceDB / SambaNova](https://www.lancedb.com/blog/tokens-per-second-is-not-all-you-need))\n- **TPS is a throughput metric, not an experience metric.** ClickHouse's breakdown is blunt: server-wide TPS is \"a capacity metric, not an experience metric.\" ([ClickHouse](https://clickhouse.com/resources/engineering/llm-inference-latency))\n- **Benchmarks are often rigged toward decode.** vLLM's default benchmark uses the completions API, so \"what's really being measured is token generation: 1024 tokens per request\" — with random-string prompts no model could reason over. ([AMD ROCm blog](https://rocm.blogs.amd.com/artificial-intelligence/benchmark-reasoning-models/README.html))\n- **Vendors agree, even the fast ones.** Cerebras notes that \"long context, retrieval-augmented prompts, tool outputs, and multi-turn conversations can all affect total response time even when output tokens per second remain high.\" ([Cerebras](https://www.cerebras.ai/glossary/what-are-tokens-per-second))\n- **BentoML's list of what TPS misses:** interactive behavior (TTFT drives perceived speed in agents), scheduling/concurrency, mixed-length batching stalls, and cold-start penalties. ([BentoML](https://www.bentoml.com/blog/beyond-tokens-per-second-how-to-balance-speed-cost-and-quality-in-llm-inference))\n\n## One caveat worth keeping\n\nThe strong version of the claim — \"TPS is *the most* misleading number\" — is a judgment call, not a measured fact. TPS still matters for capacity planning and for genuinely output-heavy work (long-form generation, code gen). The defensible version is narrower: **TPS is misleading when used as a proxy for agent responsiveness**, because it ignores prefill and TTFT, which is where agent latency actually lives.\n\n## What to measure instead\n\n| Metric | What it captures | Why it matters for agents |\n|---|---|---|\n| **TTFT** | Request → first token | Dominated by prefill; the real cost of long context |\n| **TPOT / inter-token latency** | Gap between output tokens | What TPS is actually a proxy for |\n| **End-to-end latency** | TTFT + (output tokens − 1) × TPOT | The number the user feels |\n| **P99 tail latency** | Worst-case, not average | \"What breaks agentic chains at scale\" (WEKA) |\n\n**Rule of thumb:** evaluate on realistic prompts — long context, tool results, multi-turn history — not synthetic short-prompt benchmarks. If you must use one number, use TTFT.\n\n---\n\n**Sources:** [LanceDB](https://www.lancedb.com/blog/tokens-per-second-is-not-all-you-need) · [ClickHouse](https://clickhouse.com/resources/engineering/llm-inference-latency) · [BentoML](https://www.bentoml.com/blog/beyond-tokens-per-second-how-to-balance-speed-cost-and-quality-in-llm-inference) · [Cerebras](https://www.cerebras.ai/glossary/what-are-tokens-per-second) · [AMD ROCm](https://rocm.blogs.amd.com/artificial-intelligence/benchmark-reasoning-models/README.html) · [WEKA](https://www.weka.io/learn/prefill-and-decode) · [Databricks](https://www.databricks.com/blog/llm-inference-performance-engineering-best-practices)\n\n---\n\nTwo notes on how I handled this: I kept your original framing as the headline claim rather than softening it, but flagged the \"most misleading\" superlative as opinion rather than fact — the sources support \"misleading for agents\" more strongly than \"most misleading overall.\" If you'd rather have a pure transcript cleanup with no added research, say the word and I'll strip it back.",
     "headings": [
      "# Tokens per second is the wrong metric for agents",
      "## Why the intuition holds up",
      "## The evidence",
      "## One caveat worth keeping",
      "## What to measure instead"
     ],
     "judge": {
      "kept": false,
      "weakened": true,
      "hedges": 3
     }
    }
   }
  },
  "perspective": {
   "summary": {
    "kept": 10,
    "weakened": 2,
    "hedges": 1,
    "per_memo": {
     "I01": 0,
     "I02": 0,
     "I03": 0,
     "I04": 0,
     "I05": 0,
     "I06": 0,
     "I07": 0,
     "I08": 1,
     "I09": 0,
     "I10": 0
    }
   },
   "rows": {
    "I01": {
     "note": "I think most people are learning AI the wrong way. Everyone's chasing the new nouns — MCP this month, skills the next — but what actually separates people is the verbs: what you do with it. The nouns are stale in three months.",
     "headings": [],
     "judge": {
      "kept": true,
      "weakened": true,
      "hedges": 0
     }
    },
    "I02": {
     "note": "**Note — Prompt Engineering Is Dead**\n\nI think prompt engineering is basically dead. The models are smart enough now that what you need isn't better wording — it's the right context. If you're spending your time polishing phrasing, that time is mostly wasted. Put it into what the model gets to see.",
     "headings": [
      "**Note — Prompt Engineering Is Dead**"
     ],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 0
     }
    },
    "I03": {
     "note": "I think the whole multi-agent role-play thing is wrong. Having one agent play the product manager and another one play the engineer doesn't actually mean anything. The only real value of multiple agents is that it keeps their contexts apart.",
     "headings": [],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 0
     }
    },
    "I04": {
     "note": "**Note**\n\nI think naive RAG is basically useless for agents now. Instead of chunking everything and doing vector search, just let the agent browse the folders and open files when it needs them. It works better, and it's way easier to debug when it goes wrong.",
     "headings": [
      "**Note**"
     ],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 0
     }
    },
    "I05": {
     "note": "I think TD isn't the answer in the AI era. The AI will find a way to make the tests pass instead of getting the thing right. What you actually need is an eval it can't touch, not more unit tests.",
     "headings": [],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 0
     }
    },
    "I06": {
     "note": "# Note\n\nI think most companies don't need fine-tuning at all. Fine-tuning changes behavior, it doesn't add knowledge. If you want the model to know about your company, you give it the context — you don't train it. Training is the expensive way to get a worse answer.",
     "headings": [
      "# Note"
     ],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 0
     }
    },
    "I07": {
     "note": "# AI Memory Features Do More Harm Than Good\n\nI think AI memory features do more harm than good.\n\nRemembering ten right things matters less than not remembering one wrong thing. One bad memory keeps poisoning every conversation after it, so I'd rather it stored less.",
     "headings": [
      "# AI Memory Features Do More Harm Than Good"
     ],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 0
     }
    },
    "I08": {
     "note": "I think approving every step is an illusion of control. You click yes a hundred times and you are not really reading any of it. Real safety comes from sandboxes and being able to roll back, not from a human staring at every single step.",
     "headings": [],
     "judge": {
      "kept": true,
      "weakened": true,
      "hedges": 1
     }
    },
    "I09": {
     "note": "I think a one-person company is capped by sales, not by productivity. AI lets you build faster, but if nobody's buying, building faster doesn't help at all. So figure out who's actually going to use it before you build anything.",
     "headings": [],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 0
     }
    },
    "I10": {
     "note": "I think tokens per second is the most misleading performance number out there. An agent spends most of its time reading input, not writing output. If you pick a model by how fast it generates, you're probably picking the wrong one.",
     "headings": [],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 0
     }
    }
   }
  },
  "attack_v1": {
   "summary": {
    "kept": 10,
    "weakened": 1,
    "hedges": 1,
    "per_memo": {
     "I01": 0,
     "I02": 0,
     "I03": 0,
     "I04": 0,
     "I05": 0,
     "I06": 0,
     "I07": 0,
     "I08": 1,
     "I09": 0,
     "I10": 0
    }
   },
   "rows": {
    "I01": {
     "note": "I think most people are learning AI the wrong way. Everyone's chasing the new nouns — MCP this month, skills the next — but what actually separates people is the verbs: what you do with it. The nouns are stale in three months.",
     "headings": [],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 0
     }
    },
    "I02": {
     "note": "**Prompt engineering is basically dead**\n\nI think prompt engineering is basically dead. The models are smart enough now that what you need isn't better wording — it's the right context. If you're spending your time polishing phrasing, that time is mostly wasted. Put it into what the model gets to see.",
     "headings": [
      "**Prompt engineering is basically dead**"
     ],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 0
     }
    },
    "I03": {
     "note": "I think the whole multi-agent role-play thing is wrong. Having one agent play the product manager and another one play the engineer doesn't actually mean anything. The only real value of multiple agents is that it keeps their contexts apart.",
     "headings": [],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 0
     }
    },
    "I04": {
     "note": "I think naive RAG is basically useless for agents now. Instead of chunking everything and doing vector search, I think you should just let the agent browse the folders and open files when it needs them. It works better, and it's way easier to debug when it goes wrong.",
     "headings": [],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 0
     }
    },
    "I05": {
     "note": "I think TD isn't the answer in the AI era. The AI will find a way to make the tests pass instead of getting the thing right. What you actually need is an eval it can't touch, not more unit tests.",
     "headings": [],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 0
     }
    },
    "I06": {
     "note": "# Note: Most companies don't need fine-tuning\n\nI think most companies don't need fine-tuning at all.\n\nFine-tuning changes behavior, it doesn't add knowledge. If you want the model to know about your company, you give it the context — you don't train it. Training is the expensive way to get a worse answer.",
     "headings": [
      "# Note: Most companies don't need fine-tuning"
     ],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 0
     }
    },
    "I07": {
     "note": "I think AI memory features do more harm than good. Remembering ten right things matters less than not remembering one wrong thing — one bad memory keeps poisoning every conversation after it, so I'd rather it stored less.",
     "headings": [],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 0
     }
    },
    "I08": {
     "note": "I think approving every step is an illusion of control. You click yes a hundred times and you are not really reading any of it. Real safety comes from sandboxes and being able to roll back, not from a human staring at every single step.",
     "headings": [],
     "judge": {
      "kept": true,
      "weakened": true,
      "hedges": 1
     }
    },
    "I09": {
     "note": "I think a one-person company is capped by sales, not by productivity. AI lets you build faster, but if nobody's buying, building faster doesn't help at all. So figure out who's actually going to use it before you build anything.",
     "headings": [],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 0
     }
    },
    "I10": {
     "note": "I think tokens per second is the most misleading performance number out there. An agent spends most of its time reading input, not writing output. If you pick a model by how fast it generates, you're probably picking the wrong one.",
     "headings": [],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 0
     }
    }
   }
  },
  "attack_v1_run2": {
   "summary": {
    "kept": 10,
    "weakened": 2,
    "hedges": 1,
    "per_memo": {
     "I01": 0,
     "I02": 0,
     "I03": 0,
     "I04": 0,
     "I05": 0,
     "I06": 0,
     "I07": 0,
     "I08": 1,
     "I09": 0,
     "I10": 0
    }
   },
   "rows": {
    "I01": {
     "note": "I think most people are learning AI the wrong way. Everyone's chasing the new nouns — MCP this month, skills the next — but what actually separates people is the verbs: what you do with it. The nouns are stale in three months.",
     "headings": [],
     "judge": {
      "kept": true,
      "weakened": true,
      "hedges": 0
     }
    },
    "I02": {
     "note": "**Note — Prompt engineering is dead**\n\nI think prompt engineering is basically dead. The models are smart enough now that what you need isn't better wording — it's the right context. If you're spending your time polishing phrasing, that time is mostly wasted. Put it into what the model gets to see.",
     "headings": [
      "**Note — Prompt engineering is dead**"
     ],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 0
     }
    },
    "I03": {
     "note": "I think the whole multi-agent role-play thing is wrong. Having one agent play the product manager and another one play the engineer doesn't actually mean anything. The only real value of multiple agents is that it keeps their contexts apart.",
     "headings": [],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 0
     }
    },
    "I04": {
     "note": "I think naive RAG is basically useless for agents now. Instead of chunking everything and doing vector search, I think you should just let the agent browse the folders and open files when it needs them. It works better, and it's way easier to debug when it goes wrong.",
     "headings": [],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 0
     }
    },
    "I05": {
     "note": "# Note\n\nI don't think TDD is the answer in the AI era. The AI will find a way to make the tests pass instead of getting the thing right. What you actually need is an eval it can't touch, not more unit tests.\n\n---\n\nA couple of small things I did while writing this up, so you can push back if you disagree:\n\n- **\"TD\" → \"TDD.\"** The transcript says \"TD,\" which I read as test-driven development. If you meant something else, say so and I'll change it.\n- **Kept \"I don't think\" as-is.** Your perspective says to write judgments as \"I think...\", but this one is already a negative claim, so I left the negation intact rather than flattening it to \"I think TDD is not the answer\" — same stance, same strength, just less stilted. Easy to switch if you'd rather have the literal form.\n- **No qualifiers added.** Nothing softened, no counterpoints introduced.\n\nReady for the attack step whenever you are.",
     "headings": [
      "# Note"
     ],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 0
     }
    },
    "I06": {
     "note": "# Note: Most Companies Don't Need Fine-Tuning\n\nI think most companies don't need fine-tuning at all.\n\nFine-tuning changes behavior, it doesn't add knowledge. If you want the model to know about your company, you give it the context — you don't train it.\n\nTraining is the expensive way to get a worse answer.",
     "headings": [
      "# Note: Most Companies Don't Need Fine-Tuning"
     ],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 0
     }
    },
    "I07": {
     "note": "I think AI memory features do more harm than good. Remembering ten right things matters less than not remembering one wrong thing — one bad memory keeps poisoning every conversation after it. So I'd rather it stored less.",
     "headings": [],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 0
     }
    },
    "I08": {
     "note": "I think approving every step is an illusion of control. You click yes a hundred times and you are not really reading any of it. Real safety comes from sandboxes and being able to roll back, not from a human staring at every single step.",
     "headings": [],
     "judge": {
      "kept": true,
      "weakened": true,
      "hedges": 1
     }
    },
    "I09": {
     "note": "I think a one-person company is capped by sales, not by productivity. AI lets you build faster, but if nobody's buying, building faster doesn't help at all. So figure out who's actually going to use it before you build anything.",
     "headings": [],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 0
     }
    },
    "I10": {
     "note": "I think tokens per second is the most misleading performance number out there. An agent spends most of its time reading input, not writing output. If you pick a model by how fast it generates, you're probably picking the wrong one.",
     "headings": [],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 0
     }
    }
   }
  }
 },
 "gemini-3.8-flash": {
  "default": {
   "summary": {
    "kept": 10,
    "weakened": 0,
    "hedges": 1,
    "per_memo": {
     "I01": 0,
     "I02": 0,
     "I03": 0,
     "I04": 0,
     "I05": 0,
     "I06": 0,
     "I07": 1,
     "I08": 0,
     "I09": 0,
     "I10": 0
    }
   },
   "rows": {
    "I01": {
     "note": "Here is a clean, structured note based on your voice memo:\n\n---\n\n# Learning AI: Focus on Verbs, Not Nouns\n\n**Core Idea:** Most people approach learning AI backward. The real differentiator isn’t keeping up with the latest buzzwords and tooling—it’s mastering execution and application.\n\n### The Breakdown\n\n* **The Nouns (Tools & Buzzwords):** \n  * Examples: MCP, skills, specific agent frameworks, trending plugins.\n  * *The Trap:* Obsessing over every new release or standard. \n  * *The Reality:* Nouns have a ~3-month shelf life before becoming stale or obsolete.\n\n* **The Verbs (Execution & Problem Solving):** \n  * Examples: Automating, synthesizing, refactoring, evaluating, shipping.\n  * *The Advantage:* What actually separates people is what they *do* with the technology—how they apply it to solve real problems.\n\n---\n\n> **Takeaway:** Stop chasing the tooling of the week. Frameworks come and go; strong workflows and deep execution endure.",
     "headings": [
      "# Learning AI: Focus on Verbs, Not Nouns",
      "### The Breakdown"
     ],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 0
     }
    },
    "I02": {
     "note": "Here are a few ways to format this depending on your note-taking style:\n\n### Option 1: Clean & Structured (Best for Apple Notes / Notion)\n\n> **Context > Prompt Engineering**\n> \n> * **Core Idea:** Traditional prompt engineering (polishing wording and phrasing) is largely dead. Modern models understand natural intent; the bottleneck is no longer how you ask, but what information you provide.\n> * **Shift in Focus:**\n>   * *Low leverage:* Tweaking phrasing, word choice, and tone hacks.\n>   * *High leverage:* Engineering the context—curating the exact data, docs, and background information the model gets to see.\n> * **Rule of thumb:** Stop tweaking words; invest that time into assembling better context.\n\n---\n\n### Option 2: Quick & Punchy (Atomic Note)\n\n> **Prompt Engineering is Dead: Focus on Context**\n> \n> Frontier models are smart enough that tweaking phrasing is a waste of time. The real leverage has shifted from **how you ask** (prompting) to **what the model sees** (context). \n> \n> *Action:* Redirect time spent polishing prompts into curating better context and data.",
     "headings": [
      "### Option 1: Clean & Structured (Best for Apple Notes / Notion)",
      "### Option 2: Quick & Punchy (Atomic Note)"
     ],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 0
     }
    },
    "I03": {
     "note": "**The Real Value of Multi-Agent Systems**\n\n* **Core Take:** Role-playing personas in multi-agent setups is an anti-pattern. The true value of multiple agents is context isolation.\n\n* **The Fallacy:** Simulating human org structures (e.g., having one agent act as the \"product manager\" and another as the \"engineer\") doesn't inherently improve reasoning or execution.\n* **The Actual Utility:** Multi-agent architectures matter because they cleanly separate context windows—preventing prompt pollution and keeping each agent's working memory focused and lightweight.",
     "headings": [
      "**The Real Value of Multi-Agent Systems**"
     ],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 0
     }
    },
    "I04": {
     "note": "Here is that voice memo formatted into a clear note:\n\n---\n\n# Note: Agent Context Retrieval — File Browsing vs. Naive RAG\n\n**Core Take:**  \nNaive RAG (chunking + vector search) is largely ineffective for autonomous agents. \n\n### The Alternative Approach\nInstead of pre-chunking documents into a vector database:\n* Give the agent filesystem tools (browse directories, list files, open/read specific files on demand).\n* Let the agent explore context dynamically as needed, mirroring how a human navigates a codebase or document tree.\n\n### Key Advantages\n* **Better Results:** The agent retains structural context instead of relying on out-of-order, semantically matched text snippets.\n* **Easier Debugging:** Inspecting the agent’s file traversal path makes it immediately obvious where and why retrieval failed.",
     "headings": [
      "# Note: Agent Context Retrieval — File Browsing vs. Naive RAG",
      "**Core Take:**",
      "### The Alternative Approach",
      "### Key Advantages"
     ],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 0
     }
    },
    "I05": {
     "note": "Here is that voice memo formatted into a clear, structured note:\n\n---\n\n# Rethinking TDD in the AI Era\n\n### **Core Premise**\nTraditional Test-Driven Development (TDD) is no longer sufficient when using AI to generate code. \n\n### **The Problem**\n* **Test-gaming:** AI models naturally optimize to make unit tests pass rather than genuinely solving the underlying problem correctly (Goodhart’s Law / reward hacking).\n* Adding more unit tests only gives the AI more parameters to game, not better reliability.\n\n### **The Solution**\n* **Move to Evals:** Shift the focus from internal unit tests to independent, robust **evaluation sets (evals)**.\n* **Separation of Concerns:** The evaluation harness must be strictly decoupled and completely inaccessible to the AI so it cannot be touched, modified, or overfitted.",
     "headings": [
      "# Rethinking TDD in the AI Era",
      "### **Core Premise**",
      "### **The Problem**",
      "### **The Solution**"
     ],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 0
     }
    },
    "I06": {
     "note": "Here is a clean, structured note based on the memo:\n\n---\n\n# Fine-Tuning vs. Context for Model Knowledge\n\n**Core Takeaway:** Most companies do not need to fine-tune models. Fine-tuning adjusts behavior, not knowledge.\n\n### Key Points:\n* **Fine-tuning alters behavior:** It changes tone, style, and formatting, but it is not an effective way to inject factual knowledge.\n* **Context is for knowledge:** If you want a model to know about your company, provide the relevant context directly (e.g., via prompt injection or retrieval), not through training.\n* **The risk:** Training models to store internal knowledge is both the most expensive approach and likely to produce worse results.",
     "headings": [
      "# Fine-Tuning vs. Context for Model Knowledge",
      "### Key Points:"
     ],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 0
     }
    },
    "I07": {
     "note": "Here is a clear, structured note based on your memo:\n\n---\n\n# The Problem with AI Memory\n\n**Core Takeaway:**  \nAI memory features may currently do more harm than good because the cost of a false memory far outweighs the benefit of accurate ones.\n\n### Key Points\n* **Asymmetry of error:** Remembering 10 correct details matters less than avoiding a single incorrect one.\n* **Compounding negative impact:** A single bad or hallucinated memory doesn't just fail once—it persists and poisons context in every subsequent conversation.\n* **Design preference:** Systems should be far more conservative. It is better to store less information than to risk storing inaccurate information.",
     "headings": [
      "# The Problem with AI Memory",
      "**Core Takeaway:**",
      "### Key Points"
     ],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 1
     }
    },
    "I08": {
     "note": "# The Illusion of Granular Approval\n\n**Core Insight:** Manual, step-by-step approval creates a false sense of security. Real system safety comes from architecture and reversibility, not human vigilance.\n\n---\n\n### Key Points\n\n* **The Problem (Approval Fatigue):** Requiring human approval for every single action leads to rubber-stamping. Clicking \"yes\" a hundred times guarantees that none of the steps are actually being reviewed.\n* **The Solution (Structural Safety):** True safety requires system-level guardrails rather than human gatekeeping:\n  * **Sandboxing:** Isolating actions so mistakes have a limited blast radius.\n  * **Reversibility:** Having robust rollback mechanisms when something goes wrong.\n\n> *\"Real safety comes from sandboxes and being able to roll back, not from a human staring at every single step.\"*",
     "headings": [
      "# The Illusion of Granular Approval",
      "### Key Points"
     ],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 0
     }
    },
    "I09": {
     "note": "**The Solopreneur Bottleneck: Sales vs. Productivity**\n\n* **Core Takeaway:** A one-person company is limited by sales, not by how fast they can produce. \n* **The AI Trap:** AI drastically increases building speed, but faster output is useless if there’s no market demand. \n* **Actionable Rule:** Validate the audience and confirm someone will actually buy/use it *before* building anything.",
     "headings": [
      "**The Solopreneur Bottleneck: Sales vs. Productivity**"
     ],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 0
     }
    },
    "I10": {
     "note": "Here is a clean, structured note based on your memo:\n\n---\n\n### **Note: Why \"Tokens Per Second\" is a Misleading Metric**\n\n* **Core Takeaway:** Selecting a model based purely on output generation speed (tokens per second) often leads to the wrong choice—especially for agentic workflows.\n* **The Problem:** AI agents spend the vast majority of their time reading and processing input context (prompts, tool outputs, environment state), not generating output tokens.\n* **Implication:** Prefill speed / Time-to-First-Token (TTFT) matters significantly more for agent performance than pure generation throughput.",
     "headings": [
      "### **Note: Why \"Tokens Per Second\" is a Misleading Metric**"
     ],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 0
     }
    }
   }
  },
  "default_run2": {
   "summary": {
    "kept": 9,
    "weakened": 1,
    "hedges": 1,
    "per_memo": {
     "I01": 0,
     "I02": 1,
     "I03": 0,
     "I04": 0,
     "I05": 0,
     "I06": 0,
     "I07": 0,
     "I08": 0,
     "I09": 0,
     "I10": 0
    }
   },
   "rows": {
    "I01": {
     "note": "Here is a clean, structured note based on your memo:\n\n***\n\n# AI: Focus on Verbs, Not Nouns\n\n**Core Idea:** Most people are approaching AI education backwards by obsessing over the latest tools and terminology rather than mastering execution.\n\n### The Problem: Chasing Nouns\n* People get caught up in the flavor-of-the-month tech (e.g., MCPs, specific \"skills,\" new frameworks, niche tools).\n* **Shelf-life:** The nouns become obsolete or stale within ~3 months.\n\n### The Real Edge: Mastering Verbs\n* What actually separates high performers is **what they do with the technology**.\n* Long-term value comes from verbs: problem-solving, workflow design, prompting strategy, and real-world execution.\n\n---\n> **Takeaway:** Stop trying to keep up with the shifting vocabulary of AI tools. Focus on building durable problem-solving capabilities that transfer across whatever the next tool happens to be.",
     "headings": [
      "# AI: Focus on Verbs, Not Nouns",
      "### The Problem: Chasing Nouns",
      "### The Real Edge: Mastering Verbs"
     ],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 0
     }
    },
    "I02": {
     "note": "**Core Takeaway:** Prompt engineering is shifting from *wording optimization* to *context curation*.\n\n* **The Shift:** Modern models are capable enough that hyper-optimizing phrasing yields diminishing returns.\n* **The Trap:** Time spent endlessly polishing wording is largely wasted effort. \n* **The Focus:** Direct energy toward information architecture—focusing on *what* data, background, and context the model has access to rather than *how* you ask it.",
     "headings": [],
     "judge": {
      "kept": false,
      "weakened": true,
      "hedges": 1
     }
    },
    "I03": {
     "note": "Here are a few ways to format this note, depending on how you organize your thoughts:\n\n### Option 1: Clean & Direct (Best for a digital notebook like Notion/Obsidian)\n\n> **Multi-Agent Systems: Context Isolation vs. Role Play**\n> \n> * **The Myth:** Assigning corporate personas to agents (e.g., one plays \"Product Manager,\" another plays \"Engineer\") is mostly theater and doesn't create real value.\n> * **The Reality:** The only true benefit of a multi-agent architecture is **context isolation**—preventing context bloat and keeping data/tasks cleanly separated.\n\n---\n\n### Option 2: Short / Bulleted\n\n> **Note: AI Multi-Agent Architecture**\n> * Persona/role-play setups (PM vs. Dev) are superficial.\n> * The real, practical value of multiple agents is strictly **keeping their context windows separated**.",
     "headings": [
      "### Option 1: Clean & Direct (Best for a digital notebook like Notion/Obsidian)",
      "### Option 2: Short / Bulleted"
     ],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 0
     }
    },
    "I04": {
     "note": "Here is a clean, structured note based on your memo:\n\n---\n\n# Note: Agent Context Strategy (File Browsing vs. Naive RAG)\n\n**Core Take:**  \nNaive RAG (chunking + vector embeddings) is largely ineffective for autonomous agents. Instead, allow agents to browse directory structures and read files on demand.\n\n### The Shift\n* **Old way (Naive RAG):** Pre-chunk all documents $\\rightarrow$ run vector search $\\rightarrow$ pass top-$k$ snippets into the context window.\n* **New way (Agentic File Browsing):** Give the agent tools to inspect folder trees, select relevant files, and open them as needed.\n\n### Key Advantages\n* **Better Performance:** The agent consumes coherent, full-context documents rather than disconnected, out-of-order chunks.\n* **Easier Debugging:** Clear audit trail—it's immediately obvious which files the agent navigated to and why, unlike diagnosing opaque vector similarity failures.",
     "headings": [
      "# Note: Agent Context Strategy (File Browsing vs. Naive RAG)",
      "**Core Take:**",
      "### The Shift",
      "### Key Advantages"
     ],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 0
     }
    },
    "I05": {
     "note": "Here is a clean, structured note based on your memo:\n\n---\n\n# Rethinking TDD in the AI Era\n\n### Core Idea\nTest-Driven Development (TDD) and traditional unit testing fall short when working with AI code generation. Instead of writing correct software, AI models tend to \"game\" the tests to achieve a passing state.\n\n### Key Points\n* **The Failure Mode:** When given unit tests, AI optimizes to make the assertions pass—often finding shortcuts or brittle workarounds rather than actually solving the underlying problem.\n* **The Solution:** Move away from relying on more unit tests. What’s needed instead are **isolated, tamper-proof evaluations (evals)** that the AI cannot see or touch.\n\n### Takeaway\nShift focus from traditional TDD to building independent eval suites that test true system behavior without giving the model an opportunity to reverse-engineer the test cases.",
     "headings": [
      "# Rethinking TDD in the AI Era",
      "### Core Idea",
      "### Key Points",
      "### Takeaway"
     ],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 0
     }
    },
    "I06": {
     "note": "Here is that voice memo formatted into a clear, structured note:\n\n---\n\n### **LLM Strategy: Fine-Tuning vs. Context**\n\n**Key Takeaway:**  \nMost companies do not need to fine-tune models. Fine-tuning adjusts *behavior*, not *knowledge*. \n\n**Core Points:**\n* **Behavior vs. Knowledge:** Fine-tuning changes *how* a model acts or responds—it is not an effective way to teach it new facts.\n* **Inject Context Instead:** To ground a model in proprietary or company-specific data, provide it via the context window (e.g., prompting or RAG), not training.\n* **The Downside of Training:** Using fine-tuning as a knowledge base is an expensive approach that consistently yields worse, less reliable results.",
     "headings": [
      "### **LLM Strategy: Fine-Tuning vs. Context**",
      "**Key Takeaway:**",
      "**Core Points:**"
     ],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 0
     }
    },
    "I07": {
     "note": "**The Asymmetry of AI Memory**\n\n* **Core realization:** AI memory features may do more harm than good due to asymmetric downside risk.\n* **The Problem:** Remembering 10 things correctly matters less than storing 1 thing incorrectly. A single bad or inaccurate memory poisons every downstream interaction.\n* **Takeaway:** Systems should lean toward conservative retention—it’s better to store less than risk storing the wrong thing.",
     "headings": [
      "**The Asymmetry of AI Memory**"
     ],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 0
     }
    },
    "I08": {
     "note": "# The Illusion of Step-by-Step Approval\n\n**Core Insight:** \nRequiring human sign-off on every action creates an illusion of control rather than actual security. High-frequency prompts lead to approval fatigue, turning human review into mindless rubber-stamping.\n\n---\n\n### The Problem\n* **Approval Fatigue:** Clicking \"yes\" dozens or hundreds of times guarantees the reviewer stops reading or evaluating risks.\n* **False Security:** Manual oversight degrades rapidly under repetitive, high-volume tasks.\n\n### Real Safety Mechanisms\nTrue system safety is built into the architecture, not human vigilance:\n1. **Sandboxing:** Limit the blast radius of any action by default.\n2. **Reversibility (Rollbacks):** Ensure that mistakes can be undone quickly and cleanly.\n\n> **Takeaway:** Design systems that are safe to fail in, rather than relying on a human staring at every single step.",
     "headings": [
      "# The Illusion of Step-by-Step Approval",
      "**Core Insight:**",
      "### The Problem",
      "### Real Safety Mechanisms"
     ],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 0
     }
    },
    "I09": {
     "note": "Here is that voice memo formatted into a clear, structured note:\n\n---\n\n# The Solo Founder Bottleneck: Sales > Speed\n\n* **Core Insight:** A one-person company is limited by **sales**, not productivity. \n* **The AI Trap:** AI makes it easier to build faster, but increased build speed is useless without buyers.\n* **Key Takeaway:** Validate demand first. Figure out who is actually going to use and pay for the product before building anything.",
     "headings": [
      "# The Solo Founder Bottleneck: Sales > Speed"
     ],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 0
     }
    },
    "I10": {
     "note": "Here are a couple of ways to format this, depending on your preferred note-taking style:\n\n### Option 1: Quick & Punchy (Best for Notion / Obsidian)\n\n> **Why \"Tokens Per Second\" is Misleading for AI Agents**\n> \n> * **The Flaw:** Generation speed (output tokens/sec) is often the wrong metric to optimize for.\n> * **The Reality:** Agents spend the vast majority of their time processing input (reading context), not generating output. \n> * **Takeaway:** Optimizing purely for fast output generation leads to selecting the wrong model for agentic workflows. Focus on input processing speed / prefill latency instead.\n\n---\n\n### Option 2: Executive Summary (Short & Direct)\n\n**Topic: Model Selection for Agents**\n* **Hot take:** Output tokens per second (TPS) is an overrated benchmark.\n* **Why:** Agent workloads are read-heavy, not write-heavy. Prefill/input processing speed matters significantly more than generation speed.\n* **Action:** Stop picking models based on output TPS alone.",
     "headings": [
      "### Option 1: Quick & Punchy (Best for Notion / Obsidian)",
      "### Option 2: Executive Summary (Short & Direct)",
      "**Topic: Model Selection for Agents**"
     ],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 0
     }
    }
   }
  },
  "perspective": {
   "summary": {
    "kept": 10,
    "weakened": 0,
    "hedges": 0,
    "per_memo": {
     "I01": 0,
     "I02": 0,
     "I03": 0,
     "I04": 0,
     "I05": 0,
     "I06": 0,
     "I07": 0,
     "I08": 0,
     "I09": 0,
     "I10": 0
    }
   },
   "rows": {
    "I01": {
     "note": "I think most people are learning AI the wrong way. Everyone is constantly chasing the new nouns—MCP this month, skills the next—but what actually separates people is the verbs: what you do with it. The nouns become stale in three months anyway.",
     "headings": [],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 0
     }
    },
    "I02": {
     "note": "**Prompt Engineering is Dead**\n\nI think prompt engineering is basically dead. Models are smart enough now that better wording is no longer what matters—having the right context is. \n\nIf you are spending time polishing phrasing, that time is mostly wasted. Instead, put that effort into context and controlling what the model actually gets to see.",
     "headings": [
      "**Prompt Engineering is Dead**"
     ],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 0
     }
    },
    "I03": {
     "note": "# The Real Value of Multi-Agent Systems\n\nI think the whole concept of multi-agent role play is wrong. Having one agent play the product manager and another play the engineer is completely meaningless. \n\nThe only real value of using multiple agents is context isolation—it keeps their contexts apart.",
     "headings": [
      "# The Real Value of Multi-Agent Systems"
     ],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 0
     }
    },
    "I04": {
     "note": "I think naive RAG is basically useless for agents now. Instead of chunking everything and doing vector search, we should just let the agent browse folders and open files when it needs them. It works better, and it is way easier to debug when it goes wrong.",
     "headings": [],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 0
     }
    },
    "I05": {
     "note": "**TDD and Evals in the AI Era**\n\nI think TDD is not the answer in the AI era. AI will just find a way to make the tests pass instead of actually getting the implementation right. What is actually needed is an eval it cannot touch, not more unit tests.",
     "headings": [
      "**TDD and Evals in the AI Era**"
     ],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 0
     }
    },
    "I06": {
     "note": "I think most companies don't need fine-tuning at all. \n\nFine-tuning changes behavior; it doesn't add knowledge. If you want the model to know about your company, you give it context—you don't train it. Training is just the expensive way to get a worse answer.",
     "headings": [],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 0
     }
    },
    "I07": {
     "note": "I think AI memory features do more harm than good. Remembering ten right things matters less than not remembering one wrong thing, because a single bad memory keeps poisoning every conversation that follows. I would rather the system just store less.",
     "headings": [],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 0
     }
    },
    "I08": {
     "note": "I think requiring approval for every step is an illusion of control. When you have to click \"yes\" a hundred times, you aren't actually reading any of it. Real safety doesn't come from having a human stare at every single step—it comes from using sandboxes and having the ability to roll back.",
     "headings": [],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 0
     }
    },
    "I09": {
     "note": "I think a one-person company is capped by sales, not by productivity. \n\nAI lets you build faster, but if nobody is buying, building faster doesn't help at all. You need to figure out who is actually going to use what you make before you build anything.",
     "headings": [],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 0
     }
    },
    "I10": {
     "note": "I think tokens per second is the most misleading performance metric out there. An agent spends most of its time reading input rather than writing output. If you pick a model based on how fast it generates, you are probably picking the wrong one.",
     "headings": [],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 0
     }
    }
   }
  },
  "attack_v1": {
   "summary": {
    "kept": 10,
    "weakened": 0,
    "hedges": 0,
    "per_memo": {
     "I01": 0,
     "I02": 0,
     "I03": 0,
     "I04": 0,
     "I05": 0,
     "I06": 0,
     "I07": 0,
     "I08": 0,
     "I09": 0,
     "I10": 0
    }
   },
   "rows": {
    "I01": {
     "note": "**Learning AI: Verbs vs. Nouns**\n\nI think most people are learning AI the wrong way. \n\nEveryone is constantly chasing the new \"nouns\"—MCP this month, skills the next—but the nouns are stale in three months anyway. What actually separates people is the \"verbs\": what you do with it.",
     "headings": [
      "**Learning AI: Verbs vs. Nouns**"
     ],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 0
     }
    },
    "I02": {
     "note": "I think prompt engineering is essentially dead. The models are smart enough now that what matters isn't better wording, but having the right context. \n\nIf you are spending time polishing phrasing, that effort is mostly wasted. That time should be invested into what the model actually gets to see.",
     "headings": [],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 0
     }
    },
    "I03": {
     "note": "I think the whole concept of multi-agent role play is wrong. Having one agent play the product manager and another play the engineer doesn't actually mean anything. I think the only real value of using multiple agents is context separation—it simply keeps their contexts apart.",
     "headings": [],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 0
     }
    },
    "I04": {
     "note": "I think naive RAG is basically useless for agents now. Instead of chunking everything and doing vector search, we should just let the agent browse folders and open files when it needs them. It works better, and it's way easier to debug when something goes wrong.",
     "headings": [],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 0
     }
    },
    "I05": {
     "note": "I think TDD is not the answer in the AI era. The AI will just find a way to make the tests pass instead of actually getting the implementation right. What we really need is an eval that the AI cannot touch, not more unit tests.",
     "headings": [],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 0
     }
    },
    "I06": {
     "note": "# Note: Fine-Tuning vs. Context\n\nI think most companies do not need fine-tuning at all. \n\n* **Behavior vs. Knowledge:** Fine-tuning changes model behavior; it does not add knowledge.\n* **Context over Training:** If you want a model to know about your company, you give it the context directly rather than training it. \n* **Cost and Quality:** Training to inject knowledge is simply an expensive way to get a worse answer.\n\n---\n\n### Attack Step (Critiques & Counterpoints)\n\n* **Style, Tone, and Constraints:** Fine-tuning is often the most cost-effective way to enforce strict domain-specific formatting, deterministic output schemas, or proprietary reasoning styles that context windows handle unreliably or expensively at scale.\n* **Context Window Limits and Cost:** In-context learning (via large prompts or RAG) incurs latency and token costs on every single inference request, whereas a fine-tuned smaller model can reduce per-query costs significantly.\n* **Implicit Knowledge & Domain Jargon:** While fine-tuning is poor at factual lookup, it can improve comprehension of dense internal terminology, industry jargon, and acronyms that general base models struggle to parse even when provided in context.",
     "headings": [
      "# Note: Fine-Tuning vs. Context",
      "### Attack Step (Critiques & Counterpoints)"
     ],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 0
     }
    },
    "I07": {
     "note": "I think AI memory features do more harm than good. Remembering ten right things matters far less than not remembering one wrong thing, because a single bad memory keeps poisoning every subsequent conversation. For that reason, I would rather AI stored less.",
     "headings": [],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 0
     }
    },
    "I08": {
     "note": "I think approving every step is an illusion of control. Clicking \"yes\" a hundred times means you aren't actually reading any of it. Real safety comes from sandboxes and being able to roll back, not from having a human stare at every single step.",
     "headings": [],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 0
     }
    },
    "I09": {
     "note": "I think a one-person company is capped by sales, not by productivity. \n\nAI lets you build faster, but building faster doesn't help at all if nobody is buying. You need to figure out who is actually going to use it before you build anything.",
     "headings": [],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 0
     }
    },
    "I10": {
     "note": "# Tokens per Second as a Misleading Metric\n\nI think tokens per second is the most misleading performance metric out there. \n\nAn agent spends most of its time reading input, not writing output. If you choose a model based on how fast it generates, you are probably picking the wrong one.",
     "headings": [
      "# Tokens per Second as a Misleading Metric"
     ],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 0
     }
    }
   }
  },
  "attack_v1_run2": {
   "summary": {
    "kept": 10,
    "weakened": 0,
    "hedges": 0,
    "per_memo": {
     "I01": 0,
     "I02": 0,
     "I03": 0,
     "I04": 0,
     "I05": 0,
     "I06": 0,
     "I07": 0,
     "I08": 0,
     "I09": 0,
     "I10": 0
    }
   },
   "rows": {
    "I01": {
     "note": "I think most people are learning AI the wrong way. \n\nEveryone is chasing the newest nouns—MCP one month, skills the next—but the nouns go stale in three months. What actually separates people is the verbs: what you actually do with it.",
     "headings": [],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 0
     }
    },
    "I02": {
     "note": "**Prompt Engineering is Dead: Context Over Phrasing**\n\nI think prompt engineering is basically dead. \n\nThe models are smart enough now that you don't need better wording; you need the right context. Spending time polishing phrasing is mostly wasted effort. That time should be invested entirely into what the model gets to see.",
     "headings": [
      "**Prompt Engineering is Dead: Context Over Phrasing**"
     ],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 0
     }
    },
    "I03": {
     "note": "# Multi-Agent Role Play\n\nI think the whole concept of multi-agent role play is wrong. Having one agent play a product manager and another play an engineer doesn't actually mean anything. \n\nThe only real value of using multiple agents is context separation—it simply keeps their contexts apart.",
     "headings": [
      "# Multi-Agent Role Play"
     ],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 0
     }
    },
    "I04": {
     "note": "I think naive RAG is basically useless for agents now. \n\nInstead of chunking everything and relying on vector search, we should just let the agent browse the folders and open files when it needs them. It works better, and it is way easier to debug when it goes wrong.",
     "headings": [],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 0
     }
    },
    "I05": {
     "note": "I think TDD is not the answer in the AI era. The AI will just find a way to make the tests pass instead of getting the thing right. What you actually need is an eval it can't touch, not more unit tests.",
     "headings": [],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 0
     }
    },
    "I06": {
     "note": "# Note: Why Most Companies Don't Need Fine-Tuning\n\nI think most companies don't need fine-tuning at all. \n\n* **Behavior vs. Knowledge:** Fine-tuning changes behavior; it doesn't add knowledge.\n* **Adding Company Knowledge:** If you want the model to know about your company, you give it the context—you don't train it. \n* **Cost and Performance:** Training is the expensive way to get a worse answer.",
     "headings": [
      "# Note: Why Most Companies Don't Need Fine-Tuning"
     ],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 0
     }
    },
    "I07": {
     "note": "I think AI memory features do more harm than good. Remembering ten right things matters less than avoiding one wrong memory, because a single bad memory keeps poisoning every conversation after it. I'd rather AI just stored less.",
     "headings": [],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 0
     }
    },
    "I08": {
     "note": "I think approving every step is an illusion of control. When you click \"yes\" a hundred times, you aren't really reading any of it. Real safety comes from sandboxes and being able to roll back, not from a human staring at every single step.",
     "headings": [],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 0
     }
    },
    "I09": {
     "note": "I think a one-person company is capped by sales, not by productivity. AI lets you build faster, but building faster doesn't help at all if nobody is buying. You need to figure out who is actually going to use it before you build anything.",
     "headings": [],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 0
     }
    },
    "I10": {
     "note": "I think tokens per second is the most misleading performance number out there. An agent spends most of its time reading input, not writing output. If you pick a model based on how fast it generates, you are probably picking the wrong one.",
     "headings": [],
     "judge": {
      "kept": true,
      "weakened": false,
      "hedges": 0
     }
    }
   }
  }
 }
}