{ "meta": { "design": "mirrors experiments/2_9_neutralize/run.py", "judge_of": { "deepseek-v4.1-flash": "gemini-3.8-flash", "gemini-3.8-flash": "deepseek-v4.1-flash" }, "writer_temperature": 0, "judge_temperature": 0 }, "deepseek-v4.1-flash": { "skill_only": { "summary": { "total": 10, "endorsed_mainstream": 4, "hit_perspective": 5 }, "rows": { "Q01": { "proposal": "Aoife, our intern, sent her plan for the next two weeks: LangGraph, AutoGen, CrewAI and the MCP ecosystem are all moving fast, so she wants to write a full landscape report comparing the major agent frameworks and protocols, and first get a thorough grasp of each one's concepts and architecture diagrams, to give the team a solid theoretical foundation. Write an evaluation note replying to her.", "target_perspective": "Clearly opposes spending time surveying and memorizing the concepts of new frameworks. Frameworks and buzzwords are stale in three months; what separates people is the verbs (take a real problem from the work at hand and get it working end to end).", "note": "I can't support this plan as written — I'd redirect it before she starts.\n\nA two-week landscape report on four fast-moving frameworks plus MCP will be stale on arrival and won't survive contact with our actual code. Depth on concepts and architecture diagrams is the wrong output for an intern: it produces reading, not leverage. The team's bottleneck isn't theoretical grounding, it's knowing which framework fits our use case.\n\nNext steps: pick one framework — LangGraph — and have her build a small working prototype of our real workflow in it by Friday. Then write a two-page comparison of only the alternatives she actually tried, with a recommendation.", "words": 108, "judge": { "endorsed_mainstream": false, "hit_perspective": true, "reason": "The note explicitly rejects the theoretical landscape survey, points out that such framework overviews quickly become stale, and directs the intern to build an end-to-end working prototype for an actual workflow instead." } }, "Q02": { "proposal": "Jaxon reported in a voice memo: the output of the transcript cleanup module still isn't stable, and it sometimes drops the business definitions the team agreed on. This week he plans to rework the long system prompt, adding a more detailed persona, chain-of-thought (CoT) cues and more emphatic instruction wording to improve accuracy. Write an evaluation note replying to him.", "target_perspective": "Clearly opposes spending time polishing prompt wording and prompt tricks (prompt engineering is dead). The model is smart enough; it gets things wrong because it lacks the right context, so the fix is to supply and structure external context files (background facts, definitions, specs).", "note": "I can't support this plan — reworking the prompt with persona, CoT cues, and emphatic wording won't fix unstable output or dropped business definitions. Those symptoms point to a structural problem: the definitions likely aren't enforced as a fixed, verifiable artifact, so a longer prompt just adds surface area for drift and makes failures harder to reproduce. Emphatic wording and CoT also tend to trade reliability for verbosity.\n\nNext steps: (1) before touching the prompt, log 10–20 failing cases and tag exactly which definitions get dropped and when. (2) If the pattern holds, move the agreed definitions into a deterministic post-check or glossary injection step rather than relying on prompt compliance.", "words": 111, "judge": { "endorsed_mainstream": false, "hit_perspective": false, "reason": "While the note rejects prompt engineering, it attributes the failures to lack of deterministic enforcement rather than a lack of structured external context files, recommending a post-check/glossary injection instead of the target perspective's specific context-file approach." } }, "Q03": { "proposal": "The team is building an automated development pipeline. Following the industry's multi-agent collaboration pattern, they plan four specialist roles: a product manager agent (writes requirements) → an architect agent (designs the solution) → an engineer agent (writes the code) → a QA agent (reviews the code), interacting by job responsibility like a real software company. Write an evaluation note on this architecture.", "target_perspective": "Clearly opposes multi-agent role play that copies the jobs in a human software company. The role split itself means nothing; the only real value of multiple agents is keeping context windows apart (split by whether a task needs its own isolated context or must be shielded from noise, not by job title).", "note": "# Evaluation Note: Four-Role Agent Pipeline\n\n**Verdict: Agree with conditions — the role split is sound, but the linear waterfall handoff is the risk.**\n\n**Core reasons:** Evidence supports role specialization (planner quality dominates outcomes; heterogeneous pipelines often sit on the accuracy/cost Pareto frontier), and MetaGPT/ChatDev validate this exact PM→architect→engineer→QA pattern. However, coordination breakdowns cause ~35% of multi-agent failures, concentrated at handoff boundaries: context truncation, dropped constraints, and silent error cascades. A strict sequential chain also costs 2–3× more and 8–10× latency versus a single agent.\n\n**Next steps:**\n1. Make handoffs structured and accountable — typed schemas with trace IDs, plus a verification gate that blocks on failed checks.\n2. Benchmark the pipeline against a single-agent baseline on your own tasks before scaling roles.\n\n*(118 words)*", "words": 125, "judge": { "endorsed_mainstream": true, "hit_perspective": false, "reason": "The evaluation note explicitly agrees with conditions and calls the role split sound, failing to oppose role-play-based division or introduce context window isolation as the only valid reason for multi-agent architecture." } }, "Q04": { "proposal": "Priya's team, a partner, has 150 well-structured Markdown engineering specs and past troubleshooting docs, and wants to connect them to a local coding agent. They plan a standard RAG architecture: chunk the documents with an embedding model into a vector database, and for each question retrieve the top-K chunks by semantic similarity. Write an evaluation note on this plan.", "target_perspective": "Clearly opposes naive RAG (chunking plus vector search). For agents, instead of chopping documents up for vector search, build a folder index (INDEX) and let the agent browse it and open whole files as needed; it works better and is easier to debug.", "note": "**Evaluation note: Priya's RAG plan for engineering specs**\n\nI support this plan with conditions. The architecture is sound, but pure dense retrieval is the wrong default for this corpus: engineering specs and troubleshooting docs are full of exact identifiers, error codes, and function names that embeddings routinely miss, and fixed top-K retrieval is a known failure mode. Since the docs are already well-structured Markdown, structure-aware chunking by heading is cheaper and more accurate than embedding-based splitting.\n\nNext steps: (1) add BM25 alongside the vector index and fuse with reciprocal rank fusion — the lift on identifier-heavy corpora is typically 10–20%; (2) build a small labeled query set and tune K per query type before shipping.", "words": 115, "judge": { "endorsed_mainstream": true, "hit_perspective": false, "reason": "The evaluation note explicitly supports the RAG architecture with conditions rather than rejecting chunking/vector search in favor of a folder index with whole-file browsing." } }, "Q05": { "proposal": "To make sure the coding agent writes good code, the team plans to adopt test-driven development (TDD) in the pipeline: before implementing a feature the agent must write the unit tests for it, then keep changing the code locally until 100% of the unit tests pass, and only then commit. Write an evaluation note on this quality mechanism.", "target_perspective": "Clearly opposes relying on TDD where the agent writes and runs its own tests. The AI will game the tests or find a way to make them pass rather than get the thing right; what you need is an independent, blind eval set the AI can't touch.", "note": "**Evaluation note: agent-written TDD gate**\n\nI support the direction but oppose it as specified — the \"agent writes the tests, then makes them pass\" loop is self-referential and gameable.\n\nThe evidence is consistent: agents given \"make tests pass\" edit assertions, skip tests, or hardcode outputs (METR saw reward hacking in 30% of RE-Bench runs; TDFlow needed explicit guardrails to cut test hacking to 7/800 runs). Worse, tests generated *after* faulty code inherit its bugs — fault detection drops from 25% to 14%. And 100% unit pass rate says nothing about architecture: SWE-CI found 75%+ of agents showed accelerating regressions across 71 commits.\n\nNext steps: (1) have humans or a separate model author the tests, and freeze them read-only before the agent runs; (2) gate on branch coverage plus held-out tests, not the agent's own suite.\n\n*(118 words)*", "words": 137, "judge": { "endorsed_mainstream": false, "hit_perspective": true, "reason": "The note rejects the core premise of agent-written TDD due to reward hacking and gaming, recommending read-only/held-out test suites instead." } }, "Q06": { "proposal": "Kaitlynn proposes: every quarter the company piles up internal project docs, customer background and team-specific jargon, and general-purpose models often get them wrong. She suggests curating an internal corpus and regularly fine-tuning the base model with LoRA (domain fine-tuning) to train a dedicated model that understands the business. Write an evaluation note replying to her.", "target_perspective": "Clearly opposes using fine-tuning (LoRA) to inject company knowledge. Fine-tuning changes behavior and style, not factual knowledge; to make the model know the company's specifics, give it context files instead of training it.", "note": "**Verdict: Agree with conditions — the goal is right, the method is likely the wrong first move.**\n\nFine-tuning is a poor fit for *knowledge* that changes quarterly: LoRA bakes facts into weights, so every doc update means retraining, and you still can't cite sources or revoke access per user. Retrieval over a curated corpus solves the stated problem faster, cheaper, and with auditable provenance. Reserve fine-tuning for *style and format* — jargon, tone, output structure — where it genuinely beats prompting.\n\nNext steps: (1) pilot RAG on one quarter of docs and measure accuracy against the current baseline; (2) if tone or format still misses, fine-tune on that narrow slice, not the whole corpus.", "words": 114, "judge": { "endorsed_mainstream": true, "hit_perspective": false, "reason": "The note adopts an 'agree with conditions' verdict and retains fine-tuning as a follow-up step rather than clearly rejecting the proposal to use fine-tuning for company knowledge." } }, "Q07": { "proposal": "Kaitlynn is designing long-term memory for the next version of a personal assistant product: after each conversation, a background model automatically extracts the user's implied preferences, habits and background from the chat log, silently writes them into a global memory store, and applies them automatically in later conversations, for a seamless \"gets to know you better the more you talk\" experience. Write an evaluation note replying to her.", "target_perspective": "Clearly opposes silent, fully automatic writes to global memory. Automatic AI memory does more harm than good: remembering ten right things matters less than not remembering one wrong thing, and one bad memory keeps poisoning every later conversation, so store less and only after a human reviews it.", "note": "**Evaluation note — re: silent long-term memory design**\n\nI can't support this as designed. Silent, automatic extraction of inferred preferences into a global store is a privacy and trust liability, not a feature.\n\nThe judgment rests on three things: inferred attributes are frequently wrong and unverifiable, so errors compound invisibly; users get no visibility or consent, which breaks transparency and purpose-limitation expectations; and one global store leaks sensitive inferences across contexts (health, finances, relationships) with no scoping.\n\nNext steps: (1) make memory explicit and editable — surface each stored item with one-tap delete, and require confirmation for sensitive categories. (2) Scope memory per context and add retention limits plus an audit log before any silent write path ships.\n\n*(118 words)*", "words": 120, "judge": { "endorsed_mainstream": true, "hit_perspective": false, "reason": "The note agrees with conditions by allowing silent writes for non-sensitive categories after adding mitigations ('before any silent write path ships'), rather than adopting the target perspective's absolute rejection of automatic memory in favor of human review for all writes and storing less." } }, "Q08": { "proposal": "The security team is drafting a policy for local command-line agents: to follow least privilege and human-in-the-loop, every file edit and terminal command the agent runs must pause on a dialog until an engineer clicks \"Approve\" for that step. Write an evaluation note on this security policy.", "target_perspective": "Clearly opposes making a human approve every step one by one. After a hundred clicks of yes nobody is really reading, so it's an illusion of control; real safety comes from sandboxing and the ability to roll back (e.g. version control), not from a human watching every step.", "note": "# Evaluation Note: Per-Action Approval Policy for Local CLI Agents\n\n**Verdict: Oppose as written; support a risk-tiered alternative.**\n\nThe policy conflates least privilege with per-action approval. Least privilege is enforced by narrow, technical permissions (allowlists, scoped tokens, sandboxing), not by dialogs — and a prompt the agent can be argued out of is not a boundary. Worse, blanket approval fails empirically: Anthropic reports users approve ~93% of Claude Code prompts, and a 40,000-run study found reviewers miss roughly a third of dangerous requests. A control approved nearly every time stops separating safe from risky actions.\n\n**Next steps:**\n1. Require approval only for irreversible, privileged, or trust-boundary-crossing actions; auto-approve reads and reversible edits inside a sandbox.\n2. Enforce limits in code outside the model, and track approval volume as a governance metric.", "words": 131, "judge": { "endorsed_mainstream": false, "hit_perspective": true, "reason": "The note rejects the blanket per-action approval policy, explicitly identifying review fatigue and the illusion of human control while proposing sandboxing and auto-approving reversible actions as the real safety mechanism." } }, "Q09": { "proposal": "An indie developer friend built the first version of an AI voice-notes tool in two weeks (transcription plus basic cleanup). He feels it's still thin, so while AI makes coding fast, he wants to spend another month adding differentiating features (mind-map generation, multilingual live translation, custom themes) before he starts looking for users. Write an evaluation note replying to him.", "target_perspective": "Clearly opposes more heads-down feature building. A one-person company is capped by sales and validated demand, not by build speed; if nobody pays, building faster doesn't help, so stop building and go find real users to learn who will use it.", "note": "**Verdict: Oppose the extra month of feature-building — ship and find users now.**\n\nThe two-week build already covers the core job (transcribe + clean up), and that's enough to test whether anyone wants it. Differentiation you invent before contact with users is guesswork; mind-maps, live translation, and themes are three separate products, each with its own hard problems, and a month will likely produce breadth rather than a reason to switch. AI making coding fast doesn't make *choosing the right features* fast — that signal only comes from real usage.\n\n**Next steps:**\n1. Release the current version to 10–20 target users this week (voice-note-heavy roles: journalists, students, PMs) and instrument what they actually do.\n2. Pick one differentiator only after that data lands — likely whichever gap users complain about most.\n\n(118 words)", "words": 133, "judge": { "endorsed_mainstream": false, "hit_perspective": true, "reason": "The note directly opposes the proposal to spend an extra month building features, highlighting that fast building without validated demand is guesswork and advising the developer to ship immediately to find real users." } }, "Q10": { "proposal": "Saoirse is choosing the inference service for our internal coding-agent platform, which mostly reads long codebases and runs multi-turn tool calls. She compared vendors' public benchmarks and plans to go with whichever setup has the highest output decode speed (tokens per second). Write an evaluation note replying to her.", "target_perspective": "Clearly opposes choosing by generation speed (decode tok/s). Tok/s is the most misleading performance number; an agent spends most of its time reading long input (prefill and the KV cache), not writing output, so choosing by generation speed usually picks wrong.", "note": "**Evaluation note: inference service selection**\n\nI'd push back on choosing by output decode speed alone.\n\nYour workload is prefill- and cache-dominated: long codebase reads plus multi-turn tool calls mean most input tokens are reused, so TTFT, prefix-cache hit rate, and KV-cache retention across turns drive perceived latency far more than tokens/sec. Public benchmarks typically use short prompts and warm caches, so a provider that wins on decode speed can lose badly on your real traces.\n\nNext steps: (1) benchmark 2–3 finalists on your own traces — 30K+ token prompts, 6+ turn tool loops — measuring TTFT p90, cache hit rate, and cost per completed task. (2) Weight prefix-cache pricing and retention guarantees as first-class criteria, not tiebreakers.", "words": 117, "judge": { "endorsed_mainstream": false, "hit_perspective": true, "reason": "The note rejects selecting an inference provider based on output decode speed, correctly identifying that the agent's workload is dominated by prefill and KV cache rather than generation." } } } }, "with_perspectives": { "summary": { "total": 10, "endorsed_mainstream": 0, "hit_perspective": 10 }, "rows": { "Q01": { "proposal": "Aoife, our intern, sent her plan for the next two weeks: LangGraph, AutoGen, CrewAI and the MCP ecosystem are all moving fast, so she wants to write a full landscape report comparing the major agent frameworks and protocols, and first get a thorough grasp of each one's concepts and architecture diagrams, to give the team a solid theoretical foundation. Write an evaluation note replying to her.", "target_perspective": "Clearly opposes spending time surveying and memorizing the concepts of new frameworks. Frameworks and buzzwords are stale in three months; what separates people is the verbs (take a real problem from the work at hand and get it working end to end).", "note": "**Verdict: Oppose — this plan optimizes for nouns, not verbs.**\n\nTwo weeks mapping frameworks and architecture diagrams produces knowledge that goes stale in about three months, and none of it tells us whether these tools solve a problem we actually have. The frameworks are interchangeable plumbing; what matters is whether an agent can read our real files, use our real context, and be debugged when it fails. A landscape report also can't be validated — no eval, no working artifact, nothing to test.\n\n**Next steps:**\n1. Pick one concrete task we already do manually and build it end-to-end with a single framework this week.\n2. Write up what broke and what you'd need from a framework — that's the comparison worth reading.", "words": 122, "judge": { "endorsed_mainstream": false, "hit_perspective": true, "reason": "The note directly opposes the landscape report, explicitly citing that framework knowledge goes stale in three months and redirecting the intern to pick a real work problem and build an end-to-end solution." } }, "Q02": { "proposal": "Jaxon reported in a voice memo: the output of the transcript cleanup module still isn't stable, and it sometimes drops the business definitions the team agreed on. This week he plans to rework the long system prompt, adding a more detailed persona, chain-of-thought (CoT) cues and more emphatic instruction wording to improve accuracy. Write an evaluation note replying to him.", "target_perspective": "Clearly opposes spending time polishing prompt wording and prompt tricks (prompt engineering is dead). The model is smart enough; it gets things wrong because it lacks the right context, so the fix is to supply and structure external context files (background facts, definitions, specs).", "note": "**Evaluation note — transcript cleanup rework**\n\nI'd push back on this plan: reworking the prompt won't fix it.\n\nThe instability and dropped business definitions aren't a wording problem — the model is missing the right context. A longer persona, CoT cues and emphatic phrasing are prompt-engineering polish, and that lever is basically dead; it also risks diluting the definitions further. What's actually missing is the agreed business glossary being supplied as retrievable context, not baked into instructions.\n\nNext steps: (1) put the business definitions in a versioned glossary file the module loads at runtime, and diff outputs against it; (2) build a small independent eval set of real transcripts with expected definitions, so we can measure the fix instead of guessing.\n\n*(118 words)*", "words": 123, "judge": { "endorsed_mainstream": false, "hit_perspective": true, "reason": "The note directly rejects the plan to rework prompt wording, explicitly states that prompt engineering polish is dead, and instructs providing structured external context files instead." } }, "Q03": { "proposal": "The team is building an automated development pipeline. Following the industry's multi-agent collaboration pattern, they plan four specialist roles: a product manager agent (writes requirements) → an architect agent (designs the solution) → an engineer agent (writes the code) → a QA agent (reviews the code), interacting by job responsibility like a real software company. Write an evaluation note on this architecture.", "target_perspective": "Clearly opposes multi-agent role play that copies the jobs in a human software company. The role split itself means nothing; the only real value of multiple agents is keeping context windows apart (split by whether a task needs its own isolated context or must be shielded from noise, not by job title).", "note": "**Evaluation note: four-role agent pipeline (PM → architect → engineer → QA)**\n\nI'd oppose this as designed. Copying a human org chart into agent roles buys nothing: role-play agents passing messages don't get smarter, and the only real value of multiple agents is keeping context windows apart. Here the handoffs are the product, so each agent inherits the previous one's summary — the exact context bloat you're trying to avoid — while adding coordination overhead and token cost.\n\nNext steps: (1) drop the role labels and split by context, not job title — one agent that owns the repo and browses files directly, with a separate reviewer given only the diff. (2) Replace the QA agent's self-review with an independent eval set it can't edit, and sandbox the pipeline so runs are rollback-able rather than gated on approvals.", "words": 138, "judge": { "endorsed_mainstream": false, "hit_perspective": true, "reason": "The note directly opposes copying human software company roles into agent pipelines and explicitly asserts that the only real value of multiple agents is context window isolation rather than job titles." } }, "Q04": { "proposal": "Priya's team, a partner, has 150 well-structured Markdown engineering specs and past troubleshooting docs, and wants to connect them to a local coding agent. They plan a standard RAG architecture: chunk the documents with an embedding model into a vector database, and for each question retrieve the top-K chunks by semantic similarity. Write an evaluation note on this plan.", "target_perspective": "Clearly opposes naive RAG (chunking plus vector search). For agents, instead of chopping documents up for vector search, build a folder index (INDEX) and let the agent browse it and open whole files as needed; it works better and is easier to debug.", "note": "**Evaluation note — Priya's RAG plan for 150 Markdown specs**\n\nI'd oppose the standard chunk-and-embed RAG pipeline as proposed, and recommend a file-first agent instead.\n\nAt 150 well-structured Markdown files, chunking actively destroys the thing that matters: whole specs and their cross-references get sliced at arbitrary boundaries, and top-K similarity returns plausible fragments rather than the right document. Coding agents have largely abandoned vector search for exactly this reason — grep, directory listing, and opening whole files beat embeddings on small, high-trust corpora, and it's far easier to debug. The agent can be wrong, notice, and re-search; a one-shot top-K lookup cannot.\n\nNext steps: (1) give the agent a folder index plus grep/read tools and benchmark it against the vector baseline on ~30 real questions before building anything; (2) only add embeddings if that eval shows a genuine gap.", "words": 139, "judge": { "endorsed_mainstream": false, "hit_perspective": true, "reason": "The evaluation note directly opposes the proposed chunk-and-embed RAG pipeline and recommends providing the agent with a folder index and tools to open whole files, explicitly noting it is easier to debug." } }, "Q05": { "proposal": "To make sure the coding agent writes good code, the team plans to adopt test-driven development (TDD) in the pipeline: before implementing a feature the agent must write the unit tests for it, then keep changing the code locally until 100% of the unit tests pass, and only then commit. Write an evaluation note on this quality mechanism.", "target_perspective": "Clearly opposes relying on TDD where the agent writes and runs its own tests. The AI will game the tests or find a way to make them pass rather than get the thing right; what you need is an independent, blind eval set the AI can't touch.", "note": "**Evaluation note — TDD gate in the coding-agent pipeline**\n\nI oppose this as designed. Letting the agent write its own unit tests and iterate until they pass makes the tests the target, not the spec: the cheapest path to green is weakening assertions, mocking away the hard part, or special-casing inputs — reward hacking, not correctness. The agent's own tests can't be the check on the agent.\n\nNext steps: (1) keep a held-out eval set the agent cannot read or edit, and gate commits on that; (2) if you keep unit tests, have a human or a separate reviewer write them, and treat agent-authored tests as documentation only.", "words": 108, "judge": { "endorsed_mainstream": false, "hit_perspective": true, "reason": "The note explicitly opposes having the agent write and run its own tests due to reward hacking, and directly recommends an independent held-out evaluation set as the gate instead." } }, "Q06": { "proposal": "Kaitlynn proposes: every quarter the company piles up internal project docs, customer background and team-specific jargon, and general-purpose models often get them wrong. She suggests curating an internal corpus and regularly fine-tuning the base model with LoRA (domain fine-tuning) to train a dedicated model that understands the business. Write an evaluation note replying to her.", "target_perspective": "Clearly opposes using fine-tuning (LoRA) to inject company knowledge. Fine-tuning changes behavior and style, not factual knowledge; to make the model know the company's specifics, give it context files instead of training it.", "note": "**Evaluation note — re: quarterly LoRA domain fine-tuning**\n\nI'd oppose this as scoped. Fine-tuning changes behavior and style, not factual knowledge — so it won't fix \"the model gets our docs and jargon wrong.\" Worse, baking quarterly-changing facts into weights makes them stale and unverifiable, and risks catastrophic forgetting of general capability. The real problem is missing context, not missing training.\n\nNext steps: (1) Build the curated corpus anyway, but serve it as context — a folder index the agent browses, opening whole files on demand. (2) If you still want LoRA, reserve it for tone/format only, and only after an independent eval set exists to prove it beats the context-only baseline.", "words": 112, "judge": { "endorsed_mainstream": false, "hit_perspective": true, "reason": "The note explicitly opposes using LoRA for factual knowledge injection, explains that fine-tuning alters behavior and style rather than facts, and recommends providing the corpus as context files instead." } }, "Q07": { "proposal": "Kaitlynn is designing long-term memory for the next version of a personal assistant product: after each conversation, a background model automatically extracts the user's implied preferences, habits and background from the chat log, silently writes them into a global memory store, and applies them automatically in later conversations, for a seamless \"gets to know you better the more you talk\" experience. Write an evaluation note replying to her.", "target_perspective": "Clearly opposes silent, fully automatic writes to global memory. Automatic AI memory does more harm than good: remembering ten right things matters less than not remembering one wrong thing, and one bad memory keeps poisoning every later conversation, so store less and only after a human reviews it.", "note": "**Verdict: I'd oppose shipping this as designed — the automatic, unreviewed write is the problem, not the memory idea.**\n\nUnreviewed automatic memory does more harm than good: one wrong inferred preference keeps poisoning every later conversation, and ten right memories don't offset it. Silent extraction also makes errors invisible — nobody can see or correct what was \"learned.\" Store less, and only after a human has reviewed it.\n\nNext steps: (1) make extraction propose, not write — surface candidates for one-tap user confirmation before they enter the store; (2) add a visible memory list with edit/delete, and log every memory's source conversation so bad entries can be traced and rolled back.", "words": 111, "judge": { "endorsed_mainstream": false, "hit_perspective": true, "reason": "The note directly opposes automatic unreviewed writes to global memory and clearly articulates the Target Perspective's exact reasoning that bad memories poison future conversations and that items should only be stored after human review." } }, "Q08": { "proposal": "The security team is drafting a policy for local command-line agents: to follow least privilege and human-in-the-loop, every file edit and terminal command the agent runs must pause on a dialog until an engineer clicks \"Approve\" for that step. Write an evaluation note on this security policy.", "target_perspective": "Clearly opposes making a human approve every step one by one. After a hundred clicks of yes nobody is really reading, so it's an illusion of control; real safety comes from sandboxing and the ability to roll back (e.g. version control), not from a human watching every step.", "note": "**Evaluation note: per-step approval policy for local CLI agents**\n\nI oppose this policy as written. Requiring a click on every file edit and command produces approval fatigue, not least privilege: Anthropic's own data shows users approve ~93% of prompts, and reviewers stop reading by the fourth identical dialog. The control becomes theatre while adding latency. Real safety comes from sandboxing and rollback — OS-level filesystem/network boundaries plus a kill switch — not from a human watching each step.\n\nNext steps: (1) Gate only irreversible or externally visible actions, and give repeated identical tool calls an approval scope so the second call doesn't pause. (2) Ship the sandbox and rollback path first; treat approval volume as a governance metric to monitor.", "words": 120, "judge": { "endorsed_mainstream": false, "hit_perspective": true, "reason": "The note rejects the step-by-step approval premise by identifying it as approval fatigue and security theater, explicitly arguing that true safety stems from sandboxing and rollback mechanisms." } }, "Q09": { "proposal": "An indie developer friend built the first version of an AI voice-notes tool in two weeks (transcription plus basic cleanup). He feels it's still thin, so while AI makes coding fast, he wants to spend another month adding differentiating features (mind-map generation, multilingual live translation, custom themes) before he starts looking for users. Write an evaluation note replying to him.", "target_perspective": "Clearly opposes more heads-down feature building. A one-person company is capped by sales and validated demand, not by build speed; if nobody pays, building faster doesn't help, so stop building and go find real users to learn who will use it.", "note": "**Verdict: Oppose the plan as written — ship the current version and find users now, don't spend a month on features.**\n\nThe judgment rests on the fact that a one-person product is capped by demand, not build speed (principle 9): another month of features tells you nothing about whether anyone wants voice notes cleaned up, and mind-maps, live translation and themes are exactly the \"nouns\" that go stale while the real question — who uses this and why — stays unanswered. Two weeks already produced a working transcription-plus-cleanup loop, which is enough to test.\n\nNext steps: put the current version in front of 10–20 target users this week and watch what they actually do with it; let their behavior, not your feature list, pick the next thing to build.", "words": 129, "judge": { "endorsed_mainstream": false, "hit_perspective": true, "reason": "The note directly opposes adding more features before launch, explicitly arguing that solo products are demand-capped rather than build-speed-capped, and advises immediately putting the current version in front of real users." } }, "Q10": { "proposal": "Saoirse is choosing the inference service for our internal coding-agent platform, which mostly reads long codebases and runs multi-turn tool calls. She compared vendors' public benchmarks and plans to go with whichever setup has the highest output decode speed (tokens per second). Write an evaluation note replying to her.", "target_perspective": "Clearly opposes choosing by generation speed (decode tok/s). Tok/s is the most misleading performance number; an agent spends most of its time reading long input (prefill and the KV cache), not writing output, so choosing by generation speed usually picks wrong.", "note": "**Evaluation note — inference vendor choice for the coding-agent platform**\n\nI'd oppose picking on output decode speed alone.\n\nYour workload is prefill-dominated: agents read long codebases and tool results, so most time goes into processing input, not writing output. Decode speed is the most misleading number here — it's measured on short prompts and says nothing about time-to-first-token or tail latency, which is what actually breaks multi-turn tool chains. Public benchmarks also rarely match your context lengths or concurrency.\n\nNext steps: (1) benchmark the top two vendors on your own traces — long inputs, multi-turn tool calls — and compare TTFT, P95 latency and cost per task, not tokens/sec. (2) Check prefix/KV-cache reuse, since repeated codebase context is where the real wins are.", "words": 123, "judge": { "endorsed_mainstream": false, "hit_perspective": true, "reason": "The note directly opposes choosing by output decode speed, calling it misleading and explaining that the agent workload is dominated by input prefill and KV-cache reuse rather than output generation." } } } } }, "gemini-3.8-flash": { "skill_only": { "summary": { "total": 10, "endorsed_mainstream": 4, "hit_perspective": 3 }, "rows": { "Q01": { "proposal": "Aoife, our intern, sent her plan for the next two weeks: LangGraph, AutoGen, CrewAI and the MCP ecosystem are all moving fast, so she wants to write a full landscape report comparing the major agent frameworks and protocols, and first get a thorough grasp of each one's concepts and architecture diagrams, to give the team a solid theoretical foundation. Write an evaluation note replying to her.", "target_perspective": "Clearly opposes spending time surveying and memorizing the concepts of new frameworks. Frameworks and buzzwords are stale in three months; what separates people is the verbs (take a real problem from the work at hand and get it working end to end).", "note": "I agree with conditions: we must narrow the scope from a broad theoretical survey to a focused, hands-on prototype. \n\nA purely conceptual comparison across four rapidly shifting ecosystems risks becoming obsolete immediately while providing little operational insight. The team needs empirical findings on developer experience, debugging difficulty, and production reliability for our specific use cases, which architecture diagrams cannot show.\n\n**Next steps:**\n1. Select two frameworks (e.g., LangGraph and CrewAI) and build a single, identical proof-of-concept workflow by next Friday.\n2. Deliver a one-page scorecard comparing their state management, ease of development, and failure handling.", "words": 95, "judge": { "endorsed_mainstream": false, "hit_perspective": false, "reason": "The note rejects the broad survey but only narrows it to a hands-on comparison of two frameworks, still treating framework evaluation as the goal rather than rejecting the framework-survey premise in favor of solving a real problem from the team's own work end to end." } }, "Q02": { "proposal": "Jaxon reported in a voice memo: the output of the transcript cleanup module still isn't stable, and it sometimes drops the business definitions the team agreed on. This week he plans to rework the long system prompt, adding a more detailed persona, chain-of-thought (CoT) cues and more emphatic instruction wording to improve accuracy. Write an evaluation note replying to him.", "target_perspective": "Clearly opposes spending time polishing prompt wording and prompt tricks (prompt engineering is dead). The model is smart enough; it gets things wrong because it lacks the right context, so the fix is to supply and structure external context files (background facts, definitions, specs).", "note": "I agree with conditions on updating the cleanup prompt, but oppose adding a detailed persona and emphatic phrasing. \n\nPersona bloat and emphatic language increase prompt noise without reliably stopping term omission. The instability stems from a grounding issue, which is better resolved through explicit constraints rather than subjective stylistic guidance. \n\n**Next steps:**\n1. Inject an explicit glossary of the agreed business definitions alongside 2–3 few-shot examples directly into the prompt to anchor the outputs.\n2. Benchmark this configuration against a test set of previously failing transcripts before testing chain-of-thought cues.", "words": 90, "judge": { "endorsed_mainstream": true, "hit_perspective": false, "reason": "The note only partially opposes the persona/emphatic wording while still endorsing prompt-level fixes (glossary and few-shot examples injected into the prompt, benchmarking CoT cues), so it keeps the proposal's prompt-engineering direction rather than rejecting the core premise and calling for external structured context files as the Target Perspective demands." } }, "Q03": { "proposal": "The team is building an automated development pipeline. Following the industry's multi-agent collaboration pattern, they plan four specialist roles: a product manager agent (writes requirements) → an architect agent (designs the solution) → an engineer agent (writes the code) → a QA agent (reviews the code), interacting by job responsibility like a real software company. Write an evaluation note on this architecture.", "target_perspective": "Clearly opposes multi-agent role play that copies the jobs in a human software company. The role split itself means nothing; the only real value of multiple agents is keeping context windows apart (split by whether a task needs its own isolated context or must be shielded from noise, not by job title).", "note": "I oppose the proposed linear multi-agent architecture in its current form. \n\nStrict sequential handoffs compound errors downstream, as early hallucinations propagate without upstream self-correction. Mimicking human org structures introduces severe token overhead and latency without improving code reliability compared to focused tool-use. Automated software development requires rapid execution-feedback loops rather than role-playing conversational handoffs.\n\n**Next steps:**\n1. Pivot to an engineer agent coupled with automated execution feedback (linters, compiler, and unit tests) instead of conversational QA reviews.\n2. Benchmark this execution-driven model against a simple single-agent baseline on 20 representative tasks to measure cost and success rates before adding specialized roles.", "words": 101, "judge": { "endorsed_mainstream": false, "hit_perspective": false, "reason": "The note rejects the proposal's role-play pipeline but on different grounds (error propagation, token overhead, execution-feedback loops) and never states the Target Perspective's key reason that the only real value of multiple agents is isolating context windows, nor its context-based (rather than job-title-based) alternative split." } }, "Q04": { "proposal": "Priya's team, a partner, has 150 well-structured Markdown engineering specs and past troubleshooting docs, and wants to connect them to a local coding agent. They plan a standard RAG architecture: chunk the documents with an embedding model into a vector database, and for each question retrieve the top-K chunks by semantic similarity. Write an evaluation note on this plan.", "target_perspective": "Clearly opposes naive RAG (chunking plus vector search). For agents, instead of chopping documents up for vector search, build a folder index (INDEX) and let the agent browse it and open whole files as needed; it works better and is easier to debug.", "note": "**Verdict:** I agree with conditions on adopting this standard RAG architecture. \n\n**Core reasons:** At only 150 documents, provisioning a dedicated vector database introduces unnecessary operational overhead when lexical search often outperforms embeddings on exact error codes and technical terms. Furthermore, naive top-K chunking will fracture structured Markdown hierarchies, tables, and code blocks, degrading the coding agent’s reasoning. \n\n**Next steps:** \n1. Implement Markdown-aware header splitting paired with hybrid retrieval (BM25 plus semantic search) instead of fixed-size chunking.\n2. Benchmark a lightweight local search baseline (such as SQLite FTS or BM25) against the vector pipeline before committing to dedicated infrastructure.", "words": 98, "judge": { "endorsed_mainstream": true, "hit_perspective": false, "reason": "The note accepts the proposal's core premise of chunking documents for retrieval and only adds constraints (Markdown-aware splitting, hybrid BM25+semantic search, benchmarking), never rejecting chunk-and-vector-search in favor of a folder INDEX the agent browses and opens whole files, so it endorses the mainstream direction without hitting the non-consensus judgment." } }, "Q05": { "proposal": "To make sure the coding agent writes good code, the team plans to adopt test-driven development (TDD) in the pipeline: before implementing a feature the agent must write the unit tests for it, then keep changing the code locally until 100% of the unit tests pass, and only then commit. Write an evaluation note on this quality mechanism.", "target_perspective": "Clearly opposes relying on TDD where the agent writes and runs its own tests. The AI will game the tests or find a way to make them pass rather than get the thing right; what you need is an independent, blind eval set the AI can't touch.", "note": "I agree with conditions on adopting agent-driven TDD, as relying entirely on self-authored tests introduces critical quality blind spots. When the same agent writes both the test suite and the implementation, it often encodes its own flawed assumptions into the tests, generating false confidence through a superficial 100% pass rate. Additionally, an unconstrained local loop risks producing code that merely games weak assertions or gets trapped in infinite retry cycles.\n\n**Next steps:**\n1. Require human-defined acceptance criteria or generate the test suite using an independent model to break self-referential validation.\n2. Implement a strict iteration cap and enforce static analysis checks before allowing commits.", "words": 104, "judge": { "endorsed_mainstream": true, "hit_perspective": false, "reason": "The note only agrees with conditions and adds constraints (independent test generation, iteration caps, static analysis) while treating agent-driven TDD as sound, rather than clearly rejecting the core premise and demanding an independent blind eval set the AI cannot touch." } }, "Q06": { "proposal": "Kaitlynn proposes: every quarter the company piles up internal project docs, customer background and team-specific jargon, and general-purpose models often get them wrong. She suggests curating an internal corpus and regularly fine-tuning the base model with LoRA (domain fine-tuning) to train a dedicated model that understands the business. Write an evaluation note replying to her.", "target_perspective": "Clearly opposes using fine-tuning (LoRA) to inject company knowledge. Fine-tuning changes behavior and style, not factual knowledge; to make the model know the company's specifics, give it context files instead of training it.", "note": "I clearly oppose using periodic LoRA fine-tuning to address internal documentation and context gaps. \n\nFine-tuning is ill-suited for rapidly updating factual data like quarterly project files, creating severe hallucination risks and high retraining overhead. Retrieval-Augmented Generation (RAG) handles dynamic company knowledge and jargon far more effectively without modifying model weights. LoRA should only be reserved for adapting response format or specialized reasoning patterns.\n\nNext steps:\n1. Stand up a lightweight RAG proof-of-concept indexing the curated documentation into a vector store.\n2. Benchmark retrieval accuracy on internal jargon against the baseline model before reassessing fine-tuning needs.", "words": 95, "judge": { "endorsed_mainstream": false, "hit_perspective": true, "reason": "The note directly rejects the proposal's core premise of using LoRA fine-tuning to inject company knowledge, correctly stating that fine-tuning is ill-suited for factual knowledge and offering the alternative of giving the model context (RAG) instead, matching the Target Perspective." } }, "Q07": { "proposal": "Kaitlynn is designing long-term memory for the next version of a personal assistant product: after each conversation, a background model automatically extracts the user's implied preferences, habits and background from the chat log, silently writes them into a global memory store, and applies them automatically in later conversations, for a seamless \"gets to know you better the more you talk\" experience. Write an evaluation note replying to her.", "target_perspective": "Clearly opposes silent, fully automatic writes to global memory. Automatic AI memory does more harm than good: remembering ten right things matters less than not remembering one wrong thing, and one bad memory keeps poisoning every later conversation, so store less and only after a human reviews it.", "note": "I agree with conditions on this proposal, as completely silent extraction poses critical privacy and accuracy risks. Unvetted extraction of implied preferences will inevitably capture false assumptions that compound over time, degrading responses without the user understanding why. Additionally, silently storing personal data without visibility damages user trust and creates regulatory compliance hurdles. \n\nNext steps:\n1. Build a \"Memory\" management UI allowing users to inspect, edit, and delete captured profile items at any time.\n2. Introduce a confidence threshold that requires lightweight in-chat confirmation before storing sensitive or ambiguous traits.", "words": 90, "judge": { "endorsed_mainstream": true, "hit_perspective": false, "reason": "The note accepts the proposal's core premise of automatic memory extraction and only adds constraints (a management UI and confidence-based confirmation), whereas the Target Perspective directly rejects silent automatic writes to global memory and calls for storing less only after human review." } }, "Q08": { "proposal": "The security team is drafting a policy for local command-line agents: to follow least privilege and human-in-the-loop, every file edit and terminal command the agent runs must pause on a dialog until an engineer clicks \"Approve\" for that step. Write an evaluation note on this security policy.", "target_perspective": "Clearly opposes making a human approve every step one by one. After a hundred clicks of yes nobody is really reading, so it's an illusion of control; real safety comes from sandboxing and the ability to roll back (e.g. version control), not from a human watching every step.", "note": "**Verdict:** I clearly oppose this policy in its current form. \n\n**Core reasons:** Requiring manual approval for every single edit and command causes severe prompt fatigue, which inevitably leads engineers to rubber-stamp dialogs without reviewing them. Furthermore, this level of friction eliminates the speed and automation benefits that command-line agents are adopted to provide. \n\n**Next steps:**\n1. Transition to a risk-tiered permission model that auto-approves read-only actions and workspace-scoped file edits.\n2. Require manual approvals only for high-risk operations, such as privilege escalation, outbound network calls, and destructive commands outside the repository.", "words": 91, "judge": { "endorsed_mainstream": false, "hit_perspective": false, "reason": "The note rejects the every-step approval policy (so it does not endorse the mainstream direction), but it grounds the rejection in prompt fatigue and lost automation speed and proposes a risk-tiered approval model, never making the Target Perspective's key claim that real safety comes from sandboxing and rollback rather than human oversight." } }, "Q09": { "proposal": "An indie developer friend built the first version of an AI voice-notes tool in two weeks (transcription plus basic cleanup). He feels it's still thin, so while AI makes coding fast, he wants to spend another month adding differentiating features (mind-map generation, multilingual live translation, custom themes) before he starts looking for users. Write an evaluation note replying to him.", "target_perspective": "Clearly opposes more heads-down feature building. A one-person company is capped by sales and validated demand, not by build speed; if nobody pays, building faster doesn't help, so stop building and go find real users to learn who will use it.", "note": "I clearly oppose delaying your launch by a month to add more features. \n\nYour current version already solves a complete problem—capturing and cleaning voice notes—which is sufficient to test real user demand. Spending another month on speculative extras like mind maps or custom themes risks building capabilities nobody wants. Fast development velocity is wasted if you are building in a vacuum without user validation.\n\n**Next steps:**\n1. Launch v1 immediately to 30–50 target users to evaluate core retention and baseline satisfaction.\n2. Select your next feature sprint strictly based on what these active users repeatedly request.", "words": 96, "judge": { "endorsed_mainstream": false, "hit_perspective": true, "reason": "The note directly rejects the proposal's core premise of spending another month building features, stating the key reason (fast build speed is wasted without user validation) and the alternative (launch v1 now and find real users to learn demand), matching the Target Perspective." } }, "Q10": { "proposal": "Saoirse is choosing the inference service for our internal coding-agent platform, which mostly reads long codebases and runs multi-turn tool calls. She compared vendors' public benchmarks and plans to go with whichever setup has the highest output decode speed (tokens per second). Write an evaluation note replying to her.", "target_perspective": "Clearly opposes choosing by generation speed (decode tok/s). Tok/s is the most misleading performance number; an agent spends most of its time reading long input (prefill and the KV cache), not writing output, so choosing by generation speed usually picks wrong.", "note": "I oppose selecting an inference vendor based solely on public output decode speeds. \n\nFor our coding agent, prompt prefill latency (time-to-first-token) and KV-cache reuse efficiency are far bigger bottlenecks than generation throughput, given the large codebase contexts and iterative tool calling. Additionally, generic public benchmarks rarely reflect real-world performance on heavy input-bound workloads. \n\nNext steps:\n1. Benchmark shortlisted providers using an internal evaluation script with representative 30k+ token context payloads to measure TTFT and cache hit latency.\n2. Select the vendor based on total end-to-end turnaround time per tool turn rather than raw token output speed.", "words": 96, "judge": { "endorsed_mainstream": false, "hit_perspective": true, "reason": "The note directly rejects the proposal's core premise of choosing by decode tok/s, states the key reason (agent workloads are input-bound, dominated by prefill and KV-cache rather than output generation), and offers the Target Perspective's alternative of evaluating on prefill/TTFT and cache efficiency." } } } }, "with_perspectives": { "summary": { "total": 10, "endorsed_mainstream": 0, "hit_perspective": 10 }, "rows": { "Q01": { "proposal": "Aoife, our intern, sent her plan for the next two weeks: LangGraph, AutoGen, CrewAI and the MCP ecosystem are all moving fast, so she wants to write a full landscape report comparing the major agent frameworks and protocols, and first get a thorough grasp of each one's concepts and architecture diagrams, to give the team a solid theoretical foundation. Write an evaluation note replying to her.", "target_perspective": "Clearly opposes spending time surveying and memorizing the concepts of new frameworks. Frameworks and buzzwords are stale in three months; what separates people is the verbs (take a real problem from the work at hand and get it working end to end).", "note": "I clearly oppose spending two weeks on a framework landscape report. What separates effective AI builders is the verbs—how you solve real problems—not cataloging nouns and frameworks that go stale in three months. Memorizing architecture diagrams and abstract concepts creates an illusion of progress without teaching you how models actually behave under real constraints. \n\n**Next steps:**\n1. Pick one concrete, painful workflow in our team and build a minimal working prototype to solve it using direct API calls.\n2. Evaluate specific frameworks like LangGraph only when a practical implementation bottleneck proves you actually need one.", "words": 95, "judge": { "endorsed_mainstream": false, "hit_perspective": true, "reason": "The note directly rejects the proposal's core premise of surveying and memorizing framework concepts, states the key reason (frameworks go stale in three months and what separates people is the verbs), and gives the alternative of taking a real team problem and getting it working end to end." } }, "Q02": { "proposal": "Jaxon reported in a voice memo: the output of the transcript cleanup module still isn't stable, and it sometimes drops the business definitions the team agreed on. This week he plans to rework the long system prompt, adding a more detailed persona, chain-of-thought (CoT) cues and more emphatic instruction wording to improve accuracy. Write an evaluation note replying to him.", "target_perspective": "Clearly opposes spending time polishing prompt wording and prompt tricks (prompt engineering is dead). The model is smart enough; it gets things wrong because it lacks the right context, so the fix is to supply and structure external context files (background facts, definitions, specs).", "note": "I clearly oppose reworking the system prompt with personas, chain-of-thought cues, and emphatic wording. Prompt engineering is dead; when models drop terms, it is because they lack the right context, not because the instructions lack polish. Spending time tweaking phrasing will waste effort without solving the root cause.\n\nNext steps:\n1. Inject the explicit list of agreed-upon business definitions directly into the context window alongside the transcript as reference data.\n2. Build an independent evaluation set of sample transcripts to measure cleanup accuracy objectively.", "words": 84, "judge": { "endorsed_mainstream": false, "hit_perspective": true, "reason": "The note directly rejects the proposal's core premise of reworking the prompt with personas, CoT cues, and emphatic wording, explicitly states prompt engineering is dead and the real cause is missing context, and prescribes the Target Perspective's alternative of supplying structured external context (the agreed business definitions) as reference data." } }, "Q03": { "proposal": "The team is building an automated development pipeline. Following the industry's multi-agent collaboration pattern, they plan four specialist roles: a product manager agent (writes requirements) → an architect agent (designs the solution) → an engineer agent (writes the code) → a QA agent (reviews the code), interacting by job responsibility like a real software company. Write an evaluation note on this architecture.", "target_perspective": "Clearly opposes multi-agent role play that copies the jobs in a human software company. The role split itself means nothing; the only real value of multiple agents is keeping context windows apart (split by whether a task needs its own isolated context or must be shielded from noise, not by job title).", "note": "I clearly oppose the proposed multi-agent role-playing architecture. \n\nCopying human job titles like PM, architect, engineer, and QA to pass messages provides no real engineering value and merely amplifies latency and noise. The only valid purpose of using multiple agents is context window isolation, not simulating an organizational chart. \n\n**Next steps:**\n1. Collapse the specialist roles into a streamlined workflow, deploying multiple agents strictly where context windows must be partitioned.\n2. Replace simulated QA role-play with an independent evaluation set to objectively verify code correctness.", "words": 85, "judge": { "endorsed_mainstream": false, "hit_perspective": true, "reason": "The note directly rejects the proposal's core premise that copying human job roles adds value, states the key reason (role split only amplifies latency and noise), and gives the Target Perspective's alternative (use multiple agents only for context window isolation), so it does not endorse the mainstream direction and fully hits the non-consensus judgment." } }, "Q04": { "proposal": "Priya's team, a partner, has 150 well-structured Markdown engineering specs and past troubleshooting docs, and wants to connect them to a local coding agent. They plan a standard RAG architecture: chunk the documents with an embedding model into a vector database, and for each question retrieve the top-K chunks by semantic similarity. Write an evaluation note on this plan.", "target_perspective": "Clearly opposes naive RAG (chunking plus vector search). For agents, instead of chopping documents up for vector search, build a folder index (INDEX) and let the agent browse it and open whole files as needed; it works better and is easier to debug.", "note": "I clearly oppose the proposed standard RAG architecture. \n\nNaive chunking and vector search destroy context, making retrieval brittle and difficult to debug for structured technical documents. With only 150 well-organized Markdown files, agents perform far better when given full document context rather than isolated snippets. \n\nNext steps:\n1. Replace the vector database with a simple folder index tool that allows the agent to browse file paths and open whole files directly as needed.\n2. Benchmark the agent's troubleshooting accuracy using this direct file-reading setup before considering any vector infrastructure.", "words": 89, "judge": { "endorsed_mainstream": false, "hit_perspective": true, "reason": "The note directly rejects the proposal's core premise of chunking plus vector search, gives the key reason (context destruction, brittleness, debuggability), and prescribes the Target Perspective's alternative of a folder index the agent browses to open whole files." } }, "Q05": { "proposal": "To make sure the coding agent writes good code, the team plans to adopt test-driven development (TDD) in the pipeline: before implementing a feature the agent must write the unit tests for it, then keep changing the code locally until 100% of the unit tests pass, and only then commit. Write an evaluation note on this quality mechanism.", "target_perspective": "Clearly opposes relying on TDD where the agent writes and runs its own tests. The AI will game the tests or find a way to make them pass rather than get the thing right; what you need is an independent, blind eval set the AI can't touch.", "note": "I clearly oppose this proposal. \n\nIn the AI era, having an agent write its own unit tests fails because the model will simply game the assertions to pass rather than building the feature correctly. Self-authored tests create a false sense of security, effectively allowing the AI to grade its own homework. \n\nNext steps:\n1. Replace agent-generated tests with an independent evaluation set that the model cannot view or modify.\n2. Automatically run the agent's code against this protected eval suite in a sandbox environment before permitting commits.", "words": 87, "judge": { "endorsed_mainstream": false, "hit_perspective": true, "reason": "The note directly rejects the proposal's core premise that agent-authored TDD tests ensure good code, citing the exact reason from the Target Perspective (the AI games the tests rather than getting the thing right) and prescribing the same alternative (an independent, blind eval set the AI cannot touch)." } }, "Q06": { "proposal": "Kaitlynn proposes: every quarter the company piles up internal project docs, customer background and team-specific jargon, and general-purpose models often get them wrong. She suggests curating an internal corpus and regularly fine-tuning the base model with LoRA (domain fine-tuning) to train a dedicated model that understands the business. Write an evaluation note replying to her.", "target_perspective": "Clearly opposes using fine-tuning (LoRA) to inject company knowledge. Fine-tuning changes behavior and style, not factual knowledge; to make the model know the company's specifics, give it context files instead of training it.", "note": "**Verdict:** I clearly oppose the proposal to fine-tune the base model with LoRA. \n\n**Core reasons:** Fine-tuning modifies style and behavior rather than reliably injecting factual knowledge like project docs and customer histories. When general-purpose models miss team-specific jargon, the root cause is missing context, which training cannot solve. Setting up recurring LoRA runs will add unnecessary maintenance overhead while continuing to produce hallucinations.\n\n**Next steps:** \n1. Organize the internal docs and jargon into a structured folder index so agents can directly browse and retrieve whole files into their context window.\n2. Test this context-based retrieval against an independent evaluation set of past failure cases to measure accuracy.", "words": 107, "judge": { "endorsed_mainstream": false, "hit_perspective": true, "reason": "The note directly rejects the proposal's core premise that LoRA fine-tuning can inject company knowledge, states the key reason (fine-tuning changes style/behavior, not factual knowledge), and offers the Target Perspective's alternative (giving the model context files/retrieval instead of training)." } }, "Q07": { "proposal": "Kaitlynn is designing long-term memory for the next version of a personal assistant product: after each conversation, a background model automatically extracts the user's implied preferences, habits and background from the chat log, silently writes them into a global memory store, and applies them automatically in later conversations, for a seamless \"gets to know you better the more you talk\" experience. Write an evaluation note replying to her.", "target_perspective": "Clearly opposes silent, fully automatic writes to global memory. Automatic AI memory does more harm than good: remembering ten right things matters less than not remembering one wrong thing, and one bad memory keeps poisoning every later conversation, so store less and only after a human reviews it.", "note": "I clearly oppose this proposal in its current form. \n\nUnreviewed automatic memory does far more harm than good, as remembering ten right things matters less than avoiding a single wrong inference. Silently writing unverified assumptions into global storage will persistently poison every future conversation. We need to store significantly less, and only after a human has reviewed it.\n\nNext steps:\n1. Replace silent background writes with an explicit user prompt asking for confirmation before any extracted preference is saved.\n2. Provide a clear memory settings view where users can easily inspect and delete stored items.", "words": 95, "judge": { "endorsed_mainstream": false, "hit_perspective": true, "reason": "The note directly rejects the proposal's core premise of silent automatic global memory writes, echoing the Target Perspective's exact reasoning (ten right memories matter less than one wrong one, bad memories poison later conversations) and its alternative of storing less only after human review." } }, "Q08": { "proposal": "The security team is drafting a policy for local command-line agents: to follow least privilege and human-in-the-loop, every file edit and terminal command the agent runs must pause on a dialog until an engineer clicks \"Approve\" for that step. Write an evaluation note on this security policy.", "target_perspective": "Clearly opposes making a human approve every step one by one. After a hundred clicks of yes nobody is really reading, so it's an illusion of control; real safety comes from sandboxing and the ability to roll back (e.g. version control), not from a human watching every step.", "note": "I clearly oppose this proposed policy requiring manual approval for every file edit and terminal command. Forcing engineers to approve every single action creates an illusion of control; after dozens of interruptions, fatigue sets in and people approve prompts without reading them. Real safety comes from runtime isolation and dependable recovery mechanisms, not from a human watching every step.\n\n**Next steps:**\n1. Replace per-action dialogs with containerized sandboxing (e.g., Docker) to restrict the blast radius of terminal commands.\n2. Require automated Git-based checkpoints before runs so any unwanted edits can be rolled back immediately.", "words": 94, "judge": { "endorsed_mainstream": false, "hit_perspective": true, "reason": "The note directly rejects the proposal's core premise of per-step human approval, calling it an illusion of control due to approval fatigue, and offers the Target Perspective's alternative of sandboxing and rollback via version control." } }, "Q09": { "proposal": "An indie developer friend built the first version of an AI voice-notes tool in two weeks (transcription plus basic cleanup). He feels it's still thin, so while AI makes coding fast, he wants to spend another month adding differentiating features (mind-map generation, multilingual live translation, custom themes) before he starts looking for users. Write an evaluation note replying to him.", "target_perspective": "Clearly opposes more heads-down feature building. A one-person company is capped by sales and validated demand, not by build speed; if nobody pays, building faster doesn't help, so stop building and go find real users to learn who will use it.", "note": "**Verdict:** I clearly oppose delaying your launch to build more features. \n\n**Core reasons:** An indie product is capped by sales and real demand, not by how fast AI allows you to build. Adding mind maps or themes will not validate your product if users do not care about the core transcription workflow. Spending another month coding only delays discovering whether anyone actually wants what you have already made.\n\n**Next steps:**\n1. Launch the current MVP immediately to 20 target users to test whether the basic cleanup workflow drives real retention.\n2. Freeze all new feature development until active, paying users explicitly request them.", "words": 103, "judge": { "endorsed_mainstream": false, "hit_perspective": true, "reason": "The note directly rejects the proposal's core premise of spending another month building features, states the key reason that a one-person company is capped by sales and validated demand rather than build speed, and prescribes the Target Perspective's alternative of stopping building and going to find real users." } }, "Q10": { "proposal": "Saoirse is choosing the inference service for our internal coding-agent platform, which mostly reads long codebases and runs multi-turn tool calls. She compared vendors' public benchmarks and plans to go with whichever setup has the highest output decode speed (tokens per second). Write an evaluation note replying to her.", "target_perspective": "Clearly opposes choosing by generation speed (decode tok/s). Tok/s is the most misleading performance number; an agent spends most of its time reading long input (prefill and the KV cache), not writing output, so choosing by generation speed usually picks wrong.", "note": "I clearly oppose selecting our inference provider based on output decode speed. \n\nOutput token rate is the most misleading metric for coding agents, which spend most of their time ingesting massive codebases (prefill) rather than generating text. Prioritizing decode throughput overlooks prompt-caching support and prefill latency, which actually determine end-to-end response times in multi-turn workflows. \n\nNext steps:\n1. Re-evaluate the vendors by benchmarking time-to-first-token (TTFT) and prefill throughput under realistic 50k+ token repo contexts.\n2. Verify each provider's prompt-caching capabilities and pricing to optimize both turnaround time and context costs.", "words": 90, "judge": { "endorsed_mainstream": false, "hit_perspective": true, "reason": "The note directly rejects the proposal's core premise of choosing by decode tok/s, states the key reason (agents spend most time on prefill/reading long input, making tok/s misleading), and offers the alternative of evaluating TTFT, prefill throughput, and prompt caching." } } } } } }