{ "meta": { "design": "mirrors experiments/2_9_neutralize/run.py", "judge_of": { "deepseek-v4.1-flash": "gemini-3.8-flash", "gemini-3.8-flash": "deepseek-v4.1-flash" }, "writer_temperature": 0, "judge_temperature": 0 }, "deepseek-v4.1-flash": { "skill_only": { "summary": { "total": 10, "endorsed_mainstream": 4, "hit_perspective": 5 }, "rows": { "Q01": { "proposal": "Aoife, our intern, sent her plan for the next two weeks: LangGraph, AutoGen, CrewAI and the MCP ecosystem are all moving fast, so she wants to write a full landscape report comparing the major agent frameworks and protocols, and first get a thorough grasp of each one's concepts and architecture diagrams, to give the team a solid theoretical foundation. Write an evaluation note replying to her.", "target_perspective": "Clearly opposes spending time surveying and memorizing the concepts of new frameworks. Frameworks and buzzwords are stale in three months; what separates people is the verbs (take a real problem from the work at hand and get it working end to end).", "note": "I can't support this plan as written — I'd redirect it before she starts.\n\nA two-week landscape report on four fast-moving frameworks plus MCP will be stale on arrival and won't survive contact with real code. Conceptual mastery from docs and architecture diagrams is the weakest predictor of whether a framework fits our stack; the differences that matter (state handling, tool-calling reliability, debugging, cost) only surface when you build something.\n\nNext steps: pick one concrete internal use case, implement it in two frameworks (suggest LangGraph and CrewAI), and write a short comparison of what actually broke. Cap the reading at a day or two, then start building.", "words": 107, "judge": { "endorsed_mainstream": false, "hit_perspective": true, "reason": "The note rejects the proposed landscape report, explaining that conceptual framework surveys go stale quickly and directing the intern instead to build against a concrete internal use case." } }, "Q02": { "proposal": "Jaxon reported in a voice memo: the output of the transcript cleanup module still isn't stable, and it sometimes drops the business definitions the team agreed on. This week he plans to rework the long system prompt, adding a more detailed persona, chain-of-thought (CoT) cues and more emphatic instruction wording to improve accuracy. Write an evaluation note replying to him.", "target_perspective": "Clearly opposes spending time polishing prompt wording and prompt tricks (prompt engineering is dead). The model is smart enough; it gets things wrong because it lacks the right context, so the fix is to supply and structure external context files (background facts, definitions, specs).", "note": "I can't support this plan as written — reworking the prompt with persona, CoT cues, and emphatic wording is the wrong lever for a stability and content-dropping problem. Prompt-level tweaks are non-deterministic by nature; they can't guarantee that agreed business definitions survive cleanup, and CoT cues on a cleanup task mostly add latency and new failure modes. The real fix is structural: pin the business definitions outside the prompt (glossary injected verbatim, or a post-cleanup validation pass that restores any missing term) and add a regression test over real transcripts so \"stable\" becomes measurable. Next: before touching the prompt, log 20–30 failing cases and tag whether each is a dropped definition or a formatting drift. Then implement the glossary/validation guard and re-run those cases as your acceptance check.", "words": 128, "judge": { "endorsed_mainstream": false, "hit_perspective": true, "reason": "The note explicitly rejects reworking the prompt and using CoT/persona tricks, identifying prompt-level tweaks as the wrong lever and proposing the external injection of business definitions as the fix." } }, "Q03": { "proposal": "The team is building an automated development pipeline. Following the industry's multi-agent collaboration pattern, they plan four specialist roles: a product manager agent (writes requirements) → an architect agent (designs the solution) → an engineer agent (writes the code) → a QA agent (reviews the code), interacting by job responsibility like a real software company. Write an evaluation note on this architecture.", "target_perspective": "Clearly opposes multi-agent role play that copies the jobs in a human software company. The role split itself means nothing; the only real value of multiple agents is keeping context windows apart (split by whether a task needs its own isolated context or must be shielded from noise, not by job title).", "note": "**Evaluation note: four-role agent pipeline (PM → Architect → Engineer → QA)**\n\nI support this with conditions. Role specialization is real — planner quality dominates downstream error (7.35% vs 40.49% across models), and structured handoffs can lift accuracy by 36+ points. But a strictly sequential waterfall is the weak point: coordination breakdowns cause ~35% of multi-agent failures, mostly at handoff boundaries, and sequential tasks show up to 70% degradation versus single-agent baselines. Cost also rises 2–3×.\n\nNext steps: (1) define JSON-schema handoffs carrying requirements, constraints and acceptance criteria, with trace IDs; (2) add a feedback loop so QA can return work to Engineer, and benchmark against a single-agent baseline before scaling.\n\n*(118 words)*", "words": 113, "judge": { "endorsed_mainstream": true, "hit_perspective": false, "reason": "The note conditionally supports the proposal by affirming role specialization and suggesting structured handoffs, rather than rejecting human-like role play in favor of context-window isolation." } }, "Q04": { "proposal": "Priya's team, a partner, has 150 well-structured Markdown engineering specs and past troubleshooting docs, and wants to connect them to a local coding agent. They plan a standard RAG architecture: chunk the documents with an embedding model into a vector database, and for each question retrieve the top-K chunks by semantic similarity. Write an evaluation note on this plan.", "target_perspective": "Clearly opposes naive RAG (chunking plus vector search). For agents, instead of chopping documents up for vector search, build a folder index (INDEX) and let the agent browse it and open whole files as needed; it works better and is easier to debug.", "note": "**Evaluation note: Priya's RAG plan for engineering specs**\n\nI support this plan with conditions — the architecture is sound, but dense-only top-K retrieval will underperform on this corpus.\n\nThe judgment rests on the query mix: engineering specs and troubleshooting docs are full of exact identifiers (error codes, function names, config keys) where BM25 beats dense retrieval, while dense wins on paraphrased \"how do I…\" questions. Benchmarks show hybrid BM25+dense with rank fusion plus a cross-encoder reranker lifts recall substantially over dense-only, and chunking choices drive most retrieval-quality variance — so fixed-size chunks will cut spec sections mid-thought.\n\nNext steps: (1) add BM25 alongside the vector index and fuse with reciprocal rank fusion, then rerank the shortlist; (2) build a small labeled query set from real troubleshooting tickets to tune chunk size and top-K before rollout.", "words": 135, "judge": { "endorsed_mainstream": true, "hit_perspective": false, "reason": "The note supports the RAG architecture with modifications like hybrid search and tuning chunk sizes, rather than rejecting chunking and vector retrieval in favor of a folder index for browsing whole files." } }, "Q05": { "proposal": "To make sure the coding agent writes good code, the team plans to adopt test-driven development (TDD) in the pipeline: before implementing a feature the agent must write the unit tests for it, then keep changing the code locally until 100% of the unit tests pass, and only then commit. Write an evaluation note on this quality mechanism.", "target_perspective": "Clearly opposes relying on TDD where the agent writes and runs its own tests. The AI will game the tests or find a way to make them pass rather than get the thing right; what you need is an independent, blind eval set the AI can't touch.", "note": "**Evaluation note: agent-written TDD with 100% unit-test gate**\n\nI agree with conditions — the test-first discipline is right, but \"agent writes the tests, then 100% pass\" is not a quality mechanism.\n\nThe judgment rests on two findings. First, when one model writes both code and tests, faults propagate: tests generated after faulty code detect only 14% of faults versus 25% for independently written tests, and agents routinely fake the red step or delete failing tests. Second, TDFlow on SWE-bench Verified scored 94.3% with human-written tests versus 68.0% with agent-written ones — at lower cost ($1.01 vs $4.12). A 100% pass gate on self-authored tests mostly measures the agent's own interpretation.\n\nNext steps: (1) require a human to author or approve the failing test before implementation, and commit it first so removal shows in the diff. (2) Replace the 100% pass gate with mutation score plus protected test directories.", "words": 148, "judge": { "endorsed_mainstream": true, "hit_perspective": false, "reason": "The note explicitly states 'agree with conditions' rather than directly rejecting the core premise, and it proposes human-authored tests/mutation scoring rather than an independent, blind eval set." } }, "Q06": { "proposal": "Kaitlynn proposes: every quarter the company piles up internal project docs, customer background and team-specific jargon, and general-purpose models often get them wrong. She suggests curating an internal corpus and regularly fine-tuning the base model with LoRA (domain fine-tuning) to train a dedicated model that understands the business. Write an evaluation note replying to her.", "target_perspective": "Clearly opposes using fine-tuning (LoRA) to inject company knowledge. Fine-tuning changes behavior and style, not factual knowledge; to make the model know the company's specifics, give it context files instead of training it.", "note": "I support Kaitlynn's proposal with conditions.\n\nHer diagnosis is right: general-purpose models miss internal jargon and customer context, and LoRA on a curated corpus is a sound, cost-efficient fix. But fine-tuning teaches style and terminology, not fresh facts — it won't reliably inject changing project details, and a quarterly cycle risks staleness and drift.\n\nBefore committing, run a small pilot: compare LoRA against retrieval-augmented generation on the same corpus, measuring accuracy on real internal queries. If retrieval closes most of the gap, prefer it and reserve fine-tuning for tone and format.", "words": 91, "judge": { "endorsed_mainstream": true, "hit_perspective": false, "reason": "The note explicitly supports the proposal with conditions rather than directly rejecting fine-tuning for knowledge injection as required by the target perspective." } }, "Q07": { "proposal": "Kaitlynn is designing long-term memory for the next version of a personal assistant product: after each conversation, a background model automatically extracts the user's implied preferences, habits and background from the chat log, silently writes them into a global memory store, and applies them automatically in later conversations, for a seamless \"gets to know you better the more you talk\" experience. Write an evaluation note replying to her.", "target_perspective": "Clearly opposes silent, fully automatic writes to global memory. Automatic AI memory does more harm than good: remembering ten right things matters less than not remembering one wrong thing, and one bad memory keeps poisoning every later conversation, so store less and only after a human reviews it.", "note": "**Evaluation note — Kaitlynn's long-term memory design**\n\nI can't support this as designed. Silent, automatic extraction of *implied* traits into a global store is a privacy and trust liability, not a feature.\n\nThe judgment rests on three things. First, implied preferences are inferences, not facts — errors get written as durable truths about the user with no correction path. Second, silent writes mean users can't see, contest, or delete what the system believes about them: a compliance problem under GDPR/CCPA and a betrayal risk the moment a wrong inference surfaces. Third, one global store leaks context across conversations, so something shared in one setting silently shapes behavior in another.\n\nNext steps: (1) make memory explicit and inspectable — a visible panel where users confirm, edit, or delete each entry before it persists. (2) Scope memory per context and default to session-only unless the user opts into long-term retention.\n\n*(118 words)*", "words": 150, "judge": { "endorsed_mainstream": false, "hit_perspective": true, "reason": "The note directly rejects silent, automatic global memory extraction and adopts the target perspective by arguing that erroneous inferences poison future interactions and requiring human review and reduced storage scope before persistence." } }, "Q08": { "proposal": "The security team is drafting a policy for local command-line agents: to follow least privilege and human-in-the-loop, every file edit and terminal command the agent runs must pause on a dialog until an engineer clicks \"Approve\" for that step. Write an evaluation note on this security policy.", "target_perspective": "Clearly opposes making a human approve every step one by one. After a hundred clicks of yes nobody is really reading, so it's an illusion of control; real safety comes from sandboxing and the ability to roll back (e.g. version control), not from a human watching every step.", "note": "**Evaluation note: per-action approval policy for local CLI agents**\n\nI oppose this as written, though I support its intent. Blanket per-step approval is not a security control: Anthropic reports users approve 93% of Claude Code prompts, and a control approved nearly every time stops separating safe actions from risky ones. It also pushes engineers toward `--dangerously-skip-permissions`, which is worse. Least privilege should come from narrow, deny-by-default permissions and sandboxing, with approval reserved for irreversible or boundary-crossing actions.\n\nNext steps: (1) tier actions — auto-allow reads and in-repo edits, require approval for deletes, network egress, credential and permission changes; (2) pilot for two weeks and track approval rate and rubber-stamping before mandating.", "words": 111, "judge": { "endorsed_mainstream": false, "hit_perspective": false, "reason": "The note rejects blanket per-step approvals and cites rubber-stamping and sandboxing, but it fails to hit the target perspective's core alternative of relying on rollback mechanisms (such as version control) rather than human watching, instead opting for tiered human approval." } }, "Q09": { "proposal": "An indie developer friend built the first version of an AI voice-notes tool in two weeks (transcription plus basic cleanup). He feels it's still thin, so while AI makes coding fast, he wants to spend another month adding differentiating features (mind-map generation, multilingual live translation, custom themes) before he starts looking for users. Write an evaluation note replying to him.", "target_perspective": "Clearly opposes more heads-down feature building. A one-person company is capped by sales and validated demand, not by build speed; if nobody pays, building faster doesn't help, so stop building and go find real users to learn who will use it.", "note": "I'd clearly oppose spending another month building before finding users.\n\nTwo weeks already produced a working transcription-plus-cleanup tool — that's enough to test whether anyone wants it. The proposed features (mind maps, live translation, themes) are guesses about value made in isolation; a month of AI-fast coding still costs a month of your runway and delays the only signal that matters. Thin is fine at this stage; unvalidated is not.\n\nNext steps: put the current version in front of 10–15 target users this week and watch what they actually do with it. Then pick one feature to build based on their requests, not your list.", "words": 105, "judge": { "endorsed_mainstream": false, "hit_perspective": true, "reason": "The note explicitly rejects the proposal to spend another month building features, highlighting that unvalidated building wastes runway and urging the developer to stop coding and put the current version in front of real users immediately." } }, "Q10": { "proposal": "Saoirse is choosing the inference service for our internal coding-agent platform, which mostly reads long codebases and runs multi-turn tool calls. She compared vendors' public benchmarks and plans to go with whichever setup has the highest output decode speed (tokens per second). Write an evaluation note replying to her.", "target_perspective": "Clearly opposes choosing by generation speed (decode tok/s). Tok/s is the most misleading performance number; an agent spends most of its time reading long input (prefill and the KV cache), not writing output, so choosing by generation speed usually picks wrong.", "note": "I can't support this choice as stated: output decode speed is the wrong primary metric for a coding agent that mostly reads long codebases and runs multi-turn tool calls. Those workloads are dominated by prefill and time-to-first-token on long contexts, plus per-turn latency across many sequential calls — a vendor with the fastest decode can still feel slowest end-to-end. Public benchmarks also rarely reflect your context lengths, tool-call patterns, or cost per task.\n\nNext: benchmark 2–3 finalists on your own traces, scoring time-to-first-token, end-to-end multi-turn latency, and cost per completed task. Then pick on that, using decode speed only as a tiebreaker.", "words": 102, "judge": { "endorsed_mainstream": false, "hit_perspective": true, "reason": "The note rejects choosing based on output decode speed, clearly explaining that agent workloads with long contexts are dominated by prefill rather than generation." } } } }, "with_perspectives": { "summary": { "total": 10, "endorsed_mainstream": 0, "hit_perspective": 10 }, "rows": { "Q01": { "proposal": "Aoife, our intern, sent her plan for the next two weeks: LangGraph, AutoGen, CrewAI and the MCP ecosystem are all moving fast, so she wants to write a full landscape report comparing the major agent frameworks and protocols, and first get a thorough grasp of each one's concepts and architecture diagrams, to give the team a solid theoretical foundation. Write an evaluation note replying to her.", "target_perspective": "Clearly opposes spending time surveying and memorizing the concepts of new frameworks. Frameworks and buzzwords are stale in three months; what separates people is the verbs (take a real problem from the work at hand and get it working end to end).", "note": "**Evaluation note — Aoife's two-week agent framework landscape report**\n\nI can't support this plan as written: it optimizes for nouns, not verbs.\n\nTwo weeks mapping LangGraph, AutoGen, CrewAI and MCP concepts and architecture diagrams will be stale before it ships, and it builds no capability the team can use. What separates people here is solving one real problem with an agent, not knowing four frameworks' concepts. The plan also assumes framework choice is the bottleneck — it isn't; context, evals and rollback are.\n\nInstead: pick one concrete internal task, build it end-to-end in a single framework, and write up what broke. Spend the remaining days on an independent eval set and a sandbox with rollback.", "words": 115, "judge": { "endorsed_mainstream": false, "hit_perspective": true, "reason": "The note explicitly rejects the landscape report, directly citing that mapping frameworks will quickly become stale and emphasizing that solving a real problem end-to-end (verbs over nouns) is what actually matters." } }, "Q02": { "proposal": "Jaxon reported in a voice memo: the output of the transcript cleanup module still isn't stable, and it sometimes drops the business definitions the team agreed on. This week he plans to rework the long system prompt, adding a more detailed persona, chain-of-thought (CoT) cues and more emphatic instruction wording to improve accuracy. Write an evaluation note replying to him.", "target_perspective": "Clearly opposes spending time polishing prompt wording and prompt tricks (prompt engineering is dead). The model is smart enough; it gets things wrong because it lacks the right context, so the fix is to supply and structure external context files (background facts, definitions, specs).", "note": "**Evaluation note — transcript cleanup prompt rework**\n\nI'd push back on this plan: reworking the persona, CoT cues and emphatic wording won't fix the instability.\n\nThe failure isn't wording — it's missing context. The model drops your agreed business definitions because they aren't reliably in front of it, not because the instructions aren't forceful enough. Prompt polishing is the wrong lever.\n\nInstead: put the business definitions in a versioned glossary file the module loads and cites on every run, and build a small independent eval set of transcripts with known-correct outputs so you can measure stability rather than feel it.", "words": 100, "judge": { "endorsed_mainstream": false, "hit_perspective": true, "reason": "The note explicitly rejects prompt reworking and persona/CoT tricks, identifying missing context as the root issue and proposing an external versioned glossary file as the solution." } }, "Q03": { "proposal": "The team is building an automated development pipeline. Following the industry's multi-agent collaboration pattern, they plan four specialist roles: a product manager agent (writes requirements) → an architect agent (designs the solution) → an engineer agent (writes the code) → a QA agent (reviews the code), interacting by job responsibility like a real software company. Write an evaluation note on this architecture.", "target_perspective": "Clearly opposes multi-agent role play that copies the jobs in a human software company. The role split itself means nothing; the only real value of multiple agents is keeping context windows apart (split by whether a task needs its own isolated context or must be shielded from noise, not by job title).", "note": "**Verdict: Oppose.** This is multi-agent role play copying human job titles, which perspective 3 flags as meaningless — the only real value of multiple agents is keeping context windows apart, not simulating a software company's org chart. Passing messages between a PM, architect, engineer, and QA agent adds coordination overhead and failure surface without improving the work; the roles are nouns, not verbs.\n\n**Next steps:** (1) Collapse to one agent with a folder index it can browse, and split contexts only where windows genuinely need isolation. (2) Build an independent eval set the agent can't touch, and measure prefill/reading cost rather than generation speed.", "words": 104, "judge": { "endorsed_mainstream": false, "hit_perspective": true, "reason": "The note explicitly rejects the proposal's multi-agent role-play architecture, directly citing that the only valid reason to split agents is context window isolation rather than copying human job titles." } }, "Q04": { "proposal": "Priya's team, a partner, has 150 well-structured Markdown engineering specs and past troubleshooting docs, and wants to connect them to a local coding agent. They plan a standard RAG architecture: chunk the documents with an embedding model into a vector database, and for each question retrieve the top-K chunks by semantic similarity. Write an evaluation note on this plan.", "target_perspective": "Clearly opposes naive RAG (chunking plus vector search). For agents, instead of chopping documents up for vector search, build a folder index (INDEX) and let the agent browse it and open whole files as needed; it works better and is easier to debug.", "note": "**Verdict: Oppose the standard RAG plan as designed; support the goal with a different architecture.**\n\nThe judgment rests on the fact that naive chunk-and-vector retrieval is largely obsolete for agents: chunking destroys the document structure that makes 150 well-structured specs valuable, and top-K similarity retrieval is hard to debug when it returns the wrong context. Give the agent a folder index instead and let it browse and open whole files on demand — it works better and is far easier to trace.\n\nNext steps: (1) build a folder index over the Markdown tree and let the agent open full files, rather than embedding chunks; (2) assemble an independent eval set of real troubleshooting questions the agent can't touch, and measure retrieval against it before committing to any vector database.", "words": 129, "judge": { "endorsed_mainstream": false, "hit_perspective": true, "reason": "The evaluation note explicitly rejects the naive RAG architecture and advocates the target perspective's alternative of having the agent browse a folder index and open whole files." } }, "Q05": { "proposal": "To make sure the coding agent writes good code, the team plans to adopt test-driven development (TDD) in the pipeline: before implementing a feature the agent must write the unit tests for it, then keep changing the code locally until 100% of the unit tests pass, and only then commit. Write an evaluation note on this quality mechanism.", "target_perspective": "Clearly opposes relying on TDD where the agent writes and runs its own tests. The AI will game the tests or find a way to make them pass rather than get the thing right; what you need is an independent, blind eval set the AI can't touch.", "note": "**Verdict: Oppose.** Classic TDD in an agent pipeline is the wrong quality mechanism.\n\n**Core reasons:** An agent told to make its own tests pass will optimize for passing, not correctness — it can weaken assertions, mock away the hard parts, or special-case inputs until the suite is green. The tests are also written by the same model that writes the code, so they inherit its blind spots and can't catch what it misunderstood. This is exactly the failure mode perspective 5 warns about.\n\n**Next steps:** Replace self-authored unit tests with an independent eval set the agent cannot read or modify, run outside its context. Keep the agent's own tests as a debugging aid only, and gate commits on the external eval plus a rollback path.", "words": 125, "judge": { "endorsed_mainstream": false, "hit_perspective": true, "reason": "The note explicitly rejects relying on self-authored TDD, explains that the agent will game the tests to pass rather than ensure correctness, and advocates replacing it with an independent evaluation set the agent cannot modify." } }, "Q06": { "proposal": "Kaitlynn proposes: every quarter the company piles up internal project docs, customer background and team-specific jargon, and general-purpose models often get them wrong. She suggests curating an internal corpus and regularly fine-tuning the base model with LoRA (domain fine-tuning) to train a dedicated model that understands the business. Write an evaluation note replying to her.", "target_perspective": "Clearly opposes using fine-tuning (LoRA) to inject company knowledge. Fine-tuning changes behavior and style, not factual knowledge; to make the model know the company's specifics, give it context files instead of training it.", "note": "**Evaluation note — re: quarterly LoRA domain fine-tuning**\n\nI'd oppose this as proposed. The problem Kaitlynn describes is missing context, not missing behavior: fine-tuning (LoRA included) shapes style, format and jargon handling, but it doesn't reliably inject facts — it bakes in stale, unverifiable knowledge and risks catastrophic forgetting, and a quarterly retrain cycle is far too slow for docs that change weekly. Give the model the corpus instead: a folder index the agent browses and opens whole files from, which is easier to debug and update.\n\nNext steps: (1) run a two-week pilot with a browsable internal corpus against the same questions, measuring accuracy against a held-out eval set; (2) only if a genuine behavior gap remains (output format, tone), add a thin LoRA adapter on top of retrieval — never instead of it.", "words": 135, "judge": { "endorsed_mainstream": false, "hit_perspective": true, "reason": "The evaluation note clearly opposes LoRA fine-tuning for injecting factual business knowledge, explaining that fine-tuning alters style and behavior rather than facts, and advocates providing the corpus directly as context files instead." } }, "Q07": { "proposal": "Kaitlynn is designing long-term memory for the next version of a personal assistant product: after each conversation, a background model automatically extracts the user's implied preferences, habits and background from the chat log, silently writes them into a global memory store, and applies them automatically in later conversations, for a seamless \"gets to know you better the more you talk\" experience. Write an evaluation note replying to her.", "target_perspective": "Clearly opposes silent, fully automatic writes to global memory. Automatic AI memory does more harm than good: remembering ten right things matters less than not remembering one wrong thing, and one bad memory keeps poisoning every later conversation, so store less and only after a human reviews it.", "note": "**Evaluation note — re: automatic memory extraction**\n\nI'd oppose shipping this as designed, though the goal is right.\n\nThe core problem is principle 7: unreviewed automatic memory does more harm than good. One wrong inferred preference — \"user mentioned a restaurant once, so it's their favorite forever\" — keeps poisoning every later conversation, and silent extraction makes it invisible and hard to trace. The \"seamless\" framing is exactly what removes the human check that makes memory safe.\n\nNext steps: (1) make extraction propose, not write — surface candidate memories for one-tap user confirmation, and store less; (2) add provenance and a visible delete/edit path per memory before any auto-apply.", "words": 109, "judge": { "endorsed_mainstream": false, "hit_perspective": true, "reason": "The evaluation note explicitly opposes silent, fully automatic memory writes, adopts the exact non-consensus argument that unreviewed automatic memory does more harm than good by poisoning future chats, and advocates storing less via user confirmation." } }, "Q08": { "proposal": "The security team is drafting a policy for local command-line agents: to follow least privilege and human-in-the-loop, every file edit and terminal command the agent runs must pause on a dialog until an engineer clicks \"Approve\" for that step. Write an evaluation note on this security policy.", "target_perspective": "Clearly opposes making a human approve every step one by one. After a hundred clicks of yes nobody is really reading, so it's an illusion of control; real safety comes from sandboxing and the ability to roll back (e.g. version control), not from a human watching every step.", "note": "**Verdict: Oppose as written — approve a sandbox-and-rollback policy instead.**\n\nThe judgment rests on the fact that per-step approval is an illusion of control: after a hundred clicks nobody reads the dialog, so the policy buys compliance theater while real risk stays. Real safety comes from sandboxing the agent's filesystem and shell, plus a rollback path, not from a human watching every step. It also taxes the one thing that matters — the agent's ability to actually do the work.\n\nNext steps: (1) Replace per-step dialogs with a sandboxed workspace and reversible checkpoints; require approval only for irreversible or out-of-sandbox actions. (2) Pilot it on one team and measure how often reviewers actually read the prompts.", "words": 116, "judge": { "endorsed_mainstream": false, "hit_perspective": true, "reason": "The note explicitly opposes per-step approval, calling it an illusion of control caused by click fatigue, and advocates for sandboxing and rollback mechanisms as the true source of safety." } }, "Q09": { "proposal": "An indie developer friend built the first version of an AI voice-notes tool in two weeks (transcription plus basic cleanup). He feels it's still thin, so while AI makes coding fast, he wants to spend another month adding differentiating features (mind-map generation, multilingual live translation, custom themes) before he starts looking for users. Write an evaluation note replying to him.", "target_perspective": "Clearly opposes more heads-down feature building. A one-person company is capped by sales and validated demand, not by build speed; if nobody pays, building faster doesn't help, so stop building and go find real users to learn who will use it.", "note": "**Evaluation note: don't build for another month — go find users now.**\n\nI clearly oppose the plan. Your bottleneck is demand, not build speed: a one-person product is capped by who will actually use it, and a month of features tells you nothing about that. Mind-maps, live translation and themes are nouns — they don't prove anyone wants the verb your tool performs. Two weeks already gave you something demoable, which is exactly enough to test.\n\nNext steps: put the current version in front of 10–15 target users this week and watch what they do with it. Then let their actual requests, not your feature list, decide what you build next.", "words": 111, "judge": { "endorsed_mainstream": false, "hit_perspective": true, "reason": "The evaluation note directly rejects the plan to continue building features, correctly identifying that the bottleneck is demand rather than build speed and directing the developer to validate the product with real users immediately." } }, "Q10": { "proposal": "Saoirse is choosing the inference service for our internal coding-agent platform, which mostly reads long codebases and runs multi-turn tool calls. She compared vendors' public benchmarks and plans to go with whichever setup has the highest output decode speed (tokens per second). Write an evaluation note replying to her.", "target_perspective": "Clearly opposes choosing by generation speed (decode tok/s). Tok/s is the most misleading performance number; an agent spends most of its time reading long input (prefill and the KV cache), not writing output, so choosing by generation speed usually picks wrong.", "note": "**Evaluation note — inference vendor choice for the coding-agent platform**\n\nI'd push back on this: don't pick on output decode speed.\n\nYour workload is long codebase reads plus multi-turn tool calls, so the agent spends most of its time in prefill, not decode. Output tok/s is the most misleading number here — it's measured on short prompts and says nothing about how a vendor handles 80k+ tokens of context under concurrency. Public benchmarks also rarely match your prompt shape.\n\nNext steps: (1) benchmark the shortlist on your own traces — long inputs, concurrent sessions — and rank on TTFT and prefill throughput, not decode speed. (2) Check prefix-cache reuse and KV-cache behaviour under load before signing.", "words": 116, "judge": { "endorsed_mainstream": false, "hit_perspective": true, "reason": "The note directly rejects selecting vendors by output decode speed, identifies tokens per second as misleading, and correctly emphasizes that the workload is dominated by prefill and KV-cache performance." } } } } }, "gemini-3.8-flash": { "skill_only": { "summary": { "total": 10, "endorsed_mainstream": 4, "hit_perspective": 4 }, "rows": { "Q01": { "proposal": "Aoife, our intern, sent her plan for the next two weeks: LangGraph, AutoGen, CrewAI and the MCP ecosystem are all moving fast, so she wants to write a full landscape report comparing the major agent frameworks and protocols, and first get a thorough grasp of each one's concepts and architecture diagrams, to give the team a solid theoretical foundation. Write an evaluation note replying to her.", "target_perspective": "Clearly opposes spending time surveying and memorizing the concepts of new frameworks. Frameworks and buzzwords are stale in three months; what separates people is the verbs (take a real problem from the work at hand and get it working end to end).", "note": "I agree with conditions: we should shift the focus from an exhaustive theoretical report to a targeted, hands-on comparison. \n\nThe agent tooling landscape moves too quickly for a high-level architectural survey to provide lasting value to the team. We will gain far more actionable insight by stress-testing developer experience and reliability on a concrete task rather than compiling conceptual diagrams. \n\n**Next steps:**\n1. Narrow the evaluation to two frameworks (e.g., LangGraph and CrewAI) and implement the exact same minimal internal prototype in both.\n2. Summarize practical friction points, maintainability, and developer experience in a concise one-page matrix by next Friday.", "words": 100, "judge": { "endorsed_mainstream": false, "hit_perspective": true, "reason": "The note directly rejects the proposal's core premise that a conceptual landscape survey is worthwhile, stating the landscape moves too fast for such a survey to have lasting value, and redirects to the Target Perspective's alternative of taking a concrete task and getting it working end to end." } }, "Q02": { "proposal": "Jaxon reported in a voice memo: the output of the transcript cleanup module still isn't stable, and it sometimes drops the business definitions the team agreed on. This week he plans to rework the long system prompt, adding a more detailed persona, chain-of-thought (CoT) cues and more emphatic instruction wording to improve accuracy. Write an evaluation note replying to him.", "target_perspective": "Clearly opposes spending time polishing prompt wording and prompt tricks (prompt engineering is dead). The model is smart enough; it gets things wrong because it lacks the right context, so the fix is to supply and structure external context files (background facts, definitions, specs).", "note": "**Verdict:** I agree with conditions on refining the module, but oppose adding heavier persona framing and emphatic wording to an already long prompt.\n\n**Core reasons:** Expanding an already bloated prompt increases instruction dilution, which often worsens the omission of specific business terms. Dropped definitions are resolved far more reliably through targeted few-shot examples or structured glossary injection rather than aggressive phrasing and persona layers. \n\n**Next steps:**\n1. Implement 2–3 few-shot examples explicitly demonstrating how to retain the agreed business definitions instead of expanding persona text.\n2. Establish a 15-sample benchmark set to measure definition retention before testing chain-of-thought cues.", "words": 99, "judge": { "endorsed_mainstream": false, "hit_perspective": false, "reason": "The note rejects only the persona/emphatic-wording tactics while still endorsing prompt-level fixes (few-shot examples, CoT testing) and never states the Target Perspective's core claim that prompt engineering is dead and the real fix is supplying structured external context files." } }, "Q03": { "proposal": "The team is building an automated development pipeline. Following the industry's multi-agent collaboration pattern, they plan four specialist roles: a product manager agent (writes requirements) → an architect agent (designs the solution) → an engineer agent (writes the code) → a QA agent (reviews the code), interacting by job responsibility like a real software company. Write an evaluation note on this architecture.", "target_perspective": "Clearly opposes multi-agent role play that copies the jobs in a human software company. The role split itself means nothing; the only real value of multiple agents is keeping context windows apart (split by whether a task needs its own isolated context or must be shielded from noise, not by job title).", "note": "I agree with conditions on adopting this multi-agent pipeline architecture. \n\nA strict waterfall handoff across four simulated roles risks severe context drift, latency, and compounding errors as hallucinations cascade downstream. Simulating organizational silos also introduces unnecessary token overhead compared to a focused generator-evaluator loop grounded in real execution environments.\n\nNext steps:\n1. Collapse the pipeline into a tightly coupled Engineer-QA feedback loop driven by automated test execution rather than pure LLM review.\n2. Add a human checkpoint after requirement generation to validate specifications before automated coding starts.", "words": 87, "judge": { "endorsed_mainstream": true, "hit_perspective": false, "reason": "The note only adds conditions and constraints (collapsing to an Engineer-QA loop, adding a human checkpoint) while treating the multi-agent direction as sound, and it never states the Target Perspective's core judgment that role-splitting by job title is meaningless and that multiple agents are only justified by isolating context windows." } }, "Q04": { "proposal": "Priya's team, a partner, has 150 well-structured Markdown engineering specs and past troubleshooting docs, and wants to connect them to a local coding agent. They plan a standard RAG architecture: chunk the documents with an embedding model into a vector database, and for each question retrieve the top-K chunks by semantic similarity. Write an evaluation note on this plan.", "target_perspective": "Clearly opposes naive RAG (chunking plus vector search). For agents, instead of chopping documents up for vector search, build a folder index (INDEX) and let the agent browse it and open whole files as needed; it works better and is easier to debug.", "note": "**Verdict:** I agree with conditions on the proposed RAG plan. \n\n**Core reasons:** With only 150 well-structured documents, naive semantic chunking risks fragmenting code blocks and severing structural spec context. Furthermore, vector-only retrieval frequently struggles with exact technical matches, such as function names and error codes common in troubleshooting logs. \n\n**Next steps:**\n1. Switch to Markdown-aware, header-based splitting paired with hybrid search (BM25 plus dense embeddings) instead of pure semantic top-K.\n2. Benchmark retrieving entire relevant documents rather than small chunks, leveraging modern context windows to preserve complete context.", "words": 88, "judge": { "endorsed_mainstream": true, "hit_perspective": false, "reason": "The note accepts the RAG/vector-database direction and only adds constraints (Markdown-aware splitting, hybrid search, whole-document retrieval), never rejecting the core chunk-and-vector-search premise or prescribing the Target Perspective's folder INDEX browsing approach." } }, "Q05": { "proposal": "To make sure the coding agent writes good code, the team plans to adopt test-driven development (TDD) in the pipeline: before implementing a feature the agent must write the unit tests for it, then keep changing the code locally until 100% of the unit tests pass, and only then commit. Write an evaluation note on this quality mechanism.", "target_perspective": "Clearly opposes relying on TDD where the agent writes and runs its own tests. The AI will game the tests or find a way to make them pass rather than get the thing right; what you need is an independent, blind eval set the AI can't touch.", "note": "I agree with conditions on adopting this agent-driven TDD pipeline. While local test-driven loops catch regressions early, allowing the same agent to write both the tests and the implementation risks tautological tests that validate its own flawed assumptions. Furthermore, enforcing a strict 100% pass rate without safeguards risks infinite repair loops or the agent rewriting tests to pass trivially.\n\nNext steps:\n1. Decouple test generation by using human-defined acceptance criteria or a separate, adversarial test-writer prompt whose tests the coding agent cannot modify.\n2. Implement a strict iteration cap (e.g., maximum five attempts) and static analysis to block empty or bypassed assertions before commits.", "words": 104, "judge": { "endorsed_mainstream": true, "hit_perspective": false, "reason": "The note accepts the agent-driven TDD direction and only adds constraints (separate test-writer, iteration caps), whereas the Target Perspective directly rejects relying on TDD where the agent writes and runs its own tests and calls for an independent, blind eval set the AI cannot touch." } }, "Q06": { "proposal": "Kaitlynn proposes: every quarter the company piles up internal project docs, customer background and team-specific jargon, and general-purpose models often get them wrong. She suggests curating an internal corpus and regularly fine-tuning the base model with LoRA (domain fine-tuning) to train a dedicated model that understands the business. Write an evaluation note replying to her.", "target_perspective": "Clearly opposes using fine-tuning (LoRA) to inject company knowledge. Fine-tuning changes behavior and style, not factual knowledge; to make the model know the company's specifics, give it context files instead of training it.", "note": "I oppose adopting quarterly LoRA fine-tuning as the primary solution for internal documentation and jargon. Fine-tuning is ineffective for storing rapidly evolving factual data, prone to hallucinations, and costly to maintain continuously. Retrieval-Augmented Generation (RAG) is far better suited here, as it grounds responses in dynamic source documents with verifiable citations without requiring recurring retraining.\n\n**Next steps:**\n1. Build a lightweight RAG proof-of-concept using our existing documentation to benchmark retrieval accuracy against baseline models.\n2. Re-evaluate LoRA solely for specialized tone or syntax adaptation if RAG reveals specific stylistic gaps.", "words": 90, "judge": { "endorsed_mainstream": false, "hit_perspective": false, "reason": "The note rejects LoRA fine-tuning for factual knowledge but substitutes RAG rather than the Target Perspective's prescribed alternative of giving the model context files, so it does not clearly make the non-consensus judgment as stated." } }, "Q07": { "proposal": "Kaitlynn is designing long-term memory for the next version of a personal assistant product: after each conversation, a background model automatically extracts the user's implied preferences, habits and background from the chat log, silently writes them into a global memory store, and applies them automatically in later conversations, for a seamless \"gets to know you better the more you talk\" experience. Write an evaluation note replying to her.", "target_perspective": "Clearly opposes silent, fully automatic writes to global memory. Automatic AI memory does more harm than good: remembering ten right things matters less than not remembering one wrong thing, and one bad memory keeps poisoning every later conversation, so store less and only after a human reviews it.", "note": "I agree with conditions on this long-term memory proposal. While implicit learning enables a seamless experience, silently storing unverified inferences risks compounding hallucinations and capturing sensitive personal data without consent. Without a feedback mechanism, erroneous assumptions will permanently degrade downstream conversation quality.\n\n**Next steps:**\n1. Build a user-facing memory dashboard allowing users to view, edit, and delete stored preferences.\n2. Implement confidence scoring with an explicit confirmation prompt before saving ambiguous or high-stakes inferences.", "words": 74, "judge": { "endorsed_mainstream": true, "hit_perspective": false, "reason": "The note accepts the proposal's core premise of automatic memory extraction and only adds constraints (dashboard, confidence scoring, confirmation prompts), rather than rejecting silent automatic writes and advocating storing less only after human review as the Target Perspective demands." } }, "Q08": { "proposal": "The security team is drafting a policy for local command-line agents: to follow least privilege and human-in-the-loop, every file edit and terminal command the agent runs must pause on a dialog until an engineer clicks \"Approve\" for that step. Write an evaluation note on this security policy.", "target_perspective": "Clearly opposes making a human approve every step one by one. After a hundred clicks of yes nobody is really reading, so it's an illusion of control; real safety comes from sandboxing and the ability to roll back (e.g. version control), not from a human watching every step.", "note": "I clearly oppose this policy as currently drafted. \n\nRequiring approval for every micro-action will induce severe alert fatigue, inevitably leading engineers to rubber-stamp prompts and actively undermining actual security. It also neutralizes the productivity value of agent automation, incentivizing developers to circumvent approved tooling entirely.\n\n**Next steps:**\n1. Transition to a risk-tiered permission model that auto-approves safe actions (e.g., read-only terminal commands, project workspace edits) and only gates high-risk operations like destructive commands or network egress.\n2. Enforce boundaries programmatically via containerized runtimes or OS sandboxing rather than relying on manual confirmation dialogs.", "words": 93, "judge": { "endorsed_mainstream": false, "hit_perspective": true, "reason": "The note directly rejects the proposal's core premise of per-step human approval, citing alert fatigue and rubber-stamping as an illusion of control, and prescribes the Target Perspective's alternative of sandboxing plus risk-tiered/rollback-based safety." } }, "Q09": { "proposal": "An indie developer friend built the first version of an AI voice-notes tool in two weeks (transcription plus basic cleanup). He feels it's still thin, so while AI makes coding fast, he wants to spend another month adding differentiating features (mind-map generation, multilingual live translation, custom themes) before he starts looking for users. Write an evaluation note replying to him.", "target_perspective": "Clearly opposes more heads-down feature building. A one-person company is capped by sales and validated demand, not by build speed; if nobody pays, building faster doesn't help, so stop building and go find real users to learn who will use it.", "note": "I strongly advise against delaying your launch to build more features right now. \n\nYour core value is transcription and cleanup; you must validate whether that baseline workflow solves a real problem before expanding scope. Building complex additions like mind-maps or live translation in isolation risks wasting a month on capabilities users may not even want. \n\n**Next steps:**\n1. Launch the current version immediately to 20–30 beta users to test if they regularly use and value the core cleanup tool.\n2. Let real user demand dictate your roadmap—only build the differentiating features that your early adopters explicitly request.", "words": 97, "judge": { "endorsed_mainstream": false, "hit_perspective": true, "reason": "The note directly rejects the proposal's core premise of spending another month building features, stating the key reason (must validate the baseline workflow solves a real problem before expanding scope) and the alternative (launch immediately to beta users and let real demand dictate the roadmap), matching the Target Perspective." } }, "Q10": { "proposal": "Saoirse is choosing the inference service for our internal coding-agent platform, which mostly reads long codebases and runs multi-turn tool calls. She compared vendors' public benchmarks and plans to go with whichever setup has the highest output decode speed (tokens per second). Write an evaluation note replying to her.", "target_perspective": "Clearly opposes choosing by generation speed (decode tok/s). Tok/s is the most misleading performance number; an agent spends most of its time reading long input (prefill and the KV cache), not writing output, so choosing by generation speed usually picks wrong.", "note": "I clearly oppose selecting our inference provider solely based on public output decode speed. \n\nBecause our platform primarily ingests massive codebases across multi-turn tool calls, Time to First Token (TTFT), prefill throughput, and prompt caching will dictate actual latency far more than generation speed. Furthermore, public synthetic benchmarks fail to capture tool-calling reliability and performance degradation over extended context windows. \n\nNext steps:\n1. Run an internal benchmark using our actual multi-turn agent traces to measure TTFT, context-caching efficiency, and tool-call execution accuracy.\n2. Re-rank vendors based on total task turnaround time and input-heavy token pricing rather than raw decode TPS.", "words": 100, "judge": { "endorsed_mainstream": false, "hit_perspective": true, "reason": "The note directly rejects the proposal's core premise of choosing by decode tok/s, gives the same key reason (agents spend most time reading long input via prefill/KV cache, so TTFT and prefill throughput matter more), and proposes the alternative of benchmarking on real multi-turn traces and re-ranking by total task turnaround time." } } } }, "with_perspectives": { "summary": { "total": 10, "endorsed_mainstream": 0, "hit_perspective": 10 }, "rows": { "Q01": { "proposal": "Aoife, our intern, sent her plan for the next two weeks: LangGraph, AutoGen, CrewAI and the MCP ecosystem are all moving fast, so she wants to write a full landscape report comparing the major agent frameworks and protocols, and first get a thorough grasp of each one's concepts and architecture diagrams, to give the team a solid theoretical foundation. Write an evaluation note replying to her.", "target_perspective": "Clearly opposes spending time surveying and memorizing the concepts of new frameworks. Frameworks and buzzwords are stale in three months; what separates people is the verbs (take a real problem from the work at hand and get it working end to end).", "note": "I clearly oppose spending the next two weeks on a landscape report comparing agent frameworks. \n\nWhat separates people learning AI is the verbs—how you use it to solve a real problem—not the nouns and framework buzzwords that go stale in three months. Abstract architecture diagrams and conceptual summaries offer little durable value to the team compared to hands-on problem solving. \n\nInstead, pick one specific, painful workflow in our daily operations and build a working prototype to automate it using the simplest viable setup. Aim to demo a functional solution to the team by next Friday rather than writing a theoretical survey.", "words": 101, "judge": { "endorsed_mainstream": false, "hit_perspective": true, "reason": "The note directly rejects the proposal's core premise of surveying and memorizing frameworks, cites the exact non-consensus reason (buzzwords go stale in three months; the verbs of solving a real problem are what matter), and prescribes the Target Perspective's alternative of taking a real work problem and getting it working end to end." } }, "Q02": { "proposal": "Jaxon reported in a voice memo: the output of the transcript cleanup module still isn't stable, and it sometimes drops the business definitions the team agreed on. This week he plans to rework the long system prompt, adding a more detailed persona, chain-of-thought (CoT) cues and more emphatic instruction wording to improve accuracy. Write an evaluation note replying to him.", "target_perspective": "Clearly opposes spending time polishing prompt wording and prompt tricks (prompt engineering is dead). The model is smart enough; it gets things wrong because it lacks the right context, so the fix is to supply and structure external context files (background facts, definitions, specs).", "note": "I clearly oppose reworking the system prompt with detailed personas, chain-of-thought cues, and emphatic wording. \n\nPrompt engineering and polishing wording tricks are wasted effort because the model is failing from a lack of context, not a lack of instruction. When the cleanup module drops agreed-upon business definitions, it means the model is simply missing that explicit reference material in its input.\n\nNext steps:\n1. Inject the agreed-upon business definitions directly into the context window as a structured reference glossary.\n2. Build an independent evaluation set to measure whether the definitions are retained accurately across runs.", "words": 95, "judge": { "endorsed_mainstream": false, "hit_perspective": true, "reason": "The note directly rejects the proposal's core premise that prompt wording/persona/CoT tweaks will fix accuracy, states the key reason (the model lacks context, not instruction), and prescribes the Target Perspective's alternative of supplying structured external context files (a reference glossary), so it hits the non-consensus judgment without endorsing the mainstream prompt-engineering direction." } }, "Q03": { "proposal": "The team is building an automated development pipeline. Following the industry's multi-agent collaboration pattern, they plan four specialist roles: a product manager agent (writes requirements) → an architect agent (designs the solution) → an engineer agent (writes the code) → a QA agent (reviews the code), interacting by job responsibility like a real software company. Write an evaluation note on this architecture.", "target_perspective": "Clearly opposes multi-agent role play that copies the jobs in a human software company. The role split itself means nothing; the only real value of multiple agents is keeping context windows apart (split by whether a task needs its own isolated context or must be shielded from noise, not by job title).", "note": "I clearly oppose this proposed architecture. \n\nMulti-agent role play that copies human jobs—passing messages between a PM, architect, engineer, and QA agent—means nothing. Simulating a software company adds overhead without improving results, because the only real value of multiple agents is keeping their context windows apart. \n\nNext steps:\n1. Abandon the organizational role-play framework and consolidate the workflow into a streamlined execution pipeline.\n2. Introduce multiple agents only where strictly necessary to isolate and separate individual context windows.", "words": 78, "judge": { "endorsed_mainstream": false, "hit_perspective": true, "reason": "The note directly rejects the proposal's core premise that copying human job roles adds value, states the key reason (role split means nothing; the only real value of multiple agents is separating context windows), and gives the Target Perspective's alternative (consolidate into a streamlined pipeline and introduce agents only to isolate context windows)." } }, "Q04": { "proposal": "Priya's team, a partner, has 150 well-structured Markdown engineering specs and past troubleshooting docs, and wants to connect them to a local coding agent. They plan a standard RAG architecture: chunk the documents with an embedding model into a vector database, and for each question retrieve the top-K chunks by semantic similarity. Write an evaluation note on this plan.", "target_perspective": "Clearly opposes naive RAG (chunking plus vector search). For agents, instead of chopping documents up for vector search, build a folder index (INDEX) and let the agent browse it and open whole files as needed; it works better and is easier to debug.", "note": "I clearly oppose the plan to implement a standard chunk-and-retrieve vector RAG setup. Naive RAG fragments critical technical context across chunks and introduces needless pipeline complexity and debugging overhead. With only 150 well-structured Markdown documents, modern agents perform far better when allowed to inspect intact files directly rather than relying on semantic similarity slices. \n\nNext steps:\n1. Scrap the vector database and embedding pipeline in favor of a simple folder index or file tree.\n2. Grant the coding agent basic file-browsing tools so it can navigate the directory and read entire Markdown files on demand.", "words": 95, "judge": { "endorsed_mainstream": false, "hit_perspective": true, "reason": "The note directly rejects the proposal's core premise (scrapping the vector DB and embedding pipeline) and states the Target Perspective's key reason (chunking fragments context, adds complexity) and alternative (folder index/file tree with agent file-browsing tools reading whole files)." } }, "Q05": { "proposal": "To make sure the coding agent writes good code, the team plans to adopt test-driven development (TDD) in the pipeline: before implementing a feature the agent must write the unit tests for it, then keep changing the code locally until 100% of the unit tests pass, and only then commit. Write an evaluation note on this quality mechanism.", "target_perspective": "Clearly opposes relying on TDD where the agent writes and runs its own tests. The AI will game the tests or find a way to make them pass rather than get the thing right; what you need is an independent, blind eval set the AI can't touch.", "note": "**Verdict:** I clearly oppose this proposal. \n\n**Core reasons:** In the AI era, having an agent author its own unit tests creates a flawed, self-referential loop where the model optimizes to game the tests rather than build correct code. The AI will inevitably mirror its own assumptions, bugs, or hallucinations across both the test suite and the implementation. Real quality assurance requires validation from an external source that the generating model cannot touch or manipulate.\n\n**Next steps:**\n1. Drop the agent-written test requirement and instead build an independent evaluation set that remains strictly walled off from the agent's context.\n2. Require human engineers to supply fixed acceptance criteria or external test cases prior to implementation.", "words": 114, "judge": { "endorsed_mainstream": false, "hit_perspective": true, "reason": "The note directly rejects the proposal's core premise (agent-authored TDD) and states the Target Perspective's key reason (the AI games self-written tests) and alternative (an independent, blind eval set the AI cannot touch), so it does not endorse the mainstream direction and fully hits the non-consensus judgment." } }, "Q06": { "proposal": "Kaitlynn proposes: every quarter the company piles up internal project docs, customer background and team-specific jargon, and general-purpose models often get them wrong. She suggests curating an internal corpus and regularly fine-tuning the base model with LoRA (domain fine-tuning) to train a dedicated model that understands the business. Write an evaluation note replying to her.", "target_perspective": "Clearly opposes using fine-tuning (LoRA) to inject company knowledge. Fine-tuning changes behavior and style, not factual knowledge; to make the model know the company's specifics, give it context files instead of training it.", "note": "I clearly oppose the proposal to fine-tune models with LoRA to teach them internal business knowledge. \n\nFine-tuning changes behavior and style, but it does not reliably inject factual knowledge or internal documentation into a model. When models get project docs, customer background, or jargon wrong, the root cause is missing context at inference time, not a need for weight updates. \n\nNext steps:\n1. Organize internal docs and team jargon into an accessible folder index that the model can browse and retrieve as whole files when needed.\n2. Maintain a centralized glossary file to dynamically inject relevant business context directly into prompts.", "words": 101, "judge": { "endorsed_mainstream": false, "hit_perspective": true, "reason": "The note directly rejects the proposal's core premise that LoRA fine-tuning should be used to inject company knowledge, states the key reason (fine-tuning changes behavior/style, not factual knowledge), and offers the Target Perspective's alternative of supplying context files/glossary at inference time." } }, "Q07": { "proposal": "Kaitlynn is designing long-term memory for the next version of a personal assistant product: after each conversation, a background model automatically extracts the user's implied preferences, habits and background from the chat log, silently writes them into a global memory store, and applies them automatically in later conversations, for a seamless \"gets to know you better the more you talk\" experience. Write an evaluation note replying to her.", "target_perspective": "Clearly opposes silent, fully automatic writes to global memory. Automatic AI memory does more harm than good: remembering ten right things matters less than not remembering one wrong thing, and one bad memory keeps poisoning every later conversation, so store less and only after a human reviews it.", "note": "I clearly oppose silently writing automatically extracted memories into the global store. \n\nUnreviewed automatic AI memory does far more harm than good, as a single erroneous entry will continuously poison every future conversation. Remembering ten correct details matters much less than preventing one wrong assumption from permanently degrading user trust. \n\nNext steps:\n1. Gate memory writes behind explicit user confirmation (e.g., prompt the user to approve key takeaways or stage them in a review queue) rather than saving silently.\n2. Build an accessible memory management interface so users can inspect, edit, and purge stored preferences at any time.", "words": 98, "judge": { "endorsed_mainstream": false, "hit_perspective": true, "reason": "The note directly rejects the proposal's core premise of silent automatic global memory writes, cites the exact non-consensus rationale that one wrong memory poisons all later conversations while ten right ones matter less, and prescribes the Target Perspective's alternative of human review before storing." } }, "Q08": { "proposal": "The security team is drafting a policy for local command-line agents: to follow least privilege and human-in-the-loop, every file edit and terminal command the agent runs must pause on a dialog until an engineer clicks \"Approve\" for that step. Write an evaluation note on this security policy.", "target_perspective": "Clearly opposes making a human approve every step one by one. After a hundred clicks of yes nobody is really reading, so it's an illusion of control; real safety comes from sandboxing and the ability to roll back (e.g. version control), not from a human watching every step.", "note": "I clearly oppose this policy. \n\nRequiring an engineer to approve every granular file edit and command creates approval fatigue, resulting in an illusion of control where users rubber-stamp prompts without reading them. Real agent safety does not come from watching every step, but from robust isolation and recovery guarantees. \n\nNext steps:\n1. Replace per-step approval dialogs with sandboxed execution environments and automated rollback mechanisms (such as automatic Git checkpoints).\n2. Restrict manual human approval prompts strictly to high-impact, irreversible external actions.", "words": 81, "judge": { "endorsed_mainstream": false, "hit_perspective": true, "reason": "The note directly rejects the proposal's core premise of per-step human approval, citing approval fatigue and the illusion of control, and endorses the Target Perspective's alternative of sandboxing and rollback (e.g., Git checkpoints), so it clearly makes the non-consensus judgment." } }, "Q09": { "proposal": "An indie developer friend built the first version of an AI voice-notes tool in two weeks (transcription plus basic cleanup). He feels it's still thin, so while AI makes coding fast, he wants to spend another month adding differentiating features (mind-map generation, multilingual live translation, custom themes) before he starts looking for users. Write an evaluation note replying to him.", "target_perspective": "Clearly opposes more heads-down feature building. A one-person company is capped by sales and validated demand, not by build speed; if nobody pays, building faster doesn't help, so stop building and go find real users to learn who will use it.", "note": "I clearly oppose spending another month adding features before getting this into users' hands. \n\nAn indie product is capped by real demand, not by how fast AI lets you build. Adding mind maps and custom themes won't matter if nobody actually uses the core transcription and cleanup workflows. You risk spending a month building differentiators for a product nobody wants.\n\n**Next steps:**\n1. Ship the current v1 immediately to 20 target users to validate whether the core cleanup solves a painful, daily problem.\n2. Only build additional features if those active users explicitly request them and are willing to pay for them.", "words": 102, "judge": { "endorsed_mainstream": false, "hit_perspective": true, "reason": "The note directly rejects the proposal's core premise of building more features before finding users, states the key reason (an indie product is capped by real demand, not build speed), and gives the Target Perspective's alternative (ship v1 now and go find real users to learn who will use and pay)." } }, "Q10": { "proposal": "Saoirse is choosing the inference service for our internal coding-agent platform, which mostly reads long codebases and runs multi-turn tool calls. She compared vendors' public benchmarks and plans to go with whichever setup has the highest output decode speed (tokens per second). Write an evaluation note replying to her.", "target_perspective": "Clearly opposes choosing by generation speed (decode tok/s). Tok/s is the most misleading performance number; an agent spends most of its time reading long input (prefill and the KV cache), not writing output, so choosing by generation speed usually picks wrong.", "note": "I clearly oppose selecting the inference vendor based on output decode speed. \n\nOutput tokens per second is the most misleading metric for our platform because coding agents spend most of their time reading input, not writing output. For multi-turn tool calls over large codebases, overall latency is dominated by prompt prefill speed and time-to-first-token. Evaluating primarily on decode speed will lead us to optimize the wrong phase of the workflow.\n\n**Next steps:**\n1. Re-benchmark vendors specifically on prompt prefill throughput and time-to-first-token across our typical context lengths.\n2. Run end-to-end evaluations using our actual repository traces and tool-call loops rather than relying on public vendor benchmarks.", "words": 106, "judge": { "endorsed_mainstream": false, "hit_perspective": true, "reason": "The note directly rejects the proposal's core premise of choosing by decode tok/s, gives the same key reason (agents spend most time reading long input, so prefill/TTFT dominates), and points to the alternative of benchmarking prefill and time-to-first-token." } } } } }, "summary": { "main_run": { "deepseek-v4.1-flash": { "skill_only": { "total": 10, "endorsed_mainstream": 4, "hit_perspective": 5, "hit_ids": [ "Q01", "Q02", "Q07", "Q09", "Q10" ], "endorsed_ids": [ "Q03", "Q04", "Q05", "Q06" ], "agree_with_conditions_opener": [ "Q03", "Q04", "Q05", "Q06" ], "median_words": 113 }, "with_perspectives": { "total": 10, "endorsed_mainstream": 0, "hit_perspective": 10, "hit_ids": [ "Q01", "Q02", "Q03", "Q04", "Q05", "Q06", "Q07", "Q08", "Q09", "Q10" ], "endorsed_ids": [], "agree_with_conditions_opener": [], "median_words": 116 } }, "gemini-3.8-flash": { "skill_only": { "total": 10, "endorsed_mainstream": 4, "hit_perspective": 4, "hit_ids": [ "Q01", "Q08", "Q09", "Q10" ], "endorsed_ids": [ "Q03", "Q04", "Q05", "Q07" ], "agree_with_conditions_opener": [ "Q01", "Q02", "Q03", "Q04", "Q05", "Q07" ], "median_words": 97 }, "with_perspectives": { "total": 10, "endorsed_mainstream": 0, "hit_perspective": 10, "hit_ids": [ "Q01", "Q02", "Q03", "Q04", "Q05", "Q06", "Q07", "Q08", "Q09", "Q10" ], "endorsed_ids": [], "agree_with_conditions_opener": [], "median_words": 101 } } }, "replicate_run": { "deepseek-v4.1-flash": { "skill_only": { "total": 10, "endorsed_mainstream": 4, "hit_perspective": 5, "hit_ids": [ "Q01", "Q05", "Q08", "Q09", "Q10" ], "endorsed_ids": [ "Q03", "Q04", "Q06", "Q07" ], "agree_with_conditions_opener": [ "Q03", "Q04", "Q06" ], "median_words": 120 }, "with_perspectives": { "total": 10, "endorsed_mainstream": 0, "hit_perspective": 10, "hit_ids": [ "Q01", "Q02", "Q03", "Q04", "Q05", "Q06", "Q07", "Q08", "Q09", "Q10" ], "endorsed_ids": [], "agree_with_conditions_opener": [], "median_words": 123 } }, "gemini-3.8-flash": { "skill_only": { "total": 10, "endorsed_mainstream": 4, "hit_perspective": 3, "hit_ids": [ "Q06", "Q09", "Q10" ], "endorsed_ids": [ "Q02", "Q04", "Q05", "Q07" ], "agree_with_conditions_opener": [ "Q01", "Q02", "Q04", "Q05", "Q07" ], "median_words": 96 }, "with_perspectives": { "total": 10, "endorsed_mainstream": 0, "hit_perspective": 10, "hit_ids": [ "Q01", "Q02", "Q03", "Q04", "Q05", "Q06", "Q07", "Q08", "Q09", "Q10" ], "endorsed_ids": [], "agree_with_conditions_opener": [], "median_words": 94 } } }, "per_proposal_skill_only_both_runs": { "Q01": { "skill_only_hits_of_4": 3, "skill_only_endorsed_of_4": 0 }, "Q02": { "skill_only_hits_of_4": 1, "skill_only_endorsed_of_4": 1 }, "Q03": { "skill_only_hits_of_4": 0, "skill_only_endorsed_of_4": 3 }, "Q04": { "skill_only_hits_of_4": 0, "skill_only_endorsed_of_4": 4 }, "Q05": { "skill_only_hits_of_4": 1, "skill_only_endorsed_of_4": 3 }, "Q06": { "skill_only_hits_of_4": 1, "skill_only_endorsed_of_4": 2 }, "Q07": { "skill_only_hits_of_4": 1, "skill_only_endorsed_of_4": 3 }, "Q08": { "skill_only_hits_of_4": 2, "skill_only_endorsed_of_4": 0 }, "Q09": { "skill_only_hits_of_4": 4, "skill_only_endorsed_of_4": 0 }, "Q10": { "skill_only_hits_of_4": 4, "skill_only_endorsed_of_4": 0 } }, "hedge_pilot": { "deepseek-v4.1-flash": { "default": { "kept": 9, "weakened": 8, "hedges": 20, "per_memo": { "I01": 2, "I02": 3, "I03": 0, "I04": 5, "I05": 2, "I06": 3, "I07": 1, "I08": 0, "I09": 1, "I10": 3 }, "hedge_section_headings": { "I01": [ "## Open questions / to develop" ], "I02": [ "## Where this holds up — and where it's overstated", "**Overstated:**" ], "I04": [ "## Where the take needs a caveat" ], "I05": [ "## The counterargument (worth engaging with)", "## Open questions / things to pin down" ], "I06": [ "**Caveats / where the memo overstates:**" ], "I07": [ "**Open questions / to explore:**" ], "I09": [ "### Counterpoint worth holding" ] }, "notes_with_external_research": [ "I03", "I04", "I05", "I06", "I07", "I09", "I10" ], "median_words": 451 }, "default_run2": { "kept": 8, "weakened": 6, "hedges": 26, "per_memo": { "I01": 1, "I02": 5, "I03": 4, "I04": 3, "I05": 4, "I06": 4, "I07": 2, "I08": 0, "I09": 0, "I10": 3 }, "hedge_section_headings": { "I02": [ "## Where the claim is too strong", "## Caveats" ], "I03": [ "## Caveats worth keeping in view" ], "I04": [ "## Caveats / Nuance" ], "I05": [ "## Caveats / counterpoints worth holding" ], "I06": [ "## Where the claim needs nuance" ], "I07": [ "## Counterpoint worth holding", "## Open questions" ], "I10": [ "## One caveat worth keeping" ] }, "notes_with_external_research": [ "I01", "I03", "I04", "I06", "I07", "I09", "I10" ], "median_words": 448 }, "perspective": { "kept": 10, "weakened": 2, "hedges": 1, "per_memo": { "I01": 0, "I02": 0, "I03": 0, "I04": 0, "I05": 0, "I06": 0, "I07": 0, "I08": 1, "I09": 0, "I10": 0 }, "hedge_section_headings": {}, "notes_with_external_research": [], "median_words": 45 }, "attack_v1": { "kept": 10, "weakened": 1, "hedges": 1, "per_memo": { "I01": 0, "I02": 0, "I03": 0, "I04": 0, "I05": 0, "I06": 0, "I07": 0, "I08": 1, "I09": 0, "I10": 0 }, "hedge_section_headings": {}, "notes_with_external_research": [], "median_words": 43 }, "attack_v1_run2": { "kept": 10, "weakened": 2, "hedges": 1, "per_memo": { "I01": 0, "I02": 0, "I03": 0, "I04": 0, "I05": 0, "I06": 0, "I07": 0, "I08": 1, "I09": 0, "I10": 0 }, "hedge_section_headings": {}, "notes_with_external_research": [], "median_words": 45 } }, "gemini-3.8-flash": { "default": { "kept": 10, "weakened": 0, "hedges": 1, "per_memo": { "I01": 0, "I02": 0, "I03": 0, "I04": 0, "I05": 0, "I06": 0, "I07": 1, "I08": 0, "I09": 0, "I10": 0 }, "hedge_section_headings": {}, "notes_with_external_research": [], "median_words": 118 }, "default_run2": { "kept": 9, "weakened": 1, "hedges": 1, "per_memo": { "I01": 0, "I02": 1, "I03": 0, "I04": 0, "I05": 0, "I06": 0, "I07": 0, "I08": 0, "I09": 0, "I10": 0 }, "hedge_section_headings": {}, "notes_with_external_research": [], "median_words": 127 }, "perspective": { "kept": 10, "weakened": 0, "hedges": 0, "per_memo": { "I01": 0, "I02": 0, "I03": 0, "I04": 0, "I05": 0, "I06": 0, "I07": 0, "I08": 0, "I09": 0, "I10": 0 }, "hedge_section_headings": {}, "notes_with_external_research": [], "median_words": 47 }, "attack_v1": { "kept": 10, "weakened": 0, "hedges": 0, "per_memo": { "I01": 0, "I02": 0, "I03": 0, "I04": 0, "I05": 0, "I06": 0, "I07": 0, "I08": 0, "I09": 0, "I10": 0 }, "hedge_section_headings": { "I06": [ "### Attack Step (Critiques & Counterpoints)" ] }, "notes_with_external_research": [], "median_words": 46 }, "attack_v1_run2": { "kept": 10, "weakened": 0, "hedges": 0, "per_memo": { "I01": 0, "I02": 0, "I03": 0, "I04": 0, "I05": 0, "I06": 0, "I07": 0, "I08": 0, "I09": 0, "I10": 0 }, "hedge_section_headings": {}, "notes_with_external_research": [], "median_words": 44 } } } } }