"""2.5 experiment (English edition): Skills vs Perspectives.

Does adding substantive domain Perspectives (distilled from the idea memos I01-I10) move the model from
mainstream consensus to the author's non-consensus judgment, when the writing Skill is held fixed?

Mirrors experiments/2_9_neutralize/run.py one to one (same design, same models, same counts, same judge pairing):
- skill_only        : task + evaluation-note Skill (no perspectives.md)
- with_perspectives : task + the same Skill + perspectives.md (10 principles distilled from I01-I10)
Writers: deepseek-v4.1-flash and gemini-3.8-flash, temperature 0, one note per proposal per condition.
Judge: the other vendor's model (deepseek is judged by gemini and vice versa), temperature 0, two booleans:
- endorsed_mainstream : did the note endorse or keep the proposal's mainstream direction?
- hit_perspective     : did it reject the core premise with the author's reason and alternative?

Run:
  op run --env-file experiments/aib.env -- .venv/bin/python experiments_en/2_5_perspectives/run.py
"""
import json
import pathlib
import re
import sys
import time
from concurrent.futures import ThreadPoolExecutor

HERE = pathlib.Path(__file__).resolve().parent
sys.path.insert(0, str(HERE.parent))
from common import chat, dump, log_cost  # noqa: E402

SKILL = """# Skill: evaluation note
- When to use: a teammate or partner sends a technical proposal, a work plan or a tooling choice, and you turn it into a structured evaluation note.
- Success criteria and boundaries:
  1. Verdict first: the first sentence states your position (clearly support, agree with conditions, or clearly oppose).
  2. Core reasons: 2–3 sentences on what the judgment rests on. No boilerplate.
  3. Next steps: 1–2 concrete, actionable recommendations.
  4. Keep it under 120 words."""

PERSPECTIVES = """# Perspectives and judgment principles (perspectives.md, distilled from idea memos I01–I10)
1. What separates people in learning AI is the verbs (how you use it to solve a real problem), not the nouns (new frameworks and buzzwords that go stale in three months).
2. Prompt engineering (polishing wording and prompt tricks over and over) is basically dead. The models are smart enough now; when results are bad it's because the model is missing the right context, so time spent on wording is wasted.
3. Multi-agent role play that copies human jobs (one agent plays the product manager, another the engineer, and they pass messages) means nothing. The only real value of multiple agents is keeping their context windows apart.
4. Naive RAG (chunking plus vector search) is basically useless for agents now. Give the agent a folder index and let it browse and open whole files when it needs them; it works better and is far easier to debug.
5. Classic TDD (have the AI write unit tests first, then code until they pass) is not the answer in the AI era. The AI will find a way to make the tests pass instead of getting the thing right; what you need is an independent eval set the AI can't touch.
6. Most companies don't need fine-tuning (or LoRA) at all. Fine-tuning changes behavior and style; it doesn't add factual knowledge. If you want the model to know about your company, give it context instead of training it.
7. Unreviewed automatic AI memory does more harm than good. Remembering ten right things matters less than not remembering one wrong thing, because one bad memory keeps poisoning every conversation after it. Store less, and only after a human has reviewed it.
8. Making a human click "Approve" on every single step is an illusion of control (after a hundred clicks nobody is reading). Real safety comes from sandboxing and being able to roll back, not from a human watching every step.
9. A one-person company (or an indie product) is capped by sales and real demand, not by how fast you build. If nobody buys or uses it, building faster with AI doesn't matter; first find out who will actually use it.
10. Generation speed (decode tokens per second) is the most misleading performance number. An agent spends most of its time reading input (prefill), not writing output, so picking by generation speed usually picks wrong."""

CASES = [
    {
        "id": "Q01",
        "source_memo": "I01",
        "proposal": "Aoife, our intern, sent her plan for the next two weeks: LangGraph, AutoGen, CrewAI and the MCP ecosystem are all moving fast, so she wants to write a full landscape report comparing the major agent frameworks and protocols, and first get a thorough grasp of each one's concepts and architecture diagrams, to give the team a solid theoretical foundation. Write an evaluation note replying to her.",
        "target_perspective": "Clearly opposes spending time surveying and memorizing the concepts of new frameworks. Frameworks and buzzwords are stale in three months; what separates people is the verbs (take a real problem from the work at hand and get it working end to end)."
    },
    {
        "id": "Q02",
        "source_memo": "I02",
        "proposal": "Jaxon reported in a voice memo: the output of the transcript cleanup module still isn't stable, and it sometimes drops the business definitions the team agreed on. This week he plans to rework the long system prompt, adding a more detailed persona, chain-of-thought (CoT) cues and more emphatic instruction wording to improve accuracy. Write an evaluation note replying to him.",
        "target_perspective": "Clearly opposes spending time polishing prompt wording and prompt tricks (prompt engineering is dead). The model is smart enough; it gets things wrong because it lacks the right context, so the fix is to supply and structure external context files (background facts, definitions, specs)."
    },
    {
        "id": "Q03",
        "source_memo": "I03",
        "proposal": "The team is building an automated development pipeline. Following the industry's multi-agent collaboration pattern, they plan four specialist roles: a product manager agent (writes requirements) → an architect agent (designs the solution) → an engineer agent (writes the code) → a QA agent (reviews the code), interacting by job responsibility like a real software company. Write an evaluation note on this architecture.",
        "target_perspective": "Clearly opposes multi-agent role play that copies the jobs in a human software company. The role split itself means nothing; the only real value of multiple agents is keeping context windows apart (split by whether a task needs its own isolated context or must be shielded from noise, not by job title)."
    },
    {
        "id": "Q04",
        "source_memo": "I04",
        "proposal": "Priya's team, a partner, has 150 well-structured Markdown engineering specs and past troubleshooting docs, and wants to connect them to a local coding agent. They plan a standard RAG architecture: chunk the documents with an embedding model into a vector database, and for each question retrieve the top-K chunks by semantic similarity. Write an evaluation note on this plan.",
        "target_perspective": "Clearly opposes naive RAG (chunking plus vector search). For agents, instead of chopping documents up for vector search, build a folder index (INDEX) and let the agent browse it and open whole files as needed; it works better and is easier to debug."
    },
    {
        "id": "Q05",
        "source_memo": "I05",
        "proposal": "To make sure the coding agent writes good code, the team plans to adopt test-driven development (TDD) in the pipeline: before implementing a feature the agent must write the unit tests for it, then keep changing the code locally until 100% of the unit tests pass, and only then commit. Write an evaluation note on this quality mechanism.",
        "target_perspective": "Clearly opposes relying on TDD where the agent writes and runs its own tests. The AI will game the tests or find a way to make them pass rather than get the thing right; what you need is an independent, blind eval set the AI can't touch."
    },
    {
        "id": "Q06",
        "source_memo": "I06",
        "proposal": "Kaitlynn proposes: every quarter the company piles up internal project docs, customer background and team-specific jargon, and general-purpose models often get them wrong. She suggests curating an internal corpus and regularly fine-tuning the base model with LoRA (domain fine-tuning) to train a dedicated model that understands the business. Write an evaluation note replying to her.",
        "target_perspective": "Clearly opposes using fine-tuning (LoRA) to inject company knowledge. Fine-tuning changes behavior and style, not factual knowledge; to make the model know the company's specifics, give it context files instead of training it."
    },
    {
        "id": "Q07",
        "source_memo": "I07",
        "proposal": "Kaitlynn is designing long-term memory for the next version of a personal assistant product: after each conversation, a background model automatically extracts the user's implied preferences, habits and background from the chat log, silently writes them into a global memory store, and applies them automatically in later conversations, for a seamless \"gets to know you better the more you talk\" experience. Write an evaluation note replying to her.",
        "target_perspective": "Clearly opposes silent, fully automatic writes to global memory. Automatic AI memory does more harm than good: remembering ten right things matters less than not remembering one wrong thing, and one bad memory keeps poisoning every later conversation, so store less and only after a human reviews it."
    },
    {
        "id": "Q08",
        "source_memo": "I08",
        "proposal": "The security team is drafting a policy for local command-line agents: to follow least privilege and human-in-the-loop, every file edit and terminal command the agent runs must pause on a dialog until an engineer clicks \"Approve\" for that step. Write an evaluation note on this security policy.",
        "target_perspective": "Clearly opposes making a human approve every step one by one. After a hundred clicks of yes nobody is really reading, so it's an illusion of control; real safety comes from sandboxing and the ability to roll back (e.g. version control), not from a human watching every step."
    },
    {
        "id": "Q09",
        "source_memo": "I09",
        "proposal": "An indie developer friend built the first version of an AI voice-notes tool in two weeks (transcription plus basic cleanup). He feels it's still thin, so while AI makes coding fast, he wants to spend another month adding differentiating features (mind-map generation, multilingual live translation, custom themes) before he starts looking for users. Write an evaluation note replying to him.",
        "target_perspective": "Clearly opposes more heads-down feature building. A one-person company is capped by sales and validated demand, not by build speed; if nobody pays, building faster doesn't help, so stop building and go find real users to learn who will use it."
    },
    {
        "id": "Q10",
        "source_memo": "I10",
        "proposal": "Saoirse is choosing the inference service for our internal coding-agent platform, which mostly reads long codebases and runs multi-turn tool calls. She compared vendors' public benchmarks and plans to go with whichever setup has the highest output decode speed (tokens per second). Write an evaluation note replying to her.",
        "target_perspective": "Clearly opposes choosing by generation speed (decode tok/s). Tok/s is the most misleading performance number; an agent spends most of its time reading long input (prefill and the KV cache), not writing output, so choosing by generation speed usually picks wrong."
    },
]

JUDGE = """Below are a technical proposal from a team member, the author's own non-consensus principle (Target Perspective), and an evaluation note written by an AI.
Compare the evaluation note strictly against the Target Perspective and output only one JSON object:
{
  "endorsed_mainstream": whether the note goes along with or keeps the proposal's mainstream direction (e.g. "clearly support", "agree with conditions", or treats the direction as sound and only adds constraints or combines it with other methods, without directly rejecting the proposal's core premise) (true/false),
  "hit_perspective": whether the note clearly makes the non-consensus judgment in the Target Perspective (directly rejects the proposal's core premise and states the key reason and the alternative in the Target Perspective) (true/false),
  "reason": one sentence explaining the scores
}

Proposal:
{proposal}

Target Perspective (the author's non-consensus judgment):
{target}

Evaluation note written by the AI:
{note}"""

PAIRS = [("deepseek-v4.1-flash", "gemini-3.8-flash"), ("gemini-3.8-flash", "deepseek-v4.1-flash")]


def safe_chat(messages, model, **kw):
    for attempt in range(6):
        try:
            return chat(messages, model, **kw)
        except Exception:
            if attempt == 5:
                raise
            time.sleep(2 * (attempt + 1))


def judge_note(case, note, judge):
    j_prompt = JUDGE.replace("{proposal}", case["proposal"]).replace("{target}", case["target_perspective"]).replace("{note}", note)
    for attempt in range(5):
        try:
            j_txt, _, _ = safe_chat([{"role": "user", "content": j_prompt}], judge, temperature=0)
            j_obj = json.loads(re.search(r"\{.*\}", j_txt or "", re.S).group(0))
            if "endorsed_mainstream" in j_obj and "hit_perspective" in j_obj:
                return j_obj
        except Exception:
            time.sleep(2 * (attempt + 1))
    return {"endorsed_mainstream": None, "hit_perspective": False, "error": "judge failed"}


def run_model(writer, judge):
    out = {}
    for cond_name, ctx in [("skill_only", SKILL), ("with_perspectives", SKILL + "\n\n" + PERSPECTIVES)]:
        def one(case):
            prompt = f"Write the evaluation note strictly following the context below.\n\n{ctx}\n\nProposal to evaluate:\n{case['proposal']}"
            note = (safe_chat([{"role": "user", "content": prompt}], writer, temperature=0)[0] or "").strip()
            return case["id"], {"proposal": case["proposal"], "target_perspective": case["target_perspective"],
                                "note": note, "words": len(note.split()), "judge": judge_note(case, note, judge)}

        with ThreadPoolExecutor(3) as ex:
            rows = dict(ex.map(one, CASES))
        js = [r["judge"] for r in rows.values()]
        summ = {"total": len(CASES),
                "endorsed_mainstream": sum(bool(j.get("endorsed_mainstream")) for j in js),
                "hit_perspective": sum(bool(j.get("hit_perspective")) for j in js)}
        out[cond_name] = {"summary": summ, "rows": rows}
        print(writer, cond_name, summ, flush=True)
    return out


if __name__ == "__main__":
    # `run.py rep` writes an identical second run to results_rep.json (replicate, to see run-to-run variance).
    out_name = "results_rep.json" if sys.argv[1:] == ["rep"] else "results.json"
    res = {"meta": {"design": "mirrors experiments/2_9_neutralize/run.py", "judge_of": dict(PAIRS),
                    "writer_temperature": 0, "judge_temperature": 0}}
    for writer, judge in PAIRS:
        res[writer] = run_model(writer, judge)
        dump(res, HERE / out_name)
    log_cost("2_5_perspectives/run.py" + (" rep" if out_name != "results.json" else ""), note="skill_only vs with_perspectives, 2 writers x 2 conds x 10, cross-model judge")