Build with AI · Module 3

Managing AI

Check it: trust the work without double-checking every line

Yan Wang, PhD
Describe itModule 2
Check itModule 3 · today
Choose itModule 4
Build itCapstone
Managing AIModule 3 · Yan
Coffee breakOptional Zoom team photo · camera on
Choose itModule 4 · YZ

A familiar complaint

It gets eight out of ten right

You can’t tell which two, so you double-check all ten.

First instinct

Tweak the prompt.

Second instinct

Wait for a better model.

Both assume the right setup would get all ten right.

Is that actually true?

AI is not a calculator

A calculator

2 + 2 = 4

The input decides the answer.
A miss means it’s broken.

Open work

“Clean up my notes.”

Which filler words go?Keep the casual tone?Expand “tbh”?

The input doesn’t decide the answer. Someone has to fill the gaps.

It’s useful because it fills the gaps you never specified. That’s also where it misses.

From fixing it to managing it

Management is the discipline for uncertainty

Errors come as a rate, not a list of bugs. So you decide:

How much error is acceptable?
Which errors are never acceptable?
Who does which job?
How much leash to give?
TightLoose

Trust is a dial, not an on/off switch.

Managing means handing off

You can only hand off
what you can check.

1Did it do a good job this time?
2When it fails, why?
3Can I trust it when I’m not watching?
4Can it get better on its own?

The catch: we are strong ICs

You probably don’t want to be a manager

The strongest ICs stumble hardest. What made them great turns into three traps.

See it yourselfYou can tell right from wrong at a glance, so you never write down what “right” means.
Fix it yourselfGrabbing the keyboard is faster than explaining, so next time you grab it again.
Direct it yourselfYou know the steps, so the task stops where your steps stop.
All three make you the bottleneck: firefighting, not fire prevention.

The good news

Managing AI is easier than managing people

Half of managing people
Selling the vision
Career growth
Morale

AI doesn’t need this half.

The other half
Setting standards
Building evaluations
Writing down what works

For an engineer: specs, tests and docs.

Higher up, you see less

A strategy game vs. a real king

A strategy game

Every number from every province, live.

A real king

Dispatches, filtered through layers of officials.

TransparencyDrill from a top number down to one raw item.
AcceptanceTell if it’s right without redoing the work.
SteeringFind the fault and pull it back on course.

Today’s outline

The promotion path

DirectorHow does this kind of work improve on its own?4Director
ManagerHow does it get done well without my hands on it?1See it2Fix it3Direct it
ICHow do I do this well?

Each step up hands off one habit, and only once the layer below is solid.

Question 1 of 4 · Did it do a good job this time?

I can tell if it’s right at a glance.

Stop seeing it
yourself.

Replace eyeballing with an eval and a dashboard.

1I can tell if it’s right at a glance.an eval and a dashboard
2It’s faster if I just fix it.coaching and an SOP
3I’ll just tell it each step.acceptance criteria
4I still sit in every loop.a loop that improves itself

Why measure at all

A few good examples tell you nothing

You’re blind to what you didn’t sample.Three items got better. Did the other twenty get worse?
You have no fixed reference.Today’s three aren’t last week’s three.
One complaint swings you.A single bad output feels like the whole system broke.
Defining success is harder than it looks
A target rate, not “works” or “broken.”
The whole batch, not the cases you remember.
Fixes and breakages counted apart. One change often does both.

Module 1 · Evaluation first: define success before the first prompt.

You need a repeatable way to measure: an evaluation.

Our running case

Neurotype: type freely, let AI fix it

Type as fast as you think. Don’t stop for spelling. The model reconstructs what you meant.

what you typed
Teh quarterly repport is dueon Fiday, so plrase sen me yoru numbes by Wednesda afteenoon.
what you meant
The quarterly report is due on Friday, so please send me your numbers by Wednesday afternoon.
Follow along
github.com/grapeot/neurotype
An open taskMore than one fix can look right.
Many itemsToo many to read every output.
Easy to seeRight and wrong are visible at a glance.

Stage 1 · Acceptance

Three parts of an evaluation

1

Dataset

The cases you care about, not the ones you remember.

2

Metric

Scored by code, so every change gets checked.

3

Baseline

Better than what? Doing nothing, a random guess, or today’s best method.

You define what to measure. The AI writes the code.

Stage 1 · Live

Hand over the evaluation, then run it

the ask
Build an evaluation for the corrector. Done means:
· messy/clean pairs in data/
· a script that scores any version with
  normalized edit distance (lower is better)
· a baseline: a dummy corrector that returns
  the input unchanged (no prompt, no model)
· run the baseline on the full dataset
Edit distance · lower is better
v0 · baseline (echo)—
v1 · first prompt, with the model—

v0 comes from the run on screen. v1 comes when we add the model.

Watchthe ask is a destination, not steps.

Stage 1 · Who checks

The author can’t grade its own work

A model checking itself, same context Accuracy · higher is better 75.8% before 38.1% after checking itself

Checking itself cut its accuracy in half.

GPT-3.5, commonsense QA · Huang et al., ICLR 2024
What checks it
A script when code can compute it.
A model judge when it takes judgment.
Who judges
Never the author’s context. Use a fresh one.
Judges only judge, and get calibrated against your labels.

Neurotype’s score can be computed, so a script checks it. Judgment calls go to a judge in a fresh context.

Stage 1 · Observability

A score is a compass, not a microscope

The score got worse. Why? Many small slips Or one collapse Same total drop. Different fix. The total squashes both into one number.
Layer 1 · What

Every version, one score

Are we going the right way?

Layer 2 · Why

Every case, with its rationale

Input, output, truth, and why the model chose it.

Building this viewer used to take days. With an agent it takes minutes.

Stage 1 · Live · Transparency

From the score down to one case

Layer 1 · versions v0 · baselinescore v1 · first promptscore click a version to see its cases worst first Layer 2 · one case input output truth rationale · why the model chose this
Watchwhere the rationale and the truth disagree.

Stage 1 · Noise

Know your noise floor

v1, unchanged, run three times · CER, lower is better
Run 10.081
Run 20.077
Run 30.080
Spread0.004

Nothing changed between runs. Across all versions, the spread reached 0.007: our noise floor.

One run is one sample. Only a change bigger than the spread counts.

Remember “not a calculator”? This is it, on our own data.

Stage ruleNo score, no change.

Question 2 of 4 · When it fails, why?

It’s faster if I just fix it.

Stop fixing it
yourself.

Replace patching with coaching and an SOP.

1I can tell if it’s right at a glance.an eval and a dashboard
2It’s faster if I just fix it.coaching and an SOP
3I’ll just tell it each step.acceptance criteria
4I still sit in every loop.a loop that improves itself

Stage 2 · Whack-a-mole

Why patching breeds more patches

prompt.md, week after week
Change “Teh” to “The”.
Change “repport” to “report”.
Change “Fiday” to “Friday”.
… one rule per failure

You fix one item, not the class it belongs to.

a rule in a prompt
Always capitalize “Apple”.
breaks “an apple a day”
the model applies it to every input

A prompt rule isn’t an if. You can’t see its blast radius in advance.

Patching item by item never teaches the pattern.

Stage 2 · Diagnose first

Same-looking failures, different causes

CauseWhat it meansHow to fix it
PolicyIts choice is defensible, just not ours.State the rule.
ContextThe right answer needs facts it doesn’t have.Supply the context.
AbilityIt knows what you want but does it unreliably.Spell out the procedure, or switch models.
DataThe answer can’t be recovered, or the label is wrong.Hand it to a human.
Ask the AI for the diagnosis. Your job is the right question.

Stage 2 · Live · Steering

Ask for the pattern, not the fix

1 · Ask

Diagnose, don’t fix

“Look at the worst cases. What do they have in common? Don’t change anything yet.”

2 · Decide

Some patterns are policy

Not ability. You decide what counts as right.

Slang and lowercasePasted code and URLs
3 · Fix one class

v2, then the full run

One class at a time. Read the score against the noise floor.

Stage 2 · The SOP

Write the coaching down

the loop a good SOP covers
1. Measure where you stand
2. Find one failure pattern
3. Decide what kind it is
4. Change one thing
5. Check it on the full set
6. Record what happened
yours will be worded differently

A coaching chat disappears when the chat ends. The next session starts from zero.

An SOP is a skill written as a fixed route. Test it cold: hand it to a fresh session.

live · runs in the background
Consult SOP.md and run two improvement iterations.
Back laterwe’ll check what it did.
Stage ruleFix the class, not the case.

Question 3 of 4 · Can I trust it when I’m not watching?

I’ll just tell it each step.

Stop directing it
yourself.

Replace step-by-step direction with acceptance criteria.

1I can tell if it’s right at a glance.an eval and a dashboard
2It’s faster if I just fix it.coaching and an SOP
3I’ll just tell it each step.acceptance criteria
4I still sit in every loop.a loop that improves itself

Stage 3 · The router

Why you end up as the router

Single-threaded babysitting

Outline, check. Section, check. Format, check.

Your day turns into dozens of waits and checks.

Multi-threaded babysitting

Prompt one window, jump to the next, jump back.

Busy every second, reloading context every switch.

Every handoff passes through you. The root cause: no destination.

Stage 3 · Two ways

Two ways to hand off a task

Process formulation

“Do these steps, and you’re done.”

Process certainty: you fix the route.

Outcome formulation

“Meet these criteria, and you’re done. How is up to you.”

Outcome certainty: you fix the destination.

Simple tasks: either works. Complex tasks: nobody can write “do these ten things and you’ll get promoted.”

Stage 3 · The translation story, one level deeper

When nothing defines done, AI stops at “looks done”

Steps · split, stitch, glossary, retry…

It stops when the last step runs.

Nothing in the list asks whether the post came out right. A person finds the break later, and adds one more step.

Done described · same task

It stops when the result passes.

Right format, no leftover Chinese, consistent terms. Now the agent has something to check against.

Steps can only check the steps. Only a definition of done can check the result.

Stage 3 · The supervisor

Pair the destination with a rigid supervisor

Worker “done” Supervisor checks the criteriaseparate context done not yet: keep going

It doesn’t need to be smarter than the worker. It asks the same questions every time.

Built in: /goal in Claude Code and in Codex.

Separate context, no sunk effort to defend.

Stage ruleDescribe the destination, not the steps.

Stage 3 · How much to let go

Let go on evidence, not on mood

Small pieces,checked often Longer stretches Whole tasks each step up: point to the log

Start with reversible tasks. Push one step past the boundary; pull back if it fails.

Keep a delegation log. Otherwise trust swings with your last run.

Approving every step: an illusion of control

Humans caught 13.6% of dangerous commands. An automated classifier caught 89%.

Anthropic, blind test with 1,053 people

Question 4 of 4 · Can it get better on its own?

I still sit in every loop.

Let it improve
itself.

Replace sitting in every round with a loop that improves itself.

1I can tell if it’s right at a glance.an eval and a dashboard
2It’s faster if I just fix it.coaching and an SOP
3I’ll just tell it each step.acceptance criteria
4I still sit in every loop.a loop that improves itself

Director · Hand off the manager’s loop

What the loop needs, you already built

A scoreStage 1: it can measure itself.
A rationaleStage 1: it can see why it failed.
An SOPStage 2: it knows how to fix a class.
the meta-prompt
Your goal is to improve the score. Here is the current score.
Follow SOP.md: propose one concrete change and say why.
The eval runs; you see the new score.

Director · Live

Two rounds later

Better

It found a class to fix.

Worse

A fix broke something.

Flat

Within the noise.

You decideStopAdd dataChange the metricSwitch models
It optimizes the score you show it. It learns the exam, not the job.

Director · What stays yours

The AI won’t question the framework you give it

From the website loop in Module 2

Told to rank posts by impressions, it pushed the posts that brought readers to the bottom.

It worked tirelessly inside the frame. It never asked whether impressions were the point.

Your job moves up
Boundaries
Regressions
When to stop

Director · The fuel

The data flywheel

Better product More use More data with right or wrong Sharper eval Better fixes

An eval set is a snapshot of the day you built it. Real use keeps moving.

Let normal use leave behind new examples, with the answer attached.

Only data that says right or wrong can push the flywheel.

Stage ruleIt improves itself. You decide when it stops.

The whole module on one page

What replaces each habit

1See it yourselfAn eval, a judge, a dashboard
2Fix it yourselfCoaching and an SOP
3Direct it yourselfCriteria, a supervisor, a trust ladder
4Sit in every loopA loop that improves itself, inside your bounds

This week

Your smallest next step

Before your next prompt, write one sentence: what does a good result look like?

A task you run often: twenty examples and a score.
Your capstone: a digital twin you can check, with clear permissions.
github.com/grapeot/neurotype

Checking tells you whether
an answer is right.
It can’t tell you which question deserves one.

Now · Coffee break. Optional Zoom team photo: turn your camera on.

Then · YZ: Choose it

Build with AI · Module 3 ·