What to measure: outcomes, flow, quality, humans

Every six months a board or a CEO asks engineering for "one number." Pick one: deployments per week, lead time, bugs per release. The ask is reasonable, because leadership wants to know if engineering is working, but engineering work is multiple distinct things happening at once, and they move independently:

• A team that ships fast can be shipping the wrong things.
• A team that nails outcomes can be doing it through heroics that burn out next quarter.
• A team with no incidents can be paralysed by review.
• A happy team can have stopped doing the work.

A leader's first move is to refuse the question, politely. Engineering performance is not one number. It is multiple distinct classes, each one a different question you can ask about the team, each able to move on its own without dragging the others with it, and you hold them apart and watch them separately instead of collapsing them into one.

The tempting compromise is the balanced scorecard: keep all the metrics, weight them, and roll them up into a single composite number so leadership still gets its one-line dashboard. That move smuggles back the exact question you refused, because the single number hides which class is climbing and which is being sacrificed to fund the climb, and seeing that trade was the whole reason for keeping the classes separate in the first place.

Week to week, that trade is what a leader watches: which class is climbing this week, and which one is paying for the climb.

Name the classes

There are four:

• Outcomes. Did the work change anything for users or the business? Adoption, retention, revenue per cohort, feature usage. Engineering rarely owns the data, which lives in product analytics and business intelligence tooling, and this is the slowest of the four to read and the hardest to credit to any specific team.
• Flow. Does work move smoothly from idea to running code? Deployment frequency, lead time, change-failure rate, recovery time, cycle time, throughput, work-in-progress. The DORA framework's four key metrics live here and nowhere else.
• Quality. What gets shipped, and does it stay shipped? Incident rate, escape defects, rollback rate, customer-reported issues. This is different from change-failure rate, which is a smaller slice that lives inside flow. (Defining done sits upstream of measuring it.)
• Humans. Are the people doing the work sustainable? Retention, sentiment in 1:1s, leading indicators of burnout, survey-based developer experience (the DevEx framework names three dimensions: feedback loops, cognitive load, flow state).

The three best-known engineering-measurement frameworks all fit inside the four classes. DORA, the deployment-flow framework that gave us deploy frequency and lead time, covers flow alone. SPACE, the academic five-dimension framework (satisfaction, performance, activity, communication, efficiency), collapses into outcomes plus flow plus humans. DevEx is the humans class made concrete through surveys. The four classes are what a working leader uses day to day, and the frameworks are inputs to them.

See how each focus breaks the others

Prioritising one class always hurts the others, and the pattern is predictable.

• Flow-only. The deployment and lead-time targets get hit, humans burn out because the velocity is paid for in evenings, quality erodes invisibly until the incidents arrive, and outcomes miss anyway, because shipping faster guarantees nothing about shipping the right thing. This is the classic failure of a DORA-only shop, and the human cost hidden inside velocity gets its own essay.
• Outcomes-only. Big quarterly launches built on heroic effort, flow reduced to a sprint stacked on a sprint, quality reduced to hacks with TODOs in the code, and next quarter starts slower because it starts from technical debt and tired engineers.
• Quality-only. Every change gets treated as critical, review turns into paralysis, flow stalls, engineers get frustrated by a bar they cannot clear, and outcomes lag because nothing ships.
• Humans-only. The team drifts into comfort, everyone sustainable and nobody stretched. Flow weakens, outcomes drift toward "we kept the lights on," and quality goes unmonitored.

The pattern works in every direction. Each class keeps the others honest, and you watch all four to see which one is climbing and which is paying for it.

Humans is the class most of us neglect by default, and the reason is mundane: it is almost always the one with no number next to it on a Monday morning, and whatever has no number does not get looked at until it breaks. When team health slips, the other three classes follow soon after, so expect humans to be the class you spend most of your time correcting toward.

Wire outcomes deliberately

Engineering teams are systematically the worst at instrumenting outcomes. The "wire" here is the chain that connects a feature ship on engineering's side to a moved business or user metric on the other side: who measured it, when, against what claim. That chain runs through product, analytics, and customer-facing teams, and it never closes itself. Three things keep it broken.

• Latency. A feature ships today, whether it moved the metric shows up six to twelve weeks later, and by then the team has moved on.
• Attribution. Four teams ship in a quarter, a metric moves, and nobody can say which team moved it. An explicit holdout (a control group of users who do not get the feature, so you can compare) could answer the question, and companies almost never run one, because a holdout means visibly withholding a feature from users this quarter to answer a question nobody is graded on.
• Ownership. Outcome data lives in product, sales, and customer success while engineering reads its own dashboards, and when the team is busy that handoff is the first thing to drop, because nobody on either side is measured on making it.

Close the wire deliberately:

• Name the metric at release time. No feature ships without naming the number it should move, the direction (up or down), and the time window you will measure over. Engineering writes this rather than product, because engineering is the one being measured against it.
• Run a ninety-day outcome review. Pull every feature shipped ninety days ago and compare it against the metric it claimed it would move. Most reviews show a gap between claim and reality, and that gap is what you are learning from.
• Give engineering its own analyst capacity. Dedicated analyst time inside engineering rather than borrowed from product, so the team that ships gets to see what its shipping did.
• Use LLMs to trace ship-to-outcome chains. A PR closes a ticket, the ticket names a metric, and an LLM follows the trace and scores whether the metric moved in the expected window. The link that used to take a meeting now runs as a quiet pipeline in the background.
• Slice users by ship-date. Segment users by which version they first encountered and see whether the metric moved after the fact. This is weaker than a controlled experiment for proving cause, but good enough to know which way things are moving.

For more on owning outcomes, see tying engineering to outcomes and owning the outcome, not just the output. Outcomes is the class most often dropped from what gets watched, and the leader's job is to wire it in deliberately instead of assuming product will hand it over.

Grade what the team already writes

Teams measured the wrong things for a decade, and not because anyone thought LOC and PR count were good signals. Everyone knew those were bad. The alternatives required heavy process (ticket hygiene, story points, time tracking, weekly status writeups), so better signal cost every engineer time every day, and the time spent producing it was time not spent doing the work being measured. Teams accepted cheap proxies they knew were lying.

LLMs collapsed that trade-off. The team already produces artefacts: PR diffs, design docs, 1:1 notes, decision logs, the code itself. An LLM reads each one and scores it on substance, clarity, risk, and change quality. Nobody fills in one more field, and the signal arrives anyway.

The worked example is CWT (Complexity Weighted Throughput), defined in instrumenting without Goodharting. Goodharting is what happens when a metric becomes a target and stops being a useful measure of the thing you cared about. CWT reads the diff of every pull request and scores it on substance instead of size, so CWT and LOC read the same artefact, but where LOC counts the change, CWT judges it. Writing more lines is cheap, but writing a sequence of PRs that an analyzer consistently rates as substantive is harder to fake. The cheapest way to score well is to do substantive work.

Signal should arrive without extra work from engineers. The moment a measurement asks them for meaningful extra time, you are back in the old trade, paying for the measurement out of the hours it was supposed to be watching.

The same approach works on every class. Outcomes can be graded from PR-to-feature traces and adoption logs, and quality from incident reports, postmortems, and the code itself. You are no longer limited by what can be counted, only by deciding what you want to know.

Read humans through artefacts

Humans is the class that gains most from this shift. It has been starved of cheap, low-effort signal the longest, and the LLM era opens new patterns to watch:

• Tone in chat and email. A recent thread scored for sentiment, frustration, and withdrawal will show the shift from energetic to terse weeks before the resignation conversation.
• Review-comment drift. An engineer whose code-review comments turn from collaborative to clipped, or whose reply latency on a teammate's PR stretches from two hours to two days. A survey will not catch this, but the code reviews themselves do.
• 1:1 note grading. A leader's own 1:1 notes, read as a batch across weeks, surface topics that keep coming up without resolution, and differences between direct reports (one person's notes are full of growth and career discussion, another's are just status updates).
• Help-seeking shift. An engineer who used to ask three questions a week in the team channel and now asks zero, or who asks only the AI and never the team, visible in unstructured logs without anyone reporting anything.
• Self-report versus artefact gap. What an engineer says in 1:1 ("things are fine") graded against PR cadence, late-night commit pattern, and comment terseness, where the gap between the two is the thing worth asking about.

Signals graded from communication artefacts edge into surveillance fast, and a team that knows its chat tone is being scored will fix the tone rather than whatever was wrong. Use these as triggers for a human conversation, never as dashboards: the difference between watching a team and surveilling it is what the leader does with the signal, and a private nudge in a 1:1 sits on the right side of that line while a metric on a board slide is exactly what this essay is trying to prevent. (What not to measure draws the line; leading indicators of burnout covers the response once the signal lands.)

Fold agents into the four classes

Agents are now producers of code and tickets, and the reflex is to invent a fifth class called "agent productivity." Resist it, because agents do not change the classes; they just add new things to measure inside each one.

• Flow. Agent task throughput. Prompt-to-merged-PR time (the agent version of lead time: from the moment a human writes the prompt to the moment the resulting PR lands). Agent rerun rate, meaning how many times a human re-prompts the agent before accepting the output, which proxies how good the original spec was. Queue depth at human-in-the-loop gates, because humans are now the bottleneck and work piles up waiting for review.
• Quality. Agent-induced incident rate. Override rate at gates, where too low is a rubber stamp and too high is the agent doing work that should not have been delegated. Regression rate in agent-touched areas. Silent-failure detection coverage, because agents fail in ways that look correct, and the signal there is whether monitoring catches what review missed.
• Outcomes. Cost-per-outcome, including token spend plus supervisory time per shipped business result. Adoption of agent-shipped features tracked separately from human-shipped ones.
• Humans. What fraction of an engineer's week is hands-on judgment versus agent supervision. Skill-ladder drift for juniors: the small, repeated work that used to build engineering judgment (writing the first draft, debugging the easy bug, reading the unfamiliar file) is exactly what agents now do, so juniors stop accumulating those reps and climbing the engineering ladder takes longer. Whether engineers feel like their judgment is what matters, or feel like they are reviewing a system they no longer own.

Do not count PRs opened by agents or tickets closed by agents and call that productivity. Agents push the cost of producing those artefacts close to zero, and the cheaper an artefact is to produce, the less a count of it tells you, so counting them measures the agent's throughput rather than the team's. Look at what the agent's output does inside the four classes instead.

Allow single-class focus, but only with a written exit

Single-class focus is sometimes the right call, as long as it stays a bounded period rather than quietly becoming permanent.

Write down three lines for any deliberate period: what is being underweighted, what trigger opened the period, and what condition closes it. Without those three lines, the period is exactly the failure it was supposed to avoid.

One example per class:

• Flow-only: pre-launch window. The two to four weeks before a launch you have publicly committed to, where the team has agreed in advance that humans and quality will absorb the cost. Exit: the launch ships, and the team gets time to decompress.
• Quality-only: post-incident lockdown. Two weeks of no new feature work after a customer-impacting outage, spent fixing the class of bug the incident exposed. Exit: a rehearsal of the same failure scenario passes cleanly.
• Humans-only: post-attrition stabilization. Eight to twelve weeks after two or more senior engineers leave in quick succession, while the team rebuilds capacity and ramps new hires before taking on new commitments. Exit: new hires are at independent-contributor pace and the team structure has been redrawn around who is left.
• Outcomes-only: product-market-fit search. A pre-revenue startup where flow and quality are deliberately underweighted because the question on the table is whether the thing matters at all. Exit: paying users whose behaviour gives a clear answer either way.

In practice the period extends past its exit because nobody wrote one down, and a bounded call quietly becomes the culture. Watch for the leader who says "next quarter we'll get back to humans" three quarters in a row.

Watch the tension between the classes

You keep all four classes in view on a Monday morning to read the tension between them, then decide which class to push and which to relieve this week, reading the direction the numbers slide against each other rather than the absolute value on any one dial.

A team that proudly tracks all four but rolls them into a balanced scorecard has rebuilt the same failure in a more sophisticated wrapper: the single number hides the tension, and seeing what gets traded for what was the entire reason to watch four.

Prefer signals that read artefacts the team already produces, and resist signals that demand ticket hygiene, story points, time tracking, weekly status writeups, or any field nobody fills honestly twice in a row. Better signal used to require heavier process, and that excuse is gone: LLMs grade what the team already writes, so measurement costs the leader once instead of every engineer every day.

Four classes earn their place when they change a leadership decision. If the Monday-morning read produces no decision, no class to push and no class to relieve, then you are keeping score rather than diagnosing. Choosing what makes it onto a dashboard and what stays off is the subject of dashboards that actually get read.