Instrumenting without Goodharting

Most engineering metrics that fail do not fail because the team cheated. They fail because the metric was designed in a way that made cheating the cheapest path. The classic catalogue (lines of code, pull requests merged, tickets closed, story points completed) all have the same shape: a count of surface artefacts that loosely correlates with output, until somebody starts measuring it and the correlation breaks. The list to refuse outright is its own essay; this one is about everything you do choose to instrument.

The standard advice on this is to measure outcomes rather than activities, and to use multiple metrics so no single number can be pumped. Both are correct, and both are toothless, because they describe where you want to end up rather than the move that gets you there. A team that wants a dashboard tomorrow does not get one from "measure outcomes"; it gets one from a design discipline applied to every number it considers publishing.

Treat gameability as a design property of the metric rather than an integrity problem in the team. If the metric is gameable, the team will adapt around it, because adapting is the rational response to whatever a published number rewards. When that happens, the design failed, however tempting it is to write it up as a character problem.

What follows are five moves I use when instrumenting an engineering team. They are ordered by when they get applied, but in practice I cycle through them.

Name the outcome and the decision

Before instrumenting anything, name two things out loud: the outcome the number is a proxy for, and the decision that will change when the number moves. If the second is missing, the number will get published, discussed, and never acted on, which is theatre.

A common version of this failure is the code review SLA measured as time-to-first-response. The intended outcome is shorter merge cycles, but the published number is acknowledgement latency, and no decision hangs off it: nobody re-assigns reviewers when it slips, nobody investigates a slow one. So the team adapts to the only thing the number rewards, which for a busy reviewer means a one-line acknowledgement inside the hour and then silence, because the acknowledgement is nearly free and the review it stands in for is not. The SLA stays green, merge times do not move, and the instrument has done exactly what its design asked of it.

Run the cheapest-path test

For any metric you are about to publish, ask one question: if I were rewarded purely on this number and nothing else, what is the cheapest way to move it? If the cheapest path is bad for the team, the metric is broken before it ships.

Lines of code is the canonical failure of this test, because the cheapest path to move it up is to write more verbose code. The engineer who deletes two thousand lines and ships the cleanest quarter on the team looks like a negative producer, while the junior who is still finding their voice looks elite.

Velocity points fail the same test more slowly. The cheapest path to deliver more points is to inflate what a point means, and since nobody in the estimation room has any incentive to argue a number down, a "three" quietly becomes a "five" within a quarter. After that the team's velocity stops predicting the team's capacity, and comparing across teams stops meaning anything at all.

The positive case is when the cheapest path to move the number is the thing you wanted. Customer-reported defects attributed to the surface a team owns is one example: the cheapest way to bring that number down is to ship fewer regressions on that surface, and because the source of truth lives outside the team, nobody inside can quietly lean on it. Time-from-merge-to-production under a fully automated pipeline is another, since the only real way to move it down is to improve the pipeline, which is the work you wanted anyway.

Judge the work

The field tolerated lines of code for decades not because anyone defended it, but because the alternatives were process-heavy: tickets, points, burndowns, and time-on-task all cost real hours to maintain and got gamed anyway. Counting was the only affordable instrument, so the field used it long past its expiry date.

That constraint has lifted. Models can now read a pull request, look at the diff, and produce a judgment of substance with reasonable consistency. The Goodhart problem was historically a counting problem because counting was cheap and judging was expensive. Judging is now cheap.

I built a tool on this premise, called Complexity Weighted Throughput, or CWT. It reads the diff of every pull request across every repo and scores each PR on substance rather than size. Summed per developer it gives an individual signal, and sliced by platform or org it gives a team one. Both CWT and lines of code derive from the same artefact, the code, but CWT judges the change instead of counting it. Writing more lines is cheap, while writing a sequence of pull requests an analyzer consistently rates as substantive is harder to fake, because the cheapest way to score well is to do substantive work.

I do not know whether CWT is un-gameable or just not-yet-gamed. The right assumption about any judged metric is the same one you make about any piece of code: it will need hardening, and the team will adapt to it in ways I did not anticipate. I am not claiming the metric cannot be gamed, only that the design space has opened up.

The same reframe applies to delivery-framework scorecards. A team can post improving deploy frequency and lead time for a quarter while quietly destroying itself, because counting events does not catch the cost of producing them. A judged signal (did this on-call rotation leave somebody usable on Monday) is more expensive to compute, but the cost is now within reach.

Report at the team level

Individual rollups exist for two purposes: the one-on-one conversation between a manager and the person, and the leader's own pattern-matching. They do not belong on leadership dashboards. Individual productivity dashboards have been attempted for forty years, most persistently by the people who most wanted them to work, and they do not survive contact with humans. Treat the team as the unit of measurement until you have a specific reason not to, and let the individual signal stay what it is, one input to a manager's judgment rather than a number the org publishes.

Audit the metric like code

Gameability is not a property a metric has once. It is a property the metric has against the current behaviour of the team, and team behaviour adapts, so a metric that was honest in Q1 can be theatre by Q3 with no one noticing, because the adaptation is gradual and the number stays in roughly the same range the whole time.

Re-run the cheapest-path test every quarter or two against the current state of the team, and if the cheapest path has migrated somewhere unhealthy, redesign the metric or drop it. A metric is a piece of infrastructure like any other, and it has a maintenance schedule.

When not to instrument

Some domains carry a cost of getting the design wrong that exceeds the value of the signal:

• Anything that becomes an individual performance number.
• Anything that drives compensation or ranking.
• Anything where the act of measuring changes the behaviour you were trying to observe.

The strongest move in those domains is to refuse the instrument and lean on judgment from the people closest to the work. The next strongest is to keep the instrument private to the manager and never publish it.

The AI-era replay of lines of code is already running. Acceptance rate of AI-generated suggestions is being reported as impact, with about the seriousness commit counts had twenty years ago, and it fails the cheapest-path test in the first minute: the cheapest way to move acceptance rate up is to accept more suggestions and accept them faster, and accepting a suggestion is not shipping working code, any more than accepting quickly is understanding what was accepted.