Tying engineering to user and business outcomes

Most of what your team ships will not work.

That sounds harsh, so look at the numbers from teams with enough traffic to know. When Microsoft runs a controlled experiment on a feature, roughly a third of the time the metric moves the way they hoped, a third of the time it does nothing, and a third of the time it moves the wrong way. Google and Bing report that ten to twenty percent of their experiments produce a positive result. At an early-stage startup, where you have less data and less discipline, the honest hit rate for shipped features is closer to one in five. So for every five things your team builds, four are roughly neutral or worse on the outcome they were meant to change.

Now sit with what that means if you are not measuring. You shipped five features, you and the team feel productive, and the burndown looks great, but one of those five moved the world and four did not, and you cannot tell which one. You cannot pour more into the winner because you do not know it is the winner, and you cannot kill the losers because from the inside they look exactly like the winner. At that point you are running a feature factory with the lights off, whatever the burndown says.

I have lived the version of this that is slower and more embarrassing: a whole quarter where we shipped constantly, demoed well, closed tickets, and a month after the quarter ended someone asked what any of it had done for the user, and the room went quiet. Nobody had wired up the answer. The work was real, but the line from it to anything that mattered was missing.

If you have read anything about product management in the last decade, you already know the slogan: outcomes over output. Josh Seiden's framing is the clean one. An outcome is a change in human behavior that drives a business result, and a shipped feature is only a guess that such a change will follow, so the team should be handed the outcome to chase rather than the solution to build. Most engineers half-know this by now. The trouble is that the writing almost always stops at the slogan: it tells you to care about outcomes and then leaves you standing in front of a merged pull request with no idea how to connect it to a behavior change three steps downstream. This essay is about that plumbing, the engineering-side work of building line-of-sight from a change to a result, done honestly enough that you can trust what it tells you.

And the plumbing is engineering's own work to do. A product manager can want an outcome, but only the people who wrote the change can make it emit the signal that proves it moved one, so the measurement is part of the deliverable, built into the change rather than written up afterward. A feature is not done when it merges. It is done when it can tell you whether it changed anything.

One more thing has changed, and it is why this matters more now than it did five years ago. Writing code is becoming nearly free, and a team that cannot tell whether its output worked is hard to distinguish from a fast generator of plausible noise. Hold that thought, because it reshapes everything below, and I will come back to it at the end.

Here is the plumbing.

Pick the rung you can actually move

Between a merged change and a business result there is a ladder. From the bottom up:

1. The feature shipped.
2. Adoption: whether anyone uses it.
3. A leading behavioral proxy: some action that tends to predict the result you care about.
4. The lagging business metric: retention, revenue, or cost.

The instinct is to claim the top rung, since the business metric is the version that sounds like impact when the project gets announced. "This feature will improve retention." Maybe it will, but retention moves for a dozen reasons over months, and your feature is one small input buried in the noise. You will never see your signal up there. So pick the highest rung you can both influence and instrument, and then say out loud how many rungs sit between your code and the result you ultimately care about. If your feature is meant to lift retention but the only thing you can cleanly measure is whether people complete the new flow, then completion is your outcome for now, stated together with the two rungs of inference that still sit between completion and retention. A claimed link to revenue that you cannot see tells you nothing when the next decision comes around, and a real link to completion at least tells you something you can act on.

Pre-register what counts as working

Decide what "it worked" means before you build rather than after you ship.

This is the single cheapest discipline in the whole practice and the one teams skip most. Before the work starts, write down the behavior you expect to change and the threshold that would count as success. "We expect completion of the onboarding flow to rise from sixty percent to at least seventy-five within three weeks." That sentence costs you ten minutes, and it kills the most common failure in outcome measurement, the reverse-engineered win. Without it, whatever number you observe afterward gets narrated as a success, because the human writing the narrative is the human who shipped the thing, and nobody audits a story that flatters everyone in it. With it, the number either cleared the bar you set in advance or it did not, and you no longer get to move the goalposts. The same discipline applies to reliability work: decide the reliability target before the quarter starts rather than reconstructing one after an incident. This pairs with defining done, where "outcome achieved" is part of the definition of done; here we are making that claim verifiable.

Buy the cheapest honest counterfactual you can afford

The hard question under every outcome claim is: compared to what? Completion rose to seventy-five percent, but would it have risen anyway? This is where attribution gets genuinely difficult, and I would rather concede that up front than wave it off. Outcomes have many causes, and a correlation on its own settles none of them. The honest answer to "did our work cause this" is usually "partly, probably, and I cannot fully separate it from the three other things that changed that month."

So buy the best counterfactual your stakes can justify. A staged rollout or a holdout group is the gold standard, because the users who did not get the change tell you what would have happened anyway. When you cannot run even a crude experiment, a before-and-after comparison can work, as long as you name the confounders out loud rather than pretending they do not exist. And when you have nothing but a correlation, say so, and downgrade your confidence accordingly. Match the rigor to the stakes: a small UI tweak does not need a randomized trial, but a six-month platform bet that consumes a third of your team had better come with something more than a hopeful line on a graph. The skeptics who call attribution theater are right more often than measurement enthusiasts like to admit, so claim only as much as your counterfactual supports and keep measuring anyway.

Make reliability's value survive peacetime

Reliability work has a cruel property: it is obviously valuable in wartime and invisible in peacetime. Right after an outage, when everyone has just felt the pain, nobody questions the investment. Six calm months later, when nothing has broken, that same investment looks like it is guarding against nothing, and it is the first thing cut, because the budget conversation pits features everyone can see against a non-event nobody can. This is the prevention paradox, and it is where reliability-minded engineering leaders quietly lose budget.

Make the value legible when no incident is fresh. Three moves do most of the work:

1. Hold a standing churn-cohort ratio. Measure it from your own data: users who hit an error or a slow page or an outage churn at some multiple of the rate of users who do not. Say it comes out to three times. Once you have that ratio, reliability has a permanent, quantified tie to retention that does not need a fresh outage to justify it.
2. Carry the last incident's cost as a unit price. When an incident happens, compute its true cost once, including lost revenue, support load, and accounts put at risk, then carry that figure into peacetime planning as the standing unit price of that failure mode, so the cost is quoted rather than re-argued every quarter.
3. Log near-misses as visible outcomes. Incidents caught before users noticed, degradations that auto-recovered, the error budget you burned but contained.

This is attribution by subtraction, and it generalizes to every infra and platform team whose real output is things that did not happen and hours that were not lost. Framing reliability as a product feature is its own essay; what I am after here is the attribution mechanics.

Engineer the time-to-truth

How long it takes to find out whether something worked is a design variable, and you can shorten it on purpose.

Instrument the outcome before you launch, as part of the change itself, so the clock starts at release instead of whenever someone finally remembers to add tracking. Where you have a choice, prefer outcomes that move in weeks over outcomes that move in quarters, because a faster loop lets you correct course while the work is still warm. Ship a thin slice to a small set of users to get an early read before you commit the whole team to the full build. Each of these shortens the gap between "we shipped" and "we know," and inside that gap you are steering on guesses.

Speed is not the only thing that breaks the loop. The outcome itself has to hold still long enough to read. When I first moved teams onto outcomes, the outcomes kept moving underneath them: one month the target was revenue per user, the next month pricing became the priority and the goal shifted before any signal had come back. You cannot measure against something that changes faster than your time-to-truth. So we started committing each outcome for a mid-term horizon, three to six months, long enough for the loop to close at least once before anyone was allowed to move the target. Below that horizon the team stops steering toward anything and starts reacting to whatever is loudest that week.

There is a real boundary here too. Some outcomes genuinely cannot be known inside a quarter, and pretending otherwise produces worse decisions than admitting it. When the outcome is truly unknowable on your planning horizon, that is a different problem with its own discipline, the territory of long-horizon bets.

The outcome signal is what aims the agents

Now back to the thing I asked you to hold.

When code was expensive, measurement could be treated as a reporting chore, a thing you did at the end to write the quarterly review. That era is ending: as agents drive the cost of generating code toward zero, the bottleneck moves from production to validation and comprehension, because an agent can only aim at what you can measure for it. An agent handed a tight outcome loop improves the result, and an agent handed nothing optimizes for looking done, since looking done is the only signal you left it. Once that is true, tightening the loop buys you more than generating more code does.

Seen that way, the line-of-sight you build from change to outcome is not a scorecard you show your manager but the steering input for the entire system, human and machine. A team with a tight outcome loop can point cheap, fast output at the things that move the world and turn away from the things that do not. A team with no line-of-sight loses more than a scorecard: it starts manufacturing reviewed, well-tested work that nobody can confirm ever mattered, at machine speed, and measurement stops being a report about the quarter and becomes the input that decides what the team builds next. Who answers for the outcome once you can see it is a separate question, and it belongs to owning the outcome; it only opens up once the wiring in this essay exists.