How do you measure whether AI is actually helping your engineering team?

Why “developers say they are faster” is not a measurement

In Microsoft’s 2026 survey of 20,000 workers, 66% of AI users said the tools let them spend more time on high-value work and 58% said they were producing work they could not a year ago. Those are real feelings and they matter for retention. They are not evidence that the software shipped faster or broke less. Vendors quote them because they are easy to collect. You should collect something harder.

The one large field study of an AI assistant, at a Fortune 500 support organization, did not ask agents how they felt. It counted issues resolved per hour, before and after, for about 5,000 people. That is the standard: a unit of output, measured the same way on both sides of the rollout.

The six metrics

The four you may already have. DORA’s metrics are the closest thing software delivery has to an accepted standard, and they are measured at the level of the system rather than the person, which is where AI’s effect actually shows up.

1. Deployment frequency. How often changes reach production.
2. Lead time for changes. Commit to running in production.
3. Change failure rate. The share of deployments that cause an incident or a rollback.
4. Time to restore. How long a failure takes to recover from.

AI that helps moves the first two down and holds the last two. AI that is generating plausible code faster than anyone can review it moves the first two down and the last two up, and the second pattern looks like success for about two months.

The two you need to add.

5. Revert rate of AI-authored changes. Mark AI-assisted commits (the engineering appendix of your acceptable-use policy should already require it) and track how often they are reverted or hot-fixed compared with the rest. This is the earliest warning you will get.
6. Reviewer time per change. If AI makes authoring faster and review slower, the bottleneck has moved and the team’s throughput has not changed. Most teams discover this by feel around month three. Measure it from week one.

Set the baseline first

None of this works without a “before.” Pull ninety days of data on all six metrics before the tools are rolled out, or before they are rolled out to the next team. Where the tools are already in use, take the baseline from a team that does not have them yet and roll out in stages, which is exactly how the support-agent study got a clean answer.

Report at the team level. Individual productivity measurement through AI telemetry is the fastest way to lose the trust the rollout depends on, and it measures the wrong thing anyway.

What good looks like after a quarter

• Lead time down by a measurable fraction, not a rounding error.
• Change failure rate flat or better.
• Revert rate for AI-assisted changes no worse than for the rest.
• Reviewer time per change flat, or falling because the harness catches what reviewers used to catch.
• Engineers who say the tools help, which now means something because it agrees with the numbers.

What to do if the numbers go the wrong way

Do not remove the tools. Add the gates. A revert rate that climbs is a review problem and a policy problem: which changes require a human reviewer, what the tests have to cover before merge, and what the agent is allowed to touch on its own. That is a harness, and building one is the whole point of engineering enablement.