Most teams ship a feature, watch the KPI they defined for it, and call it a win when the number moves. That instinct is reasonable, but it leaves a gap. A KPI shows whether the one number you chose moved. Product analytics looks at how users behave across the whole flow, so it can also show what happened around that number, including things no one thought to ask about. We saw this with one of our clients, a B2B company offering online services to retail sites.
The change
The team redesigned part of onboarding: they simplified the first onboarding step by moving a few required fields to the next step, on the theory that a lighter step would reduce drop-off and get more users to the point of paying.
The KPI was clear: improve completion of that specific step, and improve the overall ratio of leads that ended in a completed purchase. Working with us, the team instrumented it properly in Amplitude: every onboarding step tracked as an event, from first touch through completed purchase, alongside full activity tracking across the site.
The KPI said it worked, but something didn’t add up
After rollout, completion of the target step rose from 74% to 92%. By the metric the team set out to move, the feature was a success. But the overall lead-to-purchase conversion rate barely moved, from 24.4% to 24.1%. If the redesigned step was converting meaningfully better, that improvement should have shown up downstream too. It didn’t. A narrower analysis, watching only the target KPI, would have missed that gap or explained it away. The chart shows where it went: the extra users who got through the first step dropped off at the very next one.
Digging past the KPI
Because the instrumentation covered full user activity, not just the funnel step in question, we could go further than the top-line numbers. Rather than having someone watch session recordings one by one, Amplitude’s AI agent reviewed the session replays of users who dropped off after the redesigned step and flagged a pattern: a cluster of users hitting the same failure. That pointed us straight to the cause: one of the fields moved to the later stage had a validation error that only triggered on specific devices. It stopped those users from getting past that step, and so from reaching purchase, with no clear sign of what had gone wrong.
Why this mattered more than the feature itself
Fixing that bug had a bigger revenue impact than the onboarding redesign. Had the rollout only been judged against the step it set out to improve, it would have been marked a success while the bug kept costing purchases on affected devices, undetected.
What to instrument so you’d catch this
None of this needs an unusual setup, just a few habits that are easy to skip:
- Every step as an event. Not only the step you’re changing. Consistent event names let you compare before and after.
- The end of the funnel, not just the target step. Next to the step’s KPI, track a guardrail metric such as lead-to-purchase conversion. If the step improves and the guardrail doesn’t, that gap is your signal.
- Device, browser and operating system on every event. That way you can split a drop-off and see whether it’s concentrated in one group.
- Errors as events. Validation errors and failed form submissions are product data, not just log lines.
- Session replay on critical flows. Let AI review it, so anomalies surface without anyone having to go looking.
Where a tool like Amplitude helps
Two things made this findable in Amplitude. The instrumentation covered every step and the full user activity, not only the target step. And its AI agent reviewed the session replays, so nobody had to watch them one by one.
Not sure what your product data is hiding? See how we set up product analytics.