Verified Author Share
The trap isn’t AI. It’s the measurement system you inherited from the past, one that stopped making sense the moment your team started using its first AI tool.
Most engineering organizations are already adopting AI. Developers use GitHub Copilot, ChatGPT, Cursor, Claude Code, and many others. Analysts generate code, tests, and documentation in a fraction of the time. Delivery is accelerating.
Then sprint review arrives.
The manager looks at the dashboard, frowns, and says:
“Our Story Points dropped this month.”
That’s where the problem begins.
This article is about a silent trap affecting countless “agile” teams today, and how to escape it by measuring what actually matters.
Story Points were created with a very specific purpose: to estimate the relative complexity of work, not the effort or time required to complete it.
When a team estimates a User Story as an 8, they’re saying it’s more complex than a 3, not that it will take a specific number of hours or days.
That distinction has always mattered.
With AI, it becomes critical.
An 8-point story is still an 8-point story in terms of complexity. What changed is how long it takes to solve it. Those are completely different concepts.
Here’s what typically happens.
The team starts using AI.
Tasks become faster to complete.
During the next refinement session, estimates begin to shrink because people unconsciously start estimating based on perceived effort instead of inherent complexity. A story that used to be estimated as an 8 suddenly becomes a 3.
Velocity drops dramatically.
Leadership sees the chart and wonders:
“Is the team slowing down?”
In reality, the exact opposite happened.
The team became faster.
The measuring stick changed.
As AI increases delivery capacity, teams naturally recalibrate their estimates downward.
Velocity falls precisely when actual productivity rises.
Managing performance through velocity in this context is like measuring a car’s speed by looking in the rearview mirror.
| Question | Can Story Points Answer It? | Can Flow Metrics Answer It? |
|---|---|---|
| How much work did the team deliver? | Not directly | ✓ Throughput |
| How long does work take to complete? | No | ✓ Lead Time |
| Did AI make the team faster? | Often misleading | ✓ Throughput + Lead Time |
| How much work is currently in progress? | No | ✓ Work in Progress (WIP) |
| Where are the bottlenecks? | No | ✓ Lead Time Breakdown |
| Is waiting time becoming a problem? | No | ✓ Flow Efficiency |
| Is this piece of work more complex than another? | ✓ Yes. That’s exactly its purpose. | Not intended for that |
Story Points aren’t useless.
They’re still valuable for relative estimation and sprint planning.
The mistake is using the total number of Story Points completed as a productivity metric. That was never their intended purpose, and AI makes the distortion even greater.
Flow Metrics, widely adopted through Kanban and used by many high-performing engineering organizations, answer exactly the questions that matter in an AI-enabled environment.
The number of completed work items over a given period, whether that’s a sprint or a week.
Regardless of complexity, the question is simple:
How many valuable items did we deliver?
The elapsed time between a work item entering the system and being delivered.
When AI improves productivity, Lead Time should decrease.
That’s where the real productivity gains become visible.
Of course, speed alone isn’t enough. Delivery quality still needs to be measured. Faster development combined with lower quality simply creates more rework and larger problems later.
The number of items being worked on simultaneously.
According to Little’s Law, limiting WIP reduces Lead Time proportionally.
More capacity doesn’t mean more parallel work.
It means finishing work faster.
The percentage of total Lead Time spent doing actual work versus waiting.
Average teams often operate between 15% and 30%.
When AI is applied effectively, this percentage should increase.
You don’t need to abandon Story Points overnight.
You do need to stop treating them as a productivity metric and begin using Flow Metrics as your primary indicators.
Here’s a practical transition path.
1. Start Tracking Throughput Today: Record the date each work item is committed and the date it’s delivered. You don’t need sophisticated tools. A spreadsheet is enough. The important part is building historical data.
2. Categorize Work Items: Separate work into categories such as User Stories, Bugs, Technical Debt, Spikes. Looking at Throughput by work type reveals far more than total volume alone. If bugs dominate delivery, your “productivity” may not represent actual business value.
3. Measure Lead Time by Work Type: After three or four sprints, you’ll already have meaningful distributions. Focus on: P50 (Median), P85, If AI is creating value, the median Lead Time for User Stories should decrease. If it doesn’t, your bottleneck probably isn’t software development.
4. Establish a Before-and-After Baseline: If your AI adoption has already started, identify the transition point. Compare: Average Throughput, Average Lead Time, Variability. Those three numbers tell a much more complete story than Velocity ever could.
5. Protect WIP During the Transition: Communicate clearly with leadership: More capacity does not mean more parallel work. It means existing work finishes faster. Maintain WIP limits. Let Lead Time fall. That’s the improvement your customers actually experience.
Use this checklist to understand what’s happening inside your team.
There’s a natural adjustment period whenever AI enters the workflow.
Teams need several weeks to recalibrate their estimation habits.
During that period, total Story Points become highly unreliable as a productivity indicator.
That’s exactly when you should focus on Throughput and Lead Time.
Those metrics remain trustworthy regardless of how the team estimates work.
“There is no bad Kanban implementation. Only practices that are inappropriate for the organization’s current context.”
Kanban Maturity Model, Kanban University
The same principle applies here.
There’s no universally “wrong” productivity metric.
There are metrics that no longer fit the environment.
And the environment has changed.
One important nuance often gets overlooked.
AI doesn’t accelerate every type of work equally.
Well-defined coding tasks with established patterns often see dramatic gains.
Architectural decisions, domain discovery, stakeholder alignment, and strategic problem solving still depend heavily on human judgment.
AI helps.
It doesn’t eliminate the complexity.
That means Lead Time improvements will vary by work type.
A straightforward CRUD feature may see Lead Time reduced by 60%.
A complex legacy modernization effort may improve only marginally.
Segment your Lead Time scatterplots by both:
You’ll quickly discover where AI is creating the most value and where the real bottlenecks lie, often in reviews, approvals, or deployment rather than coding itself.
Changing metrics isn’t only a technical discussion.
It’s also a management conversation.
Leaders accustomed to Velocity need a clear narrative.
The explanation that tends to resonate is simple:
“AI is working. Our team is delivering more work, and delivering it faster. Velocity decreased because Story Points measure relative complexity, and the team naturally recalibrated estimates once effort changed.
To understand AI’s real impact, we should measure Throughput (how much we deliver), Lead Time (how quickly we deliver it), and WIP (how much work is in progress). Those metrics reflect actual operational performance.”
Show the before-and-after data.
Average Throughput.
Median Lead Time.
The numbers are usually more convincing than any theoretical explanation.
AI isn’t simply another development tool.
It changes the way engineering work gets done.
And whenever the operating model changes, the measurement model must evolve with it.
Story Points still have value.
Velocity, however, has become an increasingly unreliable proxy for productivity, arguably even before AI entered the picture.
What replaces it isn’t more complicated.
It’s simply more aligned with what actually matters:
How much value is delivered, how quickly it’s delivered, and how predictable the delivery process has become.
Throughput, Lead Time, and WIP answer those questions directly.
They don’t depend on estimation.
They don’t depend on individual developers.
They depend only on what successfully flows through the system, with AI acting as an accelerator rather than a source of measurement distortion.
The organizations that make this transition first will gain a significant advantage.
They’ll be able to demonstrate AI’s impact with evidence instead of intuition.
And they’ll identify the parts of the delivery process where AI still isn’t helping, which is often where the biggest opportunities for improvement remain.
Verified AuthorSenior Principal Agile Leader at Zallpy, with solid experience in defining and evolving scalable, sustainable agile models focused on value generation. He currently serves as Senior Principal Agile Leader at Zallpy, leading the Agility practice within the Center of Excellence, where he defines the strategic vision, governs operating models, optimizes value streams, and establishes performance and predictability metrics. Recognized as a technical and cultural reference, he acts as a mentor and coach to leaders, Agile Masters, and Scrum Masters, supporting large-scale transformations, organizational maturity development, and the consolidation of a culture of continuous improvement, systemic collaboration, and business outcome focus.
Senior Principal Agile Leader at Zallpy, with solid experience in defining and evolving scalable, sustainable agile models focused on value generation. He currently serves as Senior Principal Agile Leader at Zallpy, leading the Agility practice within the Center of Excellence, where he defines the strategic vision, governs operating models, optimizes value streams, and establishes performance and predictability metrics. Recognized as a technical and cultural reference, he acts as a mentor and coach to leaders, Agile Masters, and Scrum Masters, supporting large-scale transformations, organizational maturity development, and the consolidation of a culture of continuous improvement, systemic collaboration, and business outcome focus.
Verified Author
Verified Author
Verified Author