I’ve been struggling for quite some time figuring out the “essence of” agile metrics. What are good ones versus bad ones? Are there related sets that are relevant and how many are needed? And how much change do we need to make from our traditional metrics thinking towards agile metrics?

One thing I have discovered is that metrics need to be “different” in an agile organization or culture. Traditional measures don’t seem to work within agile teams—as I’ve often seen them to cause dysfunction (more on that later).

Another thought I’ve had is whether they should represent a trailing indicator or a leading indicator? Meaning should we measure the processes (at the beginning or leading edge) or should we more so focus on the results & learning (at the end or from the trailing edge)? I’ll explore this more next.

Leading vs. Trailing Indicators

I think of measuring estimate accuracy (estimate + actual) as a leading indicator. Why is that since you measure actuals at the end of your work?

I’m thinking of it from the point of view of what you’re trying to improve. In this case, you’re measuring estimates and actuals to improve your overall planning and estimation processes. These activities are typically front-loaded actions and are often static…meaning they are completed once at the beginning of the project.

The other challenge with this metric is taking corrective action for improvement. Effectively you can only effect a change with the next planning interval for a project. Depending on the length of that interval, improvements could be infrequent and delayed. Or depending on project success, they might not be useful at all.

So I’ve seen some agile teams that try to measure various things. Let’s take a typical example—How about we measure user story volatility in the sprint?

What we’re trying to focus on is how many times do stories change (get added, become reframed or reduced in scope, or removed entirely) within any particular sprint. To what end? Well, we’d like to improve our story grooming process and monitor how the team is delivering on their sprint planning commitments.

Ah, there’s a key – sprint planning is sort of a leading indicator, in this case you’re measuring the quality of story entry. What if we turn this into a trailing, result-oriented, measurement? Let’s measure the velocity of the team as they exit the sprint. We can measure velocity in stories produced and in story points delivered.

An even better trailing metric is to measure how the team delivered value towards their sprint goals. Meaning, did they deliver what they committed to? Not in tasks or in stories, but in content that exactly met the spirit of their customers’ needs and the level of their commitment. And did they deliver the highest priority (highest value) stories first—with lower risk and greater visibility and collaboration?

You notice that I reframed the notion of a trailing metric to a result-oriented metric. Another way of thinking about it is measuring at the “outbound side” of the teams work. That way, you’re leaving the doing of the work up to the team and simply measuring results or outcomes.

Let me provide an example of why this might be better. Let’s go back to the example I gave above. What is wrong with measuring story volatility?

On the surface it appears fine. We can see how well the Product Owner staged the sprint and also how well the team is interpreting stories for execution. For example, if they “signed up” for 10 stories, but only delivered 7 because of story (requirement) volatility, then they have a lot of improvement to do…right? Well, that could be the case.

But it could also be the case that the team trimmed those stories because they were superfluous to the customers’ needs. Meaning they were essentially waste. Or another positive interpretation of it could be the team became very creative and figured out a way to meet the spirit of the sprint goal with fewer stories. In fact, they may have even trimmed out some previous code in order to simply the application—so the overall related LOC count might be quite low too.

Can you see the point I’m making? I find that leading indicator metrics can sometimes be more prone to metrics dysfunction than trailing, results oriented metrics.

In his book, Measuring and Managing Performance in Organizations, Robert Austin coined and explored the term metrics dysfunction. In simple terms, it’s a metric whose collection influences the very behavior being measured—leading to metrics dysfunction. One of the classic examples of it is measuring software testers by the number of bugs they find. In this case, testers would report more trivial, non-germane bugs simply to hit the metric...and in the meantime they’d miss truly important bugs. Clearly not what the measurer intended!

In our example the leading metrics might influence the team towards not changing stories that need to be changed—simply to hit the metric of stories being consistent with the views from planning. I can see nothing more destructive to the agile mindset of the team than having metrics that drive negative behavior within the team—simply to meet or game the metrics.

So, what are some potentially unhealthy leading metrics to avoid within agile teams?

  • Planning quality – measuring how well you planned. An important part of this is change control, i.e. measuring how little or much change is ongoing.
  • Requirement quality – measuring whether each requirements meets some sort of baseline definition or completeness. Or that each has a specific number of acceptance tests.
  • Estimation quality – I’ve sort of beaten this down in the article—effectively anything that tries to measure estimation variance with actual effort.
  • Arbitrary results – Counting LOC produced, or Bugs found, or Requirements written—virtually anything that is materially produced by the team where you negate the quality of the result and the application of common sense in that not all results need the same level of attention and thinking.

Conversely, what are some potentially healthier trailing metrics to concentrate on within agile teams?

  • Team agitation levels – capturing multi-tasking events, # of committed hours per sprint and variance from expected levels, unexpected recruiting support for team members, training interruptions, etc.
  • Team velocity levels – trending over time, improvement trending (implying improved team dynamics), paying attention when team composition changes. Compare velocity trending relative to agitation levels above.
  • Impediment handling – #’s per team, avg. time to resolve, # that impacted the Sprint Goals. Are we removing the most critical obstacles as aggressively as possible for the team?
  • Retrospective actions – is the team improving themselves based on retrospective results, how many improvements per sprint, average time to resolve. Are they seeing the expected results from the changes?
  • Escapes per sprint – bugs found post-sprint, adherence levels to Done-ness Criteria and identifying and correcting reasons for not supporting complete done-ness.
  • Sprint Success / Failure – from the perspective of the Sprint Goal. Not so much focused at a Story or Task completeness level, but at an overall work delivered level

One of the things you’ll notice with trailing indicator metrics is that there is an inherent real-time nature to them. We want to be sampling them (observing, discussing, driving action) each and every day during each iteration or sprint.

I believe it’s this level of team awareness of their performance (metrics) and active engagement towards continuous improvement that is the key to healthy metrics in an agile context.

 

Agile Metric Categories

Another healthy discussion is what type of metrics to collect from an agile team. I’ve seen many examples that recommend the following areas as places to focus your attention:

  1. Quality Output
  2. Team Health
  3. Value Produced
  4. Throughput or Capacity

Having these categories in mind when your setting up “what to measure” within your agile teams gives you a broader and healthier base for your observations and thinking.

To give you a real world example, when I was at iContact for a time we measured the following in our agile team KPI’s (key performance indicators)—

  • # of Sprint Escapes per team, both defect escapes and process escapes—for example times when the team ignored our Done-Ness Criteria. (Quality)
  • Story cycle time / WIP; we created an aggregated metric that accommodated the different sizes of stories and examined maintaining a healthy (swarming) WIP within our teams.  (Throughput)
  • Sprint value delivery in stories / features delivered to our release plan. Our Product Owners maintained this metric—reporting and  (Value Produced)
  • Interrupt rates or churn within our teams. While we were a mature agile organization, we did suffer from excessive team member movement and  (Team Health)

We didn’t care about individual data points. Instead, we were much more interested in trending over time—either sprint by sprint or across our release tempo. We were looking for gradual but sustained improvement.

If we didn’t see that improvement, we didn’t penalize or “fire” anyone. No, instead we simply looked for root cause…asking- why was this trend happening? We were curious and always looking for ways to improve.

That, or whether the metric was indeed useful and whether we needed to change it to something better that would drive our insights and improvement.

Wrapping Up

 

You may be asking yourself why is he sharing metrics in a column focused on the Agile Project Manager? Good question.

While applying traditional metrics to agile teams can drive dysfunctional behavior, agile teams still need measures that guide their improvement. They also need balanced leadership that can protect them from overreacting to short term interpretation. This is where a seasoned Agile Project Manager can really make a difference in an agile environment.

In helping the team select a small, core set of critical measures. Also helping them to pay attention to the metrics trending and to inspect, adapt and improve based on what the data is showing them. Consider yourself the dashboard for the team that keeps them on the road, at the speed limit, and ticket free.

Thanks for listening,

Bob.