Measuring an Agile Product Owner

I went to a meeting last week and we were discussing how to measure a Product Owner.  The concept is completely understandable, but there was something about the meeting that didn’t sit well with me so I am devoting this blog to trying articulate my concerns and things that I think need to be considered.

The premise is sound  – how do you know if the Product Owner is leading you down the right path?  What if you build the wrong thing?  And how can you assess their mis-direction before precious time and resources are spent?

That is a 100% valid question and one that we should certainly spend time and thought to resolve.  The idea that there are ratio-driven metrics to measure the PO’s performance, though, seems counter-productive.


Image Source:

First, I think Product Owners have a very difficult job and I think very few people are naturally great Product Owners, if you go by the broad definition.  Product Owners are supposed to understand the market place, stay on top of the competition, interview current and prospective clients, and break down all of that data into meaningful features that will be delivered in order to drive the most business value.  They are also supposed to be experts at writing user stories, adding acceptance criteria, understanding at least some level of the technical details associated with the application, and they need to be decisive, informed and immediately available for the Scrum Team.  That is a tall order.  In fact, I recommend splitting the Product Owner’s responsibilities between the Product Owner and a Product Manager, as I referenced in a previous blog.

The metrics that were discussed in the meeting to measure the product owner were taken from a manufacturing hand-book that doesn’t seem as applicable in an Agile workplace.  It was measurements like:

  • Number of user stories that need to be re-written over number of good user stories to come up with a ratio of bad stories
  • Number of stories without acceptance criteria over all user stories
  • Amount of business value to be derived from the user story over the cost of implementing the user story

While I completely understand the intent behind these metrics, the message that it sends scares me.

Based on my experience, what makes a Scrum team effective is the team work.  In the most productive and fun efforts in my career, I was not expected to be perfect, nor did I expect that of my colleagues.  We were all expected to show up with informed opinions and ideas and work together towards an optimal solution – and we had the latitude to err and learn along the way.  We operated as a team and we would succeed or fail together.

The idea of detailed metrics for one member of the team, especially a member with job description so broad that it is almost unattainable, has the potential to divide the team and empower certain members to point fingers at other members.  That seems to defeat the whole value of a mutually-invested team.

If we need to measure a Product Owner, great.  Let’s find common metrics around the effectiveness of the TEAM that will apply to the PO and the Scrum Master.  If we find that we have a PO who is not succeeding, the best teams will not refer back to metrics to prove their point.  They will roll up their sleeves and figure out how to help.  Only after we have exhausted efforts to learn and grow should we pull out the measuring stick and beat the PO into a different job. If we create an environment of judgment and blame, we will have an even harder time attracting people to these difficult jobs.  If we create an environment of accountability, learning and collaboration, we will build great products with highly empowered people and have a ton of fun along the way.

That’s my thought, at least.  I would love to hear yours.


2 Responses to Measuring an Agile Product Owner

  1. I agree with you for the most part here. I have an issue with assuming one person is capable of juggling all of that hand making it work well. To me, measuring the product owner (if you measure them at all) should be about measuring the value delivered to production. Nothing more. A product owner’s role is to bring customer value, not to ensure he gets acceptance criteria correct the first time. I thought most of the metrics that were discussed equated to “Vanity Metrics” (to use Lean Startup terminology). They may be great to talk about, but do they really provide value to the business at the end of the day?


  2. Chris Sutton says:

    Brandon, your comparison of these types of metrics to vanity metrics is really apt. Ultimately as product owners, your success is tied directly to the success of the product. You can write perfectly valid user stories with perfect acceptance criteria, but that doesn’t matter if the features don’t matter to your customers.

    What some of these metrics seem to be getting at though is how your work affects the rest of the team. In that case, instead of using a proxy measurement technique, which assumes a “bad” user story has a negative affect on the team, you should simply go straight to the team for direct feedback. 360 feedback evaluations can be difficult, but will offer much more insight than making up some metrics.

Leave a Reply to Brandon Carlson Cancel reply

Your email address will not be published.