Output Metrics vs Outcome Metrics

In 1961, a guy called Robert McNamara entered the US Department of Defense, rich in his belief in quantitative analysis. Robert McNamara previously worked at the Ford factory and while he was there he really pioneered the use of data and systems analysis to drive efficiency in the production of cars. And he took that belief in data and analysis to his new job in the Department of Defense in the 1960s and he joined the Vietnam War. Now when he was put in the charge of the war he applied the same approach that he had learned at Ford. So he turned up to Vietnam and he started measuring things like logistics networks, which areas of the map were under US control, troop numbers, all that kind of stuff. Robert McNamara declared quite confidently that the war was going well and that we were going to win. Now spoiler alert, the US did not win the Vietnam War and one of the reasons was that the data that McNamara had available was basically just that, it was the data that he had available and what he ended up doing is ignoring all of the more important stuff like the morale, the sentiment of the Vietnamese people, all the kind of stuff that you can't really measure and that gave rise to something called the McNamara fallacy. The McNamara fallacy is something that a lot of organizations can fall into where if you over index on things that are easy to measure then you end up ignoring the things that are difficult to measure just because they're difficult to measure. (Music) Hi, my name is James Charlesworth and this is the Product Engineering podcast where we talk about the intersection between product management and engineering and today we're going to talk about measuring. How do you measure a product engineering team? Because there's different metrics that you can use and they're often split into output metrics and outcome metrics and as the McNamara fallacy shows us, output metrics are often easier to measure but they don't always tell the full story and if you only measure the things that are easy to measure then you will miss the bigger picture and you'll miss what's actually more difficult which is the outcome, the product outcomes you're trying to drive or the business outcomes that you're trying to drive and we fall into this trap because output metrics often feel good and they're often very very available in the tools that we already use. So if you're an engineer or an engineer manager then the tools you use every day can give you data and they can give you data that can cause you to fall into the McNamara fallacy. So for example, let's say you're using Jira as your project management tool. Jira can tell you things like cycle time, it can tell you burn down, velocity, story points, all that kind of stuff. GitHub can tell you how many pull requests are going in, how long they're staying in review how many lines of code, people love that when you measure lines of code. Those kinds of metrics are very very easily available to engineering leaders and they can lead you into the trap of thinking that just because those things are easy to measure that's what you should be indexing yourself on and it's often ignoring things that are more difficult to measure. So let's quickly talk about the difference between output metric and an outcome metric. An output metric is measuring how efficiently or how well you're producing things. So you know lines of code even though people hate that as a metric it is a fairly pure output metric because lines of code will tell you like how much code is being shipped to production. Door metrics will tell you how successful your changes are, how regularly they're going out and how often they need to be rolled back. Story points will tell you how effectively you're capturing those features into stories and pushing them through the board. Sprint metrics will tell you how effective your sprints are being and how accurately you're predicting the amount of work that will be shipped. These things are all output metrics because they're taking something from a feature definition into some working code and then measuring how efficient that process is. So that's a little bit like a production line. Take the example of the Ford production line. On a production line you've got lots of stages and they're similar to the stages in software So a production line you might have like assembling the chassis and then adding the engine and that sort of stuff. I don't really know how cars are made but I imagine it's a several step process like that and so by looking at all of those individual steps and how they all fit together you can measure how efficiently you're building cars in a factory and you can tweak various parts of that process to make it more efficient. So output metrics are very very useful in manufacturing but they're not as useful in product engineering and the big difference is that when a Ford makes a car they know what that car is going to be. The design has already been put in place. It's already been tested. All they need to do is make more cars more efficiently and at a higher rate but in products engineering you often don't know what features you should be building. So with all of the validation and discovery in the world you can never 100% know if the thing is going to drive the business outcomes that you want in your company. So you can't treat software engineering purely as a kind of production line system. Now that's not true for all projects. Some projects are incredibly predictable. Some projects like maybe a migration project if you're migrating a database from one technology to another that's fairly predictable. You know what the end state should look like. You know you know if you're re-platforming a product or something you know that the thing you're going to should look in a certain way because you've got a reference to what it should look like. So if you're doing a migration or re-platforming then fine treat it as a production line and measure how effectively you're doing it. But most products development is not like that. Most product development has got a lot of unknowns in it and it's not simply a case of going from A to B extremely efficiently. So you also need to consider outcome metrics. Now outcome metrics are basically derived from the business value that you're trying to get from a product. So that could be if you're a SaaS company your outcome metrics could just be your SaaS metrics like you know churn, net retention revenue, net revenue retention that sort of thing. Or they could be product metrics so product adoption, net promoter score, customer feedback metrics, feature usage, activation rate all those kinds of things that product managers are much more familiar with. They tend to be outcome metrics because if your product activation rate goes up that's probably the outcome that you want. You want more people to use your product. If you can get more people to create like stickiness inside your product so they don't want to leave it, they've got very high net promoter score that will reduce churn. So those kinds of outcome metrics do tie into the business metrics that ultimately keep businesses alive. But they are measured in a very very different way to how output metrics are measured. You can't measure churn by looking at nine lines of code. So you need to have both of them. It's horses for courses basically. You need both output metrics and outcome metrics and a lot of engineers fall into the trap of over indexing on the output metrics just because they're easier to measure. They're not just easier to measure, they're also easier to attribute as well. Because an outcome metric something like the product activation rate you know how many people are using a feature, that feature could have been built by many many teams at your organization. And so if your output like your activation rate goes up that could have been because the customer enable team did a really good job or it could be because the product team designed a really good feature or it could be because the engineering team asked the right questions. There's lots of ways that those outcomes can go up. Whereas output metrics you can break it down very very easily. So if you are looking at something like the number of pull requests that are going in or the number of story points that are happening, you can break that down to an individual team level very very effectively. You can look at a group of five or six engineers and you can compare their sprint metrics to another team sprint metrics and you can just decide that this team needs a little bit of help or this team maybe could help this other team. That's much much harder to do with outcome metrics. It's very difficult if customer feedback comes in and says we find this feature difficult to use. It's very very difficult to pin that to one individual area of your teams and say okay that feature was difficult to use we just need to go and make a tweak over here. So it's very very difficult and it's also very difficult to attribute outcome metrics. But again don't fall into the magnum r fallacy of assuming that just because they're difficult to measure and difficult to attribute that you shouldn't be measuring them because you absolutely should. Both of these metrics are important and you need to take both of them together. And product engineering really is about combining output metrics and outcome metrics. So you still have an efficient system. It still is important to measure how efficiently you're putting product out but you also need to combine that with how well are we achieving the outcomes that we want to in our product. I'll give you an example from my experience. I was once managing two engineering teams that were working on two very different projects. One of the engineering teams was working on a framework update. So if I remember right they were upgrading the version of React. So it was using the component if you know much about React like they used to be this old way of creating React components which would use like a component class-based syntax. They were migrating the code base to use hooks. So that was a massive migration project that took them about a sprint or two. I had another engineering team that were changing the navigation inside the user interface. So both of these engineering teams had a piece of work that was taking them about two weeks and they both did a fairly good job of delivering it. So the migration team that were moving this version of React, they did, they estimated it pretty correctly and I could track how efficiently the pull requests were going in, how much refactoring was happening, how much code was changing and I could track the number of bugs that were coming in as a result of that migration. So the doorware metrics for that team were easy to track. They weren't all perfect but they were easy to track. The team that was changing the navigation on the other hand, again I did have all the output metrics for them. I could see that they were doing their work but what happened is we put the navigation changes in and users absolutely hated it. It caused a lot of resentment from a lot of people. Whenever you change navigation that people have built up muscle memory around, then you can initially at least achieve a negative product outcome. So the team that were changing the navigation achieved a negative product outcome even though they had actually delivered that project on time. And that's difficult to console yourself with basically because we wanted to celebrate the success of both of these teams and the one that had done the migration project, that was very easy to celebrate the success of. We knew what we were trying to do, we got there, we tested it, it was working, congratulations, you've done that migration, you can move on to something else. The team that were changing the navigation, again they did everything they were asked to do but people hated it. And so they can't move on to something else. They then need to go back through another cycle and iterate on that navigation and fix a bunch of stuff because people didn't like it. And so that's difficult because ultimately one of those teams did exactly what you were supposed to do in terms of outcomes and the other one didn't. The other one, even though they did exactly what they were asked to do, they didn't drive the correct outcome, they were trying to make the navigation easier to use and they ended up making it harder for a lot of people. And so measuring these two teams and celebrating their success, it was incorrect to do that just based on how like efficiently they delivered their work. And that's the important thing about all kinds of delivery metrics or output metrics is they tell you how efficiently you're building things but they don't tell you anything about whether that's the right thing to be building. And so if you just celebrate like the efficiency, then you're not going to celebrate the actual changes that matter to the business. And this is how you end up with feature factories. So I said before that product engineering is not like building a car in a Ford factory and ultimately that's incredibly true and you can't treat a team as if they are a production line because then you will end up celebrating the wrong things. You will end up celebrating a production line being super efficient and they've built all their cars by 5pm and so everyone can go home happy. You can't do that with a team. If you do that, you end up with a feature factory model where your team is only there to turn requirements into working code. And particularly in the world of AI, AI can now turn requirements into working code extremely efficiently. So if you're judging your ability of your engineering team to turn some requirements into code and then AI comes along, you might think well why do I even need this engineering team? And a lot of software engineers at the minute are getting extremely nervous about that very thing because they've tied their entire purpose into the ability to move code through a factory and that factory is now automated. Robots have come in and they've started doing all those individual pieces. So it's more important now than ever to change the model about how we celebrate success and how we measure engineering teams so that we're not treating them as feature factories. We're treating them as teams combined, products and engineering teams that are driving towards an outcome and we're judging them based on the outcome that we've asked them to achieve. So the mistake I made with that team that I'd asked to change the navigation was that I was holding them accountable to how quickly they could do that project, should have been holding them accountable to how easily users can find the new navigation. That was the correct thing to hold them accountable for and I would have just given them as much time as possible until we've seen signals that they've actually meaningfully improved the product. And if it took them three times as long that's basically okay because if you say to teams it can take you a bit longer but as long as you achieve that outcome then they will naturally try and make it take a little bit quicker because they will iterate more, they'll do more experiments, they'll try and find out more before they commit too much time into things because you told them that I'm not measuring on your cycle time or the number of code like lines of code you write, I'm measuring you on when people can find this more easy to use. And that brings me on to my next point about measurement because when you are measuring outcomes it's extremely important to work out how you're going to measure them before you start doing the work. Output is fairly easy because you know that, you know you've already got the tools in place to measure things like cycle time or pull request throughput so you don't need to decide that you can just know that you've got those metrics there but outcomes you need to decide before you start investing time into something how are we going to measure this outcome because they are more difficult to measure and again just because they're more difficult to measure does not mean they can be ignored they just need a little bit more thought they need a little bit more time spent up front working out how are we going to measure the success of this project how are we going to measure the impact of this feature and then as they start working you can bring that measurement in and you can try and get the team to iterate earlier and more quickly so that they have a higher chance of achieving that outcome because after all measuring things only really matters when those measurements can change a decision. Measuring things after the fact is okay if you want to look back and do a retrospective on some work but you really want to be measuring things from day one because if you're not measuring constantly through the entire process then you're not going to be able to respond to those measurements and outcome metrics particularly are often lagging indicators so often it takes a long time for changes to show up in outcome metrics particularly around things like user feedback or net promoter score surveys and things like that user feedback won't come in until you've shipped and people have had enough time to play with the thing that you've built so they're lagging indicators or they can be lagging indicators so it's even more important to get those lagging indicators brought forward and get some of that user feedback earlier in the cycle. Output metrics are often leading indicators so you can tell every single day how much code was shipped that day you need to be able to try to get it so that every single day you can tell how far we towards the outcome that day. So I want to close out this episode by giving you a question that you should ask of your engineering team your product and engineering team and that question is did the product get better because most outcome metrics are trying to answer that question we all want to build great products and we all want those great products to be used by lots of people who stick around and stay in our product and potentially drive business revenue as well and so in order to do that we need the products to constantly be getting better and you can't tell that just by measuring how efficiently you're changing that you also need to be able to measure actually how well it is improving. So ask yourself that question do you have a way of measuring outcomes and if you don't what are those things you can do what things can you put in place that will help you measure the impact of the work you're putting out. So I'll leave you with that question I hope you're enjoying the podcast so far these first few episodes are just solo episodes with me so I can get a bit of rhythm into this but we will be having guests on and exploring some other people's opinions in the future as well so stick around pepodcast.com you can find it on all of the common streaming services my name is James Charles Wirth and I'll see you in the next one.