Accountability For Product Engineering Teams

Hi, my name is James Charlesworth and in this week's episode of the Product Engineering Podcast we're going to talk about accountability and particularly how does accountability work when applied to product engineering teams working on different pieces of work in different ways. This week's episode is just me talking to you again but subscribe to the podcast because in the future we're going to have guests come on and talk about their experiences running successful product engineering teams. So let's get into it with accountability then. You're going to hear me say this word accountability quite a lot on this episode so what is it? Well accountability is how you hold a team accountable for what you've asked them to do. So if you ask a team to build something to a product roadmap then you might say to them we would like you to build this feature to this scope. They would say we think that would take a certain amount of time you put that on a product roadmap and then after that time has passed you can review that and say did this team produce what they said they would do in that amount of time. That gives the team the ability to decide where they went wrong it gives the organization visibility into what's working and what's not working on the team and it also allows the team to celebrate their success. So teams themselves can have accountability or they should have accountability for their work just like the people that are putting the money into the team the people that are investing in it so essentially the leaders of your organization they need a degree of accountability to make sure that their investment is actually returning a positive ROI. So accountability is incredibly important and often engineering and product leaders take on a lot of that accountability themselves they are ultimately responsible for the work that their teams are doing and a lot of our jobs as engineering leaders does hinge on making sure that we've got everybody everything in the right place around us so that we can take on that accountability and we can be judged ourselves by how successful a piece of work is. But accountability can change based on the type of work that's being delivered. Let's say you're working towards a product roadmap and you've put something on your product roadmap that you think will take a certain amount of time and you've said we will be able to ship this by I don't know the end of August 20 31st of August. If the 31st of August comes around and that work has been shipped and everything has gone and we've delivered roughly exactly what we said we were going to do then that can be measured as a success and the accountability has helped make sure everybody has felt that that was a success and if it wasn't a success what can we learn from it next time and how can we get better at doing the thing that we said we were going to do. What though if the work that you're doing is actually more outcome driven and it's trying to drive some kind of product outcome or some kind of business outcome. A business outcome for a SaaS company might be something like reducing churn reducing customer churn. If you've given that to a product engineering team as their outcome and you've said to them we want to reduce customer churn by 10 percent in the next quarter then when the quarter comes to the end you can't judge that team in the same way that a roadmap driven team was being judged by. You can still have a date attached to it if they said we will reduce customer churn by 40 percent by the end of the quarter then you can still hold them accountable to whether they've actually done that or not but what you can't do is look at what they've shipped and look at what they've delivered and try and apply delivery based accountability metrics to a team that wasn't necessarily being judged on their ability to deliver they're being judged on their ability to achieve an outcome and outcomes like reducing churn can be hit with lots of different solutions and lots of different ways and it's not necessarily the case where the more you ship and the more you deliver the more you drive that outcome. You can ship a lot of features and build a lot of stuff and not really make a dent on an outcome like that equally you can do some small things that take a lot of research and a lot of experimentation but the actual delivery is relatively subtle and then have a huge effect on the outcome so you can't judge those kind of teams and hold them accountable to what they shipped and how quickly they shipped it or anything like that. If you do do that if you start putting delivery based accountability onto a team that's trying to determine outcome then what they will do is they won't have the freedom and the flexibility to do that learning and that learning is extremely important. Let's say they are trying to reduce customer churn by 40% they might have to explore five or ten different ways of doing that and they'll have to put a few different experiments up and work with a few customers and things like that they'll have to spend a lot of time researching the problem until they finally work out what they need to do to reduce churn and if you are holding them accountable to we need you to ship a certain number of things every month then what will happen is they will shortcut a lot of that learning and a lot of that experimentation because there's pressure to deliver something and if they shortcut a lot of that learning and experimentation then they ultimately will not hit their outcome as effectively as they could have done if they were given the time to experiment first. A good example of this might be because when we talk about outcomes on this podcast I always give examples of like SaaS metrics and business outcomes like churn and things like that and I have had some feedback that often it's actually the more technical projects that have those unknowns and have outcomes in them as well so it's not always about improving business metrics so something like the performance of a back-end system. A lot of people would think that performance work is fairly sort of predictable and outcome driven you identify where the bottleneck is in your performance pipeline and then you spend a certain amount of time trying to reduce that performance bottleneck and then therefore you've got some nice metrics you can judge it to but ultimately performance work often does require just as much exploration and refinement as something like custom and churn. So if you gave a product engineering team the problem of increasing performance and you gave them a performance outcome so let's say your performance outcome was something like API response times or job queue processing times or something like that then they still need to do a huge amount of experimentation they need to try different ways to beat that performance bottleneck they need to get creative there are a lot of unknowns associated with performance because at the end of the day engineers don't deliberately build things that aren't performing performance is a process of discovery so often you build something based on what you've built before and your experience comes into play and then you put it out into the world and you find that it's being used in the way that you hadn't anticipated or it's being used at scale that you hadn't really designed for so therefore you have to do something a bit new and you have to experiment and so performance work is a really good example of something that is an outcome driven thing that isn't all just product and business metrics and stuff like that so let's say you've got a team that's trying to increase the performance of your back end they're trying to reduce response times on the API now unlike a roadmap item you don't necessarily want that team to estimate and then scope and then commit to a date for doing that unless you have some kind of external factor that's causing you to need to increase the performance by a certain date that's often not the case you do often have a little bit of flexibility in when you're going to increase the performance by it just needs to be done so you can say to that team we need you to increase the reduce the response times on the API by 20 percent we need you to do it just as soon as you can do it so you give them that outcome and they'll go away and they'll explore lots of different ways to do it and they'll test out different things and they'll measure the success and gather evidence about which approach is successful and so let's say you turn around to that team in three weeks time and say what what have you shipped what have you delivered to increase performance if they turn around and say well we've done a lot of experimentation but we haven't actually fixed the performance yet we haven't actually delivered anything into production in those three weeks you need to be comfortable with not judging that team negatively by saying well you you know you haven't actually shipped anything because what they have done is they've learned it could be that on the fourth week they've done enough experimentation that they know exactly what they need to do to increase performance so they just have to ship a small amount of code and they'll radically drive that outcome so if you took a team like that and started judging them by how many things they were shipping into production every single week then what they will do is they will naturally drive they will de-optimize for learning and they will start to optimize for shipping things and if you're trying to drive an outcome you don't want to be optimized for shipping things i'll give you an example from my own experience i used to work on a sass company for the legal industry so it's a conveyancing industry here in the uk and we had two pieces of work come onto our product engineering team at roughly the same time and they were very very important in how we were holding ourselves accountable to doing them the first piece of work was to generate a document so we had a document generation functionality and we needed a new document that this platform could generate now that had a fairly well-defined scope and it also had a hard commitment we had to do this by a certain date because the document needed to be available with by a certain physical date that wasn't particularly flexible it had to happen so we knew what document we wanted to generate and we knew what it should look like because it was based on a bunch of regulatory things like there was a very very clear definition out there of what this document should have on it and we knew when we needed to deliver it so we basically did that as a road map driven development model and we held the team accountable for delivering the document the way it was supposed to be by the date they said they were going to do and that's perfectly fine that's called road map driven development you can predict that up front you can work out how much scope there is into building that you can work out you can sequence that kind you can break it down into tasks and ultimately you can deliver it in the time frame that you said you're delivering at the same time though we were also trying to tackle adoption of the platform now i don't want to be disparaging for anybody who works in the legal industry but if you've ever built software for the legal industry it can be a challenge for people to change their workflows and their patterns to adopt new software when they're working in a profession like the legal industry where it's not massively tech focused they're not they're slightly later on the adoption curve of technology essentially and so asking them to change their processes and change their workflows away from the traditional way of doing like lawyering like legal type work into using some fancy new tool can involve quite a lot of friction and that's exactly what we found when we were building this we found that we put this nice ui together and people just weren't logging into it they weren't coming back to the product in order to get the most value out of our products people needed to be using it every single day but we were measuring the adoption of it and we realized that a lot of people were only logging in once per week so we decided that we needed to fix this and we needed to fix it by making the software easier to use it needed reminders to come back in it needed to do more stuff without the user having to log in all that kind of stuff so we gave the same engineering team an outcome and we said we want you to increase the frequency which the media and user logs into this platform it's currently twice a week it needs to be every single day so that was the outcome there was no fixed scope for that there was no amount of features that we had to ship there was no delivery date for it just needed to happen as quickly as possible so it's very very different to the document generation work the document generation work had a fixed scope we knew what the document needed to be and it had a very tight deadline the driving adoption inside the product work just needed to be done as quickly as possible and we needed to get that adoption up but there was no scope we didn't even know at the start of this work how we were going to do that there were a bunch of ideas the team had had we just knew that we needed to achieve this outcome so when it came to reviewing the work at the end of the period we could look at the document generation work and we could say yeah that was successful okay the document might not have been perfect perhaps it could have done with a few more iterations perhaps there could have been a little bit more discovery about the design of the document and stuff like that but ultimately the document had all the information that it needed to have the generation worked and the output format was acceptable for our users and crucially they'd hit the date that they said they were going to hit the outcome work on the other hand it did show positive trends by a certain amount of time the same amount of time they had for the document generation and it did seem like they'd settled on a solution that was backed up by evidence and was but it wasn't quite there yet so we couldn't necessarily judge the outcome focus team with the same success metrics as the roadmap driven team because ultimately they they had they built more software because they'd tried lots and lots of different things and they've made loads of prototypes but they hadn't shipped as much software into production so the team that was building the document generation tool they had shipped less code into production but the code they'd written basically went straight through to production went through testing and went through the software development lifecycle the outcome based team a lot of the code that they wrote got thrown away because they were exploring the wrong solution essentially and so if we were to hold those teams accountable to the same thing then it wouldn't have worked because it looked like the team building that document generation staff had been completely successful whereas the outcome team hadn't but in many ways what the outcome team was trying to do was much more difficult because they were trying to kind of build the plane as they were flying it essentially they were trying to like add things to this thing and experiment as they were iterating with it they were trying out different segments on different users they had to do things like add instrumentation into the product so they could track user journeys through the products and things like that there was a lot more kind of hidden work on that team so it was unfair to turn around to them and say you hadn't shipped as many features this quarter as the team that knew exactly what they were going to do and it was a very known problem so what we had to do is essentially hold each team accountable to something different so they could both celebrate their successes but also so they could both learn for what they went wrong the learnings on the team that were building a document generation tool were really about the efficiency and the scoping and the planning process so when that team got together at the end and they discussed how the project went a lot of that retrospective feedback was oh we didn't plan things out properly we didn't have a design of early enough all this kind of stuff like the efficiency of the delivery wasn't great whereas the outcome team when we asked them to go through it theirs was completely different the kinds of things they were learning were we didn't try enough experiments we didn't target the right users with this we didn't this was a great solution and we didn't think of it early enough that kind of stuff they weren't being judged on the efficiency of what they were doing they were being judged on the actual progress outcome so both teams could ultimately learn something from it and both teams could be measured as successful in different kinds of ways but they weren't following the same model now the crucial thing to note here is that when you're holding a team accountable for some work like this it's not the actual team itself that has the accountability model so you don't have teams that are always held accountable for how many features they ship every quarter and other teams are held accountable for how effectively they've driven some kind of business outcome you attach those accountability models to the work because product engineering teams by their very nature are flexible they can pick up different kinds of work they can pick up work that involves a lot of exploration or they can pick up work that's very well defined needs to be delivered to tight deadlines and so it doesn't make sense to say that this team is always going to be held accountable to how quickly they can ship and this team over here is always going to be held accountable to you know how they can explore or drive growth in a product and that is often the trap that organizations will fall into you sometimes see organizations that will have a whole team labeled as like a growth engineering team or greenfield products discovery team or something like that and then you'll have another team that's labeled as a platform engineering team or something like that the idea being there is that the platform engineering team is doing predictable work and they can be judged on throughput and the product based or the growth team is doing discovery work and they're judged on learning but ultimately as we saw with the performance example it's possible for work to fit in both buckets on both those kinds of teams you can absolutely have a platform engineering team that takes on a project that requires a lot of exploration and a lot of iteration a lot of learning and you can have a project which doesn't involve that team actually shipping much at the start to production they're still probably writing a lot of code because they're exploring things but ultimately they're not delivering one feature per sprint or whatever it is you're judging a roadmap driven team on and they're a platform team so it doesn't make sense to say that that team are always going to be held accountable to their throughput and equally with a growth team or product engineering team they might take on a commit for a customer they might take on a feature that absolutely has to happen contractually and so it makes much more sense for them to come out of their learning mode and just work on delivering something by a fixed amount of time basically so it's important that the way that work is held accountable is a property that's attached to the work and not something that's attached to the team one product engineering team could have two pieces of work going on in parallel and each piece of work is being judged successfully by different things they could as we said with the growth team they could be trying to drive user adoption to a certain rate but they could also be working on a commit for a customer and then if they delivered some of one and some of that it's really really important that the way those two projects are judged as successful is attached to the project and not the team as a whole so you don't say this team failed because they missed this deadline or this team failed because they failed to learn on this really really vague outcome you say this project or this piece of work failed or was successful because we were measuring it based on predictability or throughput and this piece over here that failed because it was supposed to be driving an outcome and the outcome hasn't moved as much as we would like to. In fact I'm going to backtrack a little bit there I talk about failure there the opposite of success in accountability isn't failure the opposite of success is generally learning and improvement so it probably shouldn't be using the word failure it's unlikely that an engineering team will flat out fail on anything and it's really actually quite important that they don't feel like they're failing but you know sometimes these outcomes are binary sometimes you can say was the scope that we said we would commit delivered by the date we said we're going to commit it that's a yes or no question so yes it was successful or no we've learned something it's probably the better way I'll phrase that and the same with outcome driven work as well if you've asked a team to reduce customer churn by a certain amount and they've only reduced it by a part of that then that's why it's a partial success but they've learned something and that's the most important thing they've learned what direction they could be on they've learned some solutions they could put into production that will continue to drive it and ultimately they've learned then you can you can take some actions from that and you can go forward so I want to finish out by talking about reviews it's important for any leader product leader or engineering leader that we have a regular review process of the work that we're doing and the work that our teams are doing and that review process needs to be communicated in a way that the team can understand what was reviewed and what the results were and that leadership and stakeholders can also you know understand what went wrong as well and so what I would really recommend to you is to have a regular cadence of reviews and make the reviews incredibly predictable so what I mean by that is let's say you've decided at the start of a piece of work that this is an outcome driven piece of work we want to drive some kind of business metric in the organization we're going to review the progress of that outcome every month and we're going to use this format we're going to be measuring this we're going to be measuring the outcome and we're going to be asking questions like what have we learned since the last review what did we try what new evidence have we got all that kind of stuff and then crucially we're going to ask that same set of questions every single time the review cycle comes around so we're not just going to suddenly throw out a delivery focused question in there like did you hit this deadline we're going to ask the same questions and what that does is it creates comfort and the predict how we're doing towards the next review cycle and it makes it much much easier people absolutely hate it and they get really really stressed out if you judge them on something that wasn't clear that they were going to be judged on this goes for all management of engineering teams or probably all management in general but if you say to somebody I'm going to hold you accountable for reducing customer churn and then a month later you turn around and say why didn't you ship that feature when you said you were going to ship it they're going to get very frustrated with that because they all say well you didn't tell us we're being held accountable to shipping a particular feature you told us we wanted us to reduce churn so can we please focus on how well we've been reducing churn and what we've learned towards what's causing churn basically so if you suddenly spring different kinds of accountability on a team that thought they were working under one model and they thought they're being judged in one way then that can be extremely disruptive and it can cause people to disengage from the whole review process and it can cause people to just kind of quiet quit and not really care about the outcome in the first place so it's really really important that the review is tied to how a team is being held accountable and that that review process is regular and it's predictable and everybody knows what the next review process when it's going to be and what it's going to include and what they're going to be measured on so they can say whether it was successful or whether it wasn't so I hope this has been slightly educational I am going to pack a bunch of these educational episodes into the first few episodes of this podcast because it's a brand new podcast if you've got any feedback at all then it's do product engineering.com james at do product engineering.com please send me an email if you're watching this on youtube leave me a comment I really really do appreciate any kind of feedback as I shape the future of this podcast going forwards so I hope you enjoyed the episode and join me in the next one