When To Use AI In Product Engineering

There's a right way and a wrong way to use AI in product engineering. AI can obviously write code extremely quickly, but it's a mistake to use AI to just speed up a legacy delivery model when there are so many better opportunities and so many more effective ways you can use it inside of product engineering team. Hi, welcome back to the Product Engineering podcast and this yet again we're talking about AI. This is one of these topics that you really can't get away from, but I want to kind of take a step back from the whole revolution behind agentic coding and vibe coding and all that kind of stuff and talk about how you actually use AI inside a software delivery mechanism, especially when you're trying to align a team around a product outcome and not just trying to make your roadmap be delivered a little bit more quickly. AI is really really useful at speeding up certain tasks, in particular the task of writing code. So if you can give an AI system a really well detailed spec or a plan, then it's fairly trivial to have some model turn that plan into some code and it's also fairly trivial to kind of review that code and test it and ship it to production. What it can't necessarily do though is make sure that that plan is the right thing to be building inside your product. And so that's when AI should kind of work with human judgment. AI coding tools are not there to replace your judgment. They can absolutely educate you and they can help you learn about the feature you're trying to build and learn about the code base, but you still do need humans to decide whether this is the right thing to be building or not. Before AI coding tools, writing the actual code was one of the most expensive and time consuming parts of building software. So what we ended up doing as an industry is we built a process that tried to reduce the amount of risk that you were taking when you started actually writing code. So everything from like a product requirements document or kind of scope and an epic enduro and all that kind of stuff, all of that upfront planning is designed so that when you make that commitment to actually write some code and ship it to production, you're not going to be making the wrong decision. But AI actually kind of flips that on its head. It now makes it so that once you get to the point where you've got a really, really well defined piece of work that you want to do, then it's actually very, very quick to go from that to code. So what you really want to do is be focusing on are we actually building the right thing? So are we writing the correct spec for the API to work on? And that's where everybody, product managers, software engineers, leaders, managers, even like DevOps and SRE people need to start moving further up that pipeline and focus their human judgment on are we actually building the right changes inside our products that are going to drive the behaviors and our users that we want to drive. And products judgment is something that requires human beings. Despite all of these amazing fable mythos models, you still need human beings to decide what should be done inside a product. And that's for the simple reason that unlike code, when LLM can be trained on loads of code on the internet, if you're working on a software product, you probably know better than the training data that went into an LLM about the history of your product and about how your users have interacted with your product. Even like fable can't be trained on your customer calls and your interactions with your users. You know that as a human being that information still exists inside our heads. And it hasn't gone into the training data of models. It's not available on the internet, like a discovery process that you've done inside your organization that's helped you learn about what kinds of features you should be building inside your product. All of that learning still sits within your organization. You can absolutely have AI models that would go in and pull that information from documents. But often this is actually just a human experience. Often we just instinctively know based on the whole history of working on a product, what we should be doing and what we shouldn't be doing. And you still need the human beings to make those decisions. And so that goes for everybody on a product engineering team as well. Software engineers don't necessarily just know the history of the code base. They know the history of their interaction with product in the past. They remember like why products ask them to do things. They remember why they built things in a certain way in a way that is extremely difficult to explain to an agentic model. So even though AI can absolutely take some predefined requirements and ship it into code, it's really, really not very good at working out exactly what should be built because it just doesn't have that information. And you can spend days and days and weeks and months trying to teach in AI about your business and about your product and about your users. But ultimately the people that have lived that product day to day, your product managers and your engineers and your leaders, they're the people that really, really know. They're the people that really have all that context. And so having product managers and product engineers guide the AI in the right process is the correct way to use AI. Often in product engineering, we don't necessarily know the correct solution to a certain problem. We can research a problem. We can understand that it exists and we can speak to people that have that problem. And ultimately we can ask AI for a bunch of possible solutions, but ultimately we don't know at this point which of those solutions is correct. So the right way to use AI would be to explore more than one solution at the same time. Let's say you know about a problem that you know that there's a bunch of ways you can solve it. You can then go to AI and say, here's option A. Here's a brief spec for option A. Let's work together on trying to design this implementation. And then also here's option B. And then you can put both of those to maybe different sets of users. You could do split testing. You could do one and then you could do the other one. But because AI has written the code extremely quickly, you can then use AI to learn about your product. You can use AI to learn about which solutions are correct, which solutions are actually going to drive an outcome. And then ultimately at the end, you can still use AI to write the final code. But the right way to use AI is to use it as a learning tool to teach you about your products, to teach you about your users, and to explore possible interventions you can make in your product. The wrong way to use AI is to just try and collapse the product software development lifecycle into like a shorter time. Because if you do that, you're still going to have the legacy software development lifecycle where you've put all of that learning upfront into designing the perfect spec, and you're still just trying to design the perfect spec over and over again. So you actually need to use AI to help you adjust that spec as so let's say you're trying to improve the way that users flow through one part of your user interface. And there's three or four ways you could do that. Instead of trying to just put all of that into one spec and then use AI to very quickly write that, you can use AI to iterate on different approaches. You can use AI to mock up different versions, experiment with different designs and things like that, get feedback from users, and then ultimately integrate it into every part of that development lifecycle, not just the delivery stage. AI isn't just there to deliver some code to production, it's there to actually help you build that plan using actual software. So product managers have been doing this for a very long time, but usually without code. When a product manager has an idea for a solution, typically they would have worked with a designer to maybe draw up a wireframe or a mock-up, and they might have two or three different versions of the user interface that they've mocked up. And then they'll go to users and they would say, here's some ideas, here's a little thing we've built. You can give us some feedback and we'll make a decision about which of those UIs we're going to go with, and then we'll hand that to an engineering team. Now you can use AI to actually create a real version of those prototypes and put those real versions inside in front of users as well. So you can create thin production slices using AI. You can branch off your main code base or you could just use like lovable to create a separate version of your product that looks similar. And ultimately you're writing code that doesn't necessarily end in production, but you're writing code that helps you learn about the right thing that you should be building. And this is something that can be done both by product managers and by engineers as well. By putting engineers in front of your users, by moving them into a more product engineering role, then you can give engineers the ability to use AI to mock up prototypes and take them to users directly themselves. So instead of asking an engineer just to build a bunch of features and then give them to a product manager, engineers can build a bunch of features quickly with AI and they can take them to users and they can decide which of those ones are worth doubling down on. And then when they've got more context around that decision, maybe they can go to a product leader and ask them to decide, to help them decide which of those solutions they should be pursuing. And at the very end, you can still have the traditional software development lifecycle where AI is just speeding up to the delivery part. And another way to think about AI isn't we can write software more quickly, it's we can kill ideas more quickly. So people usually have a lot more ideas than they have time to implement. Let's say you recognize a problem inside your product and you've got like five different ideas of how you can solve that problem. If you can use AI to explore all five of those ideas at the start, then maybe AI can kill three of those ideas, it can find out that they're failures. In a legacy software development method, if you build the wrong feature three times or you build three of the wrong feature, that will generally be seen as a failure because you've wasted a lot of time going through the software development lifecycle and building those incorrect features. But in an AI powered product engineering model, if you can kill three ideas very, very quickly, then you've actually made progress. So you've actually tested out a bunch of stuff, you've learned what doesn't work and you can resolve around the ones that you are going to build in the file. So AI can be used to compress that feedback cycle between idea and evidence to support whether that idea is correct or not. You can use AI to explore a bunch of different possibilities and then make a much more informed decision about which of those possibilities you should go for in a way that really wasn't feasible before AI existed. When you had to have a team of engineers writing everything by hand, you had to make the right decision from the start. But now with AI, you can make the wrong decision. You can build things that are not necessarily the correct feature. You can do a lot more exploring, you can prototype a lot more ideas. And that's absolutely the right way to use AI. AI should be there to allow you to try out as many different options as possible and then make a much more informed decision. It's not just there to write the code more quickly, it's there to allow you to learn about your product. One interesting thing that you can do to diagnose how effective you're using AI is to inside your organization, try and work out how much of the code that's being written actually makes it into production. Because in the old pre-AI world, you probably find that the vast majority of your code does end up in your final product. And that's because code is very expensive to write. But now that you've got AI, it's actually not a bad thing to have a huge portion of your code not end up in the final product. That's actually a good thing. Because all of those different prototypes, all of those pull requests that ended up getting rejected, they are still teaching you something. You can create branches off of your main code base, do a load of development with AI and use that branch to try and learn something about your product and then ultimately throw that away if it was the wrong decision. So if you're finding that 100% of the code that your AI is writing is still end up in production, then it could be a sign that you're not doing enough exploration and you're not using AI in the most effective way possible. The most effective way to use AI is to try out a bunch of different things and then yeah, maybe only a percentage of those things that you try, maybe only a percentage of that code that you write actually ends up in the final product. That's generally a good thing because it was very, very cheap to write that code in the first place. So there's a little bit of an exercise for you. Have a look at your code base, work out how many pull requests are being rejected because we tried something out and we learned that it wasn't going to work. And then generally see that as like a success because of how quick it was to build those things in the first place. So have a look at your code base, work out whether you're actually using AI to drive learning and not just trying to use it to write more production code more quickly. And then you'll be able to get a sense of where you can use AI that you're not currently using it today.