What Makes AI Code Production Ready?

Using AI to build a prototype of something isn't the same as building production code that will last and stand the test of time. Often you can get something 80% of the way there using AI, and that extra 20% could take you many, many times as long than that first 80% took. So don't fall into that trap of thinking that just because you can build a prototype that looks amazing, you've got something you can ship to production. (Music) Hi, I'm James Charlesworth. Welcome back to the Product Engineering Podcast. And today we're going to talk about prototyping. And this is really one for the engineers because I know a lot of engineers have been expressing this opinion for a while now, ever since AI came around. AI is really, really good at creating a nice looking product, but the step between getting that code like out there and looking nice on the user interface, and then getting it into production and being able to support it into production is not as simple as it first seems. Now, software engineers absolutely know this, but product managers and people that aren't necessarily super, super technical from a coding point of view think that just because it looks right, it is right. Now, that doesn't necessarily mean that having an 80% of the way their prototype is a bad thing. In fact, as I'll argue many times on this podcast, being able to create a prototype that looks good and basically functions is an extremely valuable learning tool, but it's not the same thing as writing production code. So let's break down what some of those gaps are and some of those things that AI still can't necessarily do. And we'll work out where you can use prototypes and where you can use AI and where you should probably do things properly. And in fact, it's probably worth going back in time to before AI existed because this problem has always basically been there. Ever since things like React and stuff like that came out, it's been fairly trivial to create a nice looking user interface, maybe with some mock data, and then put that in front of people and say, this is what the product is going to look like. And then a lot of people fall into the trap of thinking that that really quick prototype you made is ready for production. If anyone has ever been to a hackathon, hackathons usually revolve around the smoking mirrors effect of I've made something that kind of looks like it's going to work. This is directionally where we want to go, but we haven't actually written production code. It's extremely difficult even in the age of AI to write proper hardened production supportable code in like a small amount of time, like a one day hackathon. And nothing's really changed there. AI has just turned like a one day hackathon into a 10 minute prompt exercise. Ultimately, it hasn't changed the fact that the code that you've written that might look brilliant isn't always ready to go into production. And so what are those gaps? If you've built something using claw code or you built like a user interface from scratch, what are the gaps between what you've built and what looks amazing and what's actually production ready? Well, there's many of them and they're often things like scalability, cost. A lot of the time in AI will just not really care about how much something is going to cost to support in production. So it might give you a super expensive solution that isn't necessarily going to be commercially viable. There's also things like observability and maintainability. So good software engineers will know that in order to support some software into production, you need to instrument that software with observability and metrics and things like that. So that when something breaks in production, you can very, very quickly work out what's going to break. Potentially, you can work out as it's going to break before something happens in the product. All of that observability is hidden to people that are using software. If you're a user of a web application, you probably don't realize quite how much instrumentation and observability is built into that on the back end. And so if you're being shown a prototype of something that was written with AI, you don't know whether or not there's a way for the person who built that to actually monitor what's happening on the back end. And to predict when things are going to fail and when it's going to crash. But you do need all that. In order to support something in production, it needs to be observable. It needs to be secure. That's another thing that often gets missed. It needs to be scalable. You can build things for different levels of scale. People, it's a mistake to think that scalable is a yes or no decision. Code can be scalable to the point where you want it to scale to, or it can be scalable to like a larger point. But you very rarely get a situation where code will scale from zero to infinity for the same trade-offs. Unless you've been really, really conscious about giving the AI information about how much you want it to scale, which a lot of people won't do, then what you'll end up with is a piece of software that's designed for a certain level of scale and you try and implement it to a different level of scale. Scale works both ways. So if the AI has given you a piece of software that works for a million users, but you've actually only got 100 users, that's not necessarily a good solution. That's going to cost you a lot of money to run. The infrastructure is going to be extremely complicated and expensive, and you've only got 100 users, so you're not getting that benefit. Equally, if it's built you something that will only work for 100 users and you try and roll it out to a million users, it will break there as well. So you need to be able to design for the correct scale. And again, that's one of those hidden pieces of architecture that isn't apparent in something that was just vibe-coded with AI. So you can have a product that looks functionally brilliant, but all of those questions around scalability and security and observability and maintainability, all of those are hidden and you don't know what those decisions were or if those decisions were even made in the first place. So it's really important for everyone that's building software rapidly with AI to just be aware of those kinds of decisions that need to be made in order to make something production-ready. So you have the user interface that you've built and you've maybe got some back-end functionality, then ask yourself how are we going to make sure this scales, how are we going to make sure it's secure, how are we going to support it in production, what are we going to do when something goes wrong, where the data have actually been stored, which the parties are using, all the legal stuff. All of those kinds of questions need to be answered in order to take your prototype into production. Now that doesn't necessarily mean you shouldn't do it, you absolutely should build prototypes because sometimes you don't need all of those other questions to be answered in order to get value out of a prototype. So a prototype that's been built with AI, if you're working on a product engineering team and you're working in quite an unknown environment, an AI-built prototype can give you something that's more valuable than anything else, probably more valuable than the production code and it can give you answers, it can give you learning. So if building a prototype allows you to explore an idea or it allows you to answer a really pressing question about the correct path forward in your product, then you should do it. If you want to put that into production, then you need to actually answer all of those other questions. So you need to make it scalable, you need to make it secure, you need to have observability and all of the other things. Even at a code level, it can be useful to make sure that you've got the right coding practices and the code's clean and readable and doesn't duplicate itself and all the kinds of things that software engineers care about, that's what goes into production product and that kind of stuff is completely invisible to the person using the prototype. So the solution to this is just to make it explicit what is included and what is not included in some software you've written. If you've built an application very, very quickly to test out a concept, then just make it explicit that this might not be 100% scalable, it might not be maintainable, that's okay, as long as you're saying that that's why we've done it, then you can still get the learnings and the value out of that product. Two pieces of product might look exactly the same, two pieces of software could on the surface appear exactly the same, but one of them is production ready and one of them is not and you wouldn't necessarily know from the outside whether they are or not. So that's where the skill of engineering comes in. So if a product manager, for example, is prototyping something with AI, then the role of an engineer on their team is to help them guide them down the right questions they need to answer before that's ready for production. Now I spoke on a previous episode about context and accountability. If somebody is making a decision to build something inside your product, then you do really want to empower them to be able to make that decision. But they don't always know the context behind whether that's the right decision or not. So it's the job of the organization to create a structure in which people that know about security and observability and scalability and support, those people can provide that context to the people that are going to make the decisions about whether or not it should be built. So ultimately, if you've got a product engineering team with software engineers that know all about the architecture and the correct way to scale something, and they've got product managers that maybe know the correct thing to be building, bringing those two together and allowing them to share that context will allow you to build things that are initially not production ready. And then once you know it's the right thing to build, you go make them production ready. So it's ultimately a two stage process, you build something that looks finished, and then you go and make it finished afterwards. And those two things can be very, very different levels of effort. Sometimes the amount of effort that goes into building an AI prototype is more than you think it is. So just because something looks like it was vibe coded with a one shot prompt doesn't necessarily mean that that person hasn't spent a lot of time exploring different ways. So don't disregard the amount of effort that goes into building an AI prototype because they can sometimes involve quite a lot of learning, quite a lot of iteration, quite a lot of experimentation. There's a lot of back and forth talking to an AI is still working. So don't disregard the fact that somebody's got a good prototype, just because it's not production ready. But at the same time, don't disregard all of the work that you need to do to make some software production ready. It's really, really important not to have this smoke and mirrors effect, where you've built something that looks amazing. And then everybody just assumes it can go into production. Sometimes things can, but they won't always go into production in the way that you necessarily need them to. There are a lot of platforms out there that will give you a in air quotes production ready piece of software. But that doesn't necessarily mean that it's the right production ready piece of software for your organization, for the level of scale that you're going for, for your particular regulatory or security requirements. None of that sort of stuff is guaranteed just because something looks production ready, or just because you're using a tool that has told you it's production ready. So ultimately, it comes down to a balance between over engineering and under engineering. Something that looks amazing, but doesn't have all of the scalability and the production ready stuff in it is under engineered. But something that is super, super scalable and really observable and supportable in production, but doesn't necessarily need to be at this point, that will be over engineered. If all you're trying to do is learn something, if you're trying to learn if this is the right feature to be building, then you don't necessarily want to over engineer something. It doesn't need to be fully scalable if you're only going to try it out with a handful of users. So don't over engineer stuff. Don't allow software engineers and people that really want everything to be super production ready to slow you down. Don't allow those people to stop you trying out prototypes and things with a subset of users with the outcome of learning. But equally, don't allow people that have built a prototype to just ship it to production without thinking about all those other things as well. The most important thing you can do practically is to make the definition explicit between production ready code and a prototype. So when you've got something that's been built, really genuinely document what is stopping this thing going into production today, and what do we still need to build? You don't want every new prototype to have a list of checked off items saying it's ready for production because probably that prototype took way too long to build and it's not going to give you effective learning. But obviously you don't want a prototype that has a bunch of unchecked boxes because it's not ready for production. So if this is something that resonates for you, if you've seen people trying to ship prototypes to production when they're not ready, or you've seen people over engineering a prototype because they're being told it has to be production ready when it doesn't necessarily have to be, then do let me know. My email address is james at train2code.com. I really welcome feedback on this product and this podcast, and I really, really want to have other opinions and other voices on this. I am booking guests for the future, so if you want to guest on this, then reach out to me as well. My name is James Charlesworth, and this is the Product Engineering Podcast.