49 Tech Debt Kevin: [00:00:00] Hello everyone and welcome back to the It Depends podcast I, as always, I'm your host. My name's Kevin Goldsmith and I'm delighted every other week to share some of the experiences I've had over 30 plus years in technology. Rising from developer to lead, to architect to engineering, manager to director, vp. And then for about the last 10 years, CTO, This episode. It's a big one. it's something I know is constantly coming up for folks. So, I'm surprised, I haven't done an episode just on this. I've talked around it and the subject is tech debt. A lot of times these topics come up because I've been talking to somebody, mentoring somebody, or it's come up in my own company or somebody's written in to ask about it. This one, like I said, I'm surprised I haven't talked about it in depth already [00:01:00] because it comes up a lot both in my mentoring and in my consulting and in my own company, and questions I get at conferences or questions I get for the podcast. When I talk to people about tech debt, one of the things I often talk to folks about is the idea that tech debt in itself isn't necessarily a bad thing. There's often situations where you're choosing to create tech debt, but people tend to treat it like it's a moral failure instead of a pragmatic decision made under real world constraints. And also to be fair, oftentimes it is the result of poor engineering choices or unrealistic product expectations, or unrealistic delivery dates. So it's not that it's always good, but it's not that it's always bad either. And that's important because [00:02:00] certainly if you are trying to be agile, if you're trying to be lean, if you're trying to deliver value and you have a good process for learning and a good process of iteration, you're accumulating tech debt all the time. I used to have as an email signature, under my name it would say, "creating tomorrow's tech debt today." Because honestly, if you're in a high delivery team, you're often choosing to not do the perfect version of something. You're choosing to do the pragmatic version of something, knowing full well that you will have to come back to it if you ever learned. Or read about eXtreme Programming. That's the first Agile framework I learned before even the Agile manifesto was signed. One of the things I really appreciated about that framework [00:03:00] was just constant factoring. So you're only looking a few feet ahead. You weren't looking miles ahead, you weren't planning out months of work. You were just thinking about what do I need to solve this card I'm looking at right now? And the great part about that, it seems like, oh, that's really dumb. But I had just come from Microsoft where I had worked on Windows media, where we spent a year and a half, months of spec work, months of development, months of testing. We were thinking, two years out really. And by halfway, we were realizing all the assumptions we'd made were wrong or a lot of 'em were wrong, but everything was planned out. So I'd come from that world, been doing Waterfall at Microsoft for many years. And then came to a world where we were doing XP and as the leader of that team. I wasn't building Gantt charts for years in advance, which I did at [00:04:00] Microsoft. We were just like, "Hey, this is what we're working on today. We need to do this thing. Just do that thing and let's do this thing right. Okay, now we got another card. We need to rewrite what we just did because now we have this new condition that we didn't have before." But you know what? You get really good at it. That code gets better and better and more bulletproof and more bulletproof, and then you find that parts of the code base get incredibly stable and that kind of constant reiteration moves further and further out. So I often come back to that XP experience, that eXtreme Programming experience when I think about tech debt. And I also learned a lot. Being in startups, especially early stage startups where I've made the mistake of coming in after having just been at a startup where, inherited a older code base with a ton of tech debt and spent a lot of time working through that [00:05:00] because that tech debt was actually holding the company back. I went to an early stage startup. And I'm leading engineering there. And I say, okay, well we're not gonna make that mistake. We're not gonna start with an architecture where in 10 years we're gonna have to completely rewrite and it's gonna take a really long time. We're gonna start with a much better architecture today. And instead jumped ahead way further than we needed to . Small team, just trying to get the thing off the ground. Built in massively over complicated architecture because I was trying to think of what we would need in five years and not what we would need in six months. So somewhere through that process we realized, oh gosh, okay, this is dumb. Let's just really focus on what the company needs right now. And that taught me a lot also about just how to think about architectural debt and how to think about willingly choosing. To create tech debt to solve business problems. So that's the bigger ur thoughts. I've also had to [00:06:00] deal, as I just said, I talked about joining a company that had been around for almost 10 years, that had accumulated a ton of architecture debt, a ton of code debt, and that was really hurting the company. I've been doing this thing for a long time. I've seen all flavors of good and bad technical debt. But what I've seen is that healthy teams see tech debt as a tool. Unhealthy teams see it either always urgent or never urgent, but it's not a crisis. Tech debt is a consequence of shipping. Even if you're trying to think ahead, every change you commit. Is tech debt for the future. It's just a question of when. We're gonna talk about both good and bad tech debt and how to clean up tech debt and how to prioritize tech debt and how to get support from. Oh, others who are invested in what's going on in the team, like a product manager or other folks to support putting time into tech debt, which always seems to be a [00:07:00] problem. For those of you who heard or went and listened to the recording of my talk from lead. This year there was a lot about tech debt in there because , it was a talk about the friction between product management and engineering and how to resolve it and tech debt tends to be a very central issue in that friction. So let's talk about the real leadership tension. When we talk about tech debt. It's really an umbrella covering everything from deliberate shortcuts. Somebody just trying to get something done to accidental architectural drift where just you pick a technology, it seems like the right technology, like you're thinking ahead. It's a proven technology and it just so happens you pick the wrong technology but that takes you a while to figure out. Then all of a sudden you realize, , you've been building on sand instead of building on rock, and now you [00:08:00] have challenges there. So that's what kind of what I mean by accidental architectural drift. The real challenge is telling that useful debt from the debt that signals that something deeper is wrong. There's that deliberate shortcut. That's useful debt. There's that, oops, we built on technology that's no longer being maintained, that's having more zero days. That is something deeper that is going to really have potential consequences. It's about knowing which debt matters. So the tension for leaders, if you're a lead or an EM or a director or whatever, that's the tension is you know, you have tech debt. Some debt is created intentionally for good reasons. Some tech debt is benign and safe to ignore. Some is genuinely risky and some is a symptom of maybe deeper misfunction that's completely outside. The actual [00:09:00] code. It's bad code, but it's bad code for a reason. That's not about the coder. All these look identical though when they show up as Tech Aetna backlog, which is one of the reasons why it's often very hard to get product to prioritize it because there's a ton of tech debt and all they hear is about all the tech debt that the team has, but it's not uniform and it's not all risky. And so for a product manager. It's hard to know. "Well, all I keep hearing about is all these things, but some of these things don't seem to be a problem, so how do I know any of these things are a problem?" That's one of the challenges. Engineering wants to fix the pain. Product wants to hit their dates. Executives like me, we want results. Depending on the company you're in, you want predictable results and teams stopping development work on features because they're gonna go spend three sprints working on tech debt does not look like [00:10:00] predictable results. I So how do you separate that useful debt from the harmful debt, and how do you recognize when it isn't debt at all? But a signal that your system needs attention? Every company has tech debt. Every company I've ever worked at, every company you've ever worked at, as I said, every line of code that gets checked in today into the main branch is gonna be tech debt tomorrow or in a few years. So if every company has tech debt, the mature ones know why it's there because it was a deliberate choice or it wasn't, but they understand how it happened. I mentioned that I joined this company and the company had been around for a while and I inherited a ton of tech debt. And spent a bunch of time addressing it because it was really becoming a problem for the company. That company spent years building a monolith. It wasn't the result of being [00:11:00] lazy. It was if you have a small team, monolith is the best architecture to build. So if you're an early stage company. It makes sense to build a monolith because you don't have a ton of developers. They can coordinate fairly well, and so that works great. It's easy to deploy. It is a good architecture for an early stage or a small team. We don't like it later because now you have multiple teams. They're all contributing to the same code base. They're stepping on each other's changes. They're breaking each other's changes. As the code base gets larger and larger, the deploys are taking longer and longer. There's more tests that have to run for every change. And this was the problem the company had, is it had been the right decision to start with a monolith, but as the company grew and grew and continued to add code to this monolith, they hadn't started to address those issues. [00:12:00] They hadn't started to think, okay, what's happening now? Over time, that monolith was getting slower and slower and they were having to do lots of tricks to make it fast enough for customers. It was taking forever, like a day to deploy. And only the DevOps team could do any kind of deploys because it was very error prone because of all this accumulated code. It was very fragile. Lots of things could break it, and so it was legitimately slowing the teams down. When I joined the company, that was one of the first things I saw. It was just the time it took to deploy, your cycle time was limited by the ability to deploy the code. It was the right and the right delivery expectations, but that debt constrained what the company [00:13:00] could build, and we had to address it. That became obvious almost immediately to me once I joined the company. Another company I worked at, Spotify, we would very often take on intentional debt. If you've heard my earlier talks, or read the articles or read the book where I talk about the Spotify, think it, build it, ship it, tweak it framework, which is just essentially a very lean startup way of approaching feature development. You're adding a lot of debt there, right? Because you're trying to bring value to the customer as quickly as possible. You're not trying to bring the perfect version of a feature you're trying to. Bring the MVP and you're gonna do as fast as possible. You're not necessarily choosing to write poor code, but you're choosing to not solve the bigger problem until you're sure you need to. So if you can solve the smaller problem, you solve the smaller problem, knowing that if you do this well, you're gonna have a [00:14:00] bigger problem that you're gonna have to solve. But what we would often find is that, "you know what? The smaller problem is not a problem our customers have. So, it's good that we didn't waste a lot of time solving the bigger problem, because that wouldn't have been a good use of our time." So that was by choice. We were very good at taking on intentional debt. We could do that, especially during discovery phases, because ownership was clear. Boundaries made sense. And that tech debt didn't become dangerous. It was revisited or retired naturally. Across roles and companies, the same pattern has shown up time and again, intentional debt succeeds when teams treat it as a conscious trade off and unintentional or symptomatic debt quietly accumulates until it shapes the entire system. From these experiences, I've put together a little framework for y'all that I'm gonna share. I do wanna reiterate the [00:15:00] healthiest teams take on tech debt intentionally. The worst debt I've ever seen was never a choice. It was a symptom. The framework I'm gonna share with you is a four-part lens that distinguishes the intent behind the debt and then the impact it creates. If you don't understand why the debt exists, you won't make the right decision about what to do with it. The goal is to understand the intent behind the debt so we don't misjudge the cost. Part one of this four part framework. Is intentional tech debt, what I would maybe call pragmatic tech debt. This is debt taken on by choice because you're validating an idea, you're avoiding premature optimization, you have external delivery constraints, or you're buying time to learn. Healthy engineering teams do this regularly. It only becomes problematic when it's invisible [00:16:00] or untracked. This is, to me, a sign of a healthy team is that they are taking on pragmatic tech debt, but they're also very conscious of the choices they're making and they know which of those things they're going to have to address later. A question for you when you're thinking about "is this pragmatic tech debt?" Would be, "did we take this tech debt on, on purpose? Do we have the awareness of the trade-offs we're making? Are we being deliberate?" The next type of tech debt is required tech debt. That's tech debt you must address. So that's debt that directly impacts reliability, security, and compliance, scalability, and your ability to deliver roadmap work. This kind of tech debt is not optional. Avoiding it becomes much more expensive over time. One of the [00:17:00] challenges I've seen engineers and teams have is that they want to classify all their tech debt as required tech debt. Whether it is or it isn't, because for whatever reason a developer can see. That this function that they don't like for whatever reason, but has worked forever, is someday going to be a security problem. It isn't today, but someday it really will be. Or it's a little bit of one, but they're certain it's gonna get worse or it's gonna be a performance problem someday. Isn't today, but they're convinced it's gonna be worse. The biggest fallacy I see is that teams see all tech debt as required tech debt. So a question for you to identify what is truly required tech debt is, "does this prevent the business from moving in a strategic direction?" If I go back to my experience with the monolith that [00:18:00] had outlived the benefits it brought, we couldn't ship quickly because of the monolith. We had lots of incidents because of the monolith and the architecture. It was hard to add new features. It took longer, and I could measure that. The teams could measure that. So that was clearly holding us back. It was slowing us down and as we improved upon it, as we moved more of our new development out of the monolith. Into new services, our ability to deliver improved dramatically, our stability improved dramatically. We were able to deliver more value to customers faster, which proved that this truly was required tech debt. So make sure that it doesn't feel like it, but actually is. The next type of tech debt in the framework is incidental tech [00:19:00] debt, which is safe to ignore for now. So that's tech debt that is low risk, stable, isolated, well understood, and rarely touched. Every mature code base has this. It is not the enemy and people may not like it. It may be in a language that nobody on the team writes anymore. It was a language that the company was using and stopped. And so it's this code that nobody likes 'cause it's not in the language that they like writing in. And so you wanna rewrite it, but you know what, it's worked forever. Still works. Why would you choose to change it at this point? What value does it bring, if it works? There's a lot of this kind of stuff. "Oh, I don't like the way this was written. It's too much white space, or too many comments, or The variable names are weird, or, I don't understand this logic." "Okay, sure. Yeah, I understand that. Is it working?" "Yes."[00:20:00] "Then why are you messing with it?" If teams are too willing to see all tech debt as required Tech debt. Teams have a hard time identifying tech debt as incidental tech debt. That usually comes down to us, the leaders, to ask challenging questions to make it clear that this can be ignored. The question you can ask is. "Is the cost of fixing this higher than the cost of living with it?" Knowing that if it's been working fine, you've had no problems with it. The minute you mess with it, you're probably gonna break something. You're going to introduce problems. You don't understand that it was implemented this way for a very specific reason. And so you think you are replacing it in place with something identical and you learn that, oh, it's never been identical. That's always what happens. So yeah, be very thoughtful about [00:21:00] is this tech debt, incidental tech debt? The final type of tech debt I'm gonna talk about is symptomatic tech debt, that's a signal. This is the most interesting and possibly the most insidious. Type of tech debt because this is debt that repeatedly appears because of unclear product strategy, frequent priority shifts, fractured ownership, Conway's law issues, and the lack of engineering capacity for anything but delivery. Fixing the code isn't gonna fix this. You have to fix the conditions that generate that debt. This one is a toughie because the, this system that produced this tech debt, this is why fixing the code doesn't fix it. The system is just gonna keep producing more of it. You identify that you have this tech debt in part of your platform that's [00:22:00] meeting the requirements of required tech debt. So you fix it, but because it's not the work of a developer or an incorrect assumption or technology evolving that's created this tech debt, it's a result of a poor system in your company. While you're fixing that, you're generating it everywhere else. The question here to identify it is "what system is creating this debt?" It's worth looking at each of these four types with that lens, first. Is this an isolated thing? Or is this actually indicative of a bigger systematic problem? Something about the way we do our software as opposed to a set of maybe once good, no longer good implementation decisions. So symptomatic tech debt is that last [00:23:00] kind. So let's get practical. Treat tech, debt decisions like any other trade off, deliberate, visible, and tied to business outcomes. If you heard my talk from lead Dev, either 'cause you watched on the lead dev site or you went to YouTube to look at the recording of the rehearsal. I did. If you heard that talk, one of the things I talk about to resolve disputes between product and engineering is that engineering needs to. Talk about things with impact to business outcomes. That is the language product speak. It's also the language the rest of the company speaks. If you can demonstrate business outcome value for your priorities, you can get support for those priorities. The tech debt is worth fixing has impact to business priorities. And it's important that you tie those tech debt decisions [00:24:00] to business outcomes, and that includes both that pragmatic tech debt where you're choosing to add tech debt. That comes also to that required tech debt where you're choosing to fix it and trying to get that prioritized. Tech debt decisions aren't technical decisions, they're strategic ones. Let's talk about how to evaluate tech debt. Well, let's start by identifying intentional versus accidental tech debt. Intentional is that pragmatic debt and the accidental is that required tech debt, the incidental tech debt, and the systematic tech debt. Let's start by figuring that out. You've heard me talk in prior episodes about mapping the flow of work to understand what the bottlenecks are in the system of your organization, mapping the flow work also shows you what the [00:25:00] consistent friction points that could be structural issues. Think about the effort to complete a ticket. when we talked about the system that is your organization, and we talked about mapping how work flows through that system. You do the same thing. But let's look at a ticket today. You're trying to understand the impact of this tech debt. Map the ticket. How much of the effort in completing that ticket is spent managing or working around tech debt? Consistent friction points reveal structural issues there. It's worth doing and it's worth doing across a range of tickets so that you're identifying those core problems as opposed to lots of little problems in lots of places. Assess the optionality loss, what becomes harder or impossible if we leave this alone? You've mapped a [00:26:00] problem that is a tech debt problem in your platform, in your code base, you know it's there. What becomes harder or impossible to leave it alone is this incidental tech debt that you can ignore for now, and then also look at safety and risk first. Reliability and security trump everything else. If this node of tech debt that you found doing your mapping exercise is a security issue? Is there true reliability issue, not a potential reliability issue? All right, well, now you know what you need to do. So how to reason through these decisions for intentional debt? That's that pragmatic debt. Track it, make it visible and revisit it deliberately. Come back to it on a regular basis. See if it's now becoming a problem. It was a choice you made. It may have been the right choice at a certain point. No longer the right [00:27:00] choice, sometimes. For that required debt. Prioritize it on in the strategic work and you should be able to do that because you should be able to attach, fixing it to a business outcome because you've established that correlation. For that incidental debt, which is that safe to ignore debt, leave it, just leave it unless it becomes chronic. And then for that symptomatic debt. Which is you can fix the piece of code, but you're not fixing the problem, address the upstream issues. That's a system problem. That's not a platform problem. It's tough, because I wanna say system when I'm talking about your code as well. So I'm trying to talk about the system of your organization versus the platform. So, if you hear me say system, I'm talking about. The system of your organization, not the [00:28:00] system you're coding on. I'm sorry if that's confusing. For that symptomatic debt address, the upstream issues and that strategy capacity of your team ownership and the structure. And when I say structure, I don't mean the software architecture, I mean your organizational structure that's. Potentially a fountain of tech debt that's being spewed into your code. Where teams usually go wrong here is they treat all tech debt is equally urgent and we've just identified four types of tech debt, one of which is must fix now. The other three are very shades of, was a deliberate choice, not a problem. And. Fixing the code's not gonna fix the problem. So only one of those four was, you gotta go do it now. Not all debt is the same. Treating, no debt is urgent. Just continuing to accumulate it and not really thinking about it. Also, a mistake [00:29:00] refactoring based on frustration, I don't like this code. This code was written by somebody other than me, and I think it's horrible. I wanna rewrite it. That's frustration. That's not strategy. And then ignoring structural or cultural forces that produce that recurring debt, that's that symptomatic debt or debt as a signal. That's where teams usually go wrong. You're listening to this podcast right now. If tomorrow's a working day, how do you apply this? Categorize your top three sources of recurring pain. Hopefully you know what those are. If not, figure them out. Map your system. Document intentional debt decisions, why it exists. When you're gonna revisit it, put a date on it, and a calendar reminder and what good enough means. You're deciding to be pragmatic, you're deciding to solve your current problem, not a future problem. That's great. You know that you're [00:30:00] not solving it in the best possible way. Where do you think it will fail? Yeah. When do you think that will happen? And then before that happens, set a reminder. Let's revisit this decision at this time. Look and see how things are working. If they're still working, great. We can leave it. It was a good pragmatic decision if that code is starting to come under strain. Okay. It was still a good decision. We caught it before it broke. Let's address it now. Let's prioritize that work now, because we can see that this worked for as long as we needed to, but now we need it to handle the new reality. So let's revisit. And for that symptomatic debt, it's not a coding task. You gotta start with the system of your company, and that's the question. Let's talk about the human layer of this. Most tech debt isn't about code. It's about people pressure and the incentives that we give them. [00:31:00] Engineers may feel guilty about taking shortcuts, and we as leaders, we need to normalize intentional debt as a legitimate tool. This is a decision we are choosing to make. Now, I know you wanna solve the five year problem. We really need to solve the six month problem, and then we'll learn from that and that may give us a different five year problem. Or in six months we may have realized that things are fine, or we may then need to solve the one year problem, but we are making a choice to do this. So that's where we as leaders come in. Product partners, if you're listening or you can explain to your product peers, they need clarity that some debt is an investment in learning faster. We are choosing to take some debt here that we are going to need to fix later, but what we're trying to do is learn and we're trying to learn faster. So we're choosing add this [00:32:00] debt. But we need a commitment to be able to clean it up later with the knowledge that we've taken on. Executives. So your boss or you, depending on who you are, need visibility into which debt threatens business outcomes and which doesn't. As a CTO, I am very aware of lots of tech debt in our systems. Talking to developers, I hear a lot more. What I need to understand when I'm talking to the people in the technology team is which of these things is a real risk? Which of these things is going to affect our business outcomes? Because if I'm going to make a case. Potentially, depending on how serious something is, I may need to make a case to the executive team. 'cause we may have to choose [00:33:00] to maybe achieve a goal on a longer timeline because we have to address this issue. I can't do that unless I can make a cogent business outcome discussion so that my peers understand the necessity of this. For skeptics in your team who don't like the idea that we are choosing to add tech debt, why do we make this choice? They need transparency. What's the plan? And if you say we're gonna revisit this in six months, you gotta revisit it in six months. You can't just constantly be promised, oh, we'll look at this later and then never do it. Because then people feel obligated to build that five year vision of what they think needs to happen. 'cause they'll know that if they build the six month one and you say you're gonna revisit it, that you won't, and then they know that they won't [00:34:00] ever get a chance to clean up something they're not happy about. So you need that transparency and you need to follow up when you say you're gonna follow up and you still may choose not to fix it. If it's working fine, you may choose not to fix it, but you owe the developers on your team that revisit. So teams under constant pressure, accumulate symptomatic debt, not because of the skill gaps in the team, but because the system leaves them. No alternative. If you've ever been in an organization that's just go, go, go, go, go. They're, tech debt machines, like they're just producing tons and tons of tech debt and they never get the time to come back and address it because there's always the next set of things, the new things to deliver. It is incumbent on us as engineering leaders to address those symptomatic problems because those systematic tech debt [00:35:00] fountains are also killing team morale. Nobody wants to be on a team that is being forced to make poor technical decisions for external reasons and never get the option to improve it. Mastery is an important part of happiness, autonomy, mastery, and purpose teams that are producing systematic debt. Definitely don't get to work on their mastery and honestly, probably not having a lot of autonomy either. So there's another reason why we need to address that, the problems in the system. And that is because we are absolutely going to be burning people out. If you can identify that a significant amount of your tech debt is symptomatic tech debt that it is absolutely incumbent on you to really try and address that. And again, that's not necessarily, coding isn't [00:36:00] gonna solve that problem, that's a leadership problem. Tech debt is code telling you a people story. Experience teaches you that tech debt isn't the enemy. Unmanaged debt is. Early in my career it was a very natural assumption that tech debt meant somebody made a bad decision. You look at some code, you see the comment that says, "I don't know how this works." I've been in code written by professional senior developers that has comments in it, like " fixing this for this release, but we absolutely have to come back and replace it. This will not last for long." And that release was 10 years ago or five years ago or whatever, and that code's still there. I'm sure you've seen that too. That's so common. So I thought of tech debt as somebody making a bad decision. And it's a very natural thing when you're first entering in the field, experience has shown otherwise. [00:37:00] Some debt is strategic, some debt is benign, some debt is dangerous, and some debt is a warning that the team structure or the business priorities need attention. Now I look at tech debt less as a technical issue and more as a diagnostic tool for understanding the health of the system. How much of our tech debt is intentional? How much of our tech debt is unintentional? How much of our tech debt is symptomatic? Some closing thoughts. Tech debt isn't inherently good or bad. It's a signal. And the signal only makes sense when you understand why that debt exists and what it implies before you fix tech debt, understand the intent and the system behind it. That's where the real leverage is, because tech debt doesn't derail teams. Unexamined tech debt does. If you treat every piece of debt as a defect, you'll waste time [00:38:00] if you treat every piece of debt as harmless. You'll create fragility. If you understand why the debt is there, you'll make better decisions for your team and your company. So start with the intent and then look at the system and only then decide what to fix. I hope this has been helpful. It's a bit of a philosophical approach to tech debt, but as I said. It's important to know. I hope this has been helpful. It's a bit of a philosophical approach to the subject of tech debt, but I do think that framework of understanding these different types of tech debt, being able to characterize them and then when you understand the different types. Then putting together a strategy on how to resolve them. Hopefully we'll help you in your day to day as you go forward and [00:39:00] help your team as well. Let me know if this has been helpful. Let me know if you have other challenges around tech debt that I didn't address, or specific tech debt problems. I can maybe give you some questions on trying to figure out what kind of tech debt problem you've got in your team. I wanna remind everybody, once again, I'm speaking in February at ConFoo in Montreal. I'm very much looking forward to this conference I've spoken at before, but I spoke at it during CID virtually. I haven't been able to get to Montreal for it. I love Montreal and going to Montreal, so I'm excited about that, even if it is in February, and so I'm hoping to see some of you there. If you're gonna be there, let me know. I will see you next week in the newsletter. I'll see you in two weeks back here on the podcast. Have a great week.