8: Ep. 8 | Technical Debt Prioritization

Brittany (00:00) Welcome to the Overcommitted Podcast, where we talk about our commits, our commitments, and some stuff in between. I’m your host, Brittany, joined today by Bethany and Erika We are a crew of software engineers that met at GitHub. We first connected as an onboarding group on the same team and have since scattered elsewhere. But we discovered our shared passion of learning and building cool stuff.

So we still like to come together and share what we’re learning and explore our life as developers. Whether you are committing code or committing to new challenges, we are glad you’re here. Let’s dive in. Today’s episode is going to be about technical debt. And we’re going to talk a little bit about how we prioritize it, how we decide what is worth prioritizing, and how our opinions of it have changed over time.

So let’s set the stage first and describe what technical debt is or tech debt. Really it is a classification of work that doesn’t necessarily work towards a specific feature, but is decisions that you have to make typically while building out features, often while building something very quickly that you accumulate over time like a debt that needs to eventually be either repaid or somehow offset.

So for example, if you choose to write really quickly without any unit tests, eventually that technical debt is going to come back to bite you when you don’t have adequate testing as an example. So let’s talk for a little bit first about how we prioritize technical debt repayment against new feature development. Erika, did you want to talk first?

Erika (01:34) Sure, yeah, I’m super excited about this topic because it feels very present in a lot of the work that we do and is kind of this unspoken or spoken like monster in the closet in a lot of development projects that it’s always there and it’s…

like never in my experience like directly addressed. So prioritizing it in the scope of a project typically has to serve some purpose of the project, which, you know, to be fair, makes sense. Like if there’s some kind of tech debt that’s blocking you from developing the feature or will…

unlock the feature development in some way, then that’s prioritized as a part of the project work. Some recent examples, we had some feature flagging framework changes that we needed to implement for one of our projects. And it was blocking the ability to roll out those features safely.

We tackled it as part of the project work. But unless we had that active pressing need, we probably would have pushed it off. So that’s been my experience of addressing tech debt. Otherwise, it comes in through customer support requests. And if it’s of a certain severity level,

and causing some kind of problem or a part of some kind of problem for a customer experience, it’ll get picked up as a part of some kind of first responder or shield rotation. But a lot of times, those tend to happen very quickly. It has to be completed within the span of a few days. So tech debt usually sits around for longer.

What about you, Bethany?

Bethany (03:18) Yeah, so I was recently at an offsite with my organization and they did it kind of conference style, so our VP gave a keynote on the second day and I thought she had a really interesting take on this with explaining how there’s, I believe she called it polarities, whereas in some phases of

growth, you’ve got to focus on really pushing the needle at all costs. And sometimes this is building up that tech debt and like building up that debt. But then at other times, you really need to focus on that reliability aspect and paying down that tech debt. And there’s like phases of your development cycle where you’re actively focused on that side of things. I think that

With prioritizing technical debt, if it was up to engineers, that’s probably all we would prioritize. It’s really about, Erika, to your point, leadership buy-in and product buy-in. Because if doing something isn’t going to meaningfully push your product in a certain direction, then it likely won’t ever be prioritized by the business. So you either have to find out how it will impact your product in a good way.

convince your product team or leadership team of that, or maybe advocate for gardening weeks or something like that where engineers can kind of go after things that have been bothering them and have a time boxed section where they can just go after things that they’re interested in fixing.

Brittany (04:48) I like that phases approach because I feel like it kind of speaks more to the debt word as like leverage where, you know, you’re really leveraging that tech debt so that you can move faster. And theoretically, you know, pay that off at some point in the future. Although I have yet to see a really solid example of how to do that other than like fixing something that’s directly causing a Otherwise it can be really hard to.

really hard to prioritize those things. And honestly, think also, I think as engineers, like you said, we would continuously prioritize that over anything else if we could. And I think that we kind of overvalue like the cleanliness of code. You know, I think a lot of people are like, Hey, I think that if I, if we clean this up and refactor this, we’ll be able to move so much faster. But I feel like I haven’t actually seen that like in practice translate to people actually moving faster because they always end up accumulating.

more tech debt in some other area to be able to do that. So, wow.

Erika (05:43) Yeah, definitely

it always feels like the engineers are like requesting buy-in for paying off the technical debt. But I did recently, and maybe you both already know about this, read about like, I think it was Jeff Bezos when he was…

on, well, he made some kind of mandate at Amazon about a modularization approach. And it was a top-down mandate that every team needed to modularize their service area, or otherwise they would get fired. And it really struck me as something, I mean.

questionable approach to leadership, you know, it struck me as the opposite of what I’ve a lot of times experienced where the reason for modularization is often this like circular dependencies and you know, like that affects availability and code maintainability. And so like to me that was, yeah.

like addressing the tech debt from a top-down perspective saying, look, we’re not actually going to be able to maintain this code base or this site if we don’t fix this issue. And yeah, so I guess sometimes there are experiences where it does happen in the opposite direction. Yeah, but it has to be serving a higher purpose.

side availability or, you know, maintainability over the longer term.

Brittany (07:14) That’s really interesting. I’ve never heard of a mandate from a CEO at that level for something like that. And I think there’s probably value there for having everybody pushing in one direction. think in a lot of the roles that I’ve worked in, a lot of people are a little bit afraid to tell engineers how to solve a problem. And sometimes I think having a little bit more direction that the entire company is focused towards isn’t probably a bad thing.

Like everything in software, it depends. What approaches have you found effective for preventing tech debt instead of just paying it off? Is there a way that you can just not accumulate this in the first place?

Bethany (07:48) so product managers close your ears for just a second. I personally find that, whenever I’m asked to engineer something or like asked for an estimate for how long something will take, I’ll actually go 20 % more to 50 % more. I mean, one for being able to like handle, unknowns and such.

But also, was during a project I was working on within my past company, and me and the other engineer working on it decided that we were going to allocate 20 % of our time to table stakes, we called it. And so this was like unit tests, like making sure everything made sense, like thorough code reviews.

load testing, things like that, that really made the difference for cleaning up that tech debt before it could really aggregate, especially for like a greenfield project where we’re starting out from scratch. And so I really like those kinds of approaches for like factoring in technical debt to the actual estimation.

And if things need to be cut down, then it can be a discussion from there. But if you’re just asked for an estimation, include the stuff that you think is important in that estimation.

Erika (08:59) think I’m with you. I’m terrible at adding estimations or adding on to estimations because I always get nervous that somebody’s gonna like question me even though I don’t think anyone ever has. But yeah, I think sort of what we’ve been saying like I also like the idea that like

Maybe the better approach is not to not introduce any tech debt, like minimize the amount that you introduce with the understanding that like, if you have a little bit of tech debt that you carry over into the next project, you can find time to address it.

cause yeah, the only way I’ve ever found it, not introduced any tech debt is to not write any code or move it such a slow pace that nothing ever gets really completed.

Bethany (09:47) Yeah, I totally agree. think tech debt is the most effective when you are intentionally choosing to take on that tech debt. So, I mean, similar with real debt, there’s good debt and bad debt. Bad debt tends to be the unexpected debt that comes. And then good debt is things like build your credit score or buying a house or something that brings you value.

get a house in exchange for debt, I guess. So I think the same goes for software. That flagging tech debt as it’s happening, saying, okay, make sure we’re all aligned. We are choosing to not do this in favor of going fast or getting to market sooner or X, Y, Z and making sure there’s buy-in across all.

product and design and whatnot on that decision because it should be explicit what the trade-offs are.

Brittany (10:39) I like that. Have you, switching gears a little bit, have you ever inherited a code base with significant technical debt that you didn’t create? And how did you go about approaching that?

Bethany (10:52) rewrite the service.

I know. I think whenever you inherit a code base and it is nearly impossible to work in without having the context of the people who initially built it, I all code serves a purpose and even

if it’s messier, has a lot of tech debt, it got you this far. But if the choice is that you inherit that code base and you cannot properly operate in it, then sometimes it’s just easier starting from scratch rather than trying to get all that context back or all that, clean up all that tech debt that you’re not even aware of or that you know of.

Erika (11:35) Yeah, I think I’ve been pretty lucky. Like, I haven’t necessarily worked in too many code bases that are geriatric and unwieldy. So yeah, I feel like I know, I’ve like read about this, of like how to address tech debt and like sort of the guidance is like start somewhere.

Yeah, like you can’t fix everything all at once, like make meaningful changes where you have to work and then like eventually you’ll pay off the, pay off the debt. yeah, but thankfully I haven’t had too many experiences personally working that way. How about you, Brittany?

Brittany (12:13) the name geriatric code bases. think I’m going to use that from now on. I love that. Yeah, this also doesn’t happen very often for me. think the case where this is probably going to come up is if you are inheriting a code base that isn’t like…

actively being worked on because most of the things that are actively being worked on are things that, you know, people are using regularly. And I don’t know. I have strong opinions. This kind of gets to the next question. I have strong opinions about review rates in general that continuously get stronger over time and

I feel like I’ve never been a part of one that was truly successful in the sense that net we saved more time than we would have spent trying to slowly refactor this code base and get it into a spot where we can add all the new features that we want. And so.

I think of that as like the engineer in me really wants to reach for that, but then the person who cares about like product and like serving customer features is like, no, don’t rewrite it. Don’t, don’t, don’t.

lean towards that because it’s going to take longer than you think. And like I said, I think that people kind of overvalue the cleanliness of code in terms of getting things done and the amount of bug fixes and band-aids on an existing code base that like, they might be band-aid fixes, but like they fixed an actual bug that people are going to use. like you still have to deal with probably that same bug when you rewrite it because it’s just the nature of how your service works. So.

Yeah, I think, my approach…

is yeah, to just do iterative changes and also just accept that some things are going to be hard and like, it’s never going to make sense to rewrite it. You know, some things work perfectly fine, even if you don’t really want to touch them. and I’ve definitely come across those before where it’s like, man, it really sucks to make changes in this space. And it’d be so much nicer if we could just go through and redo it all. But like that would be a multiple month project just to give feature parity to customers that could

spent doing something like delivering a new feature or something along those lines. ⁓

Erika (14:10) Yeah, I also think with like free

writing, it’s hard to communicate the value unless you have a really clear vision of what needs to happen. So like we did actually do something in my org recently that was addressing a major area of tech debt. And we had like this process like

I work on the authorization team. So a process of setting up a token for authorization for a user or an application, and the process of creating a new token for any service owner was lengthy, complicated. It took multiple days, spanned multiple pull requests. And again, we had this big initiative coming up that we were saying like,

we don’t actually think that this is going to get completed unless we address this process and make it easier. And this had been a pain point for a really, long time. And several engineers on the team had a really clear idea of what to do, get rid of the database transition, make these in memory. You know, those were kind of

the major changes and then, you know, put it all in a central location, and improve the documentation. Like, I think those were basically the biggest things that we changed and it was super successful. but I think if we hadn’t felt that pain for so long and we hadn’t, well, I don’t know if like length of time is necessary, but if we, if we didn’t have that design of

this is what is really wrong with this system and this is how we’re going to change it. I don’t think it would have been nearly as successful in the redesign. And it actually didn’t take that long. It took a few months. But yeah, so I agree with you that sometimes it’s better to rewrite it, start from scratch, but I think only if you know you’re going to make it better and like,

how you’re gonna track, like what is going to be better about it. In this case, it was like time to create a token.

Bethany (16:08) really love that, like, making sure you have data on what specifically you are looking to improve or what you’re looking to do. To, I guess, go back to the house analogy or even a car analogy. If your car has enough damage that it’s not worth it to fix it, you buy a new car. You don’t continue trying to fix a car.

that it’s just not worth it to fix. Same with the house. Sometimes you’ve gotta, there’s so many changes that you wanna make to the house that it’s as worthwhile to rebuild it from the ground up. Sometimes you do that, but a lot of times you also don’t wanna do that. You wanna just make smaller changes. So it very much depends, and I agree with that.

people do reach for rebuilds a little too much or think, this will solve all our problems when in fact it’s just going to create a class of new problems. But if you are very aligned on what you’re trying to fix and what your goals are, I think it can be very successful and very good to know what the cost is of choosing a rebuild over continuing to just build on top of this.

other service or adding incremental changes.

Erika (17:21) because I’ve also been a part of some tech deck conversations that feel like you’re, what is it, like moving the deck chairs on a sinking ship or something, like you’re like, okay, we’re doing this thing, but like, what is it actually accomplishing? Like, or, you know, we’re like changing this color arbitrarily. Like I prefer yellow over green. Okay, like it’s not really worth doing. So.

Yeah, tracking the outcome is definitely important.

Brittany (17:50) Yeah, I think the other thing that I really like about that example, other than having the clear goals, which I agree is super important before approaching a rebuild, is the time that you spent to think about the solution. know you said that you don’t necessarily, there’s not like a set amount of time, but I think that also gives you more time to digest what the actual problem is and come up with a plan that is very clear and straightforward to implement and that you’ll be able to know in advance is going to actually fix the problem.

Yeah, taking time to stop and really think about what the goals are that you’re trying to achieve and also what is like the right solution is good. We are approaching time. So I would like to ask if there are any other hot takes or changes that you’ve had over your career when it comes to thinking about technical debt. Is there anything that you think, you know?

is really key. think one thing that comes to my mind is I think that tech debt is probably the most exciting area when it comes to AI agents right now, where I think that that’s going to get me a lot more excited to try out AI agents. if I can have it address tech debt, I think that that’s something that I’m pretty excited about and actively looking into. Any other thoughts on the future or how tech debt is changing over time?

Bethany (19:07) Mine would be that I guess when I was starting my career, I very much interpreted Tech Debt as only technical improvements or like technical setbacks. So like not including tests or having a monolith, which is a bad example, but was an example from a past job. But I think there’s also another level of Tech Debt in that.

like how the code was created or how understandable code is. I think maintainability absolutely is a concern of technical debt or a trade-off of going faster or whatnot. If you’re rushing to create a feature and you don’t have time to go back and make it not as complex or not as hard to understand.

then I think that’s absolutely a side of tech debt as well. If people are afraid to go into your code, that’s a sign that you’ve got some tech debt.

Erika (19:59) Yeah, I think similarly, like I had a more narrow view of TechDat when I first started. like, I almost think of it now as like drift from the like ideal representation of your model. And yeah, that’s more conceptual than practical, but I think like,

Like you were saying, Bethany, like ideal state is that your code represents your model, like system model, business model perfectly. But you’re not going to get there probably ever. So the best thing you can do is recognize the places where there’s a drift, where there’s a mismatch. And yeah, those are your areas of technical debt.

Brittany (20:40) Nice. All right. With that, we are going to go to our fun segment. And here is what I have prepared for today. So we are going to do a word association. What I’m going to do is I’m going to call on a person and say a tech or corporate-y buzzword. And you have to say the first thing that comes to mind in like a single sentence. And we’ll see how that goes. So I’ll start.

with sorry go ahead

Erika (21:05) Is this like, so this is like free associating? Okay, cool. I can do that.

Brittany (21:08) Yep. Yep. I mean, it doesn’t

have to be tech related either, but the words that I’m choosing are all like tech buzzwords. ⁓ Yeah, I’ll start with Bethany. Okay, the first term is tech debt.

Erika (21:16) you

Bethany (21:23) a clean up or garbage collection. Yeah. Yeah.

Brittany (21:28) I like that. I like the term garbage collection applied to tech debt, actually.

Bethany (21:33) I feel I used to listen to Go Time all the time, RIP, I still wish it was here. But there was an episode where they were talking about Tech Det and they were like, some people just loved working on Tech Det only and should be like, like some people love being garbage, garbage collectors and stuff and it’s like interesting association there but.

Brittany (21:54) Yeah, it’s like a garbage collector for your projects.

Bethany (21:56) Yeah, valuable position and some people do just love it.

Brittany (21:58) true.

All right, Erika, your word is synergy.

You don’t even need to say a word, just the face was enough.

Erika (22:04) face is saying at all.

Like corporate. I’ve never heard anyone use that word in like a personal context. It’s always been like some gross, like, sorry, corporate corporate like buzz speak to be like enhancing synergy.

Brittany (22:07) Makes sense. really?

Yeah, I feel like it’s along the same lines as like flywheel. I feel like you only hear that in… Okay, Bethany. Blockchain.

Erika (22:23) Yeah.

Bethany (22:28) meme coins, I would say. It’s so unfortunate. Blockchain’s very good technology that got so, so abused.

Brittany (22:37) I agree. Erika, cloud computing.

Erika (22:40) Hmm.

expansive?

Yeah, like I feel like the clouds are super cool.

super cool development in computer technology that like really unlocked a lot of potential.

Brittany (22:50) Yeah, I agree. Bethany? AI?

Bethany (22:52) Google I.O. Do y’all remember that like a video of Sundar Panchayi that was like AI AI AI AI?

Brittany (22:59) No, actually I do. Yeah, somebody cut together all the different times in their keynote that they said the word AI.

Bethany (23:04) Yeah,

it was all that, which is sad. I work on an AI product, but that’s still what I think about that meme. That’s so funny.

Brittany (23:09) Mm-hmm. Yeah.

Erika, pair programming.

Erika (23:16) Ooh, rubber ducky.

Brittany (23:18) it. Bethany, here’s the last one. Design.

Bethany (23:21) not something I do a lot. I’m a back end. So I guess it’s more like architecture than is my design process than like UI UX designers. But I am taking that design course and I’m learning. I’m learning. Yes. ⁓

Brittany (23:34) Ooh, that’s awesome.

Erika (23:34) Woo!

Brittany (23:36) Nice. All right. Well, thank you so much for tuning in to Overcommitted. If you like what you hear, please do follow, subscribe, or whatever it is that you do on your podcast app of choice. I know they’re all different. Check us out on Blue Sky, and most of all, share with your friends. Until next week, thanks for joining us, and stay committed.