53: Scaling Search Engineering at DoorDash: From Monoliths to Custom Search Engines with Satish Saley

Brittany Ellich (00:00) Welcome to the Overcommitted Podcast, your weekly dose of real engineering conversations. I’m your host this week, Brittany, and I’m joined by…

Bethany (00:09) Hey, I’m Bethany.

Brittany Ellich (00:10) We are joined this week by Satish Saley a senior software engineer who spent years on DoorDash’s search platform team where he co-authored two major engineering transformations. First, rebuilding the search indexing pipeline with Kafka.

Flink and Elasticsearch to handle a rapidly growing catalog and later replacing Elasticsearch entirely with a custom in-house search engine built on Apache Lucene, resulting in a 50 % latency reduction and 75 % hardware cost savings. And he’s now bringing that experience scaling systems at hyper growth companies to LinkedIn. Welcome Satish.

Satish (00:50) Thank you, thank you Brittany for having me.

Brittany Ellich (00:53) Awesome. To kick us off, ⁓ Satish, what is one thing that you are currently building or obsessed with learning right now?

Satish (01:00) ⁓ Currently, I’m obsessed with learning anything that new comes out in AI world. Definitely the open-clothing that is ongoing, that is on Marauder. I have been learning that. ⁓ Yeah, I have been trying that in a few, in a sandbox environment. ⁓ Yeah, I’m more passionate towards what is new and what is the skill set that we have acquired so far and how to bridge the gap.

between these two to figure out something new.

Brittany Ellich (01:34) Nice. Do you have an open claw bot then that is is current that is running currently or.

Satish (01:39) Yes,

I have something on my old laptop. ⁓ Yeah, today I’m taking some time to ⁓ move it to Azure cloud so that it works in more scalable fashion. I’m facing some problems with my laptop, like the old laptop.

Brittany Ellich (01:56) Yeah, classic. I’ve noticed a lot more memory problems that I have recently. I wonder why. It’s weird. ⁓ Cool. ⁓ Yeah, so we wanted to talk a little bit about the experience of going from early days at a company like DoorDash, ⁓ what it looked like when you first joined. So can you take us back to the beginning of working at DoorDash and how things, what was the vibe on the team when you joined?

Satish (02:02) Yeah.

Yeah, sure. So I joined team in 2018. So the vibe was like a really start-up vibe where you like people are wearing many, many hats. You are not just joining a single team. You get to work on a lot of different things all through the tag or to the stack.

like you just name it like whether it’s a ETL pipeline or whether it’s a backend UI or like a search engine or like some consumer facing problems or some ML things. ⁓ So that was the vibe there. And in terms of scale or like in terms of search, if you talk about ⁓ the search was still being served by ⁓ the monolith application that Doordash had at that point. ⁓

Basically our team’s mission was to be the first service, be the first microservice at DoorDash, which gets extracted out of this monolith and just make it work and make it scale. So that was our mission when I joined the team.

Brittany Ellich (03:32) That’s cool. Yeah, I use DoorDash a lot, probably more than I should. And ⁓ it’s one of those things that I really love realizing like, like there’s a lot of work and effort that goes into these applications that you don’t see when you’re just using them. So ⁓ the search part of it, was really fascinating.

Satish (03:37) Okay.

Brittany Ellich (03:49) Have you, were you passionate about search going into it or is this like just the team you landed on and you just ran with it?

Satish (03:56) I never had such experience before joining DoorDash. I was originally slated for a different team altogether and the search manager at that point, he reached out to me saying like, hey, do you want to try out something in the search domain? We need some people ⁓ who has worked on microservices. So I said, okay, I can give it a shot. Let’s see, let’s learn some new things. So that’s my segue into.

Brittany Ellich (03:59) Okay.

Satish (04:25) search ⁓ as a domain. ⁓ Yeah, and I loved it there. I learned quite a lot of things throughout my journey.

Brittany Ellich (04:36) That’s cool. Bethany, do you have any experience with like scrappy initial startups of growing from there?

Bethany (04:44) Yeah, I mean it was around the same size as GitHub but less of an engineering culture, I would say. I mean…

I’ve mentioned this on the podcast before, but my initial team, which was a little designed to make a little chat app on marketing pages, when you click and so you can chat with an agent that originally was a monolith. And then they decided they wanted to break it into microservices, but not like two or three microservices, but like 20 to 30 microservices. So I think that was like trying to scale, but

maybe not in the direction we needed to scale in ⁓ given that time. So I do think it’s ⁓ when you’re talking through scaling the typical things like microservices or having like Kubernetes and things like that, it’s really important to take into account what your true scale is and whether you do need all the shiny new things. So I’ve definitely been in a position where it was…

We maybe were trying to scale too much too soon and it just made it an engineering nightmare for our for a team to maintain

Brittany Ellich (05:57) Yeah, there’s definitely a spectrum between ⁓ monoliths and microservices and it sounds like you are very much at the end of the spectrum. ⁓

Satish (06:03) Yeah.

Brittany Ellich (06:06) So Satish, one of the articles that you had shared before this, and I’ll make sure to link them, of course, in the show notes, talked about how you went from store search to item search at DoorDash, which was like a different problem. Was this part of the scaling ⁓ that you were going through or like when you were switching away from Elasticsearch? Or can you describe that a little bit?

Satish (06:28) ⁓

Yeah, so I think ⁓ this is on a high level a problem that we solved over the period of time as DoorDash grew. So DoorDash, typically when you think of DoorDash, you might think like, okay, I just get my food delivered, ⁓ but there are many more ⁓ verticals to DoorDash. Like you can get flowers delivered, you can get convenience and grocery items delivered and… ⁓

When you look at this problem from a business point of view, a restaurant probably have like 20, 30 items on the menu, but if you go to a convenience store, if you go to a Walgreens, there are like literally thousands of items, like more than 5,000, 6,000 items. So this directly translates into information retrieval problem. So where for a restaurant search, you…

might not focus that much on the items. You probably, when you think of ordering something, you think of like, okay, hey, let’s go to so and so store and like order some food from there. But for convenience and grocery items, you think about like, hey, I want to order tomatoes, I want to order eggs. That’s how your thought process naturally begins. And that is like your main… ⁓

a top of the funnel, how the queries get generated from the user’s point of view. ⁓ that’s how like we, ⁓ when we were designing the systems, we also needed to make sure that, these are not only catering towards ⁓ a restaurant ⁓ problem point of view, but these are also getting towards the item POV. And ⁓ when you’re defining the architectures, when you’re defining your constructs, those are also scalable.

when you have these mini items loaded into your retrieval engine. So yeah, so in a nutshell, would say this is like ⁓ an ongoing problem that we solved as the DoorDash grew, as the systems get matured. ⁓ But yeah, that is also, I would say, kind of one of the inputs for us to go from ⁓ a hosted search engine versus…

going into an in-house search engine.

Brittany Ellich (08:49) Did you know going into the Elasticsearch part that like this was likely something that was going to be replaced or wouldn’t scale past a certain point? Or did you, was it like a stepping stone or like a final thing when you did it?

Satish (09:01) I would say definitely a stepping stone. ⁓ So when you’re working on a technology, right? When you’re working on a retrieval engine, ⁓ definitely in a startup phase, you want to build the business. Because if you don’t have the business, then building technology doesn’t make any sense.

So you want to propel the business as fast as you can by making use of whatever is available right at the desk. And when you scale the systems, when your business grows, then you look into, you drill deep into the things like, okay, probably a better idea to ⁓ bring in some efficiency, probably a better idea to go more deeper into the technology and like bring in new technology. So that was ⁓ what I would say our mindset was. ⁓

right at the beginning, like at least in 2018. ⁓ That, okay, these are like the stepping stones that we are going to build to ⁓ propel the product further. And later on, let’s see what other people are doing. And if you just look around, there are like so many blog posts ⁓ in the e-commerce domain, ⁓ where how companies have evolved, how they ⁓ build their search infrastructure. So…

I would say DoorDash is not very different or not like an outlier in that journey. So yeah, in short, so I would say we knew these are the stepping stones that we need to pay forward. ⁓ then finally, we will have a ⁓ tech-driven or a tech-heavy platform where ⁓ you would have more ⁓ each

each of the dimensions like the product, experience, platform, everybody can go into different dimensions and dig deep into those dimensions. ⁓ So yeah, I would say definitely a step stone towards that.

Brittany Ellich (11:11) Yeah, that makes sense. And then for the rebuild suggestion, was everybody on your team bought in on, know, yeah, let’s rebuild it from scratch and do it ourselves. Or was it like a challenge to get everybody on the same page about that?

Satish (11:27) I would say, so there are like a couple of blog posts on the rebuild. Like one of the rebuild is the Kafka and another rebuild is the whole search engine. I will talk about the first rebuild, like the Kafka pipelining and how we build that. So it was really hard to get buying on that. It was, I would say it was like still in earlier days of Doordash.

⁓ where we were still heavily ⁓ focused on the product, but as a team of four or five engineers, we know what are the breaking points in the system, right? And where are we lacking ⁓ in terms of supporting the product ⁓ in an engineering fashion way? So that’s where the problem statement came in. We didn’t have a very good…

sense of pipelines, we didn’t have a very good infrastructure to iterate on a search index ⁓ in a faster way. So let’s say if someone has an idea about like, how do I want to test out a change in the index? I want to use like a different kind of analyzers. So those things like require you re-indexing a lot of data and

that used to take like months before. And that literally translates into, let’s say, a product problem like, ⁓ for example, ⁓ let’s say from the user’s point of view, when users are querying the system, you know that the users are using a lot of ⁓ gen Z slangs in the query. And…

But your system is not tuned towards that. So what do you do? So you definitely, you want to run an experiment to see, okay, how the changes are behaving. ⁓ But to get to a point where you can run that experiment, it’s taking you like one, one and a half month just to prepare the data. And on top of that, then you run the experiments for a couple of weeks and then read out. And so it’s a very long process for a startup to survive. So we need to cut this down.

in hours, the iteration on the data. So ⁓ we know this is the problem from the engineering point of view. So ⁓ it wasn’t, mean, you would need to frame the problem properly when you are presenting the problem to your leadership, right? Because if you just talk about like engineering terms like, this is taking one month, I want to cut it down to six months, six hours.

It’s hard for someone to justify, okay, so what if like from one month you are cutting it down to six months, what is six weeks, sorry, six hours, what is the benefit you are going to ⁓ give me? So you need to frame the problem ⁓ in a way that the people understand the importance of that problem and the importance of the solution. and our team was also like, I would say,

kind of new in this whole process, like whole journey, everybody’s like still learning. So it took us some while to, okay, we need to structure the problem differently. We need to understand the language of the people to whom you are presenting this problem to. yeah, so that’s how we got the buy-in for that project by presenting it to the leadership and…

working on that. like the blog post says, it was a huge success. That system lived for quite bit of time, ⁓ given the pace at which DoorDash progresses. ⁓ Yeah, and when I talk about the second part where we build the search engine, ⁓ that I would say came naturally.

by looking at the data, by looking at the pain points. At that point, ⁓ we knew we have been operating the retrieval system for almost like four, fourish years, more than fourish years. We know the problems in the system. We know the problems in Elasticsearch. We know the problems ⁓ that comes with when… ⁓

when a critical system like a retrieval system is ⁓ in a hosted environment where you can’t really control a lot of parameters which you want to control. yeah, ⁓ so that was the problem that we have, like basically scalability challenges. if ⁓ someone asks me like, hey Satish, go and make this search engines scalable 10x.

Can you do that with that technology? And the answer is no, we need to think differently about this problem. And ⁓ one ⁓ important thing that we did this time was a couple of members in our team, they had a good knowledge of Lucene and they built a very quick POC ⁓ of a search engine. And we launched it on one of the products.

And it was a huge success. So now we have all the, what I would say like theoretical data. And now we also have a practical data point where, okay, we proved that, this thing works. And now let’s go deep into it and like, let’s get funded. Let’s get funding on this project and then move forward to this. So there are a couple of approaches that happened. And I think that is also a function of the state.

at which your company is in, the state at which your team is in, ⁓ the kind of talent that you have in the team also matters quite a lot.

Brittany Ellich (17:53) Yeah, that makes sense. That makes a lot of sense. think I’ve also heard that it’s like a rule of thumb that like you have to re-architect for every order of magnitude increase typically of users. And it sounds like you went from very few users to a lot of users and like just naturally came to that point. Which makes sense.

Satish (18:10) ⁓ Yeah,

yeah, I would say so. Could be a function of the users and ⁓ like the end users and as well as the internal users ⁓ as the internal users also grows ⁓ in terms of like the use cases that they’re solving, your system needs to evolve accordingly.

Brittany Ellich (18:32) Bethany, what’s your experience with rebuilds? Have you ever had to fight for a rebuild in your career or ⁓ fight against one maybe also?

Bethany (18:40) ⁓ I’m not sure if I’ve had to necessarily fight for or against any of them. Either I was dropped in kind of in the middle of one or… ⁓

Or like, I don’t think I’ve ever been on a team where it’s been like, we desperately need to rebuild and then ⁓ we need to fight for it. But I will say I love the thought of ⁓ thinking of it as like rethinking the problem and with having realistic expectations of your goals of the system. ⁓ So for instance, I was working on a latency problem a couple months

ago and ⁓ we were able to get some very quick wins, but for the latency expected, it’s just quite frankly impossible unless we rethink the entire system because we’re… ⁓

like we have the victims of all the network hops we need to make and all the, ⁓ like that were only located in one region and things like that. So it’s, it’s interesting looking from at the problems from that perspective of, of, okay, well, we have a system that has been designed to solve this problem currently where we didn’t have latency expectations, but in a system where we do have latency expectations, what would we do differently?

⁓ Yeah, I think too, it’s, you’ve touched on this a bit, but I think it’s so important to consider ⁓ your organization structure, your culture ⁓ before you tackle a rebuild, ⁓ because I think Conway’s law ends up mattering a ton, which to listeners, Conway’s law is that your ⁓ system reflects your organizational hierarchy. ⁓ And I think so many

Satish (20:34) Yes.

Bethany (20:37) So many companies do reorgs without necessarily thinking about the…

how that impacts the system. ⁓ And so you almost have to go into a reorg the same way that you’re going into a rearchitecture because it might end up leading to that because your teams will bump heads if they’re split off and then trying to work in the same code base or same architecture and things like that. I think it’s been interesting thinking, looking at reorgs through that lens of, okay, well, what does this mean for the code or the technology?

Satish (21:10) Yeah, we are definitely ⁓ just like corollary to like building new technologies, right? Or like moving from point A to point B. ⁓ Like over this period, like I felt like ⁓ that building new technology is very easy. ⁓ The migrations are very difficult. And this is something I would say like a grunt work, which sometimes good unsung.

⁓ when people work on it and might sometimes like folks might think like, this is like something, no, this is, want to build new technology. I don’t want to do migrations. ⁓ But I think like 70 % of the work ⁓ in building new technology is going into the migration. How you are making sure that you are keeping your promises.

to your customers, or whoever it is, like end users or internal customers, and making sure that your system is better than the previous world that it was, and better in terms of latency, availability, all the system aspects, but also operability, how you are enabling your customers to get the most value out of the new system, which your old system didn’t provide. And unless you add that…

⁓ value addition. It’s really difficult for your customers to ⁓ go to the new system, motivate them to the new system using the new system. yeah, I think that is something we like whenever someone is building something new, this is something that should be on the radar and there has to be like a very specific timeline on deprecation on the movement. Otherwise I have seen like ⁓ the

⁓ certain code bases they need to maintain like okay I need to maintain this one for the old customers or like the old code pass and some new code pass the new system ⁓ that really prevents you the system being scalable

Brittany Ellich (23:25) Yeah.

Bethany (23:25) I really love that considering the customer at the end of this whole system rearchitecture or system redesign or rebuild or whatever you want to call it. ⁓ Because I think sometimes rebuilds come from maybe folks wanting a promotion or to get their name out there or from ulterior motives, not necessarily a… ⁓

nefarious or anything, but just people wanted to try a new technology, like my first team wanted to try with microservices, for instance. And so I think it’s, you can tell when a, when a rebuild is in good faith based on what that migration plan is and that there is a plan to

to roll it out and not just a plan to build the tech. Like separating that building from the rollout is absolutely the most critical part of a rebuild.

Brittany Ellich (24:24) Yeah, deprecating, decommissioning too. Like it’s not just you turn it off and then you don’t have to worry about this old service anymore. Like it’s a long tail of things that you don’t think about when you’re like, let’s just rebuild it. Yeah, I agree. ⁓ Was there any testing or anything like any data that you tried to gather like as like a proof of concept or anything like that when you were doing this in-house rebuild to prove that it would do this before you actually undertook it or? ⁓

Satish (24:30) Yeah

Brittany Ellich (24:53) Like how did you end up making that decision?

Satish (24:56) Yeah, definitely data ⁓ by launching the experiments. we had the autocomplete use case, which is on the Doordash application. So when you type in something, you get type ahead suggestions. So that particular use case ⁓ that was powered by this ⁓ newly built search engine, a POC search engine, you would call it. ⁓ And the data is like… ⁓

nothing fancy, just the standard data, right? What is your latency? ⁓ Like the type ahead is very latency sensitive. ⁓ You can’t have like ⁓ just a spinner there for the people. So it has to be very latency sensitive. ⁓ It has to be ⁓ context driven. ⁓ For example, if a user is searching

inside a convenience store. The context is different here. So if a user is typing IBU in that context, then user might be looking for ibuprofen. So you have to ⁓ complete your, you have to give your autocomplete suggestions according to the context while IBU in the global search or like in the whole world search might be something different. So.

that the context is very important. Also, ⁓ there are certain things like, ⁓ like I said, ⁓ Gen Z slangs or ⁓ some things like ⁓ if you want to look for ⁓ chicken tikka masala. So nobody types in chicken tikka masala. Everybody types in CTM. So your system needs to understand, okay, CTM is chicken tikka masala. So there are like… ⁓

a lot of nuances like this, you need to go through the data and understand those nuances and measure your new system against these nuances as well, against the tail. Definitely you will measure it against the head, but the tail is also important on ⁓ what for sessions that you are serving, ⁓ the click through rate, how many people are clicking in and clicking on your suggestions and getting the food or the items ordered.

So those kind of things, the availability aspect. ⁓ Another aspect is the refreshing the data. How fast is your system to refresh its state? ⁓ How fast it is to crunch a new set of data and refresh the search engine index ⁓ for serving? How tunable it is in terms of operator template, right? What are the plugin points that you are going to provide?

to your internal teams. ⁓ So these are some aspects on which we evaluated the new system versus the old system.

Brittany Ellich (28:02) Yeah, that makes sense. And I understand now you are at LinkedIn, is that correct? And that’s fairly recent? Has it been like a month or two or okay?

Satish (28:07) Yes.

Yes, yes, it’s real. Yeah,

like last month.

Brittany Ellich (28:15) Okay, cool. Are you working on search there or something different?

Satish (28:18) ⁓

No, I am not working on search at LinkedIn.

Brittany Ellich (28:24) Okay, nice. Yeah, you probably are still getting into it. Have you learned, is there going to be anything that you’ve like as you’re getting in, you’ve previously went from this company that, you know, was small and scaled up. Now, LinkedIn is probably pretty well scaled, I imagine at this point. ⁓ So I’m curious, like, has that changed the way that you approach things or has, you know, has there, how have you gotten up to speed on what’s happening in your new area? Anything that you’ve learned?

Satish (28:50) ⁓ Yeah, so for your last questions how I’m getting up to the speed ⁓ I think ⁓ by using a lot of AI tools So ⁓ definitely like a lot of reading but if you are curious enough to know about anything I think it’s just like a question away.

what, why, and how, and you would just know about any of the systems that is being worked. ⁓ Going from like a startup or like a mid-tier-ish like the DoorDash to LinkedIn. ⁓ So my thought process was that wherever you go in terms of software engineering, the problems are going to be the similar-ish on a very high level.

So it just like the tools are different. It’s just the way like the people do certain things are different or cut to culturally will find some things are a little different here and there. ⁓ And like some tools are like mature or ⁓ a problem that ⁓ that is out there in the world. ⁓ Big companies might have already figured out solution for that. They have like in-house solutions for it. So those things are there.

But yeah, I would say like it’s good learning opportunity as well. Like if you look at the holistic picture of a company like LinkedIn, you would see, okay, there are like certain decisions ⁓ that were made in the past, which got LinkedIn to a point where it is currently. Look at those decisions, look at how their tech stack evolved over the period of time, ⁓ in what directions that different products are moving.

So that is also very good learning experience or thought experience ⁓ for me. Yeah, so I would say, ⁓ yeah, I’m still like in kind of in the onboarding phase of it, learning through. ⁓ And one thing that I learned at DoorDash is ⁓ when I joined DoorDash, I said like, there was this big monolith and it was…

It wasn’t a pleasant experience from engineering point of view to work on monolith. And I used to think like, so my mindset at that time was like, hey, who builds monolith? Who does this kind of things? Like who, this is like a very poor engineering, who writes code like in this way. And I was like so critical of that system at that point. And as… ⁓

As I lived through the journey and I looked back, I would say like, okay, ⁓ you shouldn’t be, I mean, you don’t have to be critical of the legacy systems because those were the systems which ⁓ propelled your company, your product till that point. So they were supposed, they were doing what they were supposed to do. And it’s a…

it’s an opportunity for you to take that and mold it in a different way and write the next story for ⁓ your team, for your company. So I think that is something I ⁓ would take from my experience at DoorDash and at LinkedIn. ⁓ Definitely there are certain things which ⁓ you can question about saying, hey, ⁓ why certain things are built in certain way?

⁓ But yeah, I would look at it from the opportunity point of view, like, okay, we can ⁓ build things better from this point onwards. We can do 10x things, 100x things. ⁓ Yeah, so I would say like that is like a huge takeaway for me from my whole journey at DoorDash

Brittany Ellich (32:55) That makes sense. Sounds like you started at DoorDash pretty small and then got to see it go all through the various iterations, whereas you’re coming into LinkedIn, ⁓ you know, when things are much more established and have a lot of history to look back on. Bethany, do you have any experience, like anything that you’ve done to like help get up to speed in a space that’s already existing or like any signals that you see in a system where you’re like, this is the part where nobody likes, this is where the skeletons hide or anything like that.

Bethany (33:22) ⁓

Yeah, I mean, I think that’s typical of any engineering team. You can tell if the team is just honest about the code base and say, yeah, this is gross, or anything like that. ⁓ I think it’s, I take the Marie Kondo approach like you’re saying, Satish of ⁓ like, you thank the code for what it’s done and how it served you and move on.

And I think it, yes, you’re saying thank you. then, but let’s do it differently. ⁓ And so I think that’s always a healthy approach to take with a code base. But ⁓ I think code reviews are a really good signal for what areas are healthy versus not because people will be either willing to review your

code for a specific area if the team is very familiar about it and comfortable with it and less willing to review if it’s an area that folks haven’t touched for a minute or don’t feel comfortable with it. ⁓

So I think that culture is ⁓ an interesting one to take into account when you’re new to a team and trying to understand what areas are ⁓ more built out versus less understandable.

Brittany Ellich (34:46) Yeah, that makes sense. feel like one of the threads throughout your story too, Satish, at DoorDash is like you sort of had this experience of like build versus buy and initially you bought, you you were using the Elasticsearch as is and then decided to build something from scratch that was more custom to ⁓ your application. I think that’s something that a lot of people throughout the software industry sort of struggle with. ⁓ I’m curious, do you have any thoughts on that? Both.

Satish (35:10) Yeah.

Brittany Ellich (35:15) with your experience on it. And also now in the world of AI, I’ll ask Bethany this too, because I feel like this is a thing I hear about a lot where everybody’s like, now you can just build, just build everything. Why buy it? So any thoughts there?

Satish (35:20) You

⁓ Yeah, so I would say first ⁓ just buy ⁓ because ⁓ if you look at your goal, like if you have an idea, you want to surface it to your customers. ⁓ And what is the fastest way?

to surface it to your customers and get that feedback, right? ⁓ Definitely you don’t wanna spend time in building the building blocks, which are already available in the market. You want to just move ahead, get your idea out the door and get the feedback. So in that sense, I would definitely recommend buy ⁓ versus build. But as your system matures,

⁓ still buys a good strategy. ⁓ The build, I would say like it’s not only the build decision shouldn’t be taken only on one dimension like ⁓ money or like the pricing. ⁓ You need to think through, okay, this is costing me X money, ⁓ but what else it is costing me? Is it costing me velocity?

then whatever money it is costing, I would multiply by that 10 if it is costing you the velocity. Then in terms of operability ⁓ of that system, ⁓ what is, how much effort that the Inch team is putting in into operating that system. Not only Inch team, maybe like the product team, whoever is like ⁓ making efforts in operating into that system. So that also put like a…

I would put like a price tag on that too, ⁓ if that’s on that system. And also I would put a price tag on, okay, so you bought a system or like you rented a system via cloud provider. What is the level of support that you are getting? How efficient they are in giving you the support? Are they more knowledgeable about that system than you? ⁓

I think that if the answer is no, then yes, either you need to change the provider or you need to build your own system. yeah, so I think it’s not, would say a binary question build versus buy. So it’s more of a thought process, but yeah, definitely go with buy first and get that idea out the door.

Bethany (38:20) Yeah, I completely agree. think it’s, I mean, we’ve been hearing a lot about software as a service or SaaS companies being dead after AI because suddenly you can vibe code anything. But I would argue that with these companies or these products, you’re not paying for the code. Like we talked about earlier, the code’s easy to write. It’s the scaling, the rollout, the expertise that comes with it. I mean, so many companies have their products or their code open sourced, but still have a paid off.

offering

to host it for you, because that’s the hard part, and also understanding the code. So I don’t know. I still think that SaaS companies and products are important in this day and age of 2026. Who knows if that changes, but I think it’s still a very.

a very relevant topic to decide when you’re exploring options for scaling your company, whether you build or buy, depending how much you want to invest in expertise in these things or how much time you want your employees to spend on things that aren’t really related to your direct offering. I think that’s still relevant today.

Satish (39:33) Yeah, definitely.

Brittany Ellich (39:35) Yeah, I agree. I’m still bullish on software being important as well. ⁓ But it’s fun to read all of the things like, ⁓ like, does software need to exist anymore? Because you can just like the hard part isn’t the building, like you said, it’s the maintenance that comes with it and making sure like everything continues to work over time as it scales. So.

Satish (39:39) you

Brittany Ellich (39:56) One of the other things you said that I really liked is how much expertise they have. Is this one of many offerings that they have that is something that they’re actually an expert in or is it something that’s just something that they have available and you have learned it deeper and are contributing back to the community more than the original company, then it’s a good sign that you can probably get away with doing it yourself. So I like that.

Cool, well we are coming up on time. And so at the end of every episode, we do a fun segment. ⁓ And this week, ⁓ I thought that, you know, that we’re talking a lot about technical decisions, some things that change over time, depending on what you’re thinking. So I thought that we could do a little round robin tech confessional of a decision that you have made that maybe, you know, didn’t work out, something that you’re, you know, not proud of looking back on it.

or that you are proud of and if you would rather confess that you did something awesome, you can also do that, I suppose. ⁓ Yeah, so I we could do that. ⁓ Something that we thought was right at the time but didn’t age well, which may or may not be all software, I guess. ⁓ So I’ll start and then I’ll let you both go ⁓ with your option. ⁓ And I admittedly did not think very deeply about this before ⁓ writing it down.

Satish (41:09) Hehehe

Brittany Ellich (41:21) But I think, let’s see here.

One of the decisions I’ve made in the past, I think was like during the days, I feel like we’re out of these days now where we’re not jumping to a new JavaScript framework every 30 seconds. Like we were there for a bit when I first started ⁓ working in ⁓ software. But one of the ones that I had decided was there was one that we were using, I think it was called like handlebars ⁓ for something that we decided to put that in.

our application and turns out it’s just not the one that anybody went with or decided to continue using. And then we had to continue maintaining this for a long time. And I think not waiting a little bit more for like the technology to mature before building a funded thing on top of it was probably one of the decisions I was involved in that, you know, it was a lesson learned for future decisions. Luckily, the front end ecosystem is a little bit more. ⁓

mature now in general, so it’s easier to make those choices. that was mine. Satish, do you want to go next?

Satish (42:33) yeah, sure. ⁓ yeah. So I thought about one thing, maybe the scalability, ⁓ point. So, if you, ⁓ if you go to DoorDash, you will see like a grouping of stores, right? we call them carousels and, ⁓ one of the tech decisions, ⁓ when, when we were doing this migration from Monolith, ⁓ to our new system was that like for, for each of the carousels you, you would make.

a separate call to your microservice to build that carousel. And definitely that didn’t scale well because that one thing is that put cap on how many of these product constructs that you can have on the homepage. You can’t scale them linearly. And even though adding some tweaks in the backend, it still doesn’t work that great.

So ⁓ yeah, so that didn’t scale well. ⁓ But the next story is, ⁓ so we went back to a model, okay, wait, do this only in a single call. Okay, so retrieve everything and then build out this constructs. So that played out for a few years, then something new came up. And okay, now we also have…

Now at this point, we also need to make multiple calls to the backend. So ⁓ now you kind of settled on a hybrid approach to do it. But ⁓ the original decision was, would say, it didn’t consider ⁓ at least two years ahead of its time how the system would evolve.

But as the system matured, ⁓ now you can go crazy. mean, your system is linearly scalable. Your backend is linearly scalable. The search engine is scalable. You can just go crazy on number of calls. It doesn’t matter. ⁓ But yeah, it was a journey to get to a right state ⁓ from where we started.

Brittany Ellich (44:52) Yeah, I get that. Bethany, do you have a confession that you would like to bring to us?

Bethany (44:57) ⁓ yes. ⁓ it was probably a very quick turnaround with a technical decision to realizing it was a mistake, thankfully. ⁓ but, ⁓ I was working on a greenfield service as like the last product that I was working on for my last company. I was basically like a content tagging platform. ⁓ and, ⁓ when we were evaluating data stores, I, ⁓

I did a ton of research and I felt very convicted that DynamoDB was the best thing to go with. I went head deep into the NoSQL design patterns and I was so on board. was like, yes, this is what we’re going to be using.

Implementing it, it worked great. But then there came a time where after I was talking with the product manager, I’m like, hey, the data contract isn’t going to change much, is it? And she was like, ⁓ no, it’ll probably stay the same. Like a month in, it changed drastically. Like the ways we needed to access it changed drastically. was like, I should have called this. And on top of that, my other team members, ⁓

didn’t necessarily have the time or interest to look into how NoSQL data patterns worked. it turned out I was the only one who was able to really make changes in that space. And I was like, you know what, we’re switching. We’re going MySQL. Ripped it out in like a week and translated everything, luckily, before we had a ton of data onto it. ⁓ luckily, it wasn’t too drastic. But ⁓ yeah.

Satish (46:25) You

Brittany Ellich (46:25) No.

Bethany (46:38) Turns out engineering culture is very needed to consider in whatever you choose for your data layer. It’s not smart to choose something random that nobody has experience in. ⁓

Satish (46:48) Yeah,

definitely, Yeah.

Brittany Ellich (46:50) Shocking, yeah.

Bethany (46:51) I know, I was fairly young at the time so it was

one of those pivotal moments, it was like, alright, I’m never designing anything that I can’t back out of immediately.

Brittany Ellich (46:55) Ugh.

Satish (47:00) Okay

Brittany Ellich (47:01) smart, so the

one way versus two way door decision. Yes, yes, I love that. ⁓ Well, so teach, thank you so much for joining us. This was an awesome conversation. If folks want to find you somewhere on the internet, where do you suggest that they go?

Bethany (47:05) Absolutely.

Satish (47:06) Yeah.

⁓ I’m on LinkedIn. Yeah, I think my name is Unique Satish Saley. So you can definitely reach me out on the LinkedIn. Yeah. Thank you.

Brittany Ellich (47:29) We’ll

include that link in the show notes as well.

Satish (47:33) Thank you. Thank you for having me.

Brittany Ellich (47:35) Yeah, this is great. Awesome. Well, thank you so much for tuning in to Overcommitted. If you like what you hear, please do follow, subscribe, or do whatever it is you’re supposed to do on the podcast app of your choice. Check us out on Blue Sky and share with your friends. Until next week, goodbye.