Bethany Janos 
Brittany Ellich 
Eggyhead 
Jonathan Tamsut Erika (00:00) Welcome to the Overcommitted Podcast, your weekly dose of real engineering conversations. I’m your host this week, Erica, and I’m joined by…
Bethany (00:09) Hey, I’m Bethany.
Brittany Ellich (00:11) And I’m Brittany Ellich.
Erika (00:12) We are a group of software engineers who initially met working on a team together at GitHub and discovered we all had a passion for honing our craft through discussing our technical improvements, learning strategies, and career challenges. We continue to meet and now share this space with other engineers who genuinely love what they do. Today on Overcommitted, we are joined by Dawn McKinnon.
a technology leader with experience at Fortune 100 companies, startups, and everything in between. He’s the founder of Searchcraft, a startup focused on optimizing the developer experience of implementing search. Prior to founding Searchcraft, was CTO at a software consultancy focused on building apps for startups that were either scaling up fast or had non-technical founders. Welcome, Don.
Don MacKInnon (01:04) Thank you for having me.
Erika (01:06) We are so excited to chat with you. We have some big Rust fans, I know, in our colleague groups and can’t wait to hear more about your journey and ⁓ especially the technology at Searchcraft and some of your thoughts about the space and of course, a little bit of AI talk. ⁓ So let’s start with…
Don MacKInnon (01:26) Sure.
Erika (01:31) Searchcraft’s secret sauce, which is your Rust powered engine. So yeah, can you give us a little background on what the implementation is, some of your key technology decisions there?
Don MacKInnon (01:46) Yeah, so we kind of approached Searchcraft with this idea of trying to reduce complexity. So we felt the existing tools out there were a little too, it was too complicated for, or too complex for, to really hand off to people.
we were doing a lot of that handing off when we’re doing the software consultancy. So part of that choice to reduce complexity was to go with Rust because Rust is more efficient, uses less resources, so by definition we could reduce the size of our search clusters and there would be less to manage, right? So with Rust you don’t have to manage a JVM and have that configuration, you have to deal with on top of the application configuration.
So that was the big choice. We got a lot of performance gains out of that. And then being able to, by using Rust, we were also able to layer on additional features at like query time that you couldn’t necessarily do with other systems that we using before. So that was able to improve kind of like the relevance of our search results by doing that, yeah.
Erika (02:56) That’s very cool. So what were some of the main challenges that you faced in developing this? Maybe kind of tell us the story of start to finish and what maybe some of the bumps along the way were that you overcame.
Don MacKInnon (03:13) Would
you say when ⁓ the difficulties would be with building it in Rust or just building it in general?
Erika (03:21) building it in general? Like did you start with the rest day one or did you kind of try to think of some other options before you settled on that?
Don MacKInnon (03:30) Before we wrote it, we didn’t write any code outside of our Rust code for the engine. So we were pretty sad because we had a good period of time where we were kind of just brainstorming the structure and the stack and whatnot before we wrote any code. So it was Rust from day one. The difficulties were like I had search training.
already in Elastic, I knew pretty well how that worked. I’ve also used other Lucene-based systems before, so I know how those worked. So figuring out a way to kind of replicate a lot of that, that it’s a VM25 relevancy from the ground up in Rust, ⁓ that took some time. We do have some open source libraries that we fold into our core engine that help with that. But…
When you start from zero, there’s a lot you have to think about when you start actually writing the code. How are we going to store our indices in a way that is both performant but also reliable? How do we handle authentication through our API? So all of those things. ⁓
you know, kind of they add up over time. yeah, it’s just, it took a lot longer than we expected. So, we, took about, we built it for about two years in bootstrap mode before we spun it off into like its own company, its own product kind of thing. So, we’re building it internally at the consultancy just to kind of serve our own needs and to see if it…
was as performant as the other tools out there. so that took about two years of just working on it when we had time in between clients and whatnot. then it really, the Core Engine was done about a year and a half ago. And then last year we decided to try and test the waters to see if we could spin it off into, it’s like an actual product company, like a SaaS company, which is what we did.
So before we did that, we met with a lot of other consulting agencies and to see if this is something they would be even interested in using. And so we had a lot of those conversations and it was pretty much over like, one thing, like, yes, we would use this. We don’t like the tools that are out there on the market now. So if you had something that was lower, easier to use, less resources, better performance, we would totally use it.
So we’re like, okay, well, I think we’re onto something. Let’s try and make a company, right? So we did the whole like, fundraise thing, writing a pitch deck. We had never done that before. So like, I’ve worked with startups that were funded from BC and whatnot, but they kind of handled all that fundraising aspect of it, right? So we were just there as builders that got pulled into the project. But this is the first time myself or my co-founder had to actually do a raise ourselves. So that was.
Very, very interesting. A lot of big learning curve there, I would say.
Erika (06:30) Well, yeah, you mentioned some of the key points that you built out and, you know, I have used search platforms before, but yeah, you know, it’s one of those things that I think when I’ve integrated it, I have not thought so much about how it’s implemented. So yeah, you mentioned like indices and authentication and maybe if you could like…
just describe a little bit more of like what other problems SurfCross solves and yeah, what it does there.
Don MacKInnon (07:04) Yeah, I think one of our differentiators is we try to make search infrastructure accessible to builders of all levels, right? So for a long time, it’s kind of been gate-kept off to back-end developers, Java developers, things like that. So trying to make it accessible to more front-end developers, TypeScript teams, and also people who are maybe product managers and non-developers as well. So once it’s been integrated,
⁓ Our relevancy tools are all tunable from non-technical people through our GUI, right? So you don’t have to actually write code or do code deployments to like, okay, I want to change the weighting of fields within our search index, or I want to change synonyms out for our cinnamon library, or even changing the language of the index. You don’t need a developer to do that. You can just like…
mess with the levers yourself as a product manager. That was really important for us. Also, there’s a lot of people building with tools like MCP and Cloud Desktop and things like that. We want to make sure those people were able to use Searchcraft and not have to necessarily know how to query our API and do authentication and all that. They could just…
generate the credentials and use our MCP server and type in natural language with our API essentially.
Bethany (08:25) That’s awesome. I mean, speaking of MCP and AI, mean, I really enjoyed the article you wrote on the trust problem in AI because that is such a, such a real issue and something that I hear very consistently. mean, when I went to GopherCon this year, that was one major topic and theme that they were talking about. So I’m really curious how you can see Searchcraft fitting into
the building AI applications and making them reliable and transparent, how do you envision that flow going?
Don MacKInnon (08:58) Yeah, so.
Trust in AI, so there is an issue out there where you’ll talk to one of these eight, like, ⁓ LLMs and you’ll get answers that are maybe inaccurate, right, or they’re outdated, yeah, inaccuracy and not up to date. That’s a big problem. So one way you can kind of solve that is through that information discovery layer, so like making sure that these…
⁓ LLMs have the ability to pull relevant information out of a system, right? So like, LLMs are really great at user intent detection and understanding natural language. like, understanding what my question is asking. But if you don’t connect it to the right tools underneath it, then it’s not able to pull back the right information, right? So like, it has its initial training set of data that it knows. And then…
You guys are familiar with with rag systems essentially where you can feed more up-to-date information into that into that model and it can pull from that so through it could either do it directly through like a search craft API or through MCP and know like okay well if you’re able to pull the most relevant information because LLMs can construct the queries to get that information out of an API it can pull the most relevant information and also like the most recent information.
So that’s really critical when using these AI platforms that the correct information is getting surfaced back up.
Erika (10:25) Yeah, I’m curious. You mentioned the usability of Searchcraft for maybe non-backend developers. Is there a vision for supporting automated optimizations with agents and stuff like that as well?
Don MacKInnon (10:42) Yeah,
haven’t, ⁓ we don’t have, we are planning stuff for like, if you’re referring to like agent to agent protocol and things like that, we do have some plans that I can’t really talk about yet, but ⁓ it is definitely on our roadmap to do that, yes. We want to, whether it’s humans using our platform or like agents or other systems, like make it easier for those connections to happen, right?
At the end of the day, we look at ourselves as a way to get the most relevant information out of a system, and we don’t really prescribe if it’s a developer or an automated system or an agent or…
Bethany (11:18) I’m curious, when you started building Searchcraft, where was AI in the thought process? So is it something that happened to converge nicely, or was it something that you were building this company with, with the intent to help out with things like that from the start?
Don MacKInnon (11:35) Yeah, I mean AI wasn’t really being talked about at the time so we started building it in 2021 and we started the company in 2024 and when we started the company during our we did like a pre-seed angel raise like AI wasn’t really a topic right, but now it is and if you’re raising money you pretty much have to have an AI component and if you’re in the development space people
want to use AI tools. it’s something we’ve had to kind of like shift and like embrace as part of our platform. And I think it’s very exciting, but it wasn’t something we planned for at all.
Brittany Ellich (12:14) Yeah, that makes sense. I’m curious too. It sounds like, so I personally hate doing search. So the idea of search craft sounds really, really nice because it’s a perpetual problem. And I just never want to have to think about it and want to abstract that away to people who really care about the problem. I’m curious though, there’s some really big players in this space already, like Elasticsearch and I’m sure a bunch of others. It’s first one that comes to mind.
Don MacKInnon (12:38) Mm-hmm.
Brittany Ellich (12:40) How was it taking on a market that was already pretty established? Is this your first startup you founded?
Don MacKInnon (12:47) I mean, pretty much. I had one other bootstrap startup that was a consumer-facing product, but this is my first B2B developer tool startup and the first one where it’s like, I’m all in on it, right? So my first real one, I guess, yeah.
Brittany Ellich (13:02) Yeah.
Okay, how was it approaching a space that was already existing?
Don MacKInnon (13:10) Well, so I had a pretty good understanding of the space before we decided to even do this thing. Elasticsearch is a great tool. I think it tries to do too much. It tries to be like a one size fits all tool for search in general, but it kind of doesn’t do anything great. there’s like lots of performance issues with it that we, I’m not just saying this, but like other developers have told me like that they have to deal with.
And the space has been pretty stagnant. Like there’s a couple of big players, Elasticsearch, Algolia is another big player, but they’re only cloud. can’t self-host Algolia. So there’s a few big players. Most of them are based on Lucene, which is an older open source library that’s been around since the late 90s. So there hasn’t been a whole lot of innovation in the space, I feel like anyway. There are some other newer players that are getting into it now. And I think…
Especially with the emergence of AI being a thing in the industry, like search is kind of becoming a thing again. So it’s, I wouldn’t be surprised if you see more search companies appearing over the next year or two. But to your point earlier, like you don’t like dealing with search. I think most developers don’t, right? And I think developers should really focus on like, if you’re making a product, like focus.
your product and the problem you’re trying to solve well. You don’t need to reinvent the wheel with search. Don’t roll your own authentication. Don’t roll your own analytics. Search is also one of those things. Really shouldn’t roll yourself. Let someone who deals with that every day solve that problem, and then you worry about the problem you’re trying to solve.
Brittany Ellich (14:43) Is Searchcraft pitched at a specific size of problem? Is this something that hobbyist developers on their side projects can use Searchcraft as well as enterprise? Or what is the target market there?
Don MacKInnon (14:53) Yeah, mean,
don’t, so like anyone could use it. You can run it yourself through like a little Docker container. You can, we have a free plan. If you don’t wanna run the cluster yourself, you can use our free plan on our cloud service too. That’s totally available to people. When to pull in search? I think it’s when you get to a point when you have any significant amount of data, right?
So like if you have no data, you don’t need search. once you start, you’ll know in an application when it starts getting frustrating and there’s friction in using it when you can’t find the information you’re looking for, that’s when you need to pull in search, right? Like if I’m building a social media app, you’re gonna be unthinkable to not be able to find posts, right? Like older posts. It’s just something that you expect to be there. So I think you’ll know when you’re building it if you need search.
Brittany Ellich (15:47) That makes sense. And it sounds like you, the, sort of started with a problem at the company that you were a CTO at. Did you plan to become a founder within your career? Did you plan to like start a company at some point, or is this something that you’ve sort of been learning and adapting as you go on?
Don MacKInnon (16:04) I mean, I definitely didn’t know how to be a founder. So I’ve been learning and adapting. I’ve always wanted to kind of build my own products. I’ve done like little things on the side here and there. And my co-founder and I have talked about it like, well, we’ve built great products for other startups, but we didn’t really have say in the direction of the company itself or like what features they chose to build or bring in, right? We’re just kind of like the designers and the engineers.
behind it. So the opportunity to kind of like do our own thing was very exciting. But there’s a lot that goes into like founding a startup like you’re, you know, hiring, managing people, raising money, talking to investors, like there’s a whole thing that’s outside of just the building aspect. So for me, like the engineering part is like the easy part for my job now. I mean, I’ve gotten to the point where like,
Okay, I need to build and I need to make an integration for this language. that’s not a problem. I’ve never used this language, but we’ll figure it out. I can figure out engineering problems. Figuring out how to convince investors to invest in the company is a totally different problem that like, you know, this is, I’m new at that. So.
Erika (17:11) Do you have any advice from your experience sort of approaching this like big unknown problem for developers who are kind of going through the same experience of like how to get started, how to navigate those moments of uncertainty?
Don MacKInnon (17:26) As far as like, where do I start when I’m building or it’s like, should I make a company around this thing?
Erika (17:33) Hmm, yeah, I guess they’re valid questions. I think I’m maybe thinking of the first part where, this is, like you said, you realize that there was this big problem to solve and you’re thinking like, okay, I want to build something to fix this. I guess you said it did take longer than you expected. So maybe there was a difference in scale and…
Don MacKInnon (17:54) Mm-hmm.
Erika (17:57) you know, size of the solution. But ⁓ yeah, like I think that kind of blank page problem is familiar where you’re looking at something and there’s an empty repo, like where do I get started?
Don MacKInnon (17:59) Yeah.
I think you want to build for a problem you know, a problem that you experience. And that’s why we built this is because we were experiencing a problem. So a lot times I’ll hear like, want to make something, but I don’t know what I’m going to make. Right? Like that like repo problem. But everyone has like pain points that they encounter every day in their life, whether a developer or anybody in any profession like.
solve for the pain points that you know because you know the problem already. These are things that I deal with every day that I wish worked this way or I wish I did it this way. Solve that problem. If you look inward, you can figure out pretty quickly a problem to tackle, I think.
Erika (18:55) That’s definitely true. I feel like I’ve experienced both where I have no idea what to build and then I have a list that’s longer than I could possibly ever tackle of things that I want to solve. yeah, finding the right thing to attack on that list is definitely key. ⁓
Don MacKInnon (19:06) Mm-hmm.
And I think
choosing the tech that you choose to solve that problem with is like, you can always move fast with languages you are comfortable with, right? I think it’s more important to kind of get your idea out the door than it is to be perfect, because you can always iterate. It’s more important to prove the idea and that you’re solving the problem in the way that is correct versus, okay, is this code scalable? You can scale later.
get something shipped.
Erika (19:42) Yeah. And then what was the moment like when you realized that this was a sellable product? What was that thought process and that sort of switch?
Don MacKInnon (19:54) ⁓ Well, I think it was like like I said last year when we were having those conversations with those other development shops They kind of like validated that it’s not just a problem that we see here internally but other software consultants were having the same problem with right like having to Maintain and hand off search infrastructure is painful So if you have something lighter, it’s it’s much easier to to do that handoff with teams that aren’t search experts very like so if you work with the startup is like
you know, for TypeScript developers, asking them to maintain an Elasticsearch cluster is very, it’s a big ask, right? Like they probably will keep coming back to you. So we kind of wanted, we didn’t really think that was fair to say like, you have to keep paying me to maintain this, right? When they were supposed to be like a handoff project as a consultant. So building tools that these levels of developers can manage on their own.
Yeah, was like, truly that’s something a lot of companies will want. ⁓
Erika (20:50) Yeah.
Well, it sounds like you’re definitely making people’s lives better. And thank you so much for sharing all about your story and all about Searchcraft. I’m feeling inspired. Great. Well, we’re ending our… ⁓ We always end with a fun segment. So if I’m understanding correctly, you also had a background as a music producer.
Don MacKInnon (21:13) yeah, did I tell you guys about that? I did, yes. So in my 20s, I had my own record label and I was recording bands and whatnot. I lived in New York for a while and did that. Yeah, it was my other life before I took being a developer seriously.
Erika (21:32) All right, well today we’re gonna merge the two and we’re gonna think about walkout songs and any athlete has a walkout song that they use for their presentation at a game or whatever. And so we’re gonna think about either you, your company, or a next big release, what would that walkout song be?
So I don’t know if you have a project in mind that’s in the works. Don’t want to spoil it. You could also think about like past favorite projects. What the walkout song would be. So I know I’m dropping this and we’ll all take a turn to say ours.
Don MacKInnon (22:11) I was not prepared for this, I just want people to know.
Erika (22:15) I’ll give you some time to think while I share mine. ⁓ We currently have a project that’s been like over a year in the works to get to public preview. And so my choice is shipping up to Boston by the drop kit Murphy’s, because I want it out the door. Ship it. Let’s go tomorrow.
Brittany, do you have one in mind?
Brittany Ellich (22:35) I do actually, yeah. I also had to take some time to think about it a little bit because I’m terrible at anything music related.
I chose Livin’ on a Prayer by Bon Jovi. Mostly because of that refrain, you’re halfway there, livin’ on a prayer. I feel like everything that I’m working on right now, once it ships, it’s still only gonna be part of a larger overall solution. So every time we ship something, I’m like, but we’re still building to this big thing overall. So it’s a good reminder every time that we’re still going. ⁓
Erika (23:03) I’m glad that’s the reason and not because the
whole ship is slipping on a prayer.
Brittany Ellich (23:12) That’s true.
Yeah.
Bethany (23:14) I for one think that music and coding goes really well together. The other day I actually I’ve been listening to a lot of albums recently and I put like the album I was listening to in my PR. was like this PR was brought to you by Live 2007 Daft Punk and things like that. But lately I along with I guess everyone else in
America has been obsessed with K-pop demon hunters. So I would probably end up choosing like how it’s done or take down from from there because it just feels way more intense than it actually is and it makes me feel badass. So that would be that would be mine.
Don MacKInnon (23:54) Do I have to pick just one?
Erika (23:57) No.
Don MacKInnon (23:59) Okay, because I have two in mind. The first one’s gonna sound horrible because I’m in like fundraise mode right now. We’re trying to do a raise at the moment. So Wu-Tang Clan’s, there’s a song, Cash Rules Everything Around Me.
I’m not all about that money, but yeah, that’s right now I’m trying to raise so that’s that’s the song for right now that I played my head But as far as like our search craft the product goes I think daft punks around the world because we’re trying to be everywhere all over the place, so
Erika (24:30) love it.
Bethany (24:32) Nice.
Brittany Ellich (24:32) I feel
like that speaks to the dichotomy of being like a technical founder as well as like, you know, you’re thinking about the business side and the technical side. They go hand in hand.
Erika (24:42) Yeah.
Don MacKInnon (24:43) you
Erika (24:44) I would put those both on a playlist together too. Yeah.
Well, Dawn, if folks want to find you, where should they go?
Don MacKInnon (24:52) Sure, so my personal site’s DonMcHenon.dev and you’ll probably have to get that link in the show notes because nobody knows how to spell my last name. But otherwise, searchcraft.io is our company as well, so either place. Or I’m on Blue Sky, but DonMcHenon.dev has all my links, so.
Erika (25:08) We will definitely include that in the show notes. And with that, we’ll say thank you for tuning in to Overcommitted. And if you like what you hear, please do follow, subscribe, or do whatever it is you like to do on the podcast app of your choice. Check us out on Blue Sky and share with your friends. Until next week, goodbye.