WEBVTT 1 00:00:19.080 --> 00:00:19.660 Hey, Ben. 2 00:00:19.660 --> 00:00:21.040 Hey Matt. 3 00:00:21.040 --> 00:00:23.920 How are you? How, ugh, . 4 00:00:23.920 --> 00:00:25.840 You should leave that in. 5 00:00:25.840 --> 00:00:36.320 I, know. Alright. How are you doing? I will leave it in, won't I now? Because I'm too lazy to edit . Sorry, listener. You're gonna have to listen to that. That's awful. 6 00:00:36.320 --> 00:00:38.000 . Uh, I'm doing good. I'm doing good. I'm 7 00:00:38.000 --> 00:00:49.360 Glad to hear it. Yeah, I'm glad to hear it. Oh my gosh. Well, um, yesterday you and I were chatting with a colleague and he made a very astute observation. 8 00:00:49.360 --> 00:00:49.640 He did 9 00:00:49.640 --> 00:01:11.060 About the way that we develop software and in fact, the way the sort of group of individuals that we've surrounded ourselves with over the years, over like the last decade that we've been kind of vaguely hanging out at the same company, um, about the way that, um, that we develop or we at least approach software development. And so I thought let's talk about that today. 10 00:01:11.060 --> 00:01:46.460 Yeah, that sounds good. I'm trying to remember what his exact phrase was. I'm gonna paraphrase him a little bit here. But it, it's something like the previous companies that I had worked at, if someone came along and said, Hey, I built a message queue, they would be instantly fired, . Um, and I, I think he was kind of talking about our, uh, both, um, what's the opposite of reluctance? Uh, not aptitude, not enjoyment. There's some, it's, 11 00:01:46.460 --> 00:01:48.040 Uh, eagerness, 12 00:01:48.040 --> 00:02:22.200 Eagerness, eagerness, perfect word. Our eagerness to go and build solutions for problems that are problems that other people have attempted to solve, perhaps even in a similar way, but not similar enough for, for our tastes a lot of the time. And I know you have done this and I have done this, um, and you could very easily categorize this as not invented here syndrome, and I don't Right. I don't reject that characterization at all. Like that's very, fair. 13 00:02:22.200 --> 00:02:24.600 I think that's a character flaw that we all suffer from. 14 00:02:24.600 --> 00:02:36.060 It's very fair that this is a little bit of not invented here, but I think that we do have some rational reasons for doing these things. And maybe we can take this episode to sort of explain some of those, some of those reasons. 15 00:02:36.060 --> 00:03:21.080 Absolutely. So I think the motivating, or one of the motivating examples that we came up with was a, uh, uh, package management system that our previous company had developed called fig, which was effectively before its time, a Conan or a Conda or a insert your, like, development package needs here, right. System, which would allow you to describe your projects dependencies, go and grab them and unpack them and solve like, well, this needs this version of that, and give you errors when they weren't. And of course, there are thousands of those around. I mean, most notably, your operating system almost certainly has a variant of some kind of package management system. So, you know, you can have Debian packages or you can have, um, uh, 16 00:03:21.080 --> 00:03:21.080 snap? 17 00:03:21.080 --> 00:03:28.480 Snap. Well, nowadays you can have snap, obviously this was like 12 odd years ago. 18 00:03:28.480 --> 00:04:22.140 So I don't know that those things were necessarily around, of course nowadays people will do docker pack, docker builds and things like that. But I'm certain that there were solutions. I mean, effectively most, at least script languages have their own homegrown solution. I mean, Python has had pip forever. And I think the, the overwhelming problem that we were trying to overwhelming problem, the problem that we were trying to solve here was that there is a, a community based, at least at the time, sort of somewhat lackadaisical attitude towards, um, the exact versions of packages that came down. And so you PIP install, and at least back then you just sort of said, give me the version of this. And it just installed that version with, if it needed other packages, it would bring them in, but it would make no effort to make sure that everything worked together well. 19 00:04:22.140 --> 00:05:04.620 Nor did it guarantee that if I were to then check in some code that it PIP installed some stuff that would, my CI would get the same versions of that. And so we had this problem with floating versions would move around, and that was not a good look when you're trying to explain that, well, it worked on my machine, then we built it on CI and then we deployed it to a trading system and it did the wrong thing. So that's where, that was the sort of motivation behind it, at least the beginnings of the motivation of it. Um, and I think our colleague was like, no one else would do that. That's, that's crazy talk. You would just make something else work well enough for you and get on with your life. And I think, you know, the queue example, he said, you know, that we, someone would get fired if you came up with a message queue. 20 00:05:04.620 --> 00:06:18.440 Whereas, you know, again, we developed an internal message queue based off of like some of the papers that we'd read, but we didn't use anyone else's code for it. And again, that could be not invented here, but, I don't know if, so from my point of view, I was doing a lot of native code at this company, and I would now retrospectively justify a lot of my decisions based on how hard it is to share code. You know, it's not as straightforward as just like PIP install or Conda install nowadays of something. There's lot, normally some kind of, um, dance you have to do to make sure you get the right version of the code and its dependencies, and get the source and check it out and build it and all that kind of stuff. Which to, to, to a greater extent actually, things like VC package and Conan have now solved, but only fairly recently, but back then it was like, well, it's almost as much effort to download this package work through it's readme of how it's it's particular brand of make file or scon scripts or whatever to build the package that I want. And then find where it installs its headers to, to copy them to, oh, sorry, I'm just gonna do it myself, . So that's my retrospective justification. But you weren't in native land. How did this manifest for you? 21 00:06:18.440 --> 00:07:30.940 Uh, well, I've definitely had a few instances of this, some of which I would do over again, and some of which were, I would say, a mistake. Um, you know, the the ones where I, I really feel like we did the right thing where, um, probably more along the lines of avoiding unnecessary, they're a little smaller in scope than building like a whole package manager, right. It's more along the lines of avoiding pulling in some large tree of dependencies, right. Um, to, you know, solve a particular problem than just like, you know what, I'm just gonna write this myself. Right? Or I'm going to use what the, what the platform gives me, and I'm going to, um, enhance that in some way. I'm gonna add some additional, I'm gonna build some additional code around that in some way. Um, and like, you know, I had, I had an experience with this, like very recently, we were talking about this, uh, using some AWS services that like almost did what I wanted. 22 00:07:30.940 --> 00:08:13.860 And I was like, I spent like a couple of days like digging through the documentation, trying to figure out how to get them to do exactly what I wanted. And, you know, after a couple of days I'm like, all right, I'm just gonna build this myself. And I built a reasonable implementation in about a, a day and a half. Like, I'm actually working on that a little bit today, but it's like a day or two, right? So I spent a couple of days reading documentation, couldn't figure out how to do what I wanted. And now I've built, spent a couple of days basically reimplementing that thing, right? The trade-offs of that decision are something that you could definitely debate on either side, right? Should I have kept trying to figure out how to get what I want through configuring the service? Should I have tried to build the minimal set of things? 23 00:08:13.860 --> 00:09:20.860 Like another way to approach this problem is, all right, all right, this service has its capabilities. I'm gonna change something upstream or downstream of that to accommodate what it's capable of doing. Uh, so I can use it as is and as designed, uh, and that's gonna make other parts of my system more complex, but I'll be able to reuse this thing. That's a completely reasonable way to do it. That's not what I did. What I did was I built the exact thing that I wanted, and it doesn't do anything else. It doesn't do all the things that that service does. It only does what I need to do, and it does it in exactly the way that I want it done, and it's very testable. Right. And all of those properties for me are exactly what I want. And, uh, like I now have that code in my system and I have to deploy that code, and I have to test that code, and I have to maintain that code, and I have to carry those costs. But in my mind, those costs are lower than the costs of either adding complexity to other parts of the system in order to accommodate the way that, that this service works outta the box or continuing to bang my head against can I ever get this to work? Right? 24 00:09:20.860 --> 00:09:45.860 Right. I mean, in particular things like testability, when you're talking about something that's a service, if it doesn't provide a kind of run locally or a stubbed version, or it's not straightforward to do that, then you are definitely giving up some testability or else you're spinning up things in the real world and integration testing things. Whereas if you make it yourself, you can put the points in where you need them, the test points. 25 00:09:45.860 --> 00:11:00.050 Yeah. And it, and it's exactly that. It's, it's okay if we use this service, we're going to forever more have this future integration test cost, where the only way to be confident that we've, we've haven't broken something when we make changes to the system, is to run very expensive and potentially unreliable integration tests. And we will pay those costs basically forever. As long as this project runs. Whereas I can have a very fixed cost of writing, uh, a bit of code and testing it and using it the way that I want, building out all the stubs that I need. And then those those tests will instead be unit tests that will run quickly and reliably. And yes, I do have the ongoing cost of, um, maintaining that code if I need to change it. Um, but that's also a bit of a feature and that like, okay, I have some, I have something new that I need to be able to do. Oh, the service doesn't support that. That doesn't matter. I've built my own thing. I can, I can change it however I like. So, you know, it's, it's hard and it's hard to quantify these things, and it's hard to measure them objectively and say like, my decision was good, and here's the spreadsheet that shows it. Um, a lot of it is based on intuition about what future changes will come. And that's really hard to, you know, hard to make predictions, especially about the future, 26 00:11:00.050 --> 00:11:00.980 Especially about the future 27 00:11:00.980 --> 00:11:07.720 . Um, so, you know, it, it's, it's a judgment call at the end of the day, but it's one that I, that I definitely feel comfortable making. 28 00:11:07.720 --> 00:12:08.460 And I think that's one of those experiential things that you just have to draw on your experience of like, I have seen this before, or something that looks a bit like this before in my experience, um, it would be worthwhile doing it this way, like choosing to write it myself, taking on that complexity with open eyes of the understanding of how much work that might actually pan out to be, um. But yeah, again, that's, but that sort of feeds into the not invented here kind of stuff. And like, you know, sometimes you have to ask yourself the question, am I answering this truthfully? Like, is this something that I do need? Or is it cool to write it? And would I be excited about w working on it? Which sometimes maybe that is enough to justify it because, you know, that's, we're humans. We love what we do. We're lucky to know to have this, uh, this job where we do actually enjoy what we do. And maybe motivating it and doing it yourself is enough of a cost. If it's a day or two, like you've done, right? Maybe that's fine. If it's a six month project, maybe we should talk. 29 00:12:08.460 --> 00:13:06.260 Yeah. Right. Exactly. And, and yeah. And we were kind of talking about this, this yesterday where it, it's, I I think you can too easily discount and, you know, engineers who are, try to be hyper-rational and, you know, take their emotions out of everything and yeah. Ev like, like I think discount the value of, um, having somebody that is wakes up every day emotionally invested and motivated in a project to make sure that it works and works correctly. That's an extraordinarily powerful effect. And you shouldn't dismiss it because it's like, oh, well, why did you rebuild this thing when there's this off the shelf thing that does exactly that. It's like, well, if I use the off-the-shelf thing, I'm not gonna give a *bleep* about it, , I'm gonna, I'm gonna have it do what it does, and I'm not really gonna care too much if it's testable or if it's easy to maintain or if it works. It's just sort of this thing, and I don't, I'm not, I don't really care. Right. And there are lots of places where that's exactly what you want. They're like, I don't really want to care about this problem. I'm not interested in this problem. This isn't a valuable problem for us to solve. 30 00:13:06.260 --> 00:13:54.260 This is not important or germane enough to my product, you know, to, to care. Or, or, yeah. I mean, I have a sort of, uh, counter to that I'm working with a, uh, an open source, uh, vendor library that I'm sort of mandated to use for a particular thing that I'm doing. And, um, it doesn't quite work the way that I want to do it. And the, uh, the underlying thing that it does, I could like I say reverse engineer. I have the source code, so it's not really reverse engineering, but it's not documented the sort of underlying thing that it's doing. But I could work it out and then write my own, but I have no, no, uh, wish, no desire to do that. That sounds like an awful lot of work to reproduce what this, uh, vendor is providing us. The scope of what the vendor code does is much, much, much broader than what we need. 31 00:13:54.260 --> 00:14:33.100 So I'm picking up this giant library with all of these dependencies that it brings only to get access a tiny part of the feature. So it definitely feels like the pendulum could swing back towards just wrap this tiny part of it, but instead, I decided to go in and go, like, the thing that I actually needed to do that it doesn't do, I'm gonna do a surgical change to their open source project, fork it, and then we're gonna use that ourself, and now we can sort of take it outta the mix. And that was mainly for like testability and reproduce reproducibility reasons there. So like a binary underlying protocol that I had to either consume directly from a TCP stream or go home. And I'm like, I don't wanna have to consume this. I wanna write a test that doesn't need a TCP 32 00:14:33.100 --> 00:14:49.620 server in order to do its work. That seems silly. So, you know, there's, but I could have chosen to kind of roll my sleeves up and say, I will spend two months reimplementing this protocol, but I didn't make the call that way. 33 00:14:49.620 --> 00:14:57.460 Um, yeah. Yeah. But I mean, I definitely have also had instances where I've done this and regretted it 34 00:14:57.460 --> 00:14:59.140 , 35 00:14:59.140 --> 00:15:09.820 Um, right. It was, it was not the right choice. Um, one project in particular at, at prev-prev-co, which I know you are actually a user of, uh, the, the monitoring system that, 36 00:15:09.820 --> 00:15:11.440 Oh, yes. 37 00:15:11.440 --> 00:15:37.540 Um, we started out in that system using, uh, Mongo DB as our database. Right. And we had some problems with Mongo, and that was the early, it was like Mongo 1.0 or something. It was very early days. And, you know, I don't even know if those problems are still a problem or whatever, but we definitely had some, some pretty significant problems. And so at one point as a team, we were like *bleep* it, we're just gonna build our own database. And that was a mistake. 38 00:15:37.540 --> 00:15:45.120 I think if those words leave your mouth, you should seriously, seriously consider what you just said, . 39 00:15:45.120 --> 00:16:06.320 Yep. Yep. And we're like, it's fine. We're not, the whole point here is that we don't need a relational database. This is mostly for, I mean, the, the messages that we were, we were basically, uh, we needed a database to store messages that were coming off of a message queue. They weren't really relational in any way. Right. The, we chose Mongo because it was like a document database, and we thought it would be a good fit. 40 00:16:06.320 --> 00:16:08.480 Right. Sounds it sounds it on paper, but. 41 00:16:08.480 --> 00:16:23.140 Yeah and, and, and I don't doubt that we needed some kind of non-relational, you know, NoSQL document type database, whatever. We just should have tried something else after Mongo before immediately jumping to, we're gonna write our own database. 42 00:16:23.140 --> 00:16:24.830 But presumably you, in, in your defense, 43 00:16:24.830 --> 00:16:25.120 In Clojure. 44 00:16:25.120 --> 00:16:42.540 you had just been burned quite badly by whatever things you discovered in this nascent early version of Mongo. Yes. And so you were kind of like, uh, you know, you re on the rebound as it were from that going like, well, I'll never date again. I'm just gonna have to do this by myself. , 45 00:16:42.540 --> 00:16:46.540 Yes. Lonely me. I'm never gonna find love . So I wrote 46 00:16:46.540 --> 00:16:48.510 So, I should write my own database. 47 00:16:48.510 --> 00:17:17.820 Yeah. Yeah. And I mean, in fairness, it was one of those things, and we've talked, we talked about this yesterday, where it's sort of like, one of the advantages of doing this yourself is that you do set your path yourself on a path to incremental success. Right. Like, I know if I just put in enough time and effort, eventually I will create a solution that will solve this problem I don't know how long it's gonna take, but there's no chance that I'm going to basically wedge and have to start over from zero. Right. Or the chances of that, that are very, low. 48 00:17:17.820 --> 00:17:48.520 Very, very, very low. Unless you discover it's np hard along the way, and that's why no one else can do it. Right. But there, you know, yeah. You can make forward progress at every step, and you can set yourself a goal. You can, you can guess how long it's gonna take you, and you'll still be wrong, but you will be able to get there, . Yeah. Whereas if you are reading the documentation of a system and you are playing around with the system, there is no guarantee that you won't discover towards the end, oh, it just doesn't do the thing I need it to do. Or if it does, it doesn't do it in the way that I need it, or it's not performant enough, or it whatever. Yeah. That 49 00:17:48.520 --> 00:17:58.180 Some, some other hard constraint that basically devolves down to, well, I could fork this and do it myself, but at that point I am just kind of like building my own thing. Right. 50 00:17:58.180 --> 00:17:59.140 Right. 51 00:17:59.140 --> 00:18:20.840 So, yeah. And I mean, in, in our case, like we, we were, we had this, this situation where we had deployed the system. We were using it, it was being used in production, and Mongo was basically falling over. And we sort of had this thing of like, okay, we could try another database, but we might just be back where we are right now in two months. So let's write our own. Again, this was still a mistake. 52 00:18:20.840 --> 00:18:21.640 Okay , 53 00:18:21.640 --> 00:18:23.500 We should, we should have tried. 54 00:18:23.500 --> 00:18:26.320 I think you mean it was a learning opportunity, that you took. 55 00:18:26.320 --> 00:18:55.100 This is an opportunity for Yes. It was. The classic example of good judgment comes from experience and experience comes from bad judgment, . Um, but, uh, it was, it is definitely not something that I would do over again. Um, but I mean, in fairness, we did eventually build a database that was a solution to our problem. It's just that the ongoing cost of that and the, and the time that it took to really get it into a state where we were happy with it. Like it just, the just the trade off just wasn't there. 56 00:18:55.100 --> 00:18:56.200 It wasn't the right tradeoff 57 00:18:56.200 --> 00:19:11.240 Like, we should've We should've tried some other, we should've done. Like, I honestly, if I were to do this over again, I would do more of a set based approach, or I would say, all right, we've got four people on this team for the next two weeks, we're gonna explore four different databases. So everybody pick one and go, 58 00:19:11.240 --> 00:19:11.340 oh, that's an interesting approach. 59 00:19:11.340 --> 00:19:30.240 And, you know, share your, share your results as you go along. There's no reason to wait till the end, but like, we need to parallelize this effort and we need to try as many things as we can. And hopefully we'll find one that works. And if we don't find one after two weeks, then we'll do another two week. You know, or, or maybe we'll build it our own, or who knows. Or maybe we'll fork one. We'll take one and fork it. I don't know. 60 00:19:30.240 --> 00:20:13.580 You'll have some information, information gathering. I think I, yeah. Speaking as my, for, for myself really, um, I find that stuff really hard to do. I get so invested so quickly into something or I get very quickly like, this is rubbish, I'm not gonna be able to use it, kind of thing. And that I'll, I'll go away. So I, I, I, two weeks sounds like a long time to do that, but I think you're, you're right. You know that the approach is sound, um, going multiple ways at the same time so that, you know, everyone can meet back up at the table and sort of say, well, I hit this snag, and someone else was gonna say, oh, I hate that snag too. Maybe this is actually intractable. Maybe this thing can't be done the way that we want to do it. In any reasonable database. And that might inform your decision to say, well, let's, let's see if we can't, we can do it ourself. 61 00:20:13.580 --> 00:21:00.580 Yep. Yep, yep. So, you know, these, these kind of decisions that we make in this, this preference that we have, it's, it's definitely not obviously the right thing to do. . Um, I th I think that I can pretty easily justify some of the decisions that I made. They've turned out pretty well. Some of them are maybe a little bit more borderline, and some of them, as we were just talking about, are clearly not the right thing. Um, but it does, it does have a number of, of benefits. It has like, you know, this benefit we were talking about of like, you, you definitely tend to be more invested. You become an expert in the solution that you create, um, in a way that, I'm not gonna say it's not possible for something that you pull off the shelf, but it's just, certainly for me personally, it's significantly more difficult. 62 00:21:00.580 --> 00:21:53.060 And I feel like most people who are, who are like, think of themselves as software engineers, you know, people who build software for a living, their brains just click in better when they have have built something. It's the same, it's the same reason that you, like in school are like, we're gonna implement this algorithm. Right. And in the real world, you will never implement this algorithm. Please do not implement this algorithm. There are untold number of libraries that people much smarter than you have implemented this algorithm with all the different corner cases, and you should use their solutions and not your own, but just to make sure you actually understand how it works. We are going to implement this algorithm. Right. And I think that effect is a very powerful effect. And so when you build these things for yourself, you, you know, it's less likely you're gonna get stuck in this sort of like, yeah, this is broken and I don't understand why. Right? 63 00:21:53.060 --> 00:22:23.360 How well does that translate though, for new members of the team? Because you, that works if it was you literally oneself building the project or you know, the people that were involved in that feature, but then to somebody coming onto the team who hasn't seen it before, you've got a sort of double problem of like, it's not a standard solution, so there's no chance in heck that they would've had any experience with it and, they have to learn it as if it was a new system. Now. So I I, I'd not, I dunno if that's, um, 64 00:22:23.360 --> 00:22:52.780 No, I mean it's definitely, it definitely is a concern, but one of the things that I would say is, um, so imagine that you instead chose to use some open source tool, right? Um, and that tool is like a, a a really important part of your system, right? Like it's, it's critical. You depend on it, right? Immediately you start asking questions like, can we hire the maintainer ? Are are they, are they available? And if you build it yourself, you've done that by default. 65 00:22:52.780 --> 00:23:35.520 That is true. Right? You, you have the person right there that you're working with every day that knows the system top to bottom, and you can just ask them questions. And so as a new person on the team, you have that sort of baked in expert. Now, that expert might have a lot of constraints on their time and not, might not be able to share. And that's when you start, you know, right. Writing a bunch of documentation maybe, and then you're wondering like, why am I doing this? And so there's, there's def there's definitely reasons for, or, uh, situations in which that can fall down. But I do think one of the strengths of that, um, is that you, you, you have that person that is like, can just off the top of their head be like, oh yeah, you need to go look at this file. This is where that's controlled and you can configure it like this, or these are your options, and if you try that, that won't work. Right. Because they wrote the code, 66 00:23:35.520 --> 00:24:01.860 They, they know it. Yeah. So I guess you have a domain expert who is also again, invested, we hope, In the product. Right. And the project and, uh, is, is hopefully if you've employed the right kind of people is excited to tell people right about it and effusive and rather than like, oh, it's just Yeah. You, that, you know, Postgres just does it that way. Sorry, can't do anything about it. I It's just the way it is kind of thing, you know? 67 00:24:01.860 --> 00:24:13.380 Yeah. Yeah. And I mean, there are definitely dysfunctional forms of that where people will build these systems and then like be very reluctant to share their knowledge or write documentation or whatever 68 00:24:13.380 --> 00:24:23.600 Job security, uh, dysfunction of like, yeah, hey, yeah, I'm just the expert in blah now everyone needs blah. So, uh, I'm, I'm set for life now. 69 00:24:23.600 --> 00:24:39.940 . Yes. And, and, and there's, you know, less, less, um, pathological forms of that too, where it's just like, I'm just really busy. I can't, I can't do this. Right. I have a lot of other responsibilities. I can't spend all my time explaining to people how the system works. Right. Like, I wrote some docs, maybe you should read the docs. Read the code, I don't know. 70 00:24:39.940 --> 00:25:17.980 That's another thing as well, you know, obviously you can help yourself by documenting stuff, but like, certainly typically in organizations that I've been involved in, uh, documentation is, uh, at best out of date and at worse, actively harmfully wrong. Which is, I mean, I believe that to be the natural state of documentation, which is why I heavily discount it myself. So that's why I don't tend to read other people's documentation, which is, is sometimes is, is problematic. Cuz they'll say, well, did you read the docs? I'm like, no. Cause the docs are just wrong by , they're definitionally wrong as far as I can tell. And they're like, no, these ones are actually up to date. You're like, oh, now I feel terrible. 71 00:25:17.980 --> 00:25:18.120 . Yeah. 72 00:25:18.120 --> 00:25:18.700 I feel like a bad person. 73 00:25:18.700 --> 00:25:34.460 No, I'm, I'm, I'm mostly with you. They're the best docs are the code. Like you know, it's, it's like you should have a readme that explains some basic concepts. And if it's any more than about three or four pages or three or four screens worth, what's a page even, 74 00:25:34.460 --> 00:25:35.820 I dunno what a page is 75 00:25:35.820 --> 00:25:40.960 Nobody even knows what a page is anymore. Yeah. Three or four screens worth of text. Like it's, there's probably some lies in there. Yeah. Um, 76 00:25:40.960 --> 00:25:57.330 I will say I've had some good success with executable documentation where I've used like, like, uh, Pydoc star tests, and it's like, yeah. You know, my documentation's up to date because at least the bits that mention code, they still work and they pass the test, which is a good way. Right. But anyway, that's, I, we digress. 77 00:25:57.330 --> 00:26:48.940 Um, so one thing that came up when we were chatting yesterday was the, the sort of like, there are different mentalities to, towards making reusable components. So what we're talking about here are reusable components. We've mentioned sort of like system level things really in most of this type of stuff. You know, you mentioned a database, which feels like a, a system even though it could be implemented as more of a library, but, um, and I, well the, the vendor solution I was talking about was a big library, but it's an ecosystem of lots of libraries and they do lots of things. There's a huge amount of things that they do that I don't need them to do, which gives me complexity I don't want. But there's definitely something to be said for the kind of manicured, very well-developed and well thought about domain expert library when you actually pick up a library that's like that and you're like, I know I can trust the 78 00:26:48.940 --> 00:27:36.600 Folks behind this to have thought about everything extremely, um, thoroughly. And so I don't need to know this, the, the product well enough. Maybe I don't have this, these particular needs, um, that I would develop it myself, but I'm actually almost, I guess it's just like one could argue this is a, uh, the way you might choose to vote, right? I vote not because I want to choose everything myself, but I'm picking somebody who is smarter than me to make the decisions, like the delegation type thing. De delegation theory of, of like why you vote. And maybe I'm like, well, I need a library that does x I don't have a strong belief about how X should be done, so I'm going to pick someone else's X because I will come, it will come with their sensible choices for everything about X that are on the periphery. 79 00:27:36.600 --> 00:27:37.500 Mm-hmm. . Yeah. Yeah. 80 00:27:37.500 --> 00:28:01.760 And so that's maybe one argument against the inventing yourself where you have to like make hundreds of somewhat arbitrary decisions for things you don't necessarily care about and you aren't maybe not as invested, uh, in, in coming up with the right balance of like, should we use threading or thread pools or different processes or I don't know, but someone else has thought about this. Maybe it's configurable, maybe it isn't . Um, yeah. So there's a value there. A value that's 81 00:28:01.760 --> 00:28:53.740 Exactly. No, absolutely. And I think the thing to consider with those, with making those choices is very similar to what we've talked about with programming languages on a different episode of the podcast, which is, you know, we said before, it's like when you adopt a, when you start using a programming language, you implicitly have to adopt the values and principles of the community behind that language. And those are almost more important than the language features. If that community values testability, you will be forced to value testability by using the, the platform and the, and the APIs and the language and all the things that are built around it. If they value thread safety, if they value terse syntax, if they value a functional style, whatever it is, you are going, like, you're either gonna be fighting against the current the whole time you're using it, or you're going to have to adopt those ideas. 82 00:28:53.740 --> 00:30:16.600 And I think that you get that same effect sort of in a slightly smaller form when you're using somebody's library. And I think that you've experienced this, uh, in your, in your day job with this library, this vendor library that you're using, where the ideas and principles that they put into that library are not necessarily the ones that you would choose. Um, and there's a cost, there's a tension that is created because of that. And now you're sort of forced to make a choice of like, well, I don't really want to care about this problem, but man, I wouldn't have solved it this way if I did it. So that's, that's creating tension and this sort of sweet spot. The, the wonderful, uh, situation that we're all aiming for is when you can successfully delegate the creation of a solution to somebody and their values and principles align enough with your values and principles to where you can be like, yeah, I probably would've done it in a very similar way. This library's great. Right. Um, but that's sort of the, the, the, the, the trade off. And, and I think, you know, like that's obviously like a little bit of a gradient, right? Um, there's no, it's not a binary thing, pun intended, . Um, but, uh, the, the choices that I think we tend to make is, is would lean us a little bit more toward the side of like, you didn't solve this the way that I would want to solve it, so I'm just not gonna use it. Right? 83 00:30:16.600 --> 00:30:24.960 Yeah. And then that's definitely a symptom. And I wonder sometimes whether or not, am I just using this library wrong? Have I come at it from the wrong 84 00:30:24.960 --> 00:30:24.960 Mm-hmm 85 00:30:24.960 --> 00:31:09.020 axis, you know, and then that's when, if you know somebody who has used a library or a feature before and they go, oh yeah, you're thinking about the problem wrong. You know, you can solve the, you can solve the thing that you need to be solved, but you are thinking about it wrong. If you rephrase it this way, then it just falls naturally out of this library. And you're like, brilliant. I needed that. I was, I was going against the grain, not because, um, I fundamentally disagree with some way that the system is designed to be used or whatever, but because I didn't know and I couldn't find out how to rephrase my, uh, my problem in the right way, but sometimes you're like, I am going against the grain and I'm just fighting and fighting and fighting and fighting because it's not actually possible to use it the way that I want it to be used. 86 00:31:09.020 --> 00:31:37.200 And there is no better solution that's just like, well, sorry, that's not really designed to do the thing that you want it to do. Which is a bit more like my, my most recent experiences, they're like, they're like, well, sorry, that's just not how it works. You're like, wow. Oh, okay. I guess, I guess I'm stuck then, right? I, I've, I've made a, i I can't make another choice in this particular example. But, um, and it's not wrong. Right. You know, everyone has different values of course. You know, that's, they're, they're not necessarily wrong. They're just not the way that one would choose to do it. 87 00:31:37.200 --> 00:32:46.480 Absolutely. I mean, ev everything is a bit of a portfolio optimization where you're choosing from different trade-offs and, you know, you're, the, the, the things that you value are gonna be a function of your strengths and your weaknesses, and it's gonna be different for everybody. So yeah. But one of the things on that, that I definitely I have done and I have also seen you do is, um, you'll ask people that you know and trust perhaps in a discord channel or a Slack channel, Hey, does anyone have a good library for X? And what you're asking when you ask that question is not, can you Google this for me? Right? , what you're asking is, Hey, people that I know and trust and have similar, but not necessarily even the same sort of strengths and weaknesses, values, principles that I have, but I know what yours are, and if you tell me that library X worked for you to solve a problem, I will be able to interpolate whether it it will work for me. Right. So like I know that you, for example, value, uh, performance and, you know, uh, native fast execution and therefore if you say, Hey, this is a great library for x, I can be like, well, if my concerns are, is it gonna be fast enough? The answer will be yes. 88 00:32:46.480 --> 00:32:48.120 Right. . Right, right, 89 00:32:48.120 --> 00:32:52.440 Right. Or I can ask you and say like, Hey, was it fast enough? And you were like, absolutely. And then I'll be like, yeah, this is what we need. 90 00:32:52.440 --> 00:32:57.170 This is, and I know if I ask you about something, then it almost certainly will be testable, uh, 91 00:32:57.170 --> 00:32:58.320 Right, exactly. 92 00:32:58.320 --> 00:33:02.720 Or mock-able or have some kind of Yeah. Affordance to, to to, to testing. 93 00:33:02.720 --> 00:33:25.320 Right. And you'll be able to prepopulate and say like, yeah, it's great for this, but watch out for this thing and this will burn you and this is a problem. But otherwise it's cool. Right? Like they've, so it's like, you know, when we ask those kinds of questions to the people that we've worked with before, it's not about like, Hey, can you do a survey of all the possible tools for this for me? Yeah. It's like, can you relate your successful experience with this thing. 94 00:33:25.320 --> 00:33:26.240 Or, or 95 00:33:26.240 --> 00:33:27.680 Should I have confidence that it'll work for me? 96 00:33:27.680 --> 00:33:58.700 Or even like say, oh, uh, I don't have a good suggestion, but I have tried X and it was a catastrophe for these reasons. Which sometimes is as as valuable. In fact, it's more valuable. Right. Don't go down these blind alleys. They they are, they look like they will be, they will bear fruit, but they almost certainly won't. Now, of course, sometimes again, that person has done things wrong. I've, there have been times before now where I've, I've told people, yeah, no, that that system doesn't work that way. And then they've turned around and said, well, actually, I took a look and it does. And you're like, oh, well , today I learned. Yep, yep. . 97 00:33:58.700 --> 00:34:18.880 Yeah. That's sort of the, the, the bit about if you ask a a, an old gray beard engineer, if something, uh, doesn't work, uh, and they say, yeah, no, it doesn't work. You can sometimes discount what they say, like, that's fine. But if they say that it works, it almost certainly works. Right. Like, like unless they're just straight up lying to you. 98 00:34:18.880 --> 00:34:30.960 Yeah, it's hard to misinterpret success . Whereas defeat can sometimes be, well, again, if you rotated 90 degrees, it fits in through just perfectly actually. Just rephrase your problem. 99 00:34:30.960 --> 00:34:33.160 . Exactly. Exactly. Exactly. 100 00:34:33.160 --> 00:35:22.200 Well, I mean, it was an interesting discussion yesterday and it has been an interesting discussion today. Um, I certainly. Had never really thought about it. It was it for you and I, I think it was such a, a, a sharp observation for us both. It took us kind of by surprise that it was just offhand handed to us as like, Hey, I've noticed this thing about you, Matt and Ben, and the folks that we've worked with from the previous company, that you all have this really low threshold for reinventing things or, or, or making things yourself. You know, you don't seem to be dissuaded by how difficult it might be. And it was presented to us as almost neutral, a neutral observation, I don't think. Uh, I don't think it was meant as either a slight or a a, a blowing smoke towards us. No, no, no. 101 00:35:22.200 --> 00:35:22.440 It was 102 00:35:22.440 --> 00:35:51.260 Like, Hey, I've just noticed this really unusual thing. I've never seen it at companies I've been at before. And so it makes for a perfect conversation, topic for a podcast, and I wonder what other people listening feel. So it would be interested to hear what, uh, what observations our listener has about when, when you should build yourself, when you should investigate, and when you should, uh, just, I dunno what the other solutions are when you should just go home, . 103 00:35:51.260 --> 00:35:51.960 Yeah. 104 00:35:51.960 --> 00:35:57.140 Which seems like a perfect time to, to, to go home ourselves and call it a day. 105 00:35:57.140 --> 00:35:58.280 Sounds good. 106 00:35:58.280 --> 00:36:00.840 All right. I will see you another time. 107 00:36:00.840 --> 00:36:01.320 Yep. 108 00:36:01.320 --> 00:36:03.320 Bye for now.