WEBVTT 1 00:00:14.760 --> 00:00:19.560 Hey, Ben. 2 00:00:19.560 --> 00:00:20.560 Hey, Matt. 3 00:00:20.560 --> 00:00:21.960 How on earth are you? 4 00:00:21.960 --> 00:00:24.980 I'm doing okay. I'm still, I'm still a little sick. 5 00:00:24.980 --> 00:00:26.970 I was gonna say, you sound a bit nasally still. 6 00:00:26.970 --> 00:00:38.960 Yeah, I got, I got a little bit of not really congestion, but just sort of like, I don't know about you, but I feel like at some point in the winter, I get a cold. And this was obviously not true in the last two years of the pandemic 7 00:00:38.960 --> 00:00:40.880 Because you didn't see other humans. 8 00:00:40.880 --> 00:00:48.040 Yeah, right. But at some point in the winter, I get a cold and that cold lasts until spring. That's been my experience too. Right? 9 00:00:48.040 --> 00:01:01.700 Yeah. It's like I feel better, like I'm not really sick, but like that little lingering cough or that little sort of like, you know, rough sound in your throat. It just, the only thing that really cures it is spring. And I hate that. I don't know, really. And 10 00:01:01.700 --> 00:01:17.960 It's always, you always sound the worst after you've stopped feeling bad. That's what I might discover. So like, you feel dreadful for a couple of days and then you get no sympathy off anybody. Yeah. And then you sound all stuffy and crap the next day. And it was like, oh, are you all right? I'm like, I feel fine now, actually, but thanks for asking. 11 00:01:17.960 --> 00:01:23.300 Yeah, right. Exactly. So yeah, probably a more long-winded ans answer than you wanted for 12 00:01:23.300 --> 00:01:38.300 That was. Yeah. This is, look, this is the SYN / SYN|ACK of the podcast here. Not, not the, I I'm not actually genuine interested in how you are. I'm afraid. . I mean, I am. I am, but, 13 00:01:38.300 --> 00:01:42.440 Oh, man. Yeah. So what are we talking about today? 14 00:01:42.440 --> 00:02:08.800 That's an excellent question. Well, we were chatting yesterday at breakfast, such as it was, we, we got a coffee together as we'd kind of got on the same train. And then we got talking as we do, and then we, the, the conversation was so interesting that instead of going to our desks once we actually got to work, we sat down still wearing our coats and carried on the conversation about integration testing of all the things. I mean, 15 00:02:08.800 --> 00:02:08.900 Yeah. It 16 00:02:08.900 --> 00:02:20.900 Seems unlike us to, to be so excited about something that many people might consider very dull, but maybe not our listener who is that now going "testing? Amazing Tell me more." 17 00:02:20.900 --> 00:02:33.300 Right, right. Yeah, it's kind of shocking to me. I was looking back over some of the old episodes and it's kind of shocking to me that we haven't talked about this the best I can tell. I mean, we've definitely talked about shades of it, but, 18 00:02:33.300 --> 00:02:50.860 You know, we talked about acceptance testing, which to me Yeah. I sort of broadly think of as an integration test of, of a sort. I mean, it it is, yeah. Pretty much an end-to-end test type thing. And that's obviously has its place, but not sort of like the integration test that we, that we know and love or I've written before now where 19 00:02:50.860 --> 00:02:51.420 Yeah. Right, 20 00:02:51.420 --> 00:02:55.880 Right. Often come to regret latterly. Mm-Hmm. 21 00:02:55.880 --> 00:03:41.640 . So yeah, so I mean, basically, you know, as we were kind of talking about yesterday as you, as you said integration tests are I think a, a bit of a double-edged sword. And one of the things that has stuck with me for a very long time is a talk that JB Rainsberger gave mm-hmm. a few times in a few different venues. I think he was mostly giving this talk back in like 2008, 2009, and I think he's updated it and revised it a few times since then. But the original title of the talk was "Integration Tests are a Scam", , and the, the point, the point that he makes. And, and he has, I think he's revised this over time. I think he's changed it slightly as like "integrated tests are a scam". Okay. And he's got some, you know, some sort of caveats. 22 00:03:41.640 --> 00:04:35.300 It's a very sort of clickbait, you know, of course in this world, you know title. But it's, it, it, it makes you think, and I think the central point is, is sort of bang on, which is integration tests, which we're gonna define as tests that exercise large chunks of the system, perhaps spin up external dependencies like a database or, you know, some other sort of like network service. You know, if you had like a if your system was built out of microservices, you might spin up like a whole bunch of containers to run all your different microservices and have them all interacting with each other. You know, anything that, that talks to like a third party service you know, that that's kind of what we're defining as an integration test here. And, and usually in those kinds of tests, you're testing lots of different pieces of behavior all at once, you know. 23 00:04:35.300 --> 00:05:15.120 Right. An example of this would be like, all right, I'm gonna, I'm gonna log in as a user and I want to test to make sure that we can log in. It seems like a reasonable tester, right. Make sure users can log into our system. Right. Certainly very important. Yeah. So I'm gonna start up the web server. I'm gonna start up a database. I'm gonna populate that database with some user records. I'm gonna, you know, go to the website. I'm gonna enter in a value for the username. I'm gonna enter in a value for the password. I'm gonna hit the login button, and then I'm going to get redirected to a dashboard, a user dashboard, and I'm gonna confirm that all of that works for some definition of confirm and some definition of works, right. . and so those are the kind of tests that, that we're talking about here. 24 00:05:15.120 --> 00:06:20.820 Those are the kind of tests that I think JB was talking about in his talk. And his central point was those tests are a scam. They seem like a good idea. They seem like they give you like a little bit of a safety net and some confidence that things work and they do. But the costs of those kinds of tests are very subtle. Sometimes they can be very hidden, and they are often paid in the future with a ton of interest. And so if you build your system with the idea that you're gonna test everything through integration tests, what you often find is that over time, your tests, your test suite gets very slow and it gets very brittle and it becomes extremely difficult to make changes to your system, which was kind of the whole point of writing tests in the first place. Is that right? You want to be able to make changes to your system quickly and have confidence that you haven't broken anything. Yep. And it's like, well, you still maybe, maybe have the confidence that you, you have haven't broken anything, but you can no longer make changes to your system quickly. Right? Yeah. 25 00:06:20.820 --> 00:06:21.500 Yeah. 26 00:06:21.500 --> 00:06:38.120 And so that is why integration tests are a scam, scam, that sort of Yeah. And that idea has kind of stuck with me for a long time. And so a natural question that one would ask if you say, okay, Ben, let's take that as a premise, is what should I do instead? 27 00:06:38.120 --> 00:06:45.760 . Yeah. What should I just, should I just accept that occasionally I'm gonna make an an innocuous change and then everyone can't log in and, and that's okay. I mean, 28 00:06:45.760 --> 00:06:46.360 Exactly. 29 00:06:46.360 --> 00:06:48.340 Maybe, maybe it's okay if you're Twitter 30 00:06:48.340 --> 00:06:49.000 31 00:06:49.000 --> 00:06:54.480 Too soon, except not too soon, because by the time this goes out that no one will remember why, why? That's funny. 32 00:06:54.480 --> 00:07:05.500 Sick burn. Yeah. That's, that's true. That's a good point. But yeah, so, and that, and that is sort of what we were talking about yesterday is sort of like, okay, well if you're not gonna do that, 33 00:07:05.500 --> 00:07:05.660 What 34 00:07:05.660 --> 00:07:06.460 Are you gonna to do? 35 00:07:06.460 --> 00:07:21.690 Then? How can I just develop confidence that I can make changes and I haven't like painted myself into a corner where to, in order to make those changes, I have to like update the integration test because I've moved the login box up a little bit and that was totally not the, that's the point of my change. 36 00:07:21.690 --> 00:07:21.860 Yeah, 37 00:07:21.860 --> 00:07:26.340 Yeah. That kind of stuff. So what do I do? Tell me. Yeah. 38 00:07:26.340 --> 00:07:32.660 Well I was gonna ask you some of the things that you've done. Obviously I've done a lot of things on this, but if, if you want to throw 39 00:07:32.660 --> 00:07:55.770 No, I won't. I was playing the fool. Go on. You asked me. All right. That's, I don't, so like yeah, the, the, the, we, we, the, again, we were talking about this yesterday, which is not a, a good thing to keep bringing up on the podcast cause nobody was there yesterday, which is why we're recording this today. Yeah. But we talked a lot about transitivity of tests mm-hmm. in terms of unit tests in a lot of places. So let's take your logging in for example. Mm-Hmm. 40 00:07:55.770 --> 00:07:56.280 . 41 00:07:56.280 --> 00:08:18.840 So we have tests that say, does our interaction with a database work, can I select rows out of a database and read them back in again? Okay. That could be a test. Now that does sound integration-ey and maybe it is integration-ey but like I, I'm, I developed some amount of, of confidence that I can talk to a database that could be a first step. 42 00:08:18.840 --> 00:09:22.780 And then I go, well, now I understand how databases work. I'm gonna just get rid of the database and replace it with some kind of fake or a stub or a mock or any insert your favor, not a real thing in technology here. And then I can write the majority, the line share of my, can I access, quote the database correctly tests, using tests written against that fake stub mark, blah, whatever mm-hmm. . And now I know I'm pretty confident that I can get the user information out and maybe their sorted password, whatever it is. And then I can separately write tests that take a username and sorted password and say, is this the right user? Is this, is this the right credentials for this user? Tell me yes. No. Right. If that's how you're writing your system, obviously. And now I have a sort of transitive relationship where it's like, well, given that I trust the database code, given that I interact with the database code correctly, given that my login system works with canned examples that I've set up, and given that I've tested the interactions between those two using Mock or Fakes, now transitively, I don't necessarily have to write the code that goes to the database and checks. 43 00:09:22.780 --> 00:10:04.380 I can log into the database directly now. Mm-Hmm. , I just know the pieces work and the, the, the bits between them work and then I can keep building back and back and back until I get to the point where I have 99.9% confidence in each of the individual steps along the way, handing off correctly all the way to the web service handler or the JavaScript code. That's like when they click the login, I send this post request and I know that that will get to the end point. Cuz I have tests there and it follows all the way through to the point where I go, yes, the user is logged in and then right now, what am I giving up in that? Well, there's always, you can always come up with Machiavellian reasons why that you can break that mm-hmm. . Right. And that's fair, that's a fair fair. 44 00:10:04.380 --> 00:10:42.260 Yeah. But the, the marginal cost of dealing with the, the, the Machiavellian case compared to like the integration test cost, which is giant and as you say, brittle and often fragile and, you know, flaky. I'm, I'm definitely like, it's definitely not as certain because for each step, I'm, I'm only 99.9% sure. Let's say mm-hmm. . And so you're multiplying a bunch of 99 point nines together and maybe there's 20 of them and that quickly becomes like 97% rather than whatever. My math is not perfect here, but you know what I mean. Right. You do, you do lose something. I'm not gonna question it, but 45 00:10:42.260 --> 00:10:42.540 Yeah. Yeah. 46 00:10:42.540 --> 00:11:33.840 The, the chances of a real bug falling in that 3% crack seem diminishingly small to me, and the cost for a big integration test, or at least developing all your tests as integration tests now. Right. Maybe there's a call that says like, it is so fundamentally and critically important that users can log in, that you do have a once a day CI thing that stands up the system and does the thing or pre-deployment or even you have a human do it. Right. I, at that point, I prefer not to do that. But like, you know, if you need it then this is, there's a business case for it. But if you build a system where that's the primary thing that you, the primary way you make your tests, then you are definitely making it hard for yourself. Right. You can't iterate as quickly and it's, yes. So that would that I, I think I went all the way to the far end of the 47 00:11:33.840 --> 00:12:12.520 That's really good. I think, I think that's a really good baseline cuz I think that describes the sort of basic answer, the, the short answer to, okay, if I'm not gonna do integration tests, what I'm gonna do, well, I'm gonna break the problem down into lots of little bits and I'm gonna use the transit property to as make sure that all of these little bits can talk to each other. And each of those tests for each of those little bits can run super fast and be very reliable. Cause they're not, you know, necessarily talking to external systems or anything like that. And when they fail, they will give you a very focused answer of this bit is broken. Right? It's not like, well, something in the login process is broken. Broken, and I don't know 48 00:12:12.520 --> 00:12:38.900 What it is. Let's say somebody changes the "crypt" method. Mm-Hmm. , you know, we, we pull in a new library, it's got a different version of "crypt", it's not quite compatible, and now users can't log in. And what your, your integration test says is log in failed, and you're like mm-hmm. , why, why is login failed? And then you have to like git bisect to find the change or look through all the PRs, whatever. Whereas if your test is, I couldn't authenticate this user with this password 49 00:12:38.900 --> 00:12:39.580 Right 50 00:12:39.580 --> 00:12:44.470 Now, maybe you've got a much more localized understanding of where the problem will lie. 51 00:12:44.470 --> 00:13:03.980 Yes, yes, yes. And I think that this sort of, that sort of difference has a number of very interesting properties to it. One is it's sort of the difference between if you're doing any kind of validation, data validation, like checking individual fields versus checking like a check sum or a cryptographic SHA. 52 00:13:03.980 --> 00:13:05.240 Right? Right, right. 53 00:13:05.240 --> 00:13:47.460 Like if you just take a whole bunch of data and you're trying to be like, what are the differences between these two pieces of data? Well, if you, if you run it through diff if it's text data, you can see the differences, right? Right. Like, oh, this line is here and it's not there. If you take both of them and you do an MD five sum on them and you compare the MD five sum, it's like, well these sums are different. And it's like, well why, why did it, what happened? Now it's much easier to compare the MD five sums. It requires no understanding of the data itself. You don't have to look at it, you don't have to understand what it is, what each of the individual lines are. It's very simple, but all you really get outta that is if everything works properly, you knew that everything worked properly. 54 00:13:47.460 --> 00:14:34.420 Right. Anything breaks, you're left in this world of like, well I know it doesn't work, but I have no idea why. And so I think that is why, again, creating the bulk of your tests using this, this sort of integration style will lead to a world in which you will operate very slowly. You will be able to make changes very slowly cuz you will be constantly in this world of, hey, these two MD five sums are different and I don't understand why. Yeah. and so there, there's that, there's that property to it. The other thing that I think that it does that it's, that's kind of interesting is it does put a, a bit of a bur the, the alternative that we're proposing here where you, you break things up into small pieces and test them does put more of a burden on software engineers who are changing the system. 55 00:14:34.420 --> 00:15:31.840 Right? Right. And the burden that it puts on them is that they have to make an effort to understand when they're making a change the parts of the system that interact with that part. Right. Which is not strictly true if you say, well I'm just gonna test everything through integration test cuz I can lean on the integration test to tell me that. Right. Right, right. If I go and I make a change to the system, I can just ignore everything that it talks to and everything that talks to it and I can "trust". That the integration test will catch that. Now in practice, I will tell you that any integration tests suite that is actually comprehensive enough to test every part of a system using that approach will finish after the heat death of the universe , because the combinatorial explosion, right. Of all of the different possible code paths through all of the different possible conditionals in that system are 56 00:15:31.840 --> 00:15:33.500 Just too immense 57 00:15:33.500 --> 00:15:35.740 It's like like grains of sand in the universe. 58 00:15:35.740 --> 00:16:33.620 An interesting observation that I'd never really considered is that you are definitely reducing the dimensionality of the problem by testing unit by unit. Like I'm, I'm, I'm testing my password hashing thing. Now, of course there is an infinite, almost infinite number of inputs to my my password hashing thing. And there's, you know, essentially 64 bits, 128 bits, whatever of, of possible outputs. Right. And I can't write tests mm-hmm. , I knowingly can't write tests to cover all that. But as I'm designing that thing, I know where the bodies are buried and I can use whichever you know, ZOMBIES approach to say, you know, what if I give it an empty string, what if I give it a full string, whatever? And then I've tested it with all of those things and I'm sure it works in that situation. And now the property that I transit to the outside world is not the high dimensionality of all those possibilities. It's the did they log in? Yes or no? Yes. And then that's the one and only thing that's, that's the one thing that escapes my world. I I've, I've separated away all those degrees of freedom from like the rest of the code. 59 00:16:33.620 --> 00:17:34.800 Right, right. Yeah, absolutely. And I think one of the cool thing is, is that if you sort of do what we're talking about here, you will arrive at that sort of naturally and you will be naturally incentivized to sort of make that that contract with the outside world as small and as simple as you possibly can. So going back to our sort of database example, right. With the login, right? Like, okay, if I'm gonna break these things apart and I'm gonna have, you know, some system that interacts with the database to say, Hey, I need the, the, you know, I need to get the the salted password for this user, for example, right? Like the fundamental operation of get me a sallted password or get me any field that's related to a user has nothing to do with SQL or databases or, you know, servers or network connections at all. It's a very simple operation, right? And so there are various points in this sort of chain where you can create abstractions that are significantly simpler than the actual underlying system that's backing them, 60 00:17:34.800 --> 00:18:17.600 Right? Right. You could reasonably have said, you could have exposed the sequel to everybody, for example. Like, and then in the first case, right? Because it's convenient, Hey, select star from users where name equals quote, whatever I have in my street. That's, that's one thing. But this forces you down the, the route that says, no, no, I want to pr provide the API of bull. Does user exist? Mm-Hmm. mm-hmm. Bull does user have is check password, whatever. That's obviously terrible. This is not a great way of designing a system for anyone. But like those kinds of things, you're, you are, you're forcing the testability boundary to, to look more like what the downstream users will actually want. Because Yes. You don't wanna expose 'em to that and then you can hide that away from everyone including yourself most of the time. 61 00:18:17.600 --> 00:19:06.290 Right. Right. Right. And that has a lot of like really good design benefits that are very practical. They're practical in the sense of like, it's helping you. Right. Better, faster, more reliable tests. But it also sort of, in my experience, it actually sort of creates a, a nice sort of framework for discussion among like the people that you're working with. Right? Right. Right. Because like people have these sort of like esoteric, almost like philosophical debates about software design all the time mm-hmm. . And what actually kind of helps that, and I'm sure you've seen this with performance too, what actually kind of helps that is having a constraint, right? Saying like, oh, well we can't design it that way because it'll be slow, or we can't design it that way because it won't be testable. It actually sort of like helps people come to an agreement and say like, oh yeah, that design is better and I can see why Right. 62 00:19:06.290 --> 00:19:06.480 I've never thought about that, but yeah. 63 00:19:06.480 --> 00:19:52.840 As opposed to I want an object hierarchy here. It's, no, we should use functions as, no, we should use a list. No, we should use a set, whatever. And it's like, if you can tie that back to something concrete that everyone agrees is valuable, of course, then you can actually have like much faster, more coherent, more aligned discussions around software design because they're based on fundamental principles that you all share, right? Yeah. Yeah. And you don't get the opportunity to do that if you don't create the constraint by saying, well, we can use whatever design we want cause we're just gonna create this with an integration test anyway. Yeah. Right. Or we're just gonna test this with an integration test anyway. So it doesn't matter if you create a nice layer of abstraction over the database, you can just have the templated string with the sequel in it and, you know, inject whatever values you directly got from the UI into that script. 64 00:19:52.840 --> 00:19:55.180 Terrifying , 65 00:19:55.180 --> 00:20:26.500 Because, because that's not gonna, that's actually the easier thing for you to do, as opposed to the easier thing being, creating a nice abstraction that you can then swap out. Right. and this actually leads me to another thing that we were, we were kind of talking about, which is a technique that I do all the time, which is using the sort of test fakes that you create as part of doing this process as sort of the repository for institutional knowledge about strange errors. 66 00:20:26.500 --> 00:20:27.740 Right? Right. 67 00:20:27.740 --> 00:20:32.840 So like you build your interface to your data store, right? Right. And, 68 00:20:32.840 --> 00:20:35.840 And there were air quotes for the listeners there around the data store. Yes. 69 00:20:35.840 --> 00:21:28.180 And it's, and it's a database under the covers. The, the normal implementation is a database, right. Some thin wrapper around some, some SQL driver or something like that that you have. And maybe you even think about it, it is like a SQL based thing, right? So it's like, I'm giving this thing sequel statements and it's giving me results. I don't really understand what's going on behind that abstraction, but that's what's happening. Yeah. Like that's the interface that I have. Right. And you have your sort of real, okay, we're using Postgres, so we use the Postgres one, and then maybe there's like some, you know some really stupid stub that you create maybe where you just have like hardcoded query strings with hardcoded responses, and you can use that in some of your unit tests when you're just testing things like, yeah. Right. I just, that's not the point of this test. It's not the test if the database works or if sequels correct or anything like that. I just wanna make sure when I run this grade and I get this result, do I process it 70 00:21:28.180 --> 00:21:31.900 Correctly? Right? Yeah. Which is a very common thing to wanna do. Right. You know? Yeah, 71 00:21:31.900 --> 00:22:15.780 Exactly. Exactly. But another form of implementation that you might have there, in addition to the real one, in addition to the super dumb fake one, is a realistic one that lets you simulate error or failure modes. Right? Right. And this is something that like a lot of times you will have some strange esoteric error, like oh yeah, the cluster was rebalancing and in the middle of the rebalancing we executed this query and there was some sort of weird, you know, data partitioning condition that the database detected and it threw this era that we'd never seen before. Right. And we have no way to reproduce it because it just depends on like the timing of the database, like actually just doing its rebalancing and, but 72 00:22:15.780 --> 00:22:23.140 We wanna lock in the good behavior of if we see that again, right? Yes. Because we we fell over in prod . Exactly. Exactly. 73 00:22:23.140 --> 00:23:19.000 It created, yeah. So, so like where does that knowledge go? Right? Well, in a lot of organizations it just goes into a ticket or a document, and that's, it's sort of is that's where it is. Right. But there's an alternative here, which is you can take that hard one, hard fought knowledge where somebody had to wake up at two in the morning to deal with this thing and take it and encode it into a sort of a test double, like a fake database whose purpose is to kind of mostly behave like a real database, except it lets you puppeteer all of the different failure modes that you've seen. Yeah. So when you go and you write that code to say like, Hey, we have this one in a million error condition that only ever happened one time, and that happened 12 months ago, and I'm changing the error handling code for that right now because we're making another change. How do I know if this error handling code actually still works in 74 00:23:19.000 --> 00:23:19.780 That weird case? Yeah. 75 00:23:19.780 --> 00:24:04.120 In that weird, that one weird case, right? Well, if you encode that information into your test doubles, you can have a lot of confidence that works, right? Because you can say like, yeah, well, well we, we reproduced the error that we got in the client code by rera, that same weird error that we saw out of the database, the client code failed in the same way. So we knew that we sort of reproduced it and then we fixed the client code so that it handled that error so that it wasn't an issue. And we have a test, you know, that makes sure that like, when that client code runs in this error occurs, it, it works. So I can go back later and I can say, okay, well if I change how that's handled or I handle it in a different component, I can reuse that same fake database in a completely different component. It's like, oh yeah. When you're interacting with the, the storage system, 76 00:24:04.120 --> 00:24:04.720 You have to know.... 77 00:24:04.720 --> 00:24:07.540 Watch out for that, sometimes it'll return that error. 78 00:24:07.540 --> 00:24:09.560 Yeah. Yeah, yeah, yeah. 79 00:24:09.560 --> 00:24:28.540 Because it can happen sometimes and you can have a reasonable degree of confidence that if it happens again, you will handle it. Right. So there, there is another sort of opportunity that you get by, by, by using this approach to sort of encode that information in a way that is much more immediate than tucked away in a ticket somewhere that no one will ever absolutely look at unless something goes 80 00:24:28.540 --> 00:25:31.600 Terrible. At the very least. I mean, I, the fake stuff, I, I can see that it could work, but I I, in my experience, when those things have happened, definitely get setting out and like making a mock that walks the system through that exact system, exact set of circumstances, and then giving a test the name that explains it and maybe has the reference to the ticket where it was filed, or the page of duty alert or whatever. Mm-Hmm. is a great way of saying like, here is, here is the pile of wounds that we've recovered from. And , you know, here is our, our list of of, of, I think you said institutional knowledge, right? You know, when we, when when the gray bids go, oh, you probably hit that thing, you know, then you're gonna go, well, how was I supposed to know that? You're like, well, if you'd have written a system for which the test fell out naturally, then maybe maybe you would've found it, or maybe it would, it would've been more obvious. And certainly refactoring the code later to not handle that will be picked up. And you'll be reminded that. Yeah. Yeah. By the way, your new code, the new error handler doesn't handle the strange network split that, that we saw mm-hmm. 81 00:25:31.600 --> 00:25:43.780 mm-hmm. . And if, and if all of your if you just have a big suite of integration tests that are all talking to a real database then you can't really write those kinds of tests. Like you, you can, they're just like, they're just very expensive to write, you know, they're gonna be very, 82 00:25:43.780 --> 00:26:08.180 And in some cases they're almost impossible to puppeteer the system into a set situation where Exactly. Exactly. You can make it happen even re remotely or, or at all, you know, like some strange admin commands and you know, nobody wants to have any chance at all that, like these unusual things where you're like maybe even killing processes to try and kind of make it do the error. No, that's not code you wanna write in case it ever Yes. Somehow makes it remotely near production Right. As 83 00:26:08.180 --> 00:26:33.480 To that. Right. Right. Yeah, exactly. And then you wind up doing it, it's like, it's like very, very expensive to write those tests and run them and make sure that they work. And it's for this error that has only ever happened once. Yeah. And you start asking yourself like, yeah, we added like two minutes to our build to like, you know handle this, bring up this, this database and tease it into a state where it produced this error and then crash and then do the thing. Yeah. 84 00:26:33.480 --> 00:26:35.800 That's not, and it's flakey, 85 00:26:35.800 --> 00:26:37.940 This certainly ever happen one time. Yeah. It doesn't and it doesn't always work. Yeah. 86 00:26:37.940 --> 00:26:54.100 You can easily argue for like, well, let's just cross our fingers and hope it doesn't happen again. And which isn't a satisfying answer compared to here's a 15 line test that sets up the, the fake in the right way and calls the login method and says, yep. Look, even if we get a split during the login, we just retry. 87 00:26:54.100 --> 00:27:29.160 Yep. Done. Yep. Exactly. Exactly. So, so I, I think that there are a lot of benefits in addition to the, I think sort of more immediate benefits of the tests run faster and they're more reliable. There's a lot of like design benefits to saying like, okay, we're gonna decompose our system in this way. We're gonna be able to test it in this way. We're gonna think about it in this way. There are trade offs, like I said before. It means that you need to think a little bit more about the interactions with components when you're changing them. You have to have a sort of a wider view on the system. Right. But I think that those tradeoffs are, are very, are very 88 00:27:29.160 --> 00:28:31.430 Worth it. Yeah, I agree. So there's another sort of aspect to quote integration tests that is something that certainly I've done and I know other people do as well. And that is like, sometimes you don't know how a third party api, a third party server and how it works. Right. And so you want to kind of write Yeah. Yeah. Test in the same way that I write test to test code that I might be interacting with kind of exploratory, even if it's my own code, you know, t d D style. Like, Hey, I wonder what happens if we pass in this string to my code. Mm-Hmm. , the way that I reach to do that is to write a test that does it, even if I, it's just called testFoo and it doesn't, I'm never gonna use that test again. Right. And I'm not even gonna check it in. Like, it's a great way of interacting with your own code to be able to say like, I just wanna make, run this bit of code. But, so it seems natural to reach for testing for the same kind of exploratory testing with a real environment because maybe you don't know what happens under these circumstances. How, how 89 00:28:31.430 --> 00:29:15.500 Yeah. Absolutely. There's, there's a real power in writing tests that you intend not to keep. Right. I, I sometimes refer to this, and you've heard this before as fake make it till you fake it instead of fake it till you make it. Yep. Right. So you, you can write tests against a real system. And this, like, I've definitely done a thing where it's like, all right, I'm gonna get some read only credentials to this database. It might even be a production database cause it's got some data and it is interesting to me, but I've got some read only credentials. I'm confident that they're read only credentials, , and I'm gonna temporarily put those credentials just hardwired into my test because I don't know how this particular API works. I've never used it before. You know, I, I don't know how it works. I don't know how it breaks. I don't know what the, 90 00:29:15.500 --> 00:29:17.740 What does the error message at you look like? You know, we've got 91 00:29:17.740 --> 00:29:58.400 Yeah, exactly. Exactly. And so and I will sit down and I will write tests to sort of explore that knowing that I'm going to either dramatically change these tests before I actually go and check it in, or I'm just gonna delete them. Right. and that can be really useful, especially if you have a setup where your tests run automatically. It almost becomes kind of like a repel where you can just be typing it away in a test and be like, oh, what does this thing do? I'm gonna assert that the return value of this, you know, get user count is null. It should always be an integer. Right? Right. And then I say, oh yeah, it's 105 in the production database. Cool. What if I point it at a table that doesn't exist? Oh. It's null. It should never be null. 92 00:29:58.400 --> 00:30:38.260 Yeah. But in this case it is actually, actually no. Right. Yeah. So that kind of like exploration could be a great way when you're in that stage where we're trying to figure out how to break this thing apart, right? Because again, one of the things that, that you get from this, one of the trade-offs from this this approach is you do actually have to think about the design of this and how do I create these, like easily testable seams that's not always obvious when you're first, you know, using an API or thinking about a problem, right? Like, you have to sort of come up with that. That's your job, right? So having sort of like everything just sort of laid out in front of you in a messy way that doesn't try to do this so that you can see it all and you can interact with it all. 93 00:30:38.260 --> 00:31:29.700 You can understand like how the system works and then say like, okay, I think that if I built a little abstraction around this part, I could basically treat it like let's say for example, a a sequence, right? So I just have a sequence of lines that is coming out of this thing. I don't really know what it is, but I can create an abstraction on top of this API that will, that will basically treat it like that and then I can build my real implementation under that. And all it does is produce a sequence of lines mm-hmm. . And then, you know, lots of things are created producing like a sequence of lines, like a list of lines. So I can use that as my test double and I can start designing, basically doing the design work to tease this apart and say, okay, the code that I have that operates on a, on a sequence of lines, super easy to test, I can pass in whatever I want there. 94 00:31:29.700 --> 00:32:23.120 The API that I now know how it works, I can model as a sequence of lines. And so the wraparound that is extraordinarily simple. And then I can do some additional things where I, you know, kind of like we were saying before, maybe I write an integration test for that real implementation. Maybe what I do is I just get really heavy into mocks and I actually just like mock out the interaction with the API. That's another approach, you know, you can sort of do either way. Maybe what I do is sos something I do a lot is rather than having an integration test, I test for integration. So when my system starts up, I say, I'm gonna take all of my objects that I'm instantiating that represent real world systems that I didn't want to write unit tests for, because they would be slow and unreliable, and I'm going to make sure that they work at the point when the system starts up so that it will fail fast if they don't. 95 00:32:23.120 --> 00:33:04.660 So in the case of the database, I'm gonna make sure that I can connect to the database. I'm gonna make sure that my credentials are good. I'm gonna make sure that the tables exist, I'm gonna make sure that the tables have the scheme as I expect. Yep. And if all of that stuff is not true, when the system starts up, it doesn't really matter what's gonna happen after that. Yeah. Yeah. There's nothing else is gonna work. Nothing will work. So it's just gonna shut down and be like, Hey, I expected there to be this table and it doesn't exist, so sorry, I can't do anything. And I think that that is a much more sort of practical way to get confidence that those external dependencies really work the way that you do. Because you tend to, you tend to build them up in this sort of iterative form where, like at the moment when you're writing the code, you're pretty confident that it works. 96 00:33:04.660 --> 00:33:59.300 You know, you've done your, your make it till you fake it, or you've written a little, you know, command line test driver program to make sure that it works. Or you've, you know, stepped through with the debugger and you've made sure that it works. It's like, all right, I'm pretty confident this works. But the thing that you have no control over is the outside world. Yes. Right. Did the configuration of the database change? Did the schema change? Did other things change? And so you can try to test for those things in an integration test that runs in your ci but that test is gonna be unreliable. And it might be an unreliable in a very unfortunate way, which is if you introduce a bug that causes a problem with the database, and it's critical, you have to fix this right Now, you may wind up in a situation where it's like, oh yeah, we up the database and I need to go change this code in order to fix it, but I can't deploy that code because it depends on a continuous deployment system that actually checks the real database. Right. Which is not working, which 97 00:33:59.300 --> 00:34:00.120 Just not working. Right. And then you're 98 00:34:00.120 --> 00:34:12.840 Like, so you wind up having to be like, yeah, so you wind up being, having to do this thing like in a panic, like turning tests off Yeah. Which is like, okay, everything's on fire. You're panicking, you're turning off all the safeties. 99 00:34:12.840 --> 00:34:14.460 That doesn't seem like a good idea. 100 00:34:14.460 --> 00:34:23.920 Right. That's a situation when you, that's the very moment when you need those tests to give you the confidence that you can just very quickly get a production fix out there. And not have broken anything. 101 00:34:23.920 --> 00:34:26.680 That's an interesting observation. Yeah. I hadn't thought about that. 102 00:34:26.680 --> 00:35:12.620 Yeah. So, so that, that kind of style. Yeah. I think where you're testing your external dependencies at the sort of the last responsible moment, at the moment when you know, okay, this has to work now, it's kind of, okay, if it wasn't working before, but it has to work now. You know, that, that kind of thing can be, I think, I think a better, a better way. And then you can write those tests to be very focused, right? It's not the sort of general like, oh, do databases work at all ? Right. You know, can I, can I execute every possible query? Because my interface here is like, well, this is a general query interface, so I should test all these different things. It's like, well, no, you actually don't care about all those different things. All you care about is does this table exist? Does it have this schema? Does it have these, these types 103 00:35:12.620 --> 00:35:22.480 These are preconditions under which I've performed all my other tests. If I can prove those at runtime, now I can transitively believe that the rest of the code should work. 104 00:35:22.480 --> 00:35:42.720 Exactly. Nice. Exactly. Exactly. So, so I think that's another alternative to sort of the like, okay, I wanna write integration tests just to prove that the database is configured correctly. And it's like, okay, yeah, I can see why you'd want to do that, but rather than writing like, you know, a unit test that does that, maybe try a different approach. Yeah. 105 00:35:42.720 --> 00:35:57.660 Well, this has pretty much covered everything that and more that I had in my list below here. So, I mean, I think we should exhort folks to seriously reconsider any integration testing that they're doing with with these kinds of alternatives. 106 00:35:57.660 --> 00:35:59.180 Yeah. Might be a scam. 107 00:35:59.180 --> 00:36:04.300 Might be a scam. You heard it. Here. Might be a scam. I was gonna say you heard it here first, but No, this is what, how old is that talk? You heard it here first 108 00:36:04.300 --> 00:36:09.100 At least second . Probably like 75th . Yeah. 10 years old at leat... 109 00:36:09.100 --> 00:36:14.820 Oh dear. All right, well that's, that's how much we got our finger on the pulse here. . , 110 00:36:14.820 --> 00:36:15.660 Right? Yeah. 111 00:36:15.660 --> 00:36:20.120 Cool. All right, my friend. All right, I'll yeah. See you the next time. 112 00:36:20.120 --> 00:36:23.120 Good As always.