WEBVTT 1 00:00:16.240 --> 00:00:19.780 Hey Ben. 2 00:00:19.780 --> 00:00:20.920 Hey Matt. 3 00:00:20.920 --> 00:00:48.420 I don't know what we're going to talk about today. I'm feeling kind of like we should record a podcast, but I've had a long day of talking to other people and I'm a bit low energy, so let's talk about something interesting that's going to get me excited. What do you got? Long pause. What are we going to talk about? 4 00:00:48.420 --> 00:00:52.220 Well, we could talk about Make 5 00:00:52.220 --> 00:01:16.660 I Love Make. That's all I was very, very fashionable to make, but it's arcane, it's awkward, it's spiky to use, but it is good at a particular job and that is primary job is to already be installed on any computer you might want to use. 6 00:01:16.660 --> 00:01:21.660 That is the primary feature that Make has, is that you already have it whether you want it or not. 7 00:01:21.660 --> 00:02:48.900 Very, very likely you have it. I don't actually know nowadays if an Ubuntu say machine comes with it, but I would not be surprised to find that it is already on a system. Even if I went for a minimum install, it just sits so base. But yeah, the thing I like about Make is it's sort of self bootstrapping. Once you've got make maybe Curl, oh no, you see now we've got two things, but all right, okay, but with a very minimal set of things make can build, its everything else that you need from it. So maybe you don't like make, maybe like me, you like to use CMake and Ninja and other magical, clever things to build your software, but make can make those things happen for you. Make download the binaries and put them in a bin directory and make sure that they appear. And that is my preferred way of distributing source code for the projects that I'm involved in. One of the things I covet the most is the feedback from new members of my team or folks who are just interested in the code that I've got that they say, Hey, I just get cloned your repo and I typed Make and the whole thing built just fine out of the box. I didn't have to think about anything. I didn't have to install anything. And I'm like, yes, that's how it should be. That's how software should be. 8 00:02:48.900 --> 00:05:32.040 Yeah, absolutely. I love Make, because it can be used as a general purpose Tree of Tasks thing doer, which also can be run in parallel if you like. And with that, and as you say, it's just everywhere. You almost certainly have it whether you know it or not. And so you can get this sort of magical developer experience where you clone repo, you run make, maybe it prints out a little help menu of targets for you which you run, make, and then you just have everything magically installed and it all just works. And it's like, oh, how do I run my test? We write make test. Well how do I start up the system? Well, you run make start or make run or something like that. 9 00:05:32.040 --> 00:07:29.680 Yeah, yeah. Well, whenever you build automation like that, it serves two purposes. One is it just makes people's lives easier, full stop. It also serves as a completely authoritative and correct documentation of how the process works because that's the only way to do it is to run, make test. That's the only way that we have to run our tests. So if you want to know how to run the tests, you can read what that does and you will know for certain and you can version it with your code and change it over time. And you can have pull requests where you discuss it and be like, oh, maybe we should optimize it in this way or maybe we should offer this kind of flexibility for it. And it doesn't become this thing that you just have to know if you work on the project or living in a rotten piece of documentation somewhere that no one's updated in three years or something like that. It is like, oh yeah, how do you run the test? Well, you pull up Ben's Bash history and you find all the commands that he ran to do this, right? 10 00:07:29.680 --> 00:07:51.840 Yeah. Yeah. I found myself, especially in recent years, leaning much more toward projects that offer a sort of single binary download for their tools. A lot of the HashiCorp tools have this, and there are other projects as well where you can just go fetch this thing, unpack it for whatever architecture it is that you're using, and there you go. You've got your single binary that you could run to. 11 00:07:51.840 --> 00:08:23.560 And that factors nicely to make file thing where you're like, okay, if dot bin slash Terraform doesn't exist, then this is how to get dot bin slash terraform curl the HashiCorp endpoint and then pipe it through gunzip and then chmod and put it in dot bin and then we're done. And you're like, there you go. Alright. Now anything that needs Terraform will run that version of Terraform and it came down and then maybe you put the version number in there as well so that you can bump the version and just automatically downloads the next version and everyone's on the same version of Terraform. All that kind of good stuff, 12 00:08:23.560 --> 00:09:47.740 Right? Right. And this all gets back to make, because what is make good at, but it's like make this file exist, right? Well, how do I make that file exist? Well, here's the target that describes how you bring that file into existence at this location that you want it to be. And that can be a dependency of another thing that needs that file. And that's just essentially what you're doing when you're, it's like compilation in this context is just turning curl into bits on your disk instead of compiling a file into bits on your disk. But it's all sort of the same thing in a way. Yes. It's funny. So this is something that I've been struggling with for the last week or so actually because I had an engineer on my team conduct a little experiment with his M2 Mac where right now all their development, like you say, is on Linux. We use Mac as sort of a dumb terminal essentially for a lot of the things that we do, which is a little bit of a shame. They are pretty powerful machines, but that's just how they're set up. And all of our CI runs on Linux, all of the servers that we do run on Linux, everything else. But just out of curiosity, and maybe I shouldn't have asked this question. I had him go and run our tests on his Mac and they ran in half the time and I'm like, well, we're moving to Macs 13 00:09:47.740 --> 00:09:59.700 On a, sorry, a laptop M2, the tests run faster than our big beefy servers that we connect to. Yes, holy smoke 14 00:09:59.700 --> 00:10:05.140 A lot faster. And this project is in Java, 15 00:10:05.140 --> 00:11:50.060 Right yep 16 00:11:50.060 --> 00:12:10.320 Now there's a double cost to it. There may already be a cost because if you went back to it right now, it may have rotted in some small way, but it should be within spitting distance of working. Whereas there's sort of an unbounded amount of difference between the version, as you say, of grep or some command line flag that you're like, there is no equivalent on BSD oh oh, what are we going to do here? 17 00:12:10.320 --> 00:12:19.640 And maybe the answer is you just delete it, you haven't used it in six months, and if you need it again, well the code's not going anywhere. You can go find it, you can dust it off, you can rewrite it. 18 00:12:19.640 --> 00:12:24.620 We've spoken of this before the immutability of the history of source code. It's there to go and find again. 19 00:12:24.620 --> 00:12:25.540 Exactly. Exactly. 20 00:12:25.540 --> 00:13:05.620 Maybe is that an argument for those funny little tasks, leaning even more into the make is purely a thing doer and say the things that it does are very, very, very limited. They're restricted to running things like Python scripts for which we know are maybe more portable. Maybe I'm saying that out loud now, but now I'm not so sure because I remember having this debate with Java and another colleague of ours way back wanted to do a similar thing with Java when we were working on a system around the time actually when we were discussing whether it was C++ or Java, which I think was a long while ago, but 21 00:13:05.620 --> 00:13:06.540 It was, yeah, 22 00:13:06.540 --> 00:13:49.400 Yeah. Anyway, that time, and he was making the same point, I would love to be able to run this on a Mac or just not my desktop machine, and Java gives me that illusion, and I'm like, that's fine, but the day you come to me with a bug that's like, oh, this weird native library that we are using doesn't work at the same way on Mac as it does on Linux is the day I say stop doing that because with the particular project we were talking about, that might have been a real possibility, some esoteric networking thing or some pub sub mechanism that had a native backend that kind of feel. Whereas I think the stuff that you are talking about is more like AWS interaction, which seems like it might be the same across the two and HTTP libraries and things. 23 00:13:49.400 --> 00:13:49.940 Yeah, right 24 00:13:49.940 --> 00:14:27.300 So there's a little bit of, I mean I suppose I took the other side of what you are saying, which is like, well, no, I guess it's not actually, you said if there was a choice between, okay, we can't do this, sorry, we have to do this manually, or we have automation in Linux, you say, well, we just have the Linux. We can't do it on the Mac, but if you can get it to work on the Mac, then you would. No, that makes sense. But yeah, it's that thing of not knowing where the cross platform troubles lie until you run everything. I mean, I know this is going to sound weird, but what if you had tests? 25 00:14:27.300 --> 00:14:57.600 Well, yeah, we started out this episode with a darth of ideas and now I'm looking at the clock going and be like, if I start talking about this, we're going to run out of time. But yeah, so I mean I have definitely done things in the past where I had a lot of shell scripts that I needed to test. I wound up testing them and then what I thought was a fairly novel way, if 26 00:14:57.600 --> 00:15:03.540 I remember the way that you were tested, it might actually not solve the problem that we are seeing here. 27 00:15:03.540 --> 00:15:05.160 No, it wouldn't. Right. 28 00:15:05.160 --> 00:15:11.540 Do you want to talk a little bit about, let's segue briefly into your way of testing shell scripts 29 00:15:11.540 --> 00:15:12.100 Yeah 30 00:15:12.100 --> 00:15:14.940 And then we can talk about why it's a great idea, but it doesn't work 31 00:15:14.940 --> 00:15:24.480 And it doesn't solve this problem, right? Yeah, exactly. So the way that I did this was, and I know that there are testing libraries for Bash. That exists. 32 00:15:24.480 --> 00:15:26.300 Like actual regular testing things that work the way 33 00:15:26.300 --> 00:15:46.420 Yes, if you want to write a yes, like a regular unit style test for Bash, you can absolutely do that. I've tried those things. I find them kind of unrewarding. I don't really see the productivity gains that I get with other approaches to testing when I test bash that way. And maybe it's just that I don't know how to use them effectively, but 34 00:15:46.420 --> 00:15:47.180 That 35 00:15:47.180 --> 00:15:47.800 Has been my experience. 36 00:15:47.800 --> 00:16:04.900 Maybe you shouldn't be writing that complicated stuff in Bash in the first place, which is kind of the argument about the Python thing we just talked about. But let's assume that you've got enough little bits and pieces that are nothing more than grep and tail and a few other bits and pieces, and it's like, well, why would I, this is perfect. Sorry, I keep interrupting you. 37 00:16:04.900 --> 00:16:34.760 No, no, it's good. And so the approach that I took here was to kind of treat bash as a kind of difficult to work with text-based interaction protocol. So I wrote a bunch of Python that generated Bash and I unit tested that Python and I basically treated the Bash like, okay, this is just the protocol that you use to interact with the operating system. 38 00:16:34.760 --> 00:16:41.920 And so your tests were done at the level of the interaction with the operating system through the strings that would've passed to the interpreter. 39 00:16:41.920 --> 00:17:34.060 Exactly, exactly. And so what this wound up being is you'd have a Python program that you could run and it would basically just be like run this Python program and then pipe it into Bash because it just outputs bash. And then you can unit test that Python and you could do fun things like you could just write the output to Shellcheck, and then it'd be kind of almost an integration test of the Python to say, okay, did I miss any unit tests here? Right, is the bash that I'm generating no good. And you could compose them together so you could have a Python function that generated a bash function, and then you could call a bunch of those functions to generate a whole script and then call pipe together different bash functions to do different things. And it was kind of clever and interesting, and it gave me what I wanted in terms of the confidence that I needed 40 00:17:34.060 --> 00:17:53.060 To know that you were generating the right interactions with the operating system, which is perfect for what you were doing then. But what we're really talking about here is trying to develop confidence that the interactions with the operating system are in fact the correct interactions and have the same effect in both operating systems. And that's almost an integration test definitionally, right? 41 00:17:53.060 --> 00:23:12.960 Yes. I mean, and there might even be differences in the behavior that would otherwise be acceptable, and it would be very difficult to know what those were a priori and say, and it wouldn't even necessarily make sense to do that. It's like if you're in a world where it's like, yeah, we just deploy on Linux and develop on Linux, why would I do anything that supported BSD that wouldn't make sense? So you wouldn't design it that way. Probably the real answer to this, and I thought about this too, is don't try to lock yourself in a room with a box of pizza and a laptop and just hack this out and alright, this is going to be the weekend when you guys come in on Monday morning, we're going to be on Macs and right now we're on Linux. The way to do this, especially because the project is in Java, is to try to move more of those operational tasks into the Java world. And then when you have shrunk it down to a level where you're like, okay, it is just make and a couple of other little wrapper things and I can very quickly verify, I can clone, I can do exactly what we talked about a few minutes ago, which is I can clone the project, I can type make and it'll run and if it works, it works. And if it doesn't, well then okay, try again. You missed a spot. 42 00:23:12.960 --> 00:23:16.660 At that time, yes, I know that is not great 43 00:23:16.660 --> 00:23:52.000 Reproducibility is difficult, but I've completely lost the thread of where we were going with that now because I got ranting about Docker as is my way. But I say yeah, you were saying about other people, if you pull down someone else's repository and they either have a Docker file or for me, the worst thing is it says, first of all, pseudo app get and it lists this gigantic list of dependencies. I'm like, I don't want to install this on computer. I already have an incompatible version of this thing and I don't to install your version of it over the top of mine. 44 00:23:52.000 --> 00:26:23.240 Yes, I mean honestly, it's sort of like whenever I see that it's going into a store and they've got, I dunno pants that someone else has tried on, I realize that they're not cleaning them and they just fold 'em and they put 'em back on the shelf and it's like, this is how clothing stores work. But it's just like, I don't want to pollute my environment with whatever random packages you need for your things. 45 00:26:23.240 --> 00:27:29.360 And there's definitely this user space versus root access type considerations there. At some point something's going to have to depend on some library and I hope that the architecture of the system is enough to give you that information. You don't need to have, there's a decision to be made and this is a software design decision where it's like we could do this cool thing, but then every developer that ever works on this project is going to have to have this thing installed as root because there's no other way to do it. I wish I could say that tou should just never do that. That's just unfortunately not true. But you should not weigh those decisions lightly. You should think about what the long-term impacts of all of those things are going to be because just a tax that you're going to have to pay forever, you're never going to unwind that unless you move from Linux to Mac and you're like, ah, maybe we don't need this. 46 00:27:29.360 --> 00:27:45.430 And then you're deleting large events who uses make test three or whatever make log tail. You're like, well, let's got some complicated tail minus F thing in it, but what's also as good at doing that, I can write 10 lines of Java that'll do the same thing and then do cross platform. 47 00:27:45.430 --> 00:27:46.660 Yeah yeah 48 00:27:46.660 --> 00:27:49.240 Write once run anywhere as Java is, 49 00:27:49.240 --> 00:28:36.280 It tries to be. Actually one thing I've never done that might I wind up doing as a result of this is these days I'm pretty sure that it is possible to write a Java file with a shbang at the top and then just run it. And I had never really had the opportunity to do that. Because most of the time if I'm doing something like that, I'm reaching for Bash or something like that. But it would be interesting if some of these bash scripts that I have actually turn into Java, they can use some of the other tools that we have in the project. They could be unit tested in the same way that everyone else can, but you sort of use them as you would use a bash script. They're command line friendly. You can pipe the output. 50 00:28:36.280 --> 00:29:04.960 Frankly, even if you can't do it with a shebang directly as Java, you could very easily make that generic Java shim app that just runs itself as argv zero but within inside that jar in some internal directory or more importantly, does make Java compile. And then does that thing so that you are always making sure you're running the most up-to-date version of that and not being confused. That's an interesting thought. 51 00:29:04.960 --> 00:29:26.820 Yeah. Well, yeah, it's like an option to javac where you can say, I'm going to give you a.java file. I want you to compile it and run it in a single operation. And so you put that in your shebang and now you can just kind of write Java like you would any other interpreted language. Right. 52 00:29:26.820 --> 00:29:27.160 So one more 53 00:29:27.160 --> 00:31:02.720 I've never actually done it, so 54 00:31:02.720 --> 00:31:07.100 Why do you need a hundred megs of memory to tell me what version you are? Come on guys. 55 00:31:07.100 --> 00:31:32.280 I mean of course I'm very much aware why, because it's a much more sophisticated system, all that, but it does still grate a little bit when you're running four copies of it and you're like, what? 150 terabytes of ram? Alright, maybe not that. That's a bit except even for our beefy work machines, 150 terabytes of ram, it might be pushing. 56 00:31:32.280 --> 00:31:33.460 That's a lot of ram. I want a computer with a hundred and fity terabytes of ram. 57 00:31:33.460 --> 00:31:35.660 Yeah, that's be pretty cool. 58 00:31:35.660 --> 00:31:38.600 That sounds really cool. 59 00:31:38.600 --> 00:31:42.240 What's the largest amount of RAM you've ever had in a computer? 60 00:31:42.240 --> 00:31:43.180 A million. 61 00:31:43.180 --> 00:32:16.360 A million. That's the largest number, isn't it? I thought it was the largest number. I don't even know how much ram, I'm just going to run that now. It's like one of those things where I've stopped thinking about it for the most part. I know that some of our production machines have 512 gig, which is mind bogglingly large. That's half a terabyte. That already is like, I don't know if I have that. Oh, I do have a drive downstairs in the basement that has my videos and stuff on it that is bigger than that. But most ram hard discs I have are not that big 62 00:32:16.360 --> 00:32:26.480 At prev Preco. We had a database instance that had 192 gigs in it and I think that's probably the most that I can remember. 63 00:32:26.480 --> 00:32:45.060 I'm now just running on this. Oh, I dunno how to use a Mac. I've got a Mac now. I know we talked about this the other day and the fact that I've finally accepted my fate that the company wants me to have a Mac. All those things. I dunno how to find out how much memory it's got. I've just type free. This is exactly what we're talking about 64 00:32:45.060 --> 00:32:46.940 Right now. Yeah, right. 65 00:32:46.940 --> 00:32:50.200 DF worked for me to say how much hard drive space I've got and then I did free dash h 66 00:32:50.200 --> 00:32:56.450 BSD tools are like the bizarro world of gnu, right? Yeah. Everything is kind of the same but not really the same. 67 00:32:56.450 --> 00:32:57.300 I've just typed memory. 68 00:32:57.300 --> 00:34:47.100 It's like Solaris. Did you ever work on Solaris at all? 69 00:34:47.100 --> 00:34:47.740 What have we learned today? 70 00:34:47.740 --> 00:35:17.560 What have we learned today? Make is a great application developer interface. An adi of you should be able to clone a project and type, make, build or make or whatever it is you do, and then you've kind of got something which any developer in your organization or your friendship group or your listenership who can go, Hey, I know how to build your software. You're like, great. We've learned that Mac and Linux machines are 71 00:35:17.560 --> 00:35:18.820 Different. 72 00:35:18.820 --> 00:35:38.000 Same, same but different deceptively and we've learned that Matt doesn't know how much memory his computer has anymore. What else would our listeners need to know out of this? This is what they come for. The deep, deep insights. 73 00:35:38.000 --> 00:35:47.520 If you're building a tool that you wanted other people to use, please, please, please give me a single binary for it. I'll Love you if you do. 74 00:35:47.520 --> 00:35:49.960 Yeah, that's a good one. 75 00:35:49.960 --> 00:35:52.580 Docker sucks. 76 00:35:52.580 --> 00:35:55.560 Well, you heard it here first, everybody. 77 00:35:55.560 --> 00:35:57.280 Docker. I use Docker every day. 78 00:35:57.280 --> 00:36:22.640 Yeah, it definitely has its place. We've said that before. Bjarne Stroustrup has a quote about programming languages, which I think adequately also applies to a lot of things, including tools like Docker, and that is there are two types of programming language in the world, ones that people complain about and ones that people don't use, and I think that's true. The only reason we complain about Docker so much is actually it's pretty useful. 79 00:36:22.640 --> 00:36:24.920 Yeah, true facts. 80 00:36:24.920 --> 00:36:31.100 Alright, my friend, I think we should call it there, somehow call there too. We've talked for off an hour on absolutely nothing, 81 00:36:31.100 --> 00:36:32.560 Right 82 00:36:32.560 --> 00:36:37.860 Maybe to ourselves because maybe we lost our poor listener along the way. 83 00:36:37.860 --> 00:36:39.880 Probably so statistically likely. 84 00:36:39.880 --> 00:36:44.040 It is. Actually. It's true. Yeah. Alright, my friend. Well, until next time. 85 00:36:44.040 --> 00:36:47.040 Until next time.