WEBVTT 1 00:00:14.340 --> 00:00:19.360 Hey, Ben. 2 00:00:19.360 --> 00:00:20.160 Hi Matt. 3 00:00:20.160 --> 00:00:21.000 How are things? 4 00:00:21.000 --> 00:00:22.560 Good. Good. 5 00:00:22.560 --> 00:01:04.640 I've been thinking a bit about the tooling that we use. Um, and you and I both work in Unix environments predominantly or Linux specifically in my case, you know, Mac OS-y type things. And there's an awful lot of tool craft that we've picked up both of you and I over the years. And with certain people we've worked with in, uh, in various companies have been even better at this. I remember pairing with somebody particularly and learning a whole ton of stuff about it. I figured we should talk a bit about the kinds of things that we can do and have done and use the tooling for in the Unix shell, because I'm always surprised when I, I meet somebody who goes, how did you just do that thing? And I'm like, Oh, it's just shell. 6 00:01:04.640 --> 00:01:05.220 Right? Right. 7 00:01:05.220 --> 00:01:07.140 Let's start there. 8 00:01:07.140 --> 00:01:08.500 I think that's a great idea. 9 00:01:08.500 --> 00:01:19.020 I guess, what is your top X, where X is as many minutes as we can do Unix command line tools. The one that comes immediately comes to mind for me is using sort and uniq. 10 00:01:19.020 --> 00:01:20.720 I was just going to say the same thing, 11 00:01:20.720 --> 00:01:21.360 Get out. 12 00:01:21.360 --> 00:01:30.770 Yeah. It's like the Flathead screwdriver of, of Unix tools, right? It's, it's the thing that you use for everything when you don't know how to use anything else. 13 00:01:30.770 --> 00:02:14.720 Right, I mean, I've got a big list of things and I just want to get a sense of the data. Right? I've got log lines and I'm like, well, how, how often is this thing happening? Or how many, how often are type things of this type happening? So you might Chuck in a grep O E dash O E you know, to say only output the bit that I'm going to tell you in my little regular expression, and then you pipe it through sort, and then you pipe it through uniq dash C usually. And I guess we should explain what all those things are and why, but you end up with like a little list of, Hey, there are exactly 50 instances of this string, 48 of this string, 10 of this, and then three of those. And, you know, that's exactly what I needed to know about this log. So, um, I'm glad that we picked the same one there. That's, uh, that's a, that's a good sign. 14 00:02:14.720 --> 00:02:22.720 Yeah. That's a good tell. Yeah. You sort of like use that to make like a little histogram, I guess it's also kind of like a group by isn't that what that is in a way, 15 00:02:22.720 --> 00:03:07.040 I suppose. Yeah. It is a group by. It's very much like a group by isn't it. Cause you're saying aggregate on this key and then say, how many of them there are, it's like, you know, order by blah, sorry group by blah count star. So in the pipeline I just described the, the grep thing will takes an input and it'll find just the, a snippet that you're interested in. Now, if your files are, you just want to find the unique types counts of the lines of a file, the different contents of lines you don't need the grep part, but then you sort them to get them so that the, exactly the same lines are one after each other. Uh, so if you're, if your input is ABCA line each line, then you would end up with A,A, B, C, right. 16 00:03:07.040 --> 00:03:10.520 You're really only doing that to get the unique dash c to work. 17 00:03:10.520 --> 00:03:58.660 That's right. Yeah. That's a very good way of putting it. Yes. Yeah. I kind of wanted to it cause it's not like you want it sorted, particularly, in fact, you may resort it the end, which is what commonly one might do. So you're getting it so that the unique, which is the unique is designed to drop a subsequent or rather what's the word I'm looking for? Not subsequent, um, drop equivalent, repeated lines of a file. That's what it's designed for, but it can also say how many duplicates it encountered as it, as it went through. So when you're piping AABC through it, it sees the first A goes, that's great. That's I've only ever seen A before. Then it sees the same line again. It goes, Oh, well, I'm not going to output that line, but I'm going to count it. And then it sees B and goes, Oh, okay, that's the end of the A's I will never see another A because it presumes that the inputs are, uh, or this is what its function is. 18 00:03:58.660 --> 00:04:35.900 I shouldn't say it's presuming anything is not presuming anything. It's just what he does. So then predict, Hey, I saw two A's then it'll say I print one B and then once C and you're done. And that's wonderful. Right? And so you get a count of each of the individual inputs to a unique lines, I should say, not now. Of course, the result of that is a big list of like, Hey, I saw A once or, sorry, I saw A twice. I saw B once and I saw C once. And that's great. And then in the example, I just made up on the spot. They are in a useful order potentially, right? Cause it's like, Hey, I want the most common, but often I want to say no, just give me the top 10 of those. And so just like in SQL 19 00:04:35.900 --> 00:04:38.890 Thousands of categories, but you want to know what the most frequent one is, 20 00:04:38.890 --> 00:05:23.440 Right? Exactly. Like in SQL, if you don't give it a sort key afterwards, then the sort inside your pipeline, hip is not sorting it. Usefully. It's just an implementation detail of the way your, your group bys is working. But then you want to sort by dash a N to say, sorry, sot dash N which means sort numerically, which means it's going to interpret the first part as a number. And then you can pipe it through say less. And what we've done then is, or head minus 10 to say, just show me the first 10 of those. And so in, in what we've got grep sort uniq, another sort less. Five Unix commands one after another. And we've written a SQL query for all intents purposes on a line based text file. 21 00:05:23.440 --> 00:05:37.720 Right. So when you're like, Hey, uh, I noticed you got some user login rejections does that, how often does that happen? It's like, Oh, we got a hundred thousand of those today. Oh, wow. Is that a lot? Or does that happen every day? Then you need to go use these tools to sort of figure out if that's true. 22 00:05:37.720 --> 00:05:53.740 Right. Of course it's no substitute to actually having like metrics in your application that did it. But like, we all have been up against the gun trying to make these kinds of like what the heck's going on with my system. And we are joined by guest guest puppy in the background. I apologize for the noise. 23 00:05:53.740 --> 00:05:55.860 He's always welcome. He's adorable. 24 00:05:55.860 --> 00:06:01.320 Yeah. Gets away with it. Right. But yes, you were making a point about something technical. 25 00:06:01.320 --> 00:06:46.080 Yeah. Well, yeah, we were talking, we were like, like you said, you know, this, these are the things that you do. Like we talked about structured logging and observability in another episode. And like, if, you know, if you have some inkling ahead of time, are there things that you, although, you know, I can make some arguments for structured logging that would say like anything that you write to a unstructured log file, you can write to a structured log and it will be strictly better, but not every system has this kind of logging in place. Sometimes, you know, you start out with something and you get, you don't necessarily know what you want your structure to be or whatever it might be. So you have these things and being able to do this, it's just super useful. Plus like the logs that you're reading aren't necessarily yours, right. Sometimes they're operating system logs or other, other software's logs that you're doing this to 26 00:06:46.080 --> 00:06:49.720 It's Apache's log file that I just happened to be on the box. Right. Yeah. 27 00:06:49.720 --> 00:07:07.160 Uh huh. Right. So you, you need to have these, I feel like you need to have these skills no matter what. And of course the next one that you want with that chain of that pipeline that you're talking about is awk. Right. Cause it's like, Oh, the thing that I want is this particular column, right. And awk is like a whole programming language, 28 00:07:07.160 --> 00:07:18.240 But, but mostly it's useful for print dollar 3. Yes. Yep. I will make an argument for another tool. When you, you talk about awk first. 29 00:07:18.240 --> 00:07:23.560 I will. I was going to say there are other tools and I think I, I'm going to say I was going to say the one that I use for this is also cut. 30 00:07:23.560 --> 00:07:57.500 Cut that. Yeah. Yeah. That's where I was going cut is a little bit easier than specifying a dollar three. If you have a very clear non-space based delimiter. So for example, if you know that your, you know, your JVM dump or whatever has, Oh, it's between the second colon and then the third colon in some list of like class path or whatever, then you can like cut dash D colon, which means you use colons as the delimiter. And then you can pick three as the third thing. And that's, that's another super way of just filtering the bit of the data that you want, but awk is a full programming language. 31 00:07:57.500 --> 00:08:09.020 Yeah. Yeah. And you can, and I honestly feel bad for not having used awk for more than just print dollar, whatever. There's a few things I think I've used it for, but I always have to Google those things. 32 00:08:09.020 --> 00:08:37.820 I use it to sum up a numeric column. That's another thing that I can do. And also average, it's quite easy to write, although again, yeah, there's a little bit of stack overflow used here, but if there's something where it's columnar and I just need to go over something with a relatively straightforward thing, I can just use awk out the gate to do that kind of thing or running total or stuff like that is again, it's relatively straightforward, but that's the only bits I can remember is like begin something magical and something magical. And then the line itself. 33 00:08:37.820 --> 00:09:21.860 Of course, if you do have the benefit of a structured log, you may have written your structured log using JSON. Because a lot of people do that in which case you're probably going to want to use another command line tool called JQ, which is an amazing tool, not just for, uh, dealing with structured logs, but dealing with any sort of JSON data. If you're interacting with a web service. One of my favorite tricks to do is to take, uh, you know, I've got some web service I'm trying to explore the API and I've got the documentation there maybe. And I want to see if the documentation's right or I want to see what the actual underlying data is. And so what I'll do is I'll, I'll put together a little bash that, uh, curls that, that API and then, uh, pipes it into JQ. 34 00:09:21.860 --> 00:10:11.640 And then I'll run that whole thing in another command called watch, which is another something that we've talked about where it'll just run it every couple of seconds. Right. And now I have like a constantly refreshing view of what that, what that JSON object looks like. And I can start modifying the JQ expression to select into particular elements and explore the whole tree of the resulting JSON object, uh, in a very interactive and fast way. Right? So you sort of can walk over the whole tree and like, Ooh, this value is interesting. And this whole array of things is cool and you know, Oh, we're going to need that value. And this doesn't match up with the documentation. Um, and you know, and just a matter of few minutes, you can sort of see everything that there is to see about an API in a very interactive way. So those, those tools together, I am a huge fan of, but JQ in general is pretty fantastic. 35 00:10:11.640 --> 00:10:44.460 JQ is amazing. I think JQ is the first of the commands that we've mentioned so far there, isn't sort of a, like a BSD staple, right? Everything else is part of the Unix environment. Yeah. Uniq, sort, uh, awk. I think we've said cut. They're all like, you're almost certainly got them on your machine already. Just, just type it in, whatever you don't use, a sudo apt install sort of thing. JQ is is, is a, a separate process. I think you can probably get it from most. Um, I think I installed it on my, my Debian thing here, but it's a single static binary. You just grab it and chuck it in your bin directory. 36 00:10:44.460 --> 00:10:54.870 The fantastic thing about it, right? Like you said, um, I think I would be a little surprised if on most newer Linux distributions, you didn't get JQ out of the box, but I could be wrong. 37 00:10:54.870 --> 00:11:07.660 I'm pretty sure it needs to be installed. It's certainly. It's like one of those things that is in one of my Docker container things for competitor explorer, or like just, Hey, this is the, I expect these tools to be in my machine. It's like that it has to be there. 38 00:11:07.660 --> 00:11:23.740 Yeah. But yeah, that's it, that's I use that all the time. You can do math in it. You can do transforms in it. You can do all kinds of crazy stuff in it too. So it's sort of like, you know, if you get into sort of heavy scripting and not, not so much just exploring, you can damn near write programs in it, if you try 39 00:11:23.740 --> 00:12:43.400 Geeking out a little bit about it, it's just a really nicely written piece of software as well. It's, uh, uh, it's based on something called libjq, which is obviously comes, I say based on it, extracted from it is libjq. So if you want to do JSON parsin with like a language that, that gives you the kind of descriptive power that the JQ query language has. It compiles to an intermediate byte code, it's just, it's just sweet. It's nicely done. It's it's uh, it's, it's cool. And it's worth saying as well, that like, while the vast majority of invocations of JQ, R JQ dash capital C for me, which says enable the color, even though, uh, even though I'm about to do something that would make you want to turn it off space dot, which means select everything, Hey, I just want everything. And then I'm going to pipe it into less capital R less dash capital R, which means, Hey, less interpret, but don't try and strip out the ANSI color codes that are coming your way, because normally you're going to freak out about that. And that means I get like a pagable colored version of the, you know, syntax highlighted and pretty printed version of whatever I'm piping into JQ. So you don't have to use it to even do anything, um, like any kind of interrogation at all. You just say like, Hey, it's just a really nice, pretty printer that has supports color. Um, but then as you say, you can do dot stuff. It's, he's got his own piping internally. You can do all sorts of clever, clever trickery. It's a great, it's a great product. And yeah. So what else we got? 40 00:12:43.400 --> 00:13:39.200 On the topic of, of watch, actually another tool that's similar to that is, is this is another one that you're probably going to have to install called ENTR, uh ENTR basically uses the file system notification library that is definitely already, uh, in, in your distro, uh, to let you run a command in response to a file system change, right? So if you want to scan a directory for changes, and then whenever a file in that directory changes run, make or run JQ or run, you know, whatever your compiler of choice is or your linter or your tests or whatever it may be, you can use ENTR to do that. And it's like a really, really easy way to create a very interactive workflow with pretty much any programming language using this very simple tool. Um, you can also do it for more like things you shouldn't do. 41 00:13:39.200 --> 00:14:09.380 You can use it as a way to, you know, like, Oh, I'm going to hit this. I'm going to make this API call, or I'm going to hit this web end point whenever this file changes and post the file as well, because you don't want to actually build the event based system that you should be building. Right. You know, uh, I've seen it, I've seen it used for that, which, you know, maybe in a MacGyver duct tape baling wire situation is the right thing to do. Um, but you know, it's, it's intent. And I think it's, it's really sweet spot is, um, you know, sort of like developer tools. 42 00:14:09.380 --> 00:14:38.300 Little mini, I mean, I'm used to like NPM run watch and things like that. The various packages we'll have in there, but they're provided by Node.js services, but if you want to do it more generally, you can use the ENTR. I've seen it used on, I notify stuff that I've kind of hacked together myself as my raspberry pi development stuff. I had a thing that watched, but it was a make file target. If I don't know that there was the ENTR there, just do most of the horrible, heavy lifting of the strange protocol that you have to use to talk inotify, then this would have been great. So yeah, the ENTR that's awesome. 43 00:14:38.300 --> 00:15:07.080 The last time I really used it in anger, I was writing a Wireshark plugin. And what I would do is, is I think I was writing it in Lua. And what I would do is whenever I, I was hacking away on my Wireshark plugin. And whenever I changed the Wireshark plugin, I would have ENTR automatically run a capture that I had through Tshark and spit out the resulting, uh, processed result to get a sense of whether or not I was writing my Wireshark plugin correctly. 44 00:15:07.080 --> 00:15:21.900 Awesome. That's so cunning because yeah, I mean that you, a CI/CD style or local CI/CD, well not CD but for wireshark, which is like the last thing I would ever expect to actually have that process. That's cool. Yeah. 45 00:15:21.900 --> 00:15:37.500 It was a miracle when they added the thing that allowed you to like reload a Lua plugin in wireshark without having to quit it and start it again, which has been my go-to way of doing this kind of development and then, you know, making a pop-up happen with, Oh, I got to this point. Printf style. 46 00:15:37.500 --> 00:16:01.720 Yeah. That was a few years ago that worked out pretty well. But yeah, actually, so that leads us to another one, which of course is tshark tcpdump depending on your, your flavor there, right? Like TCP dump is usually my go-to for capture. Yeah. Normally when I'm like analyzing something, I tend to reach more for Wireshark than tshark, but I've definitely seen people with great effect use tshark both capture and analysis. 47 00:16:01.720 --> 00:16:16.760 Right, yeah. And I mean, I think this is another thing where once you've seen somebody who is good at doing this kind of thing in action debugging, a problem that you would have scratched your head for days on and finding it in a few minutes with, with, uh, a packet capture. 48 00:16:16.760 --> 00:16:21.000 We should probably tell people I was going to say, we should tell people what wireshark is. Cause I think we've been talking about it. 49 00:16:21.000 --> 00:16:23.540 Yeah, let's do that. Yeah. What's wireshark, Ben? 50 00:16:23.540 --> 00:16:35.360 What's a wireshark? Yeah. So, um, when you communicate over the network or actually, I mean, it can be, you can use Wireshark on like USB devices and stuff like that. 51 00:16:35.360 --> 00:16:36.460 Bluetooth and stuff. Yeah. 52 00:16:36.460 --> 00:17:02.260 So if you're, if you're doing any sort of communication protocol and in general, it's probably worth asking the question. Can I see this in Wireshark? Why would you want to see it wireshark? Well, because Wireshark will show you all of the bytes, everything that you're sending back and forth between computer a and computer B or device a device B, and allow you to apply filters to them, to sort of shrink them down to the stuff that you care about, transform the raw bites into something that's a little bit more meaningful and readable. 53 00:17:02.260 --> 00:17:13.600 I was going to say, the raw bytes is a bit of underselling of wireshark, wireshark understands almost every protocol known to mankind. It's kind of like the C3PO of things and show you what it means almost always. 54 00:17:13.600 --> 00:18:03.480 Yeah. Yeah. No, that's a great analogy of it. Yeah. That's all the, all of the, and you know, those were the thing I was saying was I was writing earlier was, uh, um, um, or Wireshark plugin, like being able to see like at a, you know, at the various OSI levels or whatever it might be for your particular thing, like what are the messages that are going back and forth here? It's an incredibly powerful tool. And it adds a level of observability to anytime, anywhere that you're basically connecting two devices or two computers together, uh, or multiple computers together. So it's, it's something that I've used. And I know you've used a whole ton to troubleshoot all kinds of problems. And if you're not familiar with it, I highly suggest you give it a try because I know like, what's your, what's your, uh, do you have, do you have a cool Wireshark story? Like we never would have found this, but for Wireshark kind of a thing? 55 00:18:03.480 --> 00:18:33.340 I do. I don't know that I can talk about it publicly, unfortunately, but, um, yeah, very unusual behaviors in esoteric networking devices before now, the have, uh, have been traced back to either like hardware issues or, or similar, on that subject actually, and in a similar vein. And I'd realize we're doing like all the two minutes on each of these tools, we can easily do a whole episode on, 56 00:18:33.340 --> 00:18:36.240 Oh, this is like a lightning talk. Yeah, that's true. Yeah. 57 00:18:36.240 --> 00:18:59.360 Right. But in a similar vein to Wireshark, TCP dump and things like that, system tap and strace or strace is probably the one people are most familiar with this. This is the, uh, Snoop on a process, just like you were snooping on a network connection between processes in Wireshark strace will say, Hey, I can run another process or I can attach to another process and say, what all operating system calls are you doing? 58 00:18:59.360 --> 00:19:40.720 And I want to look at the parameters that come in and the parameters the operating system gives you back. And that can give you an, an awful read on an awfully deep read or awful awful assessment, really deep read on what a process is doing. And that's super, super useful. When you have, for example, a process that you don't understand why it's in a weird state, you can attach the strace to it and go, Oh, it's, it's in a, it's waiting on a, an event. What is event? Oh, it's file descriptor 37. What is file descriptor 37? And then you can go and look, and this is something I noted for another part of the, the talk. In Linux, you can go and look at that processes information in slash proc. So I would say, why are you blocked? Wait, you're trying to read from 39, what is file descriptor 39? 59 00:19:40.720 --> 00:19:44.420 So I will go to proc slash proc slash and then the pid of the process. 60 00:19:44.420 --> 00:19:46.520 And that's just mounted in the file system, slash proc, right? 61 00:19:46.520 --> 00:20:23.440 It's just a magical, file system. Exactly. And within that file system is a bunch of useful information. And then there was one directory per process. You can go into that directory and there's a bunch of files that aren't really files. They're magic that talk to the kernel. And then you can like, look at, for example, all of the open file handles. All of the open file handles are appear as symlinks between a numbered file, like 37 in the case of the thing I've just been talking about. And it will be symlink to either the actual file on disk, or it will be symlink to a special magic looking thing that will tell you I'm a socket, or I'm a, I'm a, a unix pipe, that kind of stuff. 62 00:20:23.440 --> 00:21:00.860 But you know, maybe you won't know what that is at that point. Maybe you'll have to give up at that point, but it gives you, Hey, I'm blocked on the network at some level, and then you might crack out Wireshark and go, well, what are you blocked waiting for? Right. And can I see anything before this point? Um, but that it works only, you know, works on your own software. If you happen to, if that can give you a hint as to, well, I think the only places where it could be blocked on reading from a file is here and here. Okay. That's where we're wedged. Um, but more importantly, it works on other people's software. So if you're stuck with why on earth, does this esoteric binary that what a vendor has given me, what on earth is it doing here? strace is a fine way to find out. 63 00:21:00.860 --> 00:21:02.040 So what's system tap then? 64 00:21:02.040 --> 00:22:03.920 System tap. Yeah. So system tap is like, um, strace plus plus. System tap allows you to write small programs that get injected into the kernel and run effectively in a sanitized, safe, uh, environment within the kernel on behalf of various other parts of the operating system. And so you can kind of trap and filter, operating system calls, various kernel events that can happen that are a layer below, even what like strace can see. So, you know, Hey, I'm, uh, I had to allocate a new 4k page of Ram. And so that's an event that happens in the kernel. It's like, Oh, well, that's interesting to me. I want you to run this bit of code and do something when that happens. It has a bunch of, um, useful, uh, scripts that you can crib from to write from. But the story that ends up with the punchline and system trace, uh, sorry, system tap, found the issue was, um, a latency spike in a trading system I was working on and the latency spike. 65 00:22:03.920 --> 00:22:46.580 We traced back to, uh, exactly what I just described like that we, we, there was a counter that went up, which was like, Hey, the number of filesystem, uh, sorry, the number of page faults has gone up. So if you access an area of memory, you haven't accessed before. It's a page fault. The operating system has to decide what to do very often. It says, Oh, is part of your heap that I just didn't give you the actual physical memory for yet. So I'm just going to find a spare 4k page that was free before, swap it in there. And then you can go on your merry way and you can continue with your life. And that's great, right? And that, that allows you to say, allocate me 10 gig of Ram. And you don't actually get 10 gig of Ram instantly the operating system just says, here's a space 66 00:22:46.580 --> 00:23:26.520 that's 10 gig wide. Every time you look at a bit inside of there, I'm going to kind of pop in some 4k pages for you. And you don't, you can't tell the difference, but it takes a little bit longer to access it. The first time. There's also a major page fault, which is like what people think of when you think about virtual memory, which is like swapping to disk. So this is like, Hey, I ran out of memory. And this was a page that I was talking to before I've been reading and writing from it. Maybe it contains my executable itself. And the operating system says, Hey, I'm a bit stuffed, stuck from memory right now. I'm gonna write this out to desk, or I'm going to throw it away, knowing that I can load it back again from disk. And then when you hit that page, it goes, Oh, Oh, Oh no. 67 00:23:26.520 --> 00:24:15.660 Uh, right now I need to find this for you. And it's much longer obviously to actually go and get it from disk than it is to just find a bit of physical memory and say, Oh, that's yours now. Right? Background set. We were having issues where we were losing packets. We're dropping packets under a very high load. And, um, it, long story turned out to be something that we had presumed was pre faulting. That is we'd specifically asked the vendor code to touch every 4k page in the block of Ram. We'd give them specifically. So that, that faulting, that, that, um, minor page had happened for every single block. Now it turns out there are better ways of doing it than that, but that's how the vendor implemented it. Um, we'd asked for this flag to be set on like a two gig buffer of Ram that we knew was really important to us. 68 00:24:15.660 --> 00:25:00.780 Nobody else should touch, but unbeknownst to us that wasn't happening. And so every time the process went to access a new area of this two gig, uh, memory, the first time it had to do a minor page fault, which again is really, really, really fast these days, but it requires taking out a lock, a process level lock because you're about to monkey with the page table and move things around and map memory around. And so it was blocking on that lock. We discovered we, we, we system tap was like, no, every time we get here, um, this is what the call stack looks like. And we were able to look up the call stack. I can go, Oh my gosh, this is the actual kernel area, the kernel code that's being called. And it's trying to take out this lock and it sat there. That's where it is when we're spending all of this time. 69 00:25:00.780 --> 00:25:01.460 Interesting. 70 00:25:01.460 --> 00:25:43.140 I've just taken 10 minutes to talk about, tell a war story about this, but system tap gave us the facility. Right? Right. Um, as it happened, the, the vendor that we were working with, um, uh, opened, had open source the source, which was amazing. And it was really, really valuable to us. And I was able to find the part where you sort of set the flag and said, Hey, please, can you fault this stuff in? And they'd written the code, which essentially said for I, in number of pages, uh, and then literally the C code of parens char star, the memory address, you know, int temp equals that, right? So that is read a byte from that memory and put it into a temporary register, a temporary variable, I should say, 71 00:25:43.140 --> 00:25:44.320 And then it got optimized out? 72 00:25:44.320 --> 00:25:47.780 Of course it, got optimized out the components. Like you're not doing anything with that go away. 73 00:25:47.780 --> 00:26:29.080 And so it got optimized out. And this was one of the first times, this is a long, long time ago, but the compiler Explorer is up and around. It was the first time that I remember sending through other sheepishly, uh, a patch to them. And as a compiler Explorer link that showed that their code on a modern compiler. This was obviously written for like GCC 4, which didn't do it, um, you got optimized away. So anyway, the happy ending was we were able to fix that and move on with, but system tap was what allowed us to find it system tap can also do stuff like how often are you being descheduled, that's another really good, good, sweet spot for it. If you're like, Hey, I'm running my process. And I, um, I think I'm using the CPU all the time, but every now and then something happens and it takes longer. 74 00:26:29.080 --> 00:27:37.620 You can say, well, okay, what's happening on this? Oh, it's a sibling CPU is sending you a TLB shoot down, which it sounds like a really complicated sequence of words. And it is, but it's like one of these weird things that can happen between nodes in the system, which is totally unobservable, otherwise, right. There are some counters you could look at in proc interrupt or whatever, but if you want to know, no, you actually got de scheduled because it's really important thing inside the kernel had to run. But my stuff's more important than you. Anyway, system tap brilliant, very difficult to kind of get on with. Unfortunately it's not as cool as, as like the uniq and sort world of things, but it is, it is a useful thing to have in your arsenal. I know that there is dtrace on other systems, and I think there's a port of dtrace to Linux and there's some stuff using the turning it back circle. So the BSD packet filtering API is another kernel compiled sort of safe, um, system, which you can use to specify packet, capture filters. Like, Hey, I want to see packets like this and it gets compiled and run in kernel so that you aren't spending time going in and out of kernel space. This is when you, when you're doing your, uh, you know, TCP dump or your yeah. 75 00:27:37.620 --> 00:27:41.780 Is that separate from libpcap or is that what libpcap is? 76 00:27:41.780 --> 00:28:29.440 Uh, it's separate from the, I dunno if libpcap uses it under the hood. I think I'm not actually, I'm not a hundred percent certain on that, but I know, I know that there's a Barclay packet capture, filter syntax thing is, is a thing. And it's slightly, it's more restrictive than for example, what you can type into Wireshark if you've ever seen the difference between the two. So yeah, maybe it is the same as what pcap does anyway, that that's, there's an E BPF, which is the extended Barclay packet filter that can do more than packet pack filtering. It can do essentially what a system tap can do as far as I understand, but I haven't looked at it for a while. So I'm sure listeners are a home listener. Our listener is grinding their teeth kind of going, no, that's not how it works at all. In which case I invite you to email us or tweet at us and tell us where we're going wrong with that. But, but yeah. Okay. I'm going to get off my little soap box over the exciting systems level tools that I play with, 77 00:28:29.440 --> 00:29:21.620 Ah, system tap. I mean, it's, so there's a whole other category actually, uh, of tools. And this is definitely higher level than system tap, but, but maybe equally useful, certainly more commonly useful, which is all the process management stuff. So like ps, pstree, kill, top, all that stuff that it's like, okay, I have this, this machine it's running all of these. Why is my computer so damn slow? And what can I do about it? Right? Like you, you, you bring up some, you know, you launch some tool or you bring up some webpage or whatever, and it's like, ah, everything's super slow now what's going on? Well, it's like, you know, first things first, you probably run top and see, okay, what is using all the CPU? What is using all the memory? If you're not coming from a Linux environment, this is like, you know, task manager or, um, what's the Mac one. I forget is it also task manager? 78 00:29:21.620 --> 00:29:25.340 I have literally no idea, you know, my position on Macintosh. Yes 79 00:29:25.340 --> 00:29:46.240 I do. Um, but yeah, you know, it's, so if you want to see everything that's running and how much memory it's using and how much CPU it's using, how much it's virtual memory it's using, uh, what it's command line arguments are even, and the controls to send it signals to kill it or stop it or whatever you might need top can do that. There's another variant of this, which I sometimes use called H top. I don't know if you're an htop fan? 80 00:29:46.240 --> 00:30:01.620 Yeah. I mean, it's a bit new fangled for an old fart like me. It's got colors and bars and CPU things. I don't understand it. It wants to use the F keys. What are the f keys are my domain! Leave them alone! No, but no, htop it is a fine tool too. 81 00:30:01.620 --> 00:31:04.420 Yeah. Yeah. But that's that, you know, sort of what the hell is going on with this computer? Why is it so slow? Why, uh, why, why is this stuff not working? Um, and then once you have all that, you might want to dive a little bit deeper into it and you can do it one of two ways you can just run top. And I honestly, half the time I feel like I just run top, I see what's wrong. And then I, and then I jump out of top and I go use something like pstree, maybe a ps with a grep or something like that. Um, to, to dive deeper, you can do a lot of that stuff in top and also htop, right? Like all those things are there to like, I want to see the tree of processes and their threads, so I can see like, you know, which processes now spawn like a bajillionty threads and what the hell are those threads doing? And why are you screwing up my machine, all these threads. Um, and you know, you can do it all like that. I honestly, I find myself oftentimes in situations where if I'm deploying something, not using Docker, which is my preferred way to do this stuff. 82 00:31:04.420 --> 00:31:07.960 Are you trying to, are you trying to enrage me here, right? Yeah. Carry on. Yes. Yes. 83 00:31:07.960 --> 00:31:10.140 That's a whole, we're going to do a whole episode on that. 84 00:31:10.140 --> 00:31:14.880 You you're teasing me a whole episode on, on the appropriate use of Docker. 85 00:31:14.880 --> 00:32:14.360 Yeah. If I'm yes. The appropriate use of Docker very, well put, but if I'm deploying something not using Docker, there's a trade-off there. And one of the trade offs is that I have to generally do the process management myself. So if I want to turn the damn thing off and be very sure that it's off, I need to make sure that all of those processes have actually stopped. And that means the whole tree of processes, not just the top level bash script that kicked it all off. Right? Yep. And in those scenarios, I generally find myself using pstree to look at the tree of processes and look at all of their PIDs and look at what process groups they're in. And combining that with a kill that like kills a whole group of processes or kills a single process. And it's like, all right, I'm gonna kill this top-level process. Does it correctly kill the ones below it? Well, I'm going to find out by killing it. Right. Or I'm just going to send another signal to it. That's another great thing about kill is that you don't have to just use it for kill. You can use it for any signal. Um, you can use it for hup. You can use it for, uh, the user signals. Yes. 86 00:32:14.360 --> 00:32:29.880 Let's just take, the hup is hanging up. It's like the equivalent of disconnecting, this the signal you used to be given when like the modem was disconnected from the serial connection, but we now use it to mean, Hey, gracefully, shut down, please. Probably I'd like you to whatever you'd like. Yeah, 87 00:32:29.880 --> 00:33:31.560 You can kind of, it's like the definition of it has gotten, it's almost like another user signal. It's like user three at this point. Right. I I've seen it used for, um, the one place where I've seen it used it sorta kind of made philosophical sense to me is check to see if your connections are stale, right? Like, like you, you hup a process when you, you know, your computer has been woken back up from sleep. And who knows if the TCP connections that previously had still connected? Um, yeah. You can hup a process and be like, Hey, and if it's, if it's written to handle that signal, it might go interrogate all of its sockets and make sure that they really are connected, you know, send a heartbeat or do some other things to make sure that everything is still good. Um, but yeah, if you, if you're doing things with signal handling and you want user2 to mean something, you can use, kill to send user2, to your process, just make sure you get the command arguments right. And don't accidentally try to send, uh, terminate to both your PID the mytical pid. 88 00:33:31.560 --> 00:33:50.680 Segv is a fun one. I like, I like confusing people by sending, you know, Hey, your thing crashed. And I'm like, Oh yeah. Well, especially if you got automated reporting of crashes, which I've done before now, you just kill minus segv. That'll take you a while to work out that one. How do we die here? All right. Enough. I shouldn't be giving people ideas. 89 00:33:50.680 --> 00:33:52.900 You're giving people all your practical jokes. 90 00:33:52.900 --> 00:34:01.420 Sigbus Other funny things. Yeah. This is too early for the April. Maybe it isn't, it's too late. Probably for the April. Fool's one. 91 00:34:01.420 --> 00:34:05.260 Yeah. Oh man. Next year. Um, 92 00:34:05.260 --> 00:34:21.520 Yes, it was a ton of stuff you can do with yeah. PS and top. And we've already talked about, I mean, both of those use both, um, command, uh, operating system, um, API calls and they actually look inside slash proc most of the time, as far as I can understand it, to understand what's going on. 93 00:34:21.520 --> 00:34:29.440 So we should talk about something that's bad. We've been talking about all this good stuff. I want to talk about something that is, I drives me crazy. 94 00:34:29.440 --> 00:35:35.720 All right. Well, let's get to that in a second. I want to just finish off on a couple of things. Cause I'm looking at the hastily scrawled notes that I did while I was in a meeting before this, before this recording. And, um, you mentioned awk, which we already said is a full programming language. There are a couple of other awk, like, which is a strange thing. So sed is a great thing for just doing relatively straightforward stream editing that is text replacement or with some minimal state. I mean, I typically use it just to literally change like one thing in a line of, uh, files, right? One line of a file. And then perl, of course, which is a full programming language as everyone's heard, but like Perl to me is spelled perl dash pi space, dash E split space quote, and then a small replacement string. And that is the Pearl in place, re run this on a bunch of files in place, replace them with the result of having run them through the Perl script, what I am about to type in. And so that is what I use to do big refactors where my automated tools give up on me. I will do find dash name. Oh, we didn't talk about find, Oh no, 95 00:35:35.720 --> 00:35:40.640 Yeah. We know we're going to miss some, we're going to publish this and someone's gonna be like, you didn't talk about this. And we're like, Oh my God. 96 00:35:40.640 --> 00:36:17.720 Uh, you know, instead of cat you know and gzip and gunzip. Yeah. But um, you find a bunch of files in there. Like here, all the CPP files here are the header files. I'm now going to replace this string and I'm going to use perl dash pi dash e to replace them in place. And then I can rely on git, ha ha, that's a whole other great tool, um, to, to tell me, did I do this right? Or not, by like me doing a git diff and seeing what did I change? Okay. That looks great. Commit that run my tests, obviously. So those are the other things I wanted to talk about in this thing, before we move on to the, what we've done, the good, now it's time to do the bad, bad. I have to think of what is, what constitutes the ugly to finish the finish. Boring. You gotta have the 97 00:36:17.720 --> 00:36:38.920 Sergio, uh, Sergio Leone. Isn't that who, I don't know the spaghetti westerns. Well, yeah. So the precursor to this though, I think, I think is talking a little bit about like bash and shell check and the sort of, you know, some of the bash flags, because it is in doing that, that you run into the problem that we're about to talk about and it drives you insane, 98 00:36:38.920 --> 00:37:37.900 Can I talk about, uh, a good thing in bash first of all bash. And in fact, anything that uses gnu read line, gnu readline is like the text input, any almost anything you type into in a unix command line interactively is using readline. And so that's when you, you know, you can press up and down and go through your history and, you know, control backwards and forwards to move between words and that kind of stuff. And this is the thing that, that I tell people, they're like, what? I didn't know, you could do that. And then it's life altering in some cases. And that is all period, press all period, to toggle through the last command line arguments that you specified. So this, I know, it sounds like, what would you need that for? But like you do imagine you're doing LS or cat a file, right? You catting the file to just, or lessing the file just to see whether or not this is the file that I want to delete. Right. I'm checking the contents of that. Yeah. I can definitely delete this. And then you're going to do RM space and then the same file. And you better be sure you type the same thing. And again, because you're going to delete the wrong file, otherwise, 99 00:37:37.900 --> 00:37:44.280 Because the file is some, gnarly GUID with, uh, you know, uh, some other identifier stuck on the end of it. Yeah, exactly. 100 00:37:44.280 --> 00:38:35.740 Exactly. Now one way to do this of course, is to go up arrow and then replace the word less with RM. And you can do little of these tricks with carrot, carrot, less carrot, RM or whatever. But I like to look at the thing first, stare, stare it in the eye and say, this is definitely what I meant to do. So alt period will do. We'll bring that second, like the command line argument to the previous function to call into, uh, under the carrot. And then you can keep pressing alt period and it goes through all the other ones. And so if it's not the last one, but the one before one before, it's like uparrow, but taking a command and a quick plug for alternate shells. Because if we're gonna talk about bash, I'm just going to quickly say, other shells are available. My shell of choice is fish, and fish, um, uses, um, we'll do partial matching on whatever you've partly typed in before you hit all period, if you hit all period on empty space. 101 00:38:35.740 --> 00:39:03.800 It'll do exactly like bash. But if you're like, no, um, I just want to less the, Oh, I dunno. Some log file. I know it's got log written in the middle of you just have a log alt period, and then you can keep hitting alt period, like searching through any argument you've passed to any command in its history that has the word log in it, which is just more beautiful. Okay. Enough ranting about both alt period and fish. You talked about shell check, which is another useful utility. But it's useful because you need it, not because it doesn't anything valuable necessarily. 102 00:39:03.800 --> 00:39:12.100 So when you learn all these wonderful commands and you discover the power that is being able to do all of this stuff. It's almost like being a wizard. 103 00:39:12.100 --> 00:39:19.360 It's exactly like being a wizard about this the other day. It's like being a programmer is like a wizard. Yeah. Carry on. Sorry. 104 00:39:19.360 --> 00:39:22.460 It's the closest thing you're going to get. I tell my kids that all the time. 105 00:39:22.460 --> 00:39:23.380 Does it work? 106 00:39:23.380 --> 00:39:26.000 Eh, You know, kinda. 107 00:39:26.000 --> 00:39:28.260 I think your eldest is more technical minded. 108 00:39:28.260 --> 00:39:32.920 They're buying it. That's going to be interesting to see how that shakes out actually. 109 00:39:32.920 --> 00:39:35.720 Right. Um, anyway. Wizards! Wizards using bash. 110 00:39:35.720 --> 00:40:12.820 So once you discover all these magical spells that you can cast in the terminal that give you all these powers, the next thing that you're going to want to do is automate them, right? Because you don't want to have to actually be around to do all this stuff and type it all out by hand. You want to start automating it. And when you do that, you're inevitably going to start writing bash scripts, and then you're going to write bad, bash scripts. Cause that's what happens when you start writing bash scripts. And the thing to help you with that is shell check, because shell check will tell you everything that your bash scripts are doing wrong and you will gladly thank it for that. Uh, and this combines extremely well with all the other tools that we were just talking about earlier, like watch and ENTR, where you're like, Oh, I'm writing this bash script. 111 00:40:12.820 --> 00:40:55.800 I'm gonna use the ENTR to run shell check on it. Every time I changes that I can never make a mistake. At least not one that shell check would check. Right. It's great. Um, you probably will find conventions that you do at least I do. When it comes to bash where you want to toggle things, the set command in bash can be used to turn things on and off. And there are things like, Hey, if you see an undeclared variable and I try to use it, please fail. Don't just keep going. Uh, also if you encounter any error, please fail. Don't just keep going. Uh, and the sort of the magical combination or like, um, I guess the other one in this little shibboleth that I always put at the top of my bash scripts is if a, if you have a pipe, so you have a series of commands connect to a pipe, 112 00:40:55.800 --> 00:40:57.240 I think we have the same shibboleth. 113 00:40:57.240 --> 00:40:58.240 The first, yes. Set e u o pipefail. Is that? 114 00:40:58.240 --> 00:41:01.020 Pipe file, that's the one. Yep. 115 00:41:01.020 --> 00:41:07.440 Yup. So if the first command in, in a pipe fails, please don't just carry on to the next one. You can just stop right there 116 00:41:07.440 --> 00:41:32.080 Or, indeed, if any, anything in the pipeline dies, then consider the whole command to be dead. Exactly. Occasionally you need to turn that off for very small areas of things because you're trying to do stuff and it doesn't. Yeah. But, but yes, it's I, until this moment, I hadn't really registered that. I thought C plus plus had the monopoly on terrible defaults for things, but it turns out bash is there with the same field. 117 00:41:32.080 --> 00:41:41.200 If you think of bash as a programming language. It is way in the top. 10 of terrible. Why would you do this? Why would you do this? Uh, but it's a shell, 118 00:41:41.200 --> 00:41:57.900 It's a whole bunch of magical things, rules like, Oh yeah. If you want to do, you know, dollar at is like the, all of the arguments. But if you put dollar in quotes, it quotes each one individually because that's magically useful obviously, because when you want to pass, but not, if you just do dollar one, you know, 119 00:41:57.900 --> 00:42:35.640 Or dollar star, which is subtly different, right? Yeah, exactly. It's one of those terrible questions that you could do. What's the difference between dollar at and dollar star. I don't want this job anymore. You guys, really? This is what you're going to throw at me, I'm out, I'm out, I'm out. Um, yeah, but no. So, so you start, you, you learn the magical spells of bash. You learn all these wonderful commands, you start automating things. And then, and then you will come to hate what we hate, which is the activate pattern. Because the activate pattern breaks all of this. 120 00:42:35.640 --> 00:42:53.360 We've talked about this so much and I was ranting about it like earlier this week. And you said, Oh, we should do an episode on that. And now you've sprung it on me. And I haven't had chance to build up the head of steam and bile. No, we should talk about when it's useful and when it's not useful. And I should try and be productive and not just knee-jerk angry, but 121 00:42:53.360 --> 00:42:55.460 So what is the activate pattern Matt? 122 00:42:55.460 --> 00:43:46.080 Well, to me, the activate pattern is the management of an environment that you're going to be running commands in by manipulating global variables, like the path like magical variables. That mean things to various, uh, applications you might be running or programs you might be running like the Python path or, um, other things like that. And essentially you're saying, I would like to have the convenience of pretending that my computer looks at this particular way. And the way in order to I'm going to do that is I'm going to run a shell script, which is either going to mutate my environment. It is, or it's going to fire up a new environment with those things preset. And now I can program away into my heart's content and my compiler will be GCC seven and my Python will be python three nine that comes from over here. And everything's wonderful and beautiful. 123 00:43:46.080 --> 00:44:16.410 And that's, that's great. And it's a powerful way of having an environment, you know, without using something like Docker that looks different from the default one, you might get on the computer so I can see the allure of it. And I understand why it came into being, but what it does is it gives you a hybrid promise because parts of the system come from the real system, parts of the system come from the magical environment variables you're sharing. And it's very hard to get decent into, um, uh, between what's the thing. 124 00:44:16.410 --> 00:44:16.980 Intersubjective? 125 00:44:16.980 --> 00:44:38.940 Intersubjective is the magic word there experience because you're like, Oh yeah, I'm running this thing. And if you are unbeknownst, doesn't like, so often we're trying to help out folks who, uh, are saying, Hey, I'm typing this stuff in and it's not making any sense. If you don't realize that there's some magic that rerouting all of the normal things that you might do in a shell to other magical things, then you're really, really stuck. 126 00:44:38.940 --> 00:45:18.400 You know, your type. Well, what type, which GCC you have to think like, well, hang on a sec, what do you mean? Oh yeah, that says, opt bin some magical thing, some giant thing in your dot file directory. And you're like, Whoa, Whoa, where's that coming from now? Now I don't know where I am anymore. I don't understand your computer, your computer, your operating system has been subverted in a very deep and scary way. And so that's, that's what I, that's one of the reasons though, that's the sort of pragmatic reason why I'm against it. It just makes it difficult to debug because people will forget to say, Oh yeah, I'm using some magical Ruby switcher that this magically switches Ruby, every time I change directory. So that's one thing. 127 00:45:18.400 --> 00:45:27.160 it's also a global variable and who likes global variables. Right. You know, you're setting something which is, is, yeah. If you got some things to say about it before, I kind of explain. 128 00:45:27.160 --> 00:45:31.300 I mean, I think we could both do a whole other podcast on why this is a bad idea, but it's like. 129 00:45:31.300 --> 00:45:32.920 A whole podcast? Oh episode. 130 00:45:32.920 --> 00:45:44.340 No I meant a whole other like a 12 series one hour, each breaking down why the activate pattern is terrible. 131 00:45:44.340 --> 00:45:55.840 I'd be interested. Cause obviously I, I I've read just ranted for a good 10 minutes or so about this. What, what's your feelings if I missed anything or if I communicated, what do you think? 132 00:45:55.840 --> 00:46:42.640 I think I can describe what I don't like about it in a slightly different way, but my feelings are basically the same. When I first got out of school. The shop that I worked for was combination Windows and Solaris. All the servers were Solaris and all of the desktop machines were Windows. And that was my first real introduction to the Unix environment. My school had a Solaris lab and I did some like very basic things there, but it's like, it's not until you're getting paid to do a job that you were like, I really should learn how this stuff works. And that's when I first started learning these tools, which I have to say, by the way, investment in the, in the Unix tool chain is the one of the best, if not the best technological investment I've made in my career, it's held up for throughout my, literally my entire career. 133 00:46:42.640 --> 00:47:32.500 And it is so useful. And it trans it sort of like moves with me from job to job, to job. And it's just one of the very best investments I've made in anything, in any technology ever so strong endorse. But, um, when I was first out of school working in Solaris and I started writing these kinds of automated scripts, right. And my boss at the time was like, never rely on the path. And I'm like, why it's there? Like just, why would you not do that? And he's like, it's a global variable, never rely on the path. If you're calling a command, make sure that you have the full path to that command. And if for some reason you can't get the full path to that command, normalize it to a full path and then print it out so that, you know, what command you were actually running when the, when the stupid thing ran right to me, the activate pattern is going in the opposite direction of that philosophy. 134 00:47:32.500 --> 00:48:39.720 It's not only are we going to rely on the path, we're going to rely on these 10 other magic global variables that you have no idea what they are and they could change at any time. And it's just, it's going in the opposite direction. And I very quickly learned in those early days, why you don't rely on the path because can make some bad mistakes that way, you know, running the wrong version of things, especially when you start getting into like working. Uh, and this was later obviously, but like I've gotten burned by like, Oh, I wrote this bash script for Mac OS, but with the core utils package installed. So it has the gnu utilities instead of the BSD ones. But the BSD ones in this environment, were further up the path and the other ones. So you got a different version of sed, which did a totally different thing and your thing broke. Right? And so for me, that activate pattern is just, it's just doubling down on a bad idea, which is I'm going to mutate my global environment. Then I'm going to rely on that. And especially when you're automating these kinds of things, like there's not a human being there to see that the thing went terribly wrong a lot of the times, right? Like, so you really want to like, just be, I don't know, hyper paranoid that you, you know, what commands you're calling. 135 00:48:39.720 --> 00:49:03.520 Wow. You've that, that is gone in an even more militant direction away from it then that I'm prepared to do. I mean, like, as, as you know, we, our company is open-sourced a little magical, uh, sort of installer. Installer light thing that does do some of this stuff, right. It does say, well, okay, if we put this in the bin and you can have these particular things, but so in a way I felt that I've slightly argued myself against, uh, that program. 136 00:49:03.520 --> 00:49:06.280 I love that program! 137 00:49:06.280 --> 00:49:25.100 But the idea of that's a project called Ozy, um, which we can put in the show notes, or we can forget people can probably can't Google it because it's a terrible, terrible name to Google for. O Z Y. Yeah. But it's, um, but it's based off of dot files, isn't it? So it doesn't rely on path tricks so much. 138 00:49:25.100 --> 00:50:28.980 It has one path trick, right at the beginning just says, okay, this is where I'm going to find the ozy binary. And this is how we're going to do things. But other than that, um, it lets you install things like JQ and other bits and pieces based off of, well, here's a dot file in this directory. This is the version of JQ you get here. Now, again, that's sort of spooky action at a distance, but it's not a global variable. It's a local variable. It's maybe it's an obvious because it scopes to the current directory and say like, for example, although we haven't used it in compiler Explorer, we've talked about it because it's super convenient to say, look, we all know that we need the same version of Terraform in order for us and the other admins to be able to like administrate the site and not forever be playing, upgrade tennis with each other. Um, and so having a dot ozy dot yaml, which says, no, Terraform is version of this done. It means that even if I'm in that directory, I'm just going to get the right version of Terraform. That's kind of a nice thing, but it shares some characteristics. It's got some, you know, there's the, the siren song of like, when it's so convenient, I just typed Terraform and I get the right version of Terraform. Well, as opposed to, 139 00:50:28.980 --> 00:50:57.320 So that's a very clear trade off though? Right. Like, and I will say when I use ozy and I automate things that use ozy, I'm always sure to not rely on ozy bin, the bin directory for ozy being in the path. I always explicitly call, you know, home dot Ozzie slash bin JQ. Yeah. Whenever I want to call the command, right. Because I'm saying, I want the ozy version, not the one that's got installed in the operating system, because I want the version that's in my yaml file 140 00:50:57.320 --> 00:51:31.640 For exactly this reason, Ozy supports a command, which I used in some makefile somewhere, which says, Hey, here are the name of all of the things that I'm going to use. First of all, make sure that they're downloaded and installed and then print out all their paths. And then you can just use that as they, or rather it prints out the path that everything is installed in so that you can ensure and put that in front of all of the subsequent commands. So it kind of is like your one touch point with the, just tell me where things are going to be. Tell me where JQ is going to be. Tell me where, um, Terraform or whatever else. I feel like we've gone slightly off track from that. That'd be by, by bringing up Ozy because, but 141 00:51:31.640 --> 00:51:49.680 Well, you know, it's in a way it's related. Cause a lot of what we use Ozy for is managing these command line tools that we're all talking about, right? Like it's, it's an, it's another way to sort of manage them and install them. And especially when you're building bash scripts and other automation on top of them, you want to make sure that you have the versions that you think you have running in different environments and all of that. So 142 00:51:49.680 --> 00:52:45.320 Kind of it, right. We want to make sure that everybody has the same experience, but that same experience isn't predicated on magical activation. I think that's that the other thing is like this whole idea of like source activate dot sh, is a user specific step you have to do, that's now polluted that terminal until you unactivate it in some way. And obviously your prompt changes usually. And you've kind of like, Hey, look, you're in a magic world now. And I don't like that. How about it just works. Always like go into the directory and say, okay, if I I'm just going to run Terraform, Oh it's terrible. 0.17. Of course it is because I'm in the compiler explorer repository. That's what it needs. Yeah. So yeah, th th I suppose then it's not so much against the idea of managing carefully managing the environment that you want to run things. And it's this almost like the way that it's achieved, it's achieved through magical state action at a distance right. 143 00:52:45.320 --> 00:52:58.240 Environment variables in particular like mutating environment variables as a part of an activate process just is like, I kind of want to depend on as few environment variables like user okay. Home. Sure. 144 00:52:58.240 --> 00:52:59.360 Locale. 145 00:52:59.360 --> 00:53:00.400 That's about it. 146 00:53:00.400 --> 00:53:03.400 That's another one. Actually that was always the one. 147 00:53:03.400 --> 00:53:04.920 Locale? Eh, still even....locale, right? TZ? 148 00:53:04.920 --> 00:53:38.920 So a friend at a previous company would have his locale set to, I can't remember what it was now, but it was like something which is different from one else. But it means, it means that things with capital letters appeared like in a different space from where I'm used to it. He said, no, I've just always wanted it this way. I'm like, I can't use your computer. But also again, it's another thing that shows up stuff like, um, scripts that were like doing LS pipe, you know, head minus one to find like the first thing would not necessarily work. If there was the locale was set differently because LS sorts things based on the locale. Global variables are everywhere, man. It's, it's not a good thing. 149 00:53:38.920 --> 00:53:43.980 Ah, well that's ugly. That's it. You found the ugly man. Ugly. Good. 150 00:53:43.980 --> 00:54:29.680 We had the bad, we had the ugly. So let's do a quick conclusion then. Um, so there's a ton of tools that will be already installed in almost any Unix system. And obviously we've been talking about Unix the whole time. I'm aware that windows has something called PowerShell. That is conceptually sounds cooler because it's object based rather than line based. Having looked at a chunk of power ship, power shell code. I don't get it yet. I don't. And I'm sure it's much more powerful than I understand. And in fact, I know it is, but don't have any experience. So I can't really help you out there. But like anyway, Unix, small command line utilities is probably already there. We've we've talked about PS. We talked about top. We talked about the slash proc file system. We talked about uniq and sort, and particularly that how you can use it to build a group by pattern. 151 00:54:29.680 --> 00:54:30.400 strace 152 00:54:30.400 --> 00:55:03.460 column is another one actually column, which we didn't talk about column takes inputs. Um, and then it will find, it will columnize the inputs, assuming that they're separated by, uh, spaces, or if you do column dash T it uses tabs. Um, and it'll turn, turn like outputs that were like, well, the dollar $1, $2 three spaces into, into dollar one, and then there's enough white space to make it so that all the dollar twos line up and then all the dollar threes lineups. So, you know, it's makes a nice little table in your, in your browser browser, in your shell 153 00:55:03.460 --> 00:55:27.700 Columns good. Sed, awk, perl, Um, cut all sorts of do the kind of manipulation that can be useful. You talked about TCP dump system tap, uh, strace Wireshark, uh, watch. We talked about grep, obviously. Uh, what else? What, are there any other tools I just want to, before we give up on this thing, there's, there's. 154 00:55:27.700 --> 00:55:28.660 dmessage? 155 00:55:28.660 --> 00:56:02.660 dmesg. Oh my golly. All right. Yeah. That's a great one to finish with this D message is the fantastic how to, uh, uh, out sysadmin, the sysadmin nine times out of 10. Right? So in my experience, when you've reached the point where something really, really odd is happening on a computer, and you've probably already pinged the sysadmin team that, that help administrate your system and say, look, we've something funny is going on. It's taking a lot longer to read files nine times out of 10 typing dmesg, which dumps the current kernels ring buffer of like the most recent things that have happened. 156 00:56:02.660 --> 00:56:44.200 Noteworthy things have happened. Um, you'll find in the last half dozen lines, probably the clue, if it's something that's really weird. So the things that, that I normally find that, that the trip this up are things like any sec faults and anything on the system will be reported there. So you can say, Hey, that's weird. Some kernel process say faulted just now, Oh, I bet you it's related. Or you'll see, Oh, CPU overheating, throttling it back. You're like, Oh, I wonder if there's a problem with the cooling or you'll see imminent, smart drive, failure alert, all these kinds of things that you hope will be automated away, but dmesg appears. And then you appear like, you know, what's going on in the system. But in fact, you don't, you just read the last three lines of dmesg. So that's, that's, that's my DMS pitch. Uh, then if you wanna add to that? 157 00:56:44.200 --> 00:56:47.200 You're a wizard Harry.