WEBVTT 1 00:00:00.160 --> 00:00:01.680 Hi, I'm Matt Godbolt. 2 00:00:01.680 --> 00:00:02.780 And I'm Ben Rady. 3 00:00:02.780 --> 00:00:26.540 And this is Two's Complement, a programming podcast. Today we're delighted to have our very first guest: James Grenning joins us. Hi James. 4 00:00:26.540 --> 00:00:28.000 Hi, Matt. How are you doing? 5 00:00:28.000 --> 00:00:29.120 Very well. Thanks. 6 00:00:29.120 --> 00:00:29.680 Hi Ben. 7 00:00:29.680 --> 00:00:30.640 How's it going? 8 00:00:30.640 --> 00:00:31.540 Long time. 9 00:00:31.540 --> 00:00:32.380 Yeah. 10 00:00:32.380 --> 00:00:35.060 So, James, do you want to give yourself a little introduction? 11 00:00:35.060 --> 00:01:20.800 Sure. Um, yeah, I'll give you, I'll try to make it short. But, uh, in high school I had a friend that was programming and it looked like a terrible thing and involved punch cards and rooms with no windows. So I tried to stay away from that. And then in college, I bumped into it as part of engineering school and found out it was actually fun. And then, uh, well here I am, you know, 42 years later still enjoying it, have been lucky to run into a lot of good people along the way, well like Ben there, we were working together about almost two decades ago, I think not, maybe not quite gosh, ran into Bob Martin early in my career. We worked together at the same company and after I did some work at a big company for about 15 years and then ended up consulting. So I like to say I've been unemployed since 1996. 12 00:01:20.800 --> 00:01:43.820 And then somehow Bob said, let's go to Snowbird. And I said, Oh yeah, I'd love to go skiing. And then there was that agile manifesto thing. So I got involved in that totally by accident. Can talk about that later. And, uh, my background is embedded systems and I learned, C back before they had all the bugs out of the compiler and in a better system, C is used a lot. 13 00:01:43.820 --> 00:01:49.320 When did they get the last bug out? No one told me. 14 00:01:49.320 --> 00:02:06.080 I mean, the bugs are still winning, but at any rate. So, uh, and then, now what I do mostly is try to help people learn how to do test driven development in embedded systems. And that's, uh,I wrote a book about that and, that's my business. 15 00:02:06.080 --> 00:02:45.140 And that's kind of why you are an obvious choice for us when Ben suggested, "Hey, I know a guy". Ben and I've been talking about testing over the last few episodes and in particular, testing and C and C++, and why it's different from everything else. Why? I mean, maybe it isn't as different as I think it is, but certainly in my experience, compared to some other languages where it's very easy to install mocking, write tests around the edges of things, all those kinds of things where C++ you tend to have to design your system a bit differently, which we've discussed to death, but I would be interested in your experiences. But the first question I have really is what makes something an embedded system? How do you define embedded? 16 00:02:45.140 --> 00:02:51.760 Good question by the way for me so that my market is as big as possible, it involves a lot. 17 00:02:51.760 --> 00:02:53.960 So is my cell phone an embedded system? 18 00:02:53.960 --> 00:03:04.560 Yes. I mean, certainly there are embedded aspects to your phone and anything that you kind of look at. Well, actually phone is kind of hard to distinguish from a computer these days. 19 00:03:04.560 --> 00:03:07.880 Or a supercomputer already 10 years ago, right? 20 00:03:07.880 --> 00:03:28.360 Yeah, sure. But if you looked at like the early, uh, cell phones and such, they of course had computers in them and you would have certainly thought that was an embedded system because you recognize it as a phone, not as a computer. And so for my definition is kind of colloquial, I suppose, if you don't recognize the thing as a computer, it's an embedded system. 21 00:03:28.360 --> 00:03:28.800 Got it. 22 00:03:28.800 --> 00:03:41.160 Now that means it could be quite big. It could be, you know, the, a phone, a camera, all kinds of things have computers buried in them, cars, millions of lines of code in cars. It's like really scary. 23 00:03:41.160 --> 00:04:43.820 But yeah, so a car could be an embedded system. And in fact, yeah, some of us have cars that are very much explicitly, you know, self drivable things that are just terrifying. But, but, um, I was trying to explain to my, my kids earlier, actually I said about what embedded system was, and I was pointing at like the oven and there's a little clock display. And I said, that's an embedded system. And then, you know, and then I was sort of working my way up and that's what made me think is a cell phone, an embedded system. I know, you know, the modem in it is, has a base band firmware, and it's doing some crazy RF manipulation stuff, and that definitely fits my mental model of what an embedded system is. But the phone itself with all the, the arm CPU and doing the computy things, it's still share some of the aspects with what I think of as a traditional embedded system. That is, it doesn't have a keyboard I can type on. I'm not developing on it. And I can't just write my tests on my, my computer and have them run on it in the same way necessarily that they do on the actual device itself, which I think is maybe what makes embedded testing different from other testing? Or am I barking up the wrong tree there? 24 00:04:43.820 --> 00:05:01.420 I'm not sure what tree your barked up, but, uh, you, I was off like thinking about, can I SSH into my phone? It's like, you know, it is a little Linux box after all, but, uh, you know, so, but you know, so are you, were you asking me like, is, uh, testing of embedded systems different than testing anything else? 25 00:05:01.420 --> 00:05:05.020 Right. I suppose that's yeah. That's, that's the question I have. 26 00:05:05.020 --> 00:05:38.160 Uh, well, yeah, of course it's different, but it's also the same. And, uh, you know, so I came back to, uh, well, we were working on a fairly large embedded system at Xerox in the late nineties. It was one of, uh, Object Mentor, Bob Martin's company's clients. And it would have filled up the office I'm in. And it, it was, you know, taller than me and, you know, longer than my arm span. So, you know, over six feet tall and over six feet wide, and then you can chain them together. 27 00:05:38.160 --> 00:05:39.140 Wow. 28 00:05:39.140 --> 00:05:41.820 And that was an embedded system. 29 00:05:41.820 --> 00:05:48.580 I'm used to being smaller than, sorry, the embedded system being smaller than me. That's, that's another thing in my sort of mental checklist of is it embedded 30 00:05:48.580 --> 00:06:41.960 But you didn't, look at it as...it was a printer, right. You didn't look at it as a computer and it has all the problems of an embedded system. So what's the problem with an embedded system? One of the problems is that you can't just fire up the compiler on it, for instance, you know, like my laptop, computer, my Mac, I can run the compiler on that. Maybe you can run the compiler on your, on your Android phone. I'm not sure probably could, um, it probably does, right when it's updating and stuff. I don't know. So typically people doing embedded systems, they have cross compilation involved. So, you know, they're working on one system, you know, a Mac or Linux box or a PC, usually with specialized tools that really aren't C anymore in some cases because the Silicon vendors have done you a favor and extended the language. Yeah. 31 00:06:41.960 --> 00:06:44.100 Yeah. There's some air quotes there, for the... 32 00:06:44.100 --> 00:07:09.700 There's an air quote there. Um, you know, is it a handcuff to their Silicon when you start to use their special features? Like they'll start to make, uh, addresses, I'm sorry, they'll make uh, registers in the device appear like global variables. And now the compiler does something special. Annotate things say, well, this is not just a plain old function. It's an interrupt function, put a special keyword on it. That's like, it kind of breaks it up, but there's ways to get around all that 33 00:07:09.700 --> 00:07:32.530 I suppose that's really where that my piques, my interest, because that's, that's the kind of thing that is harder to test. I would imagine is if you have got, as you say, like a, some apparent global variable that has to be a global variable, because really it's a system register on your embedded system that controls, I dunno, the LEDs that are lighting up or something like this. 34 00:07:32.530 --> 00:07:33.020 Serial port or whatever it might be... 35 00:07:33.020 --> 00:07:53.520 Exactly. Yeah. Right. So how does one go about testing? So let's pick a say serial port. That's a great example because it's going to be driven by higher level bits of code, but ultimately you're going to be sending stuff to a shift register or doing some magical thing. Maybe there's an interrupt associated with it. What kind of things are you thinking about when you're designing for test or just testing a piece of code that talks to a serial port? 36 00:07:53.520 --> 00:08:39.000 Well, it's been about 40 years since I did that, by the way, but that doesn't matter. I still remember. Okay. You know, so if you were, if low-level code doing one thing, you know, it's job is to wait until something happens on the serial port and that has nothing to do until then. So it's just, I'm going to make it as simple as possible. No interrupt for starters. Uh, the software might be sitting there saying, is there a character, is there a character, is there a character, is there a character, and then maybe every now, and then it would go and change the LED so that you knew that something was happening. Right. You know, every thousand cycles through there or something, maybe it changes the LED. I don't know what the right cycle, what the right duty rate for that would be today. Yeah. But then once it said, there is a character there, then you would go read it and then you would do something with it, you know? 37 00:08:39.000 --> 00:09:31.180 So, uh, you know, a single purpose driver for a UART or something, let's just say, it's waiting for a line of text. So it would grab that character. And it would say, are you a carriage return? Are you an enter button? And you'd say no. And you put it into the queue, you put it into a circular buffer, whatever it is, are you one? And until all that happens, and as soon as that happens and something else would take over, right. So that would be, you might build a little thing that knew how to take characters from a device and put them into FIFO. So something else could take it out. Right. You know, so separation of concerns, you know, lots of people doing embedded systems are, well, I've got a first, I often tell a joke and hopefully I won't alienate anybody that might be listening that will be offended by this joke. But, um, what's the definition of embedded software? 38 00:09:31.180 --> 00:09:31.880 I don't know. 39 00:09:31.880 --> 00:09:34.300 I don't know. What's the definition of embedded software? 40 00:09:34.300 --> 00:09:36.080 Software written by double E's, 41 00:09:36.080 --> 00:09:42.600 I see why you said that was contentious....in good humor, please. I know... 42 00:09:42.600 --> 00:09:49.460 Just kidding everybody. Hey, I, I studied EE, so It's like, right. 43 00:09:49.460 --> 00:10:03.360 But so you, you, you were talking about separation of concerns. And so presumably the, the sort of the take home there is that first of all, to make something, to be made something testable, one has to separate one's concerns and then you were kind of alluding to something perhaps... 44 00:10:03.360 --> 00:10:05.580 That's right. Yeah 45 00:10:05.580 --> 00:10:11.320 Things that maybe by default, that's not what happens when people, write the code as you've described. 46 00:10:11.320 --> 00:10:28.240 So again, not to sound like a, you know, there's a lot to know in this world and to, uh, get an app to work is amazing. Okay. Uh, I like to say being able to get an app to work means you pass the aptitude test. 47 00:10:28.240 --> 00:10:32.460 Oooo, dad jokes coming out [inaudible] 48 00:10:32.460 --> 00:10:59.500 Spelled app-titude. OK, now it's amazing to be able to do it at all. Now then the next thing is, can you do it such that someone else could actually look at your work and know what you did? Or could you even look at it yourself two weeks later and know what you did, I keep wondering who keeps breaking into my Ruby on rails website and messing it up. But I kind of know who that is, you know, so I'm the only one with the keys. So it's gotta be me. Right. Um, 49 00:10:59.500 --> 00:11:02.120 it's that previous you, darn him. 50 00:11:02.120 --> 00:11:22.660 So knowing, uh, discovering really in emergent design, right? Discovering where the boundaries in a system should be. You know? So in the beginning, I, I see all kinds of fun stuff when I work with my clients, uh, about 12 years ago, this happened with one of my clients. 51 00:11:22.660 --> 00:12:10.820 They said, well, you know, after your training class, you're going to stay with us for two days. And, uh, you're going to show, you know, we're going to use it on some of our code while you're still here. And it's like, okay, that sounds like a good idea. I usually tried to sell that and I kind of gave up because nobody wanted to buy extra consulting time. So just took the class. And this, this client flew people in from around the world to participate in the workshop. And then we, uh, then we ended with a couple of day and a half of in their code. And, um, while we ran into some crazy stuff in their code, so we, it worked well. So I started going for this company to numerous places around the world and one place I was at, they brought their code to the workshop and it was one file with main, in it, and about 5,000 lines. 52 00:12:10.820 --> 00:12:11.400 Wow. 53 00:12:11.400 --> 00:12:19.280 And it worked! And it's doing this critical job and, but there was absolutely no separation of concerns, 54 00:12:19.280 --> 00:12:30.240 Wow, as employee that would make my heart sink. But as a consultant, does that make you think, hooray, I can come in here. I can save you from yourself. 55 00:12:30.240 --> 00:13:19.860 Well, it means that, um, I can probably keep working as long as I feel like it pretty sure. As long as it continues to be fun. Uh, and it is great fun. Now that also opened my eyes because I thought I would show people TTD in toy problems in a clean room environment, right. Green, a Greenfield. And they would be able to integrate that into the way they think, and then be able to take it to work. And then they started showing me their actual code and kind of some of the problems, because I didn't actually run into those compilers that are helping you...more air quotes here...Um, and, uh, you know, one guy said, Oh, does this mean I don't have to test my code? He was looking for an out, it's like, no, we don't give up that easy. The preprocessor is an amazing Swiss army knife. 56 00:13:19.860 --> 00:13:40.300 It's a yes. And like a Swiss army knife. You can cut yourself quite badly with it as well. But obviously some, sometimes it's the only thing you've got when you're up against the, as you say, vendor, strange extensions. And you're like, well, I can replace stdio.h with my own one, maybe, and then we can record what the printf was doing or whatever heinous trickery. 57 00:13:40.300 --> 00:13:41.900 Ah, yes, you've.... 58 00:13:41.900 --> 00:14:28.320 I've been around. I've seen a few of the things you could do, but I'm interested what other, you know, that that's, obviously we can talk about that in a minute specifically, but, uh, you know, you're, you're saying that there are some tricks in the bag that maybe are not obvious that it takes somebody who's had the experience like yourself of knowing how to unpick that 5,000 line file into testable components and the kind of techniques that you can use. And the fact that it is kind of worth it as well at the end of it. All, which I think when you are, you don't know those techniques and when you're not really sure that it's going to be beneficial, it's really hard to convince yourself to do the work or to moreover like Ben, you would often say at this point, or to convince the people that pay you that it's worth doing the work 59 00:14:28.320 --> 00:14:54.460 Yeah. Yep. Yeah. Certainly something we were talking about, uh, in another episode is, you know, when is the right moment to sort of point out the elephant in the room and say, Hey, you know, every time we make a change to this code, we introduce two bugs. So every time we fix a bug, we get two more bugs. So if we keep doing this, eventually we're going to have an infinite number of bugs. And if we don't change what we're doing, then that's not going to work out well for anybody. And so, you know, 60 00:14:54.460 --> 00:14:55.820 That's right. 61 00:14:55.820 --> 00:15:40.220 I don't know about, about you James, certainly when I was consulting and sort of, for some period after that, when I was, I would be mentoring or talking to people about testing, the most common question I would get is, you know, Hey, I have this legacy system, I have this old system, I have the 5,000 line main method. How do I do the things that you're talking about? And you know, my unfortunate response is, that's probably like the hardest thing that you can do with testing right, is, is retrofitting that existing system. Um, but there's always a place to start, you know, I never want to give people the impression that there's, that you shouldn't even try. Right. Th there's always a way to start pulling things apart. There's always a way to start making more testable areas of your code. 62 00:15:40.220 --> 00:15:44.280 Oh yeah. There is. Especially with the Swiss army knife in your pocket. 63 00:15:44.280 --> 00:16:00.200 Yeah. And sometimes you got to pull a few dirty tricks to get there. Right. And like, if, if that's what it takes, then that's what it takes. But, um, I would love to hear some of your experiences sort of rescuing some of those code bases. Like what, what tricks were you able to pull, uh, to get some testable pieces of code out 64 00:16:00.200 --> 00:16:52.380 Tricks is not the right term. I usually call them engineering techniques. I mean, I mean, hacks, right? And then I have a whole webpage dedicated to this, by the way. And it, uh, it's uh, how do you get your legacy C into a test harness is the name of it. It's it's on my website and it's because of, uh, that trip around the world I did with this one customer. Every time I went, I saw something different and I would write an article about it. And that way, the next time I ran into it, I could say to the person in the workshop, it has, it's like, Oh, you've got this problem, go read this article. And then I could manage a group of 20 people if some of them are off reading articles, right? So there's some things there and you have jokingly, I call them now we're going to learn the engineering techniques. 65 00:16:52.380 --> 00:17:44.140 And then I mean hack, to get past this problem. And don't...Don't think [that] I think this is a good idea for you to do and leave it in your code because you did this hack. It means you need to go and fix something later, but let's just see if we can get your code under test first, this is the, uh, the mindset would be don't change the production code. Can I do something external in a build or, you know, the make file or the one of the Swiss army knife techniques is to, uh, put a include file on the command line and shove it into the beginning of the file. Right? And so if, if I've got a, uh, one of these deeply embedded processors that has changed the language and introduced like, uh, interrupt as a key word and the regular C compiler says, interrupt, what the hell is that? 66 00:17:44.140 --> 00:18:23.860 Okay. Game over. You can put a forced include in that turns the interrupt into nothing. And now that line of code isn't the problem. There'll be another one. There'll be a problem in a little while something else. Oh, that line that thinks it's the register. Well, I can put...in the forced include file we'll start to collect all my little hacks. Right. I start to put those in and, uh, you know, if it wanted something capital, IN...kind of looks like a macro, um, it's actually a register on a certain device. I can turn IN into a integer. Okay. Now later I get a little later, I get a linker error. 67 00:18:23.860 --> 00:18:31.820 Right. And this is to have like a, some kind of, uh, to be able to build this on a non embedded compiler, is that the, the sort of... 68 00:18:31.820 --> 00:18:40.840 Oh. Yeah. Cause yeah. Cause if some of these deeply embedded things, you're not going to fit a test harness in there you're you, maybe you can get assert and printf maybe, 69 00:18:40.840 --> 00:18:55.540 but the first step is to make sure you can run it on your own computer, uh, as part of your regular testing. And this is one of the techniques that you could employ to bring embedded code into a world where you can build it on your windows machine or your Linux machine or whatever. 70 00:18:55.540 --> 00:19:20.000 Yeah. Right. That's right. Yep. And, uh, yeah, just teasing the code off of its environment. Uh, if you could, if people actually thought about portability, which of course nobody does. Cause what they're thinking about is we got to get this product to work. Uh, now one of the things with hardware is it changes so fast. Um, you might have to re redo an embedded system because the part that you depend on in your, in your product became obsolete. 71 00:19:20.000 --> 00:19:20.880 Gosh... 72 00:19:20.880 --> 00:19:39.360 well now w we never wanted to change the product, but, uh, TI decided, decided to stop selling this part. So now we got to redesign the product, or no, we could save 10 cents if we redesign the product, okay, then you go and spend a half, a million dollars of non-recurring engineering to do that. It might be worth it, you know, so 73 00:19:39.360 --> 00:20:25.280 That's almost points at a better design, makes it easier both to test which we're moving away from modular separation of concerns or the things that we sort of talked about. But if, for example, I have to change either my audio output device of my little embedded system, because it's 10 cents cheaper to use a different DSP or a different, I'm going to wave my hands a lot here. And you're going to give me a lot of, but if that is encapsulated as the audio subsystem in a nice way, that makes it, first of all, easy to test without actually having an audio subsystem. It also means it easier perhaps to just switch it out and say, well, I can write the test of my new audio system and drop it in, and we're good to go. Have you observed that? Is that a, is that a thing that can happen? 74 00:20:25.280 --> 00:21:05.720 Yeah. There's a, there's a name for this. It's called hardware abstraction layer. Okay. Um, it's an old idea which gets ignored a lot. And then I was listening to your last two podcasts and you guys were talking about virtual functions are too expensive to use sometimes. Um, so you can just imagine the kind of conversations you get in where, Oh, we need to add a function call here. It's like, I can't have a function called there. And gosh, you know, so I mean, a regular old direct function call. I can't have a direct. This needs to be a monolith of 5,000 lines with go-tos. I'm sorry, I don't have a stack. It's like, is this true? Yes, this can be true. 75 00:21:05.720 --> 00:21:50.400 Oh my gosh. I mean, I live in a luxury land of, of mostly modern compilers when you start talking about the vendors that you're too. And so I make all these, these grand claims in, in presentations, I do about the amazing things that compilers can do if you give them the right information. So sometimes that kind of thing, it becomes a moot because, Oh, no, the inliner can see through 12 levels of indirection. And you know, that that function call goes away and it's just, you know, register access plus plus or whatever. But in the kind of compilers you're dealing with, that may not be true. You may be talking about a 15 year old compiler with tons of magic, extra keywords that aren't properly supported by any kind of optimization within the compiler. So you have to think about these in a very different way. That's a really interesting thought. Gosh, that's terrifying. 76 00:21:50.400 --> 00:22:43.020 It was kinda funny. Uh, I went to Llewellyn Falco's, uh, approval testing course a couple of months ago. And, and, uh, on the last day I asked him, would you like me to show you one of the things I used for getting code into, you know, legacy code into the build. And they wanted me to show us, so we stayed after the day, and then he had this revelation about five minutes in he goes, Oh man, I've been so lucky. All my code actually builds right away. You know, he's working. And it's like, I mean, one of the givens for your world is it sometimes the code won't even build and it's like, that's right. You. So I've started automating the legacy code build process. And I started toying with this a while ago. I was, uh, so my typical, uh, engagement with the client is I'll go and do a training class that lasts two or three days. 77 00:22:43.020 --> 00:23:13.840 Now I've kind of shortened it to two because people are overwhelmed at that point. After two days, anyway, their brains are like kind of ready to explode with things that they'd never even considered before. Um, and then we spend the next day or two in their code and, you know, so the first two days are in this ideal world where, um, you know, the new code and we can see what the future could be. Right. And so you're talking about how do we help people motivate to, uh, to start to change? And, uh, you're familiar with cyber dojo? 78 00:23:13.840 --> 00:23:18.640 I think I am is that I'm sort of like, uh, uh, web based, um, typing code, 79 00:23:18.640 --> 00:23:23.700 Compiler thing that you're working on. Is it right? Compiler Explorer, I mean 80 00:23:23.700 --> 00:23:33.820 Sort of goals, I think in terms of like, they allow you, well, I didn't use to be able to run code, but I think code cyber dojo is a thing which lets you actually write and run the code and that's kind of the focus 81 00:23:33.820 --> 00:24:13.540 That's right. Yeah. Yeah. So I use it for my training. It's really revolutionized my, uh, my business. And uh, so what that means is that people, when they are in cyber dojo, it's a virtual environment. Uh, I have a cloud server that runs it and I've customized it with my own code and that sort of thing. So I start people off on simple problems and they find out how hard it is to get simple code to work, even if, you know, if there's nothing else going on, you know, so we build a FIFO and you know, what's that like? And you know, can people actually incrementally grow a FIFO from a series of test cases? What we're trying to do, you know, it starts out empty, then you put something in, it's not empty. You can know what the sequence is. 82 00:24:13.540 --> 00:24:20.800 We've talking about a software FIFO here, just boring data structure. FIFO not like an actual hardware FIFO. You're just saying no 83 00:24:20.800 --> 00:24:27.560 Software. You know, the, the interrupt routine happens. You pull the character off, off the device, you put it in a FIFO, something else, consumes it later. 84 00:24:27.560 --> 00:24:36.740 Got it. And as you say, you're opening people's eyes to just the sheer number of things that can go wrong. If you start TDD designing a FIFO from nothing? 85 00:24:36.740 --> 00:24:36.740 From nothing. 86 00:24:36.740 --> 00:24:52.580 and then working up and, and that's, and by that point, you've also, you've extracted all of the embedded specific issues. And you're just saying, let's learn about testing or how one could write this code, first of all, without thinking about that, gosh. 87 00:24:52.580 --> 00:24:57.480 Yeah. It might've been called from and interrupt routine but that doesn't matter. 88 00:24:57.480 --> 00:24:57.880 Which brings it's own fun, but... 89 00:24:57.880 --> 00:25:35.860 Separation of concerns. Yeah. Yeah. And so they get to see over over three, uh, cycles of, uh, I show that present a problem, uh, demo a little bit, have them do an exercise and a debrief. We do that three times. Um, they could to see, you know, how hard it is to get code, to work at all. And even when you're making little tiny changes. Right. And, uh, and then we go and then it's like, okay, but what about our code now? What do we do? Right. And guess what the first problem is, everybody's got these builds that take hours, or, you know, you guys were talking about build time last time. Weren't you? 90 00:25:35.860 --> 00:25:37.480 We were, we were, yeah. 91 00:25:37.480 --> 00:25:38.620 I listened to that. Yeah. 92 00:25:38.620 --> 00:25:40.650 It will not be the last time we talk about build times. 93 00:25:40.650 --> 00:26:26.460 Yeah like eight seconds, I think was your guys' number. And you were talking about this rule of eight or something, right? Eight 80 and yeah. Okay. Um, yeah, so I, I asked people beforehand to complete a questionnaire about how long their build time is. And when they tell me that they don't have time for TTD, I say, Hmm, well I'm spoiling it, if any of your listeners are going to come into my class. Oh, Oh, you know, so, um, you don't have time for TDD, but you don't mind waiting a half hour for a build. Uh, you got a lot of time for TDD. You got, you got lots and lots of time you, what your problem is. You're building everything at once. You have a production build, you don't have any tests builds. You think a build is something that's permanent, a build is something you just need, you might need just for today. 94 00:26:26.460 --> 00:26:36.560 Or so we start to do that. And actually then the problem is I've never built this code in GCC or visual studio, whatever they're choosing to use for their, for their thing. And it's like.. 95 00:26:36.560 --> 00:26:43.460 I was just going to ask how many people do you run into that have never tried to build their code for anything that the target platform does that happen? 96 00:26:43.460 --> 00:27:13.400 You could look at my survey on my website and see, I think I actually cut it off to the only, the first hundred entries. But, uh, just because I don't know if I didn't have an implemented the next button yet and the database is getting kind of big, it was kind of bogging my system a little bit and I don't, and I'm cheap. So I didn't feel like buying a more expensive web server, but you know, Amazon, the next price up would have cost me 10. Nevermind. I should stop talking about my cheapness. 97 00:27:13.400 --> 00:27:17.200 You're just efficient. Engineering efficiency. 98 00:27:17.200 --> 00:28:08.960 Most, most people, uh, that are doing embedded system very few are actually testing off target. Now I'm starting to see more of it and oftentimes it's the person whose idea was to come to my course and to bring his colleagues with them. Right. And so, um, it's more common to only build for the, uh, the target system. Right. And, and I think it back to my first 20 years of development and that's what we did. We, you know, we, we might've run the compiler on it, but we never tried to run it. We tried to run little pieces of it. And then as soon as we had the target, we never ran it anywhere but the target after that, and part of the story is the targets have problems. You know, they're built by humans too, and they've got bugs. And so, you know, I kind of wanted to be able to do finger-pointing more accurately. It's like, well, my code, I know does this. And that's, you know, 99 00:28:08.960 --> 00:29:08.180 The, the idea that people haven't used multiple compilers on their code is...it's always useful. I mean, like as a, as a developer, who's Mo mainly in the service space, if you just point clang and clang, tidy, static analysis and GCC, and all the different, various variants of GCC different versions, you always learn something new and you turn all the warnings on. And there's just so much information that the compilers can give you about your own code. Um, and if you've only ever pointed one, and especially if you've only done, like you say, like a release build, because, um, very often I, as I understand it, these embedded systems don't have the same amount of memory or storage as, uh, uh, as you would want, if you were to do an unoptimized build or even I've had, um, people arguing for not using some abstractions in C++ for embedded systems, quite reasonably, because if you don't build them in the full level of, of optimization, then those, those abstractions don't go away. 100 00:29:08.180 --> 00:29:54.660 The compiler isn't able to optimize them out. And then you miss like timing windows for hardware access and stuff like that. So it does seem like a very different world for the target device in release build, versus having the opportunity to build it with, uh, like the memory sanitizers that we now have access to on development machines and Valgrind and it's like, and all of those other things that come with it, and it seems like a no brainer, but obviously without knowing the techniques that you were talking about, you know, without like force, including some compatibility header, it seems insurmountable to take your 5,000 line code that every third line is, is using some register keyword in a very suspicious way and get it building on your windows machine and running it. 101 00:29:54.660 --> 00:30:26.380 How much easier in your experience, especially with embedded systems. If you start out, you have the luxury of a Greenfield project and you start out with enough people, you know, part of the team, that's building it, where they sort of, you know, they buy into this philosophy of, okay, we're going to write tests and we're going to have, you know, we're going to target multiple platforms and we're going to, you know, set up the environment so that we can build and run these tests quickly. How, how much of an improvement is that over trying to retrofit it later in your experience? 102 00:30:26.380 --> 00:30:32.840 I was lucky in my younger days to be in a lot of Greenfield things, because nobody knew how to program computers, then 103 00:30:32.840 --> 00:30:40.900 I'm laughing, making you upset. You're making yourself sound very, very old, which you're not. So please carry on. Well, 104 00:30:40.900 --> 00:30:46.760 I'm older than like 95% of all the programmers probably at this point, right. People that are practicing 105 00:30:46.760 --> 00:30:51.550 That might, that might, kinda always be true though. Isn't there like a curve? 106 00:30:51.550 --> 00:30:55.660 Yeah. I think, I think Bob was Bob Martin was saying that, uh, every five years it doubled. 107 00:30:55.660 --> 00:30:57.760 Yeah, right. Exactly. 108 00:30:57.760 --> 00:31:07.100 yeah. Yeah. So pretty much almost everybody has less than five years experience. Right, right. Yep. Yep. So, uh, okay. What was I talking about? 109 00:31:07.100 --> 00:31:12.820 We're talking about whether, if you starting Greenfield vs retrofitting later and the relative costs there. 110 00:31:12.820 --> 00:32:00.300 All right. Yeah. Okay. So, uh, well most of the world doesn't have that luxury because they're having to deal with somebody else's thing that they already did, or they're moving it to a new platform because the hardware keeps changing. Right. And so I it's really common to run into, you know, one of the questions I'll ask and of course is okay, who's got five-year-old code. You see the hands go up, who's got ten-year-old code, who's got 20 year old code. Who's got 30. Or if you're at like a telecom company, it's like, we've got 50 year old code, you know, it could be. Yeah. I mean also probably C, you're not going to have any, C older than 50 years, you're going to have 40, you can have 30 and 40 year old C code. Right. It's possible for sure right now. Right. Um, so a lot of people live in that now, uh, for sure it's easier starting from a Greenfield. 111 00:32:00.300 --> 00:32:32.860 And it also helps if you know what to do, because a lot of people don't really know what to do in that situation because they don't get it very often. Um, you know, it's like, Oh, well, you know, when you first write tests, when you have no code, the tests are kind of stupid and trivial and testing, ridiculous stuff. Okay. It doesn't take long for the code to go beyond that. Right. Um, you guys familiar with the zombie, uh, acronym? 112 00:32:32.860 --> 00:32:34.460 No. 113 00:32:34.460 --> 00:32:35.220 No, what is zombie? 114 00:32:35.220 --> 00:33:03.340 So, so that's, uh, I've got an article about it. TDD guided by zombies. It's on my blog and, uh, zombie stands for zero one many. And then, okay. It's a two dimensional acronym or maybe a multi-dimensional acronym. Zero one, many is a linear part. And then the other side of the matrix are boundaries, interfaces, exceptional situations and keep it simple and basically starting off in something and TDD is find the zero tests first. 115 00:33:03.340 --> 00:33:52.000 So it metaphorically, it's good with something like a collection, because, well, what if the collection is empty? What can you do? You can check to see if it's empty. You can make sure it's not full. You know, if I'm looking at let's use a circular buffer, because that's what the that's, what's in the article. Um, and then you're kind of out of stuff to do at that point. And then what can we do next? We can put something in, Oh, then it's not empty anymore. And it's probably not full because it's probably not a one element thing. Right. And so you kind of grow your way in while you're doing this. You're paying attention to different things, interfaces in the beginning test, you're really trying to get your interfaces. Right. So they're convenient to use, right. It's trivial to get them to pass because you don't really need that much code to say return true. It's empty, you know, and most people go, Oh, that's such a worthless test and your code is wrong too. So why are you so happy right now? 116 00:33:52.000 --> 00:33:53.820 Yeah but that test saves you later, right? 117 00:33:53.820 --> 00:33:54.080 Yeah, yeah. 118 00:33:54.080 --> 00:33:59.000 When you add the code that makes it possibly not return true, but you...you don't. 119 00:33:59.000 --> 00:34:13.090 That's right. Yeah. So I'll hear, I'll hear from people after this first experience, they'll say, well, it's just against my principles to write code that I know is wrong. It's like, okay, but you don't mind writing code that you don't know is wrong. 120 00:34:13.090 --> 00:34:17.340 You write code that's wrong all the time because we all do 121 00:34:17.340 --> 00:35:11.860 You wrote code that's wrong and you were okay not knowing about that. Okay. Um, but the, uh, yeah, certainly easier with that. I did this other conference talk, which, uh, what is it called? It's a tracer bullets and zombies or something like that. And the idea with the tracer bullets, you know, it's a pragmatic programmer idea. I'm just using their term. Uh, but you go start looking for the unknowns and, you know, it said, you know, Oh, we're going to build this. Uh, I was working on a project for my brother IOT radios. And, uh, yeah, I didn't know anything about IOT radios and pretty much, there's a long list of all the stuff I didn't know anything about. And he's asking me to help him. So it's like, sure, this'll be fun. What you might do is you might go work on the business part of the problem because you understand that and you'd be comfortable with that, right? 122 00:35:11.860 --> 00:35:57.740 Oh, when we take the, uh, uh, reading, we convert it to a PSI and then we can do these formulas in PSI. We all, we know what those already were, so we could have worked on that, except all the risk in this project is all the things we don't know. And so with the tracer bullet approach is you go start looking for those unknowns and try to build a walking skeleton of, you know, can I get a pressure reading from next to the fire hydrant, uh, onto the tablet for the guy through all the layers, you know, which involved, uh, IOT devices and, uh, well, it's about nine layers by the time you're done of stuff. I didn't know about, you know, when I first started, I only thought there were five layers of things. I didn't know. And then by the time we finished, there were nine. 123 00:35:57.740 --> 00:36:44.200 Um, and we could prove the concept then though now, uh, where I'm getting to at this. And what's interesting is that all those little boundaries, the code that's in between is just like hard. You know, it's like a hard coded, uh, well, um, you know, none of the error checking is really there. It's just about, can I transport something from the bottom, all the way to the top and the little things you learned along the way. And now after you get, once you've proven, you can do that. That pretty much the high risk stuff has gone. And now you can say, well, we get a reading once every second. And you know, from 10 sensors, we get readings. Once every second, you gotta collect those together and present them as a JSON package to the web browser. Right. You know, so there's, now once you've proved that you could do anything, then you start to do the stuff, which is pure software. 124 00:36:44.200 --> 00:37:06.800 And so you're shooting the tracer bullets to find out that all the stuff you don't know and getting something to work on this edge where you're not going to bother writing tests for that, because you need an integration test, not a, a unit test. And then you try to minimize that thing that's outside your control and maximize the thing that is pure software, where you can play the zombies game. 125 00:37:06.800 --> 00:37:07.560 Right, right, right. 126 00:37:07.560 --> 00:37:23.720 Okay. You know, so, right. So if I only had one set, if I had no sensors reporting, what would I report to the web browser? Nothing. If I had one sensor, what would I report if I got to it? You know, what, whatever, you know, so you can start to, uh, grow the stuff in between. Right. And, uh, 127 00:37:23.720 --> 00:38:08.420 Because we talk about, you know, engineering techniques, AKA hacks for testing embedded systems in strange and interesting ways. It's amazing how many, so, so many of those techniques are so portable from one programming language to another, from one library to another, from one environment to another, you know, once you start learning the sort of basic philosophy of, okay, I'm going to, I'm going to test this with, uh, with null values with zero values, with empty lists. And yes, the tests will initially look very dumb, but they're going to lay a foundation that I can then build on. As I add more complicated things, you know, techniques like you're talking about like the tracer bullet, you know, sometimes people call that the steel thread where you cut through all of the different layers of the system defining sort of simple versions of the API as you go. 128 00:38:08.420 --> 00:38:50.720 And because those are testable, you're confident that you can change later. It's like, Oh, well we have to support this and this and this. Yeah. But don't worry about that yet with the tests in place, we can change our mind. We don't have to get the perfect design upfront because we know we are confident that we can change it later and not break anything. And it's those techniques, like some of the techniques that you learn are just not portable. Like I can tell you terrible things that I've done to test asynchronous Python in the last six months. That pretty much are not useful in any other situation, unless you're using asynchronous Python and that's it. But there are lots of these techniques that you learn that can be applied in lots of different environments. And you sort of get a lot of bang for your buck when you're, when you're sort of learning those techniques. 129 00:38:50.720 --> 00:39:02.200 And you remember all those years ago when we were together in Texas and, uh, I was wanting to do something with a socket. I've never done it before. And so I was talking to you and he said, Oh, it's easy. Just do this, this and this. You remember that? 130 00:39:02.200 --> 00:39:05.880 You know I don't remember that? But I'm really glad that whatever it was that I said was good. 131 00:39:05.880 --> 00:39:13.100 Yeah. Well, so you, you helped me. So we're trying to adapt fitness to this, uh, thing. It's Sabre Sabre Airllines. 132 00:39:13.100 --> 00:39:15.240 Oh, yeah. I remember. 133 00:39:15.240 --> 00:39:50.460 Right. I was trying to listen in trying to intercept the traffic and record it. And I'd never done anything with the socket before. So now people ask me, well, if I want to test drive code with a socket, what do I do? It's like you wrap the socket up in something and you integration test that, and then you build yourself a abstraction layer over socket. What are you trying to do with your socket? I'm trying to send messages from one thing to another. Okay. Your application shouldn't be about sockets. Your applications should be about getting something from something that delivered you one and right. The thing underneath there happens to be a socket. 134 00:39:50.460 --> 00:39:51.120 Absolutely. 135 00:39:51.120 --> 00:40:05.280 Yeah. And so you helped me down that and I, that, uh, knowing what to do when there's something that's outside, your control is hide it. And I saw, um, Matt's making the, I thought he was trying to get us to end.... 136 00:40:05.280 --> 00:40:08.930 No no no, I'm saying how much I love this, I think it's just wonderful to hear the stories. 137 00:40:08.930 --> 00:40:09.980 Manual heart emoji there.... 138 00:40:09.980 --> 00:40:20.560 For me it's just interesting to hear like, stories of an early Ben. So I'm tapping you up here to learn about my friend through your interactions two decades ago. 139 00:40:20.560 --> 00:40:27.320 Ben was great. He knew way more Java than I did. So that was good. 140 00:40:27.320 --> 00:41:17.640 That was back in the Java days, man. I mean, it's, and you know, we were, we were talking at the time about like those abstractions, you, you can create them for testing, but they, they make your code more reusable. Whether you like it or not. It's like, it it's like inflicting reusability on you in a good way where it's like, you know, if you create abstractions that hide. Yeah. Okay. This is a socket and we made that for testing, but you know, maybe it could be a file and you wouldn't even know the difference. Right. Um, or like a, you know, basic thing is like standard in standard out, right? Like I'm reading from a socket. Could I also read from standard in? Yes. If you're, if you, if you design those abstractions to be minimal and simple, so you sometimes get that for free. And that's, you know, one of the great benefits, benefits of testing that again, I think cuts across all different environments, all different languages. Uh, it just, it just helps you no matter where, where you're working. 141 00:41:17.640 --> 00:41:51.520 Right. You know, you run into people that say, Oh, no test driven development couldn't possibly work for us because we're special because we do, we do this. It's like, okay, you're special. I, yes, you are. But you know what? It doesn't matter. Okay. So the problem solving techniques, you know, the problem in an embedded system is you depend on hardware, the problem with a web based applications, you depend on the database. Right? What do we do? We break the dependency, isolate the right. Okay. So the ideas are portable. So you got lots to learn from different people. 142 00:41:51.520 --> 00:42:11.820 Matt, you mentioned memory checking. So CPPUtests, the, uh, the open source, uh, test harness that we use a lot for embedded systems built on C++ we built in memory leak detection into it. And it's extremely important because everything leaks pretty much if you're not paying attention. 143 00:42:11.820 --> 00:42:21.120 Right. And I mean, especially if you're dealing with older C compilers that don't have some of the new and nice things that have come with C++, which have closed a lot of the doors, but 144 00:42:21.120 --> 00:42:42.300 Yeah. Then you got to know how to work those too and someone might not want to pay for them. Right. I mean, from because you're talking about constrained environments, right. You were, I know in one of your earlier episodes, you're saying, well, we, we can't afford the virtual function. Okay. I remember the first time that happened now, your compiler Explorer would have been really useful to me back then. 145 00:42:42.300 --> 00:42:46.760 Right. That's kind of why it exists is exactly because of those types of things. So yeah. 146 00:42:46.760 --> 00:42:48.160 Is that why it exists? Okay. 147 00:42:48.160 --> 00:42:55.840 Answering those questions of like, can we do this thing? I don't know if we can afford that. Let's have a look at what it does. Yeah. But yes. And you're, you're at Motorolla... 148 00:42:55.840 --> 00:43:09.440 Yeah. I'm at Motorola 20 years ago. And, uh, um, I'm proposing that. We put an interface on something with virtual functions and the lead engineer says, no, we can't use virtual functions here. It's like, why not? Because our document says, we can't. 149 00:43:09.440 --> 00:43:31.560 And I'm exaggerating a little bit. And I said, well, why does your document say you can't just because they're too big. Oh, okay. And I went off and did an experiment and I'll use a little air. So the virtual function dispatch was about this big and the direct function was about this big. But guess what? The alternative was a switch case, which was THIS big. Right. 150 00:43:31.560 --> 00:43:39.500 Right. For the purposes of the listeners, the switch case was considerably larger than the, either of the other two things that James gestured at us, 151 00:43:39.500 --> 00:44:17.500 It took two switch elements to, uh, exceed the virtual function dispatch. So it's like, you can't just look at the micro. You gotta look at the big picture. Absolutely. Now the last thing I want to just mention, because it's, if there's anybody in this kind of world, trying to drag existing code into a test harness, and I wouldn't mind if anybody wanted to contribute to it, it's a thing I called gen X fakes. And I mean, no disrespect for gen Xers. It's Gen- Xfakes and X fakes stands for exploding fakes 152 00:44:17.500 --> 00:44:18.480 That sounds painful. 153 00:44:18.480 --> 00:45:06.760 I mean, exploding, fake. So I'll make this fast. Cause when you're in an embedded world, we never got over to the what about legacy code part? So first what would be possible right? Is kind of how I start my course out. And then we have to deal with the real world after that. And your mileage may vary, but pulling code under to get under test. First off you say include something. You can't find the include file. Okay. Then you've got to go find out where the include file is and put it in your, in your test case. And then now it can recognize your code. And then of course your include file won't build because there's other include files missing. So you go to this loop for a long time, I call this crash to pass and it's up on my, uh, my blog to eventually you get to linker errors. 154 00:45:06.760 --> 00:46:02.000 And what do you get to linker errors? Um, this, this whole process, very incremental way to drag code into no batch mode. Just one error at a time when you get to linker errors, you get like a hundred linker errors, right? And I'm only trying to test like one little function out of this thousand line file. And I want to test a hundred lines of it. Let's say. And if I got, I'm just going to pick some numbers, 20 unresolved, external references that are unique. I really probably only need two of them to be resolved into real fakes, spies or fakes or mocks or something. But the other ones I've got to get the linker to be happy. So I need something for them too. So gen X fakes, we'll look at the linker output and produce a bunch of little test stubs that if you happen to execute them, it will fail the it'll exit, the test. 155 00:46:02.000 --> 00:46:35.620 Abort the test. Got it. So this is a way of, of, of essentially, as you say, taking the, the things that you know that you called, you think you don't need and generating essentially the stubs that say, ah, we're done. Yeah, like you say, they explode, they are born, they print print and they fail the test. And it's like, no, you, you need to either stub this out or fake it out or mock it out. Or you need to remove the dependency on the, in that function. And this allows you to link a subset of your, you build and link against the tiny subset of your giant program in order to test just that one small element of the program. That's amazing. 156 00:46:35.620 --> 00:47:25.860 Right. And so, yeah. So if you had a thousand line or a 10,000 line file, I've seen a hundred thousand line files in my legacy adventures. I know, you know, to drag that under test is, you know, it's going to have so many linker errors people are going to, they're going to keep wanting to give up. Right. And so, uh, now gen X fakes, I've added to it a little script because the there's a series of things that you do a lot of, like what includes path do I need to add to my test build? Uh, so if it gets the error that says, can't find include file, it goes and runs a find command on your directory structure and then produces a little line that, that you could add to your make file. It's like here, put this in your make file. Right? And now it can find that file. And you know, there's a series of things like that, that. 157 00:47:25.860 --> 00:47:52.620 Like a tool kit for making it as straightforward as possible to get to a point when you don't have a sane build system, like, you know, I'm, I'm used to having these big sprawling projects that have been set up by me to be sensible and like sub contained and like, everything can just include itself and whatever. But if you're not in that luxurious world where you just put in the header file, you're like, no, this is a great way of taking just one small subset and building out a fast test. So you're talking about... 158 00:47:52.620 --> 00:48:04.720 you know, start to get us there and then, and then you have people, uh, well, okay, so that's a start. And then three years later, somebody will send me an email. It's like, Oh, it's really great. We did the stuff you talked about. And now our code base is completely different and really great 159 00:48:04.720 --> 00:48:06.880 Makes you feel good. To get to that point for sure. 160 00:48:06.880 --> 00:48:15.600 Well, because of three years of, of not, they weren't trying to do that, but as they went to different places, they followed the boy scout rule, which has make some improvements, 161 00:48:15.600 --> 00:48:17.320 make it a little better... 162 00:48:17.320 --> 00:48:29.780 make it a little bit better. And if you choose always to just take another shortcut, then it's always going to get worse. But if you know, more times than not, you choose to make a little bit better than eventually it's going to get better. So, yeah. 163 00:48:29.780 --> 00:49:33.700 Yeah. That's, it really, that's the only way to sort of revive those legacy systems. At least in my experiences, you just got to change your philosophy, which can be hard for other reasons, but it takes no real technical work, change your philosophy. And then just apply that philosophy going forward, doing the things, accomplishing the goals that you would try to accomplish otherwise. Right? Like we have products that we need to get out, we have features we need to add, we have bugs we need to fix. Yeah. Do all those same things. Just apply a different technique. And eventually you'll find yourself in a very different place, which, you know, it's a little bit like boiling a frog. You don't notice that it happens with sort of looking back on it and go, wow, we really made some serious changes here. It's kind of amazing. So speaking of looking back, uh, I wanted to, to ask you about, uh, the sort of the upcoming 20 year anniversary of the signing of the agile manifesto. For me, you know, we've kind of talked about this on, on earlier episodes of the show that agile movement XP in particular for me was, was a, uh, a welcome change from, uh, other processes that I was using when I first got out of school, right? 164 00:49:33.700 --> 00:50:36.760 Like we I worked at this place, it was using the rational unified process and had these sort of very formal mechanisms for defining requirements and writing code. And we never did any sort of automated testing. It was all was kind of manual testing. You know, we had manual testers that would verify our code. And for me agile and, and XP in particular was a movement toward more engineering focused tasks because at the end of the day, software is written by programmers. Well, sometimes it's written by double E's, but usually software is written by programmers. And so orienting the, the, the project orienting the team, orienting all of these things around what the programmers are doing was at the time a novel idea, but, you know, in, to me made a lot of sense. So I just kind of wanted to ask you, you know, over that timeframe, you know, did agile turn out to be the thing that you expected? What was different about it? What sort of like, you know, looking back at the world as you saw 20 years ago, what would you have predicted that has actually happened today? 165 00:50:36.760 --> 00:50:38.340 Nothing. 166 00:50:38.340 --> 00:50:41.420 Nothing? 167 00:50:41.420 --> 00:50:42.620 Wow. 168 00:50:42.620 --> 00:51:53.300 So, so what, uh, okay, so there's, that's a short answer. Um, has, what was the expectation at the time the expectation was who cares? Um, you know, will anyone care? We had no idea that anybody would care about those four bullet points and by the way, there was no signing of anything. So there was the writing of those four bullet points and then, uh, uh, debating of the 12 practices that backed it up over email afterwards, um, which I kind of watched that was really learning from these guys at this time. Right? So I'm there as a 20 year person having done, uh, stuff, you know, helping my former employer learn waterfall and, you know, trying to get them to go from chaos to something that, that, you know, we could start to get some quality. And, um, then the 90's were, uh, trying to, um, improve processes, but also, uh, um, iterative was starting to be talked about a little bit more with the object oriented programming, you know, evolutionary type stuff was starting to be talked about. 169 00:51:53.300 --> 00:52:04.080 Um, and I think, uh, late nineties, when the words extreme programming were uttered, it's like, what does that mean? Cause extreme sports were big, then it's like, Oh, we'll have to wear elbow pads, 170 00:52:04.080 --> 00:52:04.600 Climb up a mountain with a laptop. 171 00:52:04.600 --> 00:53:09.220 Yeah. Right. And, uh, you know, so we were, uh, an object mentor, Bob Martin's company. We were learning from Kent and Ward and, uh, Ron and, uh, Martin Fowler about how to, how to do these things. And so we're about three years, two years into it at a Snowbird. And, um, you know, if somebody asks me to go skiing and Snowbird, what am I going to say? Okay. Right. I'm going to say yes. All right. And so, yeah. Right. Okay. So that was, and then you're going to go hang out with these guys that you respect and talk about software development. So that's what, uh, the meeting was it, you know, the couple of things that, uh, I forgot, but a friend of mine who I did a lot of consulting with out there at the time John Johanson, a few years later, he reminded me, he said, you know, after that meeting, you said that you brought the note cards, this was something that us XP people would do as part of the XP, extreme programming contingent of that, that meeting. 172 00:53:09.220 --> 00:54:23.100 And, um, you know, these all could be false memories by now having talked about this stuff and made them a little more interesting each time you talk about it. But, uh, so, uh, my friend John said, yeah, you told me like, right after that meeting, that you toss the note cards out. And one of the techniques that we were using in those days, well, actually in the eighties, uh, as part of total quality management at the company, Bob Martin and I worked together at, uh, total quality management was about structured problem solving and communicating in a way that was non-threatening and that sort of thing. And so we would do brainstorming on note cards, you know, and you'd say, well, what are we, why you know, what do we want out of what's the problem with software development? Or what do we want? And so, um, we started writing on note cards and then seeing if there was any agreement, right. And so, uh, then it'd probably move quickly to the, to the whiteboard. And it turned out to be, you know, well, having a plan isn't bad, but there's something, you know, plans change. So we, we should have a plan, but we shouldn't not change. So, you know, the whole something over the other thing that, that conversation started happening. 173 00:54:23.100 --> 00:54:29.340 Yeah. That particular phrasing in the agile manifesto of like, these are okay, but this is better. Right. 174 00:54:29.340 --> 00:55:24.060 And most readers of that now, um, if they're, you know, kind of against the idea of agile, only read the other side and say, well, of course, other against plan. So, you know, get rid of that. Right. You know, so, um, but it was really about, uh, uh, you know, trying to see how we could collaborate better now, like you, Ben, I was in there because I thought this was a better process. I was really kind of into the process oriented thing. I'm an engineer. So, um, I'm thinking about problem solving and effective ways of problem solving. And, you know, so I'm there thinking that now she goes, oh the process. Yeah. It's important, but it's more important that people you're with and yeah. Yeah. The first bullet point was about people and it was just kind of like, I came here to talk about extreme programming. They're talking about people being, playing nice together and, uh, you know, 175 00:55:24.060 --> 00:55:29.560 There's so many software engineering problems turn out to be people problems. As I think it's taken me 20 years, 176 00:55:29.560 --> 00:55:35.060 That's a Weinberg thing. Right? Yeah. All tech, there are no technical problems are only people problems, I think. 177 00:55:35.060 --> 00:55:40.560 Yeah. You going to become more and more aware of that, the more you do more you program with people. 178 00:55:40.560 --> 00:55:54.240 Right. Yeah. So, uh, so that was kind of interesting. And then the fact that it became popular, like a friend of mine, uh, you remember Lowell, of course, Ben, right? From Object Mentor. He was a business manager. 179 00:55:54.240 --> 00:55:55.240 Uh, yeah. Yeah. 180 00:55:55.240 --> 00:56:18.150 So like about six years after that, um, we were talking and, you know, I was looking at, uh, starting my own company because I wanted to get embedded people doing, um, TTD and stuff. I couldn't get them to come to Object Mentor and also said to me, James, you know, you're part of the agile manifesto. That's like instant credibility. It's like, what, Why? 181 00:56:18.150 --> 00:56:20.760 I just wanted to go skiing. 182 00:56:20.760 --> 00:56:23.420 Yeah. Come on. It's like, well, you were there though. 183 00:56:23.420 --> 00:56:25.400 It's like right place, right time. 184 00:56:25.400 --> 00:57:48.680 It's turned out to be a really good thing for the resume. Um, and, um, you know, so right after starting to learn about extreme programming, I was working with a company doing embedded stuff, and I had 20 years of embedded. And so I just started experimenting with them and it turned out it was a good idea. They're so fantastic. Now, as far as how agile has done in hindsight, I was pretty pumped about how it was doing 10 years ago, because I thought there was a lot of improvement happening and I'm kind of disenchanted because, I think we're back. I think we're in the dogma following rather than problem solving mode. And, uh, you know, scrum is wildly popular. If you go to a scrum gathering, there are no engineers there. Which, which, I mean, except for me, they invited me to go and talk, talk, and then I complained about there not being any engineers. It's like, where are your engineers? It's like, well, they don't like scrum. I mean, that would say that, um, if you, if you go online and you read about what engineers think of agile, they think it's horrible because this is, here's my rant is it's not focused on the engineering skills is not what the, the world has done. And if you remember, did you ever go to one of the extreme programming immersions back in the old days? 185 00:57:48.680 --> 00:57:52.980 No. I didn't have the travel budget back then. 186 00:57:52.980 --> 00:58:52.020 Yeah. Okay. So, uh, the way we, the first extreme programing immersion was, uh, well filled with, you know, people that already knew it and loved it, but we tried there, not everybody, but we, I think we tried to start from the top-down management practices down to engineering practice. And it was, if, if everybody didn't already want to be there, it would have been a disaster. It didn't work. And then we started teaching the engineering practices first. And, you know, by Thursday, when you introduce stories and planning, it's like, Oh, big deal. It's like right now, we know now we know what a little piece of work is. And if you start the other way, and this was one of my gripes about agile is it starts the other way, the hard way. And it alienates, the engineering people have been learning how to do stuff, you know, by planning and, you know, not writing code until it's too late, right. Instead of starting sooner and learning sooner. Right. And so, uh, 187 00:58:52.020 --> 00:59:57.300 For me, the most amazing thing about XP in particular was that the realization that if you follow this set of engineering practices, which are things like test or development pair programming, to a certain extent, continuous integration, there's a whole bunch of other stuff that you don't have to do, like project management, stuff that you don't have to do. And that is extremely threatening, if your job is "project manager", right. That's not what you want to hear. Right? You want to, you want to hear the opposite of that. And so that's, I think part of the problem is, is that the reason that agile has kind of gotten away from that being, you know, not just to say it's rooted in engineering practices or should be rooted engineering practices, it's almost underselling it. It is like 95% engineering practices. And once you have that foundation in place, you only need to add another little 5% to make the whole thing work. Right. It doesn't take that much. It just takes some note cards and a couple of conversations and an understanding of what you're trying to build. And you can do amazing things without having to add a whole bunch of cost. 188 00:59:57.300 --> 01:00:03.020 You could probably even be successful with JIRA. At that point. 189 01:00:03.020 --> 01:00:05.340 Let's not take this too far... 190 01:00:05.340 --> 01:00:09.470 I'm talking out, I'm talking about the side of my mouth. I don't know anything about Jira, but I do hear people complain about it. 191 01:00:09.470 --> 01:00:16.180 People complain about whatever tracking and management system you put in place. 192 01:00:16.180 --> 01:00:41.680 Yeah. Yeah. It's true. I still have to try to explain to people it's like the whole idea behind the story card is that it's a promise to have a conversation. Like it's about the conversation and we're going to talk about this. We don't need, we don't need to have like this huge JIRA system for capturing all the requirements, putting all the things and figuring everything out. That's not what this, that's not what a story is. Right. The story is, we're going to talk about it, but now I'm just ranting. 193 01:00:41.680 --> 01:00:44.520 Thank you for not calling it a user story. 194 01:00:44.520 --> 01:00:46.580 Ah, yeah, 195 01:00:46.580 --> 01:00:51.560 Because it's really a story. That, that word is like, it's unfortunate. 196 01:00:51.560 --> 01:01:08.920 Yeah. There's, there's a lot of, I think there's a lot of that. Yeah. I would call it nuance, but it's really the most important part. But a lot of that stuff that kind of got lost over the years, um, which is a shame, but you know, some of us still know what it is. Some of us still know how it's supposed to work. 197 01:01:08.920 --> 01:01:49.260 I was out with these guys. Um, uh, but, uh, I'm out to dinner and after a conference with a business guy and, um, we were talking about agile and said, yeah, people are only doing the easy half of agile. And then yeah. So I only doing the easy half of agile. And then he said to me, well, what's the hard half? It's the technical practices that are the hard half. And he goes, no, that's not the hard half. The hard half is what people half. And it's like, I hit myself in the head. It's like, Oh, damn, there's three halves, You've got the planning half, you've got the people half, and you've got the technical half. It's like, Oh, okay, 198 01:01:49.260 --> 01:01:51.930 We're going to need a bigger boat. 199 01:01:51.930 --> 01:02:13.240 That sounds like an absolutely perfect place to stop the conversation that we could obviously keep it on for many hours. There's a lot of interesting things to talk about here, but before we go, James, do you want to let our listeners know how they can find you online? Like you mentioned your blog a few times, um, any other way of contacting you? 200 01:02:13.240 --> 01:02:28.440 Okay. So my website is wingman-sw.com. Wingman software. And, uh, I wrote a book Test Driven Development for Embedded C. If you search for that, you'll find me. @Jwgrenning on Twitter. Those are ways to find me. 201 01:02:28.440 --> 01:02:46.200 Fantastic. Well, thank you so much for coming on and talking to us, this has been so interesting enlightening and certainly lovely to hear other people struggling with different types of testing and C, and then obviously to hear about the agile stuff too, is just mind blowing for me. So thank you so much for being with us. 202 01:02:46.200 --> 01:03:02.600 Well, Hey, it was great to talk to, uh, two guys that get it. And, uh, I think, uh, I really liked your first two episodes of your podcast. I liked the little game, the game theme. That's cool. One player two player. 203 01:03:02.600 --> 01:03:11.600 One day, someone will figure out how to actually get to the podcast from the main page, but I'm not holding my breath.