Matt Godbolt: Hey, Ben.
Ben Rady: Hey Matt.
Matt Godbolt: How are you doing today?
Ben Rady: I'm doing alright.
Matt Godbolt: I'm, I'm a bit croaky as you can hear. I'm at the tail end of, uh, one of the many colds, which thankfully was not Covid. Um, but in a family where folks have just gone back to school, there's, there's zoo of microorganisms that get reintroduced to you after a summer break. And one of those was the, the common cold and left me without a voice, which is like the second worst thing ever. If you could stop me from typing, that would be, you know, the end of my life. I think right there that'll be, you know, couldn't, can't talk, can't type what's the point of it, of it all anymore. Right. But, um, yeah. So, um, we discussed this before, before we started recording that, you know, we're gonna try and pick a topic that would let you talk and , we are now one minute into this recording, and I'm the only one who's talked so far other than you , so I'm not doing very well
Ben Rady: At this. So the plan is going perfectly exactly as we designed.
Matt Godbolt: So what do you wanna talk about?
Ben Rady: Uh, I thought we could talk a little bit about, uh, the differences between and the implications of iterative and incremental development.
Matt Godbolt: Okay. Now see, those sound very similar to me. They both start with an I, which, and whereas me all know, all synonyms start with the same letter, so they're immediately the same word, but they, they feel like they should be, they should be the same.
Ben Rady: Uh, yes.
Matt Godbolt: So can you define them for me? Imagine I didn't know what they were already. It's easy to do because
Ben Rady: It's true. I, uh, yes. And, and, and in fairness, I think pretty much everybody that uses these terms and thinks about these concepts, we'll mix them up from time to time. They're very related, but they're not the same thing. And it is sometimes very helpful to be like, no, no, no, no, we're not gonna do this incrementally, we're gonna do this iteratively. And if you haven't had a discussion about what those words mean, then it sounds like, no, no, no, we're not gonna do this we're gonna do this.
Matt Godbolt: Right, yeah!
Ben Rady: And it says, okay, well that what, what did you just say?
Matt Godbolt: There's a subtle distinction in those two words. But they're lost upon me, so, tell me.
Ben Rady: Right. Right, right, right. So the, the best analogy that I've heard for this is like, imagine you want to paint a painting and you want to paint, you know, uh, you know, you have some sort of vague vision for the painting that you want to paint, but you want to paint a painting. So the, uh, the incremental way to do it would be to paint exactly the painting that you want one small piece at a time. So you, if it's a portrait, maybe you'd paint the face and then you'd paint the hair, and then you'd paint the body and then you'd paint the background. Right. Uh, the iterative way to do that would maybe to start with a drawing, and then you do like a drawing with some color. Right. You throw out the drawing and you use the drawing with some color. Or maybe you do what a lot of famous artists do, and you actually paint over top of your drawing.
Matt Godbolt: Right? Right.
Ben Rady: But you do a drawing and then you add some color, and then you add more stuff on top of that, and then you sort of refine the, the lines and, and make it more clear, or, or, you know, maybe even if you're, uh, an impressionist, maybe you intentionally make it more blurry or whatever it's that you're trying to do to achieve the outcome of your painting. Right. Right.
Matt Godbolt: Okay.
Ben Rady: Um, and that is, that is the difference, that is I think, a good example of the difference between, uh, an an iterative approach where you're sort of doing the same thing over and over again and making it better every time.
Matt Godbolt: Right.
Ben Rady: Versus an incremental approach where you're building big things by building lots of little, small things.
Matt Godbolt: Got it. That makes perfect sense. Um, and it maps, now you've said it, I know we discussed this literally for 10 seconds before we started recording, that really strikes a chord with exactly what I'm trying to do in my day job right now, which is like, I am definitely going for an iterative approach over an incremental, because I really, really, really want to see the sketch of the whole system.
Ben Rady: Mm-Hmm.
Matt Godbolt: Some very vague sense before I commit to saying this is the right way to do it. And then go back over and like, color in between the lines and, and maybe replace little parts of it as opposed to saying, here's my six month plan and week two of, of, of, of month two, we'll be doing this. This is like the next incremental chunk of the work. And I can't predict that right now, but what I can do is do everything in six days, , and then try doing it. And, um, everything over six weeks and then maybe six months, that kind of feel.
Ben Rady: . . Yeah. And it's, and it's tricky. And, and I think it makes it even more confusing when people talk about this. 'cause quite often people work, uh, doing kind of both of these things at the same time where they will do, uh, an an iterative approach where they will, you know, sketch out sort of little, you know, the, the sort of steel thread or the skeleton where you
Matt Godbolt: Tracer bullet where
Ben Rady: You have like, all, all Yeah. You sort like all the little pieces and then like, we'll sometimes iterate on each of those pieces individually.
Matt Godbolt: .
Ben Rady: And then also we'll sometimes work through them incrementally. Right? So you'll like build out your skeleton, and then once you, uh, and then then once you have the skeleton, you'll be like, okay, now we're actually gonna go through and incrementally improve each part of the skeleton until it, uh, is up to the, like, the standard that we want, or maybe sufficient to deploy or sufficient for some particular use case. Uh, and then you come back and you then refine those things iteratively again. And so the, this whole iterative and incremental phrase just kind of gets lumped together.
Matt Godbolt: Mm-Hmm.
Ben Rady: Because people tend to, uh, do both of those things. But it is sometimes like very important to make a very clear distinction about like, you know, are we doing this incrementally or are we doing it iteratively? Right. So, like in general, in my mind, if you're not really sure what the problem is in the first place, you want to do it iteratively.
Matt Godbolt: Mm-Hmm.
Ben Rady: Right? Like, you want to make sure that you actually understand like, am I trying to make a portrait of a woman or am I trying to draw, you know, a cartoon, right? Like, I don't, I don't actually even know yet. Right. Like, I'm not sure. Um, whereas certainly if you're going to try to subdivide the work, uh, one way to do that is to think about the, uh, sort of incremental steps that you can take and can they even be subdivided? So there are certain situations in which they can't, like one increment depends on the next,
Matt Godbolt: Right? Yep.
Ben Rady: Right? So you can't actually subdivide the work, but sometimes there are ways where you can, you can break things down into steps and you can actually do some of those steps ahead of time. This is maybe like a little bit like a, a branch predictor in a way. Where it's sort of like, we're
Matt Godbolt: Absolute, yeah.
Ben Rady: Gonna have to do all these things in order, but we probably are gonna have to do that one. So we're gonna lead the target a little bit, and we're gonna start doing that first, even though, you know, technically
Matt Godbolt: or, or because I have resources that I, I, you know, my team is four, four strong. And although we don't have a cl totally clear idea about what the, the project as a whole is gonna look like, I'm
Ben Rady: mm-Hmm,
Matt Godbolt: 80% sure we need something which looks a bit like X, so we might as well start doing X now. And maybe that's sounds more like a hybrid approach then when you've broken into pieces that you can, I guess this is what you were saying earlier, you have like the iterative and incremental, you can sort of like iterate on, uh, each individual component having like decided to do them. , uh, yeah. Sorry, I've confused myself. . Yeah. Yeah, yeah,
Ben Rady: Yeah. And you can do that serially, or you can do it in parallel. You can, you can incre you can iterate on each increment one at a time through all the increments. Yeah.
Matt Godbolt: Yeah.
Ben Rady: You can try to break up the work across multiple increments and hope that they all tie together. And, you know, sometimes you can get that right. You can iterate on all of the increments taking a pass over, all of them, one at a time, which is basically just iterative development.
Matt Godbolt: Right. You know,
Ben Rady: You're just doing a whole Passover, the whole thing in each, each time. Um, and you can, and again, and this is why people use this phrase, iterative, incremental, because you're almost certainly doing some kind of things, some
Matt Godbolt: Combination of them. Right. And I suppose
Ben Rady: Without even necessarily realizing it,
Matt Godbolt: I suppose there's sort of a, a, a fractal or recursive thing here where you might say, I'm gonna hack together the worst possible version of this thing. Uh, and I'm gonna say over the weekend that wouldn't, you know, but like in a way where you'll say like, okay, one person's gonna go off and spend a couple of days spiking, uh, genuinely spiking out the whole thing and kind of going like, yeah, we're gonna need a reliable messaging system. And so I just use TCP for now, but that won't scale because of X. We, we need, you know, I need to be able to set notify people, but I'm just using printf right now. Um, you know, monitoring was non-existent, but I'm sort of printing out a heartbeat every two seconds. And if, if it goes down maybe, and, but I got it working and now I know what the components are, and so now Mm-Hmm, , I can break it into the four pieces, and then we can do take two, four at a time, agree roughly on what the, the handover parts are.
Matt Godbolt: And then we are parallelizing four ways. And then, and it could potentially continue to break down where you're like, oh, actually are our reliable transport mechanism needs. Actually, it's a, it's the size of a whole project. It turns out we didn't think it was, yeah. We were just gonna buy in Kafka or whatever, but, um that's not working for us for whatever reason. We need to bring in some exotic magical hardware that does something, whatever. You know, that's, that's the kind of discovery part of it. But that is an incre. No, that's an iteration. See, I can see why you get confused. Right. I'm actually having a problem now. Even though it made sense when you described it right at the beginning.
Ben Rady: I know, I know. It's almost unfortunate that that's like the term that people kind of came up with for these sorts of things. Like, you, you like removing the alliteration would've been nice.
Matt Godbolt: Yeah. I guess progress, you know, in terms of, I, as soon as you started describing this, I was thinking the different, you know, you're used to painting analogy here, which is, which is a good one. And you know, one, one thing that, um, some old school, you remember modem level, um, image formats would, they would send it to you progressively. That's like, they would send you every fourth line of the image so that you could kind of fill in the gaps on the client side while you were waiting for your 14 K four modem to download the rest of the png, right? . But you could see something and it was a blurry version of it, and then it would refine as time went on, and that, that was referred to as progressive, like a progressive download of the, the image.
Matt Godbolt: And so that kind of feels to me a little bit like, again, um, iterating on the image and continuing to, to progressively refine it over and over again. Similarly, like, you know, fractal rendering systems typically did this thing where they would like do a big pass over the whole screen to generate your mandelbrot. Right. And then they would go back in and fill in where all the detail was. 'cause I mean, that was different reasons, but meant you got like a pretty picture straight away and then, you know, you left it longer and then you could print it out and see some really high resolution thing. And that's a similar deal. So it would've been nice to have used, as you say, not an alliterative or even similar sounding word.
Ben Rady: Yes, yes. Exactly. Exactly. Um, but yeah, I mean, I think all of these, these things get, uh, even if you can't keep the words straight in your head, and as I said before, I, I mix them up all the time too, keeping the concepts straight in your head is important. And it's important when you're kind of in the, the stage, which I, I think you're right now where you're planning out a whole bunch of work and trying to figure out like, okay, how are we going to do things like discover the, the, uh, the unknown unknowns? Yeah. The rumsfeldian unknowns.
Matt Godbolt: Rumsfeldian. Yeah.
Ben Rady: Yes. The things that we don't know, we don't know. Right. Um, because, you know, whenever you're building any sort of sophist, you know, reasonably sized system, they're going to be those things, right. There's gonna probably be a lot of those things, and especially in the early stages, like your goals are in many cases, to kind of use your development process as a catalyst for discovering them, right? Like, you want to find forcing functions where it's like, well, if we can make this thing talk to that thing, then we'll know that this part, that there's no more unknown unknowns there. Right? . . And so like, if you were doing things, um, purely incrementally, right? You might not do it in that order, right? You may not work on that piece first because it like, doesn't deliver any value by itself. Right?
Matt Godbolt: Right.
Ben Rady: Um, but because you're trying to do it, you're trying to discover this thing, you wind up building this out, and it becomes this sort of literal little iterative slice where you're sort of like, okay, we get em to talk to each other. What if we try this? If we tried that, what if we tried this? What if we tried that? . And then you finally get something that works, and now you've kind of like knocked out all of the unknown unknowns from that section of the problem. Right?
Matt Godbolt: Right.
Ben Rady: Um, so I mean, you know, like some of this stuff, if you are the type of programmer who, you know, just wants to be handed a list of things to do and doesn't really want to think about how they all fit together, uh, and doesn't, isn't really responsible for, you know, coming up with a, with an architecture or a large scale plan for, for projects you might not think about very frequently, and you might not care. You just be like, incremental, iterative, I don't know. I'll just write code. Like, why are you making this complicated?
Matt Godbolt: Right, right.
Ben Rady: Um, but if you're trying to organize the work of many people, or if you're trying to build systems that are, uh, kind of like inherently risky, like no one knows that you can actually accomplish what it is that you're trying to do, then you can save a lot of time and money by thinking about like the order in which, and the sort of progressive, uh, quality of which you are going to organize your work.
Matt Godbolt: Yeah, that makes an awful lot of sense to me. Yeah. And, and as you say, it couldn't be more apropos for my current situation where I am exactly doing that right now, and people are asking me, left, right, and center, you know, what do we do? What, what needs to be done? And, and my, my my, uh, sort of ostriching of it is like, we're just gonna build something first, and then we can argue about whether or not X or Y is possible or not. Let's do a minimum, you know, this is a minimum viable product style approach. Which again, it has that same feeling. It's like, can this be accomplished? And in doing so, what do we learn about what the things that we need to, uh, to, to continue doing are, are, or what things we need to, um, work on subsequently? Sorry, that's, this is gonna be a fun to edit one.
Ben Rady: I've got a whole bunch of footsteps that are showing up on my audio, so, I don't even know what to tell you at this point. Right.
Matt Godbolt: We might, we might, there won't be. So listener if you hear anything weird going on in the background there. That was, uh, my bad attempt at editing as much as we can. The noisy world that we live in.
Ben Rady: I feel like I should apologize to editing Matt, right?
Matt Godbolt: Oh, yeah, yeah. Well, that's right. Editing Matt would be fine. He's, he's, uh, he's off in the future. He's, uh, he's having a great time. He's probably on the train, actually. I've started editing these on the train and it's been a quite a good experience, although it has put in sharp relief the, uh, the, so I, I wear my, my, um, Bluetooth headphones for, for doing it on my laptop, and I run audacity.
Ben Rady: Mm-Hmm. .
Matt Godbolt: This is a complete non-sequitur from what we're talking about before. But maybe it's interesting to people, I dunno,
Ben Rady: That's this whole, this whole podcast is a non-sequitur.
Matt Godbolt: It is, isn't it? Um, so I put my headphones in and you get exposed to the lag between your laptop and the headphones. You forget, there's so much buffering going on. You don't notice it on your phone when you hit play and you put your phone away in your pocket and, you know, music starts playing and you know, or your alarm goes off and you pull out the phone, you're like, oh, yeah, whatever. Um, but when you are like staring at a, um, waveform of what you said and you're look like trying to line up things and kind of go, all right, when, when was the, um, start? Is it here or here? Although that's usually pretty obvious to see. It's really obvious that there's a gap between where the thing says it's playing and where you, what you can hear and you kind of like stop it and the music carries on the music, the, the sound carries on a bit longer. So that was an interesting discovery, but it doesn't affect materially affect the editing system, editing setup.
Ben Rady: Yeah. Um, One other point I could probably make about this, um, is that I think one of the dangers of working in this way, which I think is how most modern software development happens, but you know, as opposed to a non incremental and non iterative approach where you would basically be like, uh, we're gonna plan out all the software that we're gonna write and then we're gonna write it. Right? Right. And that certainly is very, if you want, if you want to optimize for breaking the work apart into lots of, you know, independent groups and doing it in parallel, if assuming you get the first part right, that makes that incredibly easy, right? Like you just say, we're, this is all the software we're going to write, and team or person A is gonna write this part, you write this part, you write this part, uh, and then you integrate them all together at the end and it works great, right? Um,
Matt Godbolt: Never been a problem.
Ben Rady: But assuming. Yes. Exactly. So assuming that you're not building software in 1983, uh, and you're not doing that, then one of the traps that you can fall into is thinking of, and I think this is more a problem with iterative development than it is incremental development. And I'm gonna tax the brains of our listeners to recall what what those things mean.
Matt Godbolt: Which one's, which
Ben Rady: Again, very, very briefly here, but I think this is more of a problem with iterative development, where you can fall into a trap of basically just writing terrible code, right? Because you're just like, oh, well, this is just the first iteration and we're gonna come back and we're gonna clean this up. And one of the things that I have learned working iteratively on many different projects, many different times, is that you'll be very well served by operating under the assumption that this will be the last iteration you will ever get to do.
Matt Godbolt: Wow. Right?
Ben Rady: So what, whatever capabilities and whatever quality and whatever level of, uh, sophistication you decide that you want to achieve in this iteration, you should figure out what that is and you should stick to it. And you should not fool yourself into thinking that, oh, we'll fix this later. Right? So if you build the capabilities into the software that are sufficient to meet some need, your expectation should be that it will immediately be used to fulfill that need and that there's not going to be some other period of time where you can come back and, you know, document all the things or refactor the code or write the tests or fix all the corner cases that are gonna show up every once in a while. That's just not how this stuff works, right? Like when you're working,
Matt Godbolt: As soon as it's good enough, it will end up in production
Ben Rady: As soon as it's like you're working iteratively and you're making it better and better and better. And there's going to come a point where it's like, oh, this is good enough now, and that's the production system, and you might get another chance to do more, uh, iterative, uh, improvements to it, but you might not.
Matt Godbolt: And so you have to, it's treat it seriously, you know, although it's a, yeah, don't be thinking it's a prototype. That's perhaps one thing you could do where you sort of put everything together and go, well, I just hack it together,
Ben Rady: Yeah
Matt Godbolt: Treat it as If it's like, Uh, Uh, a cut down. But so as you say, suddenly you'd be okay going to production.
Ben Rady: Yeah. And I, and I mean, I think this, it really informs like how you spend your time in those iterative steps. Like, again, it kind of gets back to like, what are you doing with these iterative steps? Are you trying to like find those rumfeldian unknowns? Because if you are, don't spend the time to like integrate it with the other thing that's not particularly relevant, that won't give you that rumfeldian unknown or like write the, you know, main function that'll actually like, deploy it or anything like that, or, or things like that. Um, because two reasons, one is that's not what you're trying to figure out right now. And two is if you make it too production-y, it might actually accidentally go into production. So spend your time on the things that you're actually trying to prove out and make sure, I mean, there's sort of a, a gray area-ish way to say this where you sort of like, don't make it too good, I guess is sort of almost how I would say it.
Matt Godbolt: Be careful. Yeah. Be careful how good you make it.
Ben Rady: Be careful you don't make it too good too quickly in the wrong dimensions accidentally. Right. Is kind of how I would would say that, right? Like,
Matt Godbolt: Yeah
Ben Rady: Very intentional about what your iterations are and what it is that you're trying to figure out and don't spend your time on things. And you might have people that are telling you, it's like, yeah, can you, can you deploy that to our server? It's like, no, I'm not gonna work on that yet. We, we know we can deploy it. That's not what we're gonna do. Now I generally am a person who says deploy first, and that's a, a process that I like to follow, but you gotta think about the environment that you're in and, and sort of like what you're actually trying to achieve with these, with these iterations.
Matt Godbolt: No, exactly.
Ben Rady: So you, you kind of have to go into this, or at least I, my philosophy at this point is you, you have to go into this with this idea that when you finish with one of those iterative steps, that might be the last, and you have to be able to walk away from that code and, and be like, yep, this is going into production like this, or this is never gonna be touched again for six months because we've discovered that there are unknowns here that we don't, can't deal with, and we're just not going to do this project at all. And so I'm just gonna put this on the shelf and I'm never gonna worry about it again until, you know, three years later and they're like, oh remember when, we did that project. Let's go back to that.
Matt Godbolt: Yes. Yeah, yeah, yeah. But I guess you have to be prepared for that, right? You have to go into that sort of mentally and emotionally ready to say like, Hey, we might discover when we're doing this process that there's actually the kraken sleepeth under the, under the waves here, and oh, now we, we, we dunno how to solve this and now we can't do the project as a whole
Ben Rady: Mm-Hmm, . And I feel like that's less of a problem when you're doing things incrementally, right? Like if you have some system that you're trying to build and, you know, let's just say four components and you build components one, two, and three and you haven't built four yet, it doesn't work yet, right? Like maybe you've done an incremental and iterative approach and you've got like three very well-built components and you've got one sort of stubby prototype-y component that maybe if like we don't lean on it too hard, it could maybe go into production. But generally if you're doing things in a purely, uh, incremental way, you don't have that problem because until you have all of the increments, you don't have a running piece of software,
Matt Godbolt: Right? Right. And so like in, in, in the situation the system that I'm looking at right now, the sort of access that I'm not worrying about is performance. So I'm just like,
Ben Rady: Mm-Hmm, ,
Matt Godbolt: That is a place I can come back and refine later on, but I do need it to be functionally not necessarily correct, but functionally in the right ballpark and show that the capabilities could be added to. And then once we've got to a some point in the system, it's like, well now it's just expanding out in a different direction. It's like expanding horizontally in terms of like the features. Well, we can do, you can do add ads and subtracts now it's like, yeah, but ads and subtracts feel like the same kind of thing. There's no surprises in subtracting as well as adding and multiplying and dividing or whatever you are doing right. But showing that you can have something which can do computations of that flavor is good enough to be able to say, I can get this to production. Everyone think it's fine. Um, and now, um, we can, uh, refine that by running another, uh, an an iteration over the top of it.
Ben Rady: Yep. Uh, one more, I think one last point on this one place where, uh, incremental development can be especially powerful is when you have an existing system and you're trying to add new capabilities to it, right? So you've got the system, it works, it's in production, whatever you're trying to do, and you're trying to figure out like, okay, we, we've got 35 different things we could add to this. What should we do next? Well, what you probably don't want to do is an iterative pass across all 35 things, right? , what you probably want to do is pick one thing and build that one thing incrementally and add that to the system and get that into production, and then go back and be like, all right, we've done one of 35. What's next? Well, actually we came up with another 20, so now there are 64 things, right? Right,
Matt Godbolt: Right, right.
Ben Rady: Which of the 64 things should we do now? Oh, we'll do this one. Okay, cool. Yeah. Right. So, so that, uh, that can be another sort of, uh, valuable way to think about, um, organizing that type of work is you could say like, in the early days, we're maybe gonna do more iterative development. And then once we've got a a working functional system, we're gonna switch to a more incremental approach. And, you know, again, that's not to say that you don't mix and match these, that you don't do iterations on your increments.
Matt Godbolt: Right, of course.
Ben Rady: But when you're, when you're sort of looking at the, the total, you know, scope of work, like it can sometimes be useful to sort of think about those things in, in two different modes.
Matt Godbolt: No, it makes, makes a lot of sense. Well, today I learned the difference between those two words that I'm not even gonna say out loud because I'm gonna get them the wrong way round again, incremental and iterative or
Ben Rady: Yes.
Matt Godbolt: Refining and uh, uh, and building piecemeal. I dunno if we can come up with better terms than this, but
Ben Rady: Yeah. What, okay. Yeah. So let's think about this. What, what should we actually call these things? I like progressive, like for, for so iterative and progressive to me, like I like that substitution, right? So you could just say progressive and incremental.
Matt Godbolt: Yeah. Yeah. Right.
Ben Rady: I think that's, I I like that better. Progressive and incremental.
Matt Godbolt: Progressive has other overtones though. That's the thing probably that,
Ben Rady: You know, you think of insurance.
Matt Godbolt: Yeah. Either that or Yeah. Or, or else, uh, someone with maybe, uh, on the political spectrum or politically Yeah, yeah. Which sometimes yeah. So
Ben Rady: Progressive enhancement. That's too long though. Yeah. I mean, anything would be better than iterative and incremental. They're just like, they're basically like the same word. The levenshtein distance between those two words is not high enough.
Matt Godbolt: Exactly. They're the same, but they are very, they're different, but yeah. Well, anyway, I don't think we need to solve the naming problem right now because, you know, as we know, that's one of the, the two hardest problems in computer science.
Ben Rady: Uh, you, you forgot the third one off by one errors.
Matt Godbolt: Oh, .
Ben Rady: Yeah. No, that's, I agree. Agreed.
Matt Godbolt: I think that's probably a good place to, to stop this today. And, uh, yeah.
Ben Rady: Yeah. We'll, uh, let's call here.
Matt Godbolt: Well catch up next time.
Ben Rady: Sounds good.