Matt Godbolt: Hey, Ben.
Ben Rady: Hey Matt.
Matt Godbolt: We should really not be talking over the intro that I edited out. We should be more professional about this thing.
Ben Rady: I don't know. It can be a good source of outtakes.
Matt Godbolt: It's better than me singing, I suppose. It gives me the cold opens and cold closes, which is worthwhile.
Ben Rady: We'll do the cold open at the open. That'll be a new interesting way to do a cold open.
Matt Godbolt: A new interesting way to actually just launch into one and go, "Oh, I suppose we should press the button to make the theme music play now." I was actually thinking about arranging the theme music for clarinet, cello and drums so that me and my kids could play it together. But, you know, given that they think everything I do is the most terrible, boring thing to do, I don't imagine that will go through.
Matt Godbolt: But anyway, that's not what we're here to talk about today.
Ben Rady: We can talk about anything that we want. That's the amazing thing about this podcast. No one can stop us.
Matt Godbolt: That's very true. What power we have! You and I already discussed this when we went out with friends last night. On the walk to the bar, you suggested this topic and we had a little pre-game. So I think we should talk about it - and that topic is about hands.
Matt Godbolt: And you've spoken about this as one of your little aphorisms and sayings.
Ben Rady: So the one that we're going to talk about today is "dirty hands are right." This comes from an internet thing - this is before memes, I feel like.
Matt Godbolt: I mean, memes in their cultural, appropriated sense, as opposed to the scientific sort of Richard Dawkins-ish memes.
Ben Rady: The Cult of Done, yes. This was like a set of ideas invented by a person named Bre Pettis. They basically sat down one day and came up with approximately 10 rules for getting things done. It took them like 25 minutes. They're like, "I'm just gonna knock this out and these are gonna be my rules." I actually have the poster over my-
Matt Godbolt: I was going to say there's a- I can't quite see it because of the blur, but I know it's there from having been to your house. It's the Done Manifesto, right?
Ben Rady: Yes, the Done Manifesto. The Done Manifesto is the manifesto of the Cult of Done. This was introduced to me by a friend of mine back in, gosh, 2008 something like that. The thing that I love about it is it literally was created in like 20 minutes.
Ben Rady: It's not super well thought out, but that's kind of the point - it's about getting things done. It doesn't have to be super well thought out. It just needs to be done. Then you can move on from it, learn from it, improve it. But if you don't ever finish it, then you can't do those things.
Ben Rady: You can Google this, listener - the Cult of Done, the Done Manifesto. One of the things in the Cult of Done, I'm going to try to quote this directly from the manifesto: "People without dirty hands are wrong. Doing something makes you right."
Matt Godbolt: That's actually interesting because that's not how you've said it to me before.
Ben Rady: Yeah, I've said it as "dirty hands are right." That's the shorter version of it. 04:21.77 Matt Godbolt Right, but there is there's a subtle difference.
Ben Rady: I wanted to talk about this because I've repeated it many times since learning it, and my thinking has become more nuanced, perhaps like the original quote. It has shifted because I think if you want to apply this in an effective and useful way, you need to really understand what it means to have dirty hands. Being able to walk around saying "dirty hands are right" is a great way to make a lot of mistakes very quickly if you're not careful about wielding that power.
Matt Godbolt: Got it, so should we talk about that simplified version first? Forgive me for simplifying further, but what this essentially says is: if somebody's in the middle of doing something and they say "we should do it this way," and you aren't in the middle of doing it - trust them. They're probably the right person to be doing it. Don't throw stuff in from the peanut gallery just because you might have an opinion. Don't bikeshed. Unless you're doing it yourself, you have no horse in this race. Shut up and let them get on with it, right?
Ben Rady: Exactly right.
Matt Godbolt: And that's how I've taken it from all the things. From that point of view, I understand why it gets things done, because it means you don't have this constant interruption from other people saying, "Have you thought about doing this?" And you're like, "No, I haven't. And I'm not going to." I chose to do it this way, and I will therefore get this thing done, and then we will move on to something else.
Matt Godbolt: Obviously, everyone can think of reasons why this isn't always true, but it's not a bad first approximation for a startup environment like we've been in for a while. Let's just say: if you're interested in doing it and you're already doing it, you keep on trucking and we'll trust you to get on with it.
Ben Rady: Absolutely right. If your goal is to get things done, this is very effective. If your goal is to be SOX-compliant, this is maybe not the approach you want to use. But I think it's actually more interesting to talk about the exceptions. When are dirty hands wrong?
Matt Godbolt: When you're preparing food - that's right! It's a slightly silly example, but...
Ben Rady: When you're preparing food is a great example of when dirty hands are wrong. You're in the kitchen, you just took the trash out and you're like, "I'm gonna cut up this chicken anyway..."
Ben Rady: The SOX-compliance thing is definitely like, yeah, you can get things done and then you're not going to be compliant. But one of the more important nuances that I've added into this, especially in the last five years, is understanding the distinction between the hands that are currently dirty because they are building a thing, and the hands of the people who will be expected to support that thing. If those are not the same hands, then that is a situation in which dirty hands can be wrong.
Matt Godbolt: Interesting. This is like the typical support engineer versus productionization engineer scenario. At Google, you have the SREs who are supporting it. Way back in time, it was like engineers threw it over the fence, and the SREs just had to deal with the fires. They pushed back, and eventually, it was like: "No, before you can throw this at me, the SRE has to sign off on it," or "You also get to go on the pager duty rotation for that thing" so that everybody learns. Hopefully that distills a correctness out of the process. Just because "Hey, I'm the guy who wrote it, I'm right" - that doesn't follow when I'm not the person who's going to be woken up at 4:00 in the morning when the memory leak finally crashes the machine.
Ben Rady: Right. The problem with that transition of "I'm going to build something with dirty hands and then take my filthy dirty hands and use them to hand this thing to you" - just mixing metaphors here for fun...
Ben Rady: It's like "free as in goldfish."
Matt Godbolt: I've never heard that before, is that yours?
Ben Rady: No, I got that from somewhere. I forget where.
Matt Godbolt: Free as in goldfish, yeah - free as in puppy. Here you go, this is yours now, but it comes with a massive amount of responsibility.
Ben Rady: Right. I've given you this thing and really it's like a curse. Now you are responsible for this, and all of my mistakes have become your mistakes, which is great for me and terrible for you.
Matt Godbolt: Yeah. Absolutely awful for you.
Ben Rady: I think the main way to avoid this organizationally is just: don't do that. Don't let engineers absolve themselves of their own sins by passing unsupported software off onto other people. I'm going to couch this a little bit - I don't understand how to apply the software engineering techniques that I know and have been successful with to that kind of SRE model, where you have software engineers that build software and then hand it off to SREs to support them.
Ben Rady: You can certainly build software that way. I'm not saying you can't. I'm just saying I don't know how to take all of the useful, effective techniques that I have and fit them into that model. They all just completely fall over when you're like, "Yeah, I'm just going to build a thing and then hand it off to these people, and then I'm going to go onto my next project and never worry about it again."
Ben Rady: So, one way to avoid this failure of "dirty hands are right" and allow dirty hands to be right is to just don't let people do that. Or maybe more realistically, don't structure your organization to do that on purpose.
Matt Godbolt: It is, as I say - perhaps we should talk about the original phrasing in a second, but I can think of another example where dirty hands may not always be right. I'm definitely trying this on, thinking this through. Folks who are inexperienced, either at the particular task at hand or at software engineering in general...
Matt Godbolt: I mean, I can't help but feel like maybe I'm just a grumpy old man who wants to tell these young whippersnappers what to do.
Ben Rady: As previously stated, this is our podcast and we get to say whatever we want.
Matt Godbolt: I know, but it doesn't come without repercussions. I like to be thoughtful about these things, as you are as well. So with that in mind, it is the case that you can have a junior developer or somebody who isn't a software engineer develop something, and you look at it as a seasoned engineer or somebody with more domain experience and go, "Wow, you've really made that hard on yourself."
Matt Godbolt: And at that point, those dirty hands, they're like, "Well, we've just done it this way." This is where we are now. It's hard to argue that they're right. I mean, they're right as much as they've got something done, presumably. Maybe that's how you get out of this. Maybe it's the wiggle words - it's like, well, they're done with V1, and now we need to think about this again as a V2.
Ben Rady: I think that point is absolutely correct. Any time that you put someone in a situation where they're not capable of doing the task that you've given to them, you need to think about how you're gonna make it possible for them to succeed. If you want to operate an organization with this mantra of "dirty hands are right," then when you give someone a task they may not be capable of doing, you need to find a way to support them. That cannot be what we were just talking about where you've got some seasoned engineers saying, "Did you think about this? Have you considered this? Have you bikeshedded this?"
Matt Godbolt: Like either at the very end of a pull request when you've designed a system and got it all wrong, or as you say, fly-by - literally someone posts in Slack and says, "Hey, I finished my thing," and then everyone takes a pot shot at it. That's not a good feeling.
Matt Godbolt: So you have to get the seasoned engineers' hands equally dirty. Is that where you're going?
Ben Rady: Yes, and in the right way. It depends on what you're trying to do here. I think most software engineering organizations value developing the skills of their engineers. Not all do, but the better ones do. You can use this as an opportunity to teach those junior engineers the things that they don't know that make their dirty hands decisions bad.
Ben Rady: And you have to do it in a way where you're not just inflicting your help on them. You need to give them a goal, tell them that this is too much for them.
Ben Rady: This is this game on hard mode.
Ben Rady: And if I just left you alone, you wouldn't be able to do it well. You're smart, maybe you're going to figure out how to do it, but you're going to make a lot of mistakes if we just give you no help.
Ben Rady: So we're going to give you some help, and we're going to structure it in this way.
Ben Rady: The particular way that you structure it depends on the person, the organization, and a lot of other factors. The key element here is if you just overwhelm somebody with a task that is way too hard for them, they're going to compensate for that in ways that are bad for everybody.
Ben Rady: And one of those ways is to make a bunch of dirty hand decisions that the organization is going to be very upset that they made.
Matt Godbolt: This comes to things you've given presentations about at work - you've talked about the zone of proximal learning. Is that the right term?
Ben Rady: Zone of proximal development, yes.
Matt Godbolt: Right, which is like the Goldilocks zone of giving someone something they couldn't achieve on their own, but with a little bit of help, they can achieve. That's the best place to be learning - stretching yourself a little bit without being overwhelmed.
Ben Rady: You want to be whelmed. Not overwhelmed, not underwhelmed, you want to be whelmed. That's when you're learning.
Matt Godbolt: Obviously, it requires a certain amount of forethought to give them a task which is stretching, and that you have somebody available to be equally dirty handed in the correct way.
Matt Godbolt: That doesn't always happen though. I'm thinking now in more general terms where folks who are smart and wanted to just get a job done have gone off by themselves and gotten that job done. Then you're stuck with the decisions they made. Like, pragmatically, we needed a solution to store our files, so I made a directory on the network share, and I stored 250,000 files in one directory. And it works for me.
Ben Rady: That's exactly right. That's not a junior software engineer. It might be like a senior analyst who learned a little bit of Python just so they can automate some parts of their job and then built this thing.
Ben Rady: And you come along one day and you realize that the operation of the entire company depends on this untested 1000 line Python script where half the code is commented out.
Matt Godbolt: I remember the very first day I interviewed at the finance job that brought me to America. I was speaking with one of the people in their London office and I said, "Oh, what are you doing?" expecting to hear some kind of cool debugging story. They said, "Yeah, I'm debugging this spreadsheet."
Ben Rady: When I'm trying to explain to people that are not programmers why it's so important to reduce complexity in software, and why we spend so much time with techniques and tools trying to reduce the complexity of our software, they ask why this is so important.
Ben Rady: I tell them to imagine they had a spreadsheet that was responsible for their entire company, and there was a whole bunch of stuff in it that they didn't understand, but they were tasked to change it.
Ben Rady: Our code base is 150,000 lines. Imagine that spreadsheet was 150,000 lines and not 150,000 lines of data - 150,000 lines of functions.
Ben Rady: Formulas that refer to other formulas that refer to other formulas. That's what we're dealing with. And if we don't keep it clean so that people understand how to change it, we'll not be able to change it.
Ben Rady: So the next time you come to us and say "we need to add this thing," I'm going to be like "no, I can't."
Ben Rady: I literally can't because every time I change this cell, all these other cells break.
Matt Godbolt: And we don't understand it and it's too complicated.
Ben Rady: But to your point - you have that person that is not a junior engineer eager to learn, that wants to be involved. It is a person who is just trying to do their other job, which is not software engineering, and they have found a way to automate some things. They asked ChatGPT a question and they paste the answer into whatever.py.
Ben Rady: I think that is absolutely a case where dirty hands are not right.
Ben Rady: Interestingly, this is a facet of another situation in which dirty hands can be wrong. And the hands in question in this case can be someone like us, a very seasoned professional software engineer who's spent a lot of time doing this. And it's when the scenario is flipped.
Ben Rady: Imagine that you have the interest rate calculation code, and you don't really understand the calculation that's going on here. There's some analyst or an accountant who is an expert in pricing this thing.
Ben Rady: Maybe not an industry calculation, but like a pricing algorithm.
Matt Godbolt: About pricing options or something complicated.
Ben Rady: If you go off and try to build that solution without really involving that other person, you're almost certainly not going to do it right. Or you're going to design it in a way where maybe it fits the one particular use case that you thought about, but if you want to adapt it or use it for something else, it won't work.
Matt Godbolt: You're missing the context to be able to generalize it in a meaningful way.
Ben Rady: And that has other dimensions too. Like imagine a security aspect, or a privacy aspect where you may be an accomplished software engineer, but if you don't understand the legal ramifications of whether or not we obfuscate this field or that field, or the international law around cookie storage.
Ben Rady: Then you can be a very good software engineer and have dirty hands and be very wrong.
Matt Godbolt: So really, the dirtiness is a function of responsibility and skill. There's a threshold by which you have to have a certain amount of skill and responsibility to be able to claim the dirty hand mantle. All the things we talked about until you brought in the analyst or the security expert have been about programmers having dirty hands, and if they've got enough seniority or experience then they can prevent the bike shedding that could happen when you have four or five other smart people with reasonable ways of doing it.
Matt Godbolt: Your analyst isn't necessarily a good programmer and so their hands don't necessarily qualify for the dirtiness in this case.
Matt Godbolt: But equally, we're not saying this is unique to programmers.
Matt Godbolt: It's just like, if this isn't your main role, if you're not expected to be doing this, then maybe you don't count in the dirty hand analogy.
Matt Godbolt: Let's go back to reading the original thing, because I think we've got the inverse of the actual phrase.
Ben Rady: It says "people without dirty hands are wrong."
Ben Rady: Doing something makes you right.
Matt Godbolt: If we just take that first half, that sort of says, if you've got no skin in the game at all, then you just can't be right by definition.
Matt Godbolt: That stops the peanut gallery. That stops the pot shots from other people coming along saying, "Have you ever written a database before? No?"
Matt Godbolt: Then stop telling me how to manage files because I'm doing it right now.
Ben Rady: That is the condensed version. The second sentence is "dirty hands are right. Doing something makes you right."
Ben Rady: But focusing on that first part is probably the more universally true thing. If you're not involved in something, don't just offer your opinion because you feel like you want to offer your opinion. If you don't have any skin in the game, if you're not the person who's going to get rewarded if it succeeds or yelled at if it fails, then be careful.
Matt Godbolt: Unsolicited, because obviously, if somebody comes to you and says, "Look, you don't know what I'm doing here, but can I use you as my rubber duck to debug or to bounce ideas off of?"
Ben Rady: We're going to give you some dirty gloves that you can put on.
Matt Godbolt: So the original thesis of this discussion as we walked to Monks Pub last night was, "When are dirty hands wrong?"
Matt Godbolt: It's interesting because it's definitely something that we have trotted out over the last few years as being like, "Let's just stop bike shedding about it."
Ben Rady: I think it's a really good way to think about the importance of building software in teams. A lot of organizations like to build cross-functional teams.
Ben Rady: You'll have a team made up of two or three software engineers, maybe a trader, an analyst, and a risk expert. Or you'll have a product team with a UX expert, a couple of front-end developers, a couple of back-end developers, and a security expert.
Ben Rady: These cross-functional teams, if you structure and align them right, saying "you're going to build this thing together, you need to work together, you need to come to consensus" - that's when it works.
Ben Rady: The one senior engineer by themselves isn't going to know the pricing algorithm, but when you put that person on the same team as someone who has that knowledge and you have everyone working together toward a common goal, that's when it works.
Ben Rady: You collectively have dirty hands. You collectively have ownership of the outcome.
Ben Rady: If it's not going to work, then you're all going to suffer. And if it does work, then you all succeed. That's a much better model.
Ben Rady: For organizational staffing and resourcing reasons, that can be difficult to achieve. Sometimes those individuals might be very limited in their time or numbers, and you can't put them on every team because their goals get divided.
Ben Rady: You can't always do it. But when you can, it solves a lot of the problems with dirty hands. You get all the capabilities and skills needed to solve a problem.
Ben Rady: Together and pointed in the same direction. And as long as people aren't jerks to each other, generally magic will happen.
Ben Rady: Whereas if you have a structure that's more like that SRE structure...
Ben Rady: Where it's "you're not on the same team, you get to go home at night and you don't" - that kind of approach. Then this attitude can be very dysfunctional.
Ben Rady: If you think "I'm doing this and I'm right."
Ben Rady: I don't even know how you do something like that and not have it blow up in your face.
Ben Rady: I just don't understand how you could apply these ideas to that. Maybe someone out there has done it. Maybe they'll tell us.
Matt Godbolt: I think we've pretty much covered the dirtiness of hands. If you've got skin in the game and you have someone on your team that can help you, or you have the skills yourself, or maybe you are helping someone else, then dirty hands are the right approach.
Matt Godbolt: But it's not always the case that just because you're doing a task, you get a free pass on the Dirty Hands rule. If it's not your area of expertise or if you're out of your depth, that needs to be considered. And obviously, if you're preparing food, you should always have clean hands.
Ben Rady: And sing the alphabet song while you wash your hands, people.
Matt Godbolt: Given where we are now, I'm imagining the theme tune is playing over in the edit. So I guess on that note, we should call it quits and chat next time.
Ben Rady: Sounds good.