Matt Godbolt: Hey, Ben.
Ben Rady: Hey Matt.
Matt Godbolt: How you doing buddy?
Ben Rady: I'm doing pretty good.
Matt Godbolt: It's been a while. We've been talking a lot about testing recently. So what do you think, uh, to mixing it up a little bit?
Ben Rady: Yeah, we should, we should not talk about testing today.
Matt Godbolt: I mean, we are quickly becoming, uh, all test all the time podcast, which wasn't really the original intention of this podcast, although, and importantly,
Ben Rady: It's mostly my fault.
Matt Godbolt: Well, I mean, it's kind of your thing, right? You are, you're the test person, or a test person
Ben Rady: Yes, I'm very testy
Matt Godbolt: Testing. At times. No, never testing, never testing. And we, we, when we were discussing this podcast initially, like you and I were thinking, the thing that we would like to do is bring along kind of like audiences that would by default follow us for the kind of things we're known for, and then show them something different, you know, get some people on that can talk about something that maybe they aren't as familiar with or, um, just dig into something that's just interesting to us. Right? I mean, that's, that was the idea. And then we started talking about testing and we carried on, keep talking about testing because there's just so much to talk about. And it's really not very well understood and not very well implemented. And you know, you, and I've seen a lot of it done well and not have it done, not so well. So let's talk about something else.
Ben Rady: Yeah. I want to hear about the, uh, chip reverse engineering stuff that you've been looking at for, Oh, I don't know. Probably quite a while now, right?
Matt Godbolt: Well, yeah. It's one of my, one of my hobbies has led me down a very strange path. And so I, I grew up in like the late eighties, early nineties with a particular, a computer from Britain. And for whatever reason, I've ended up taking a trip down memory lane and writing an emulator for that computer. And the thing about writing an emulator is it's an amazing way of understanding fundamentally what a computer is and how a computer works and how the peripherals work. And if, if, for example, you don't have an instinctive, um, understanding of that, or if you've never actually written an interrupt handler and frankly who has, or that level of, of things, but you're interested in that side of, of computing, like how they actually work. You can go a lot far worse than writing an emulator. It's a great way of giving you that exposure while in the familiar, warm embrace of a modern IDE and sort of debuggers and all the things that come along nowadays. And in particular, I was the, the, the emulator that I wrote for the BBC micro, which was a computer in my school when I was 10, 11, uh, the emulator that I wrote, I chose to write it in JavaScript, which is not noted as a systems programming language. I think it's, fair to say,
Ben Rady: It does have that reputation as not a systems language.
Matt Godbolt: Right. Um, but what it is is, is, you know, universal and more importantly that you can put it in a website and then you can play your video games from your childhood in your web browser. And what's not to like about that, right? But the thing about emulation is that there is a kind of a period at the beginning when you're writing a bunch of code and you're kind of doing it blind, you're just kind of taking it on faith pretty much that you're emulating this particular facet of the CPU. And you've got a big table of these things, right? There's a huge list in your lap, um, of the, the, the opcodes that the CPU runs. And you're like going through them one by one, or you'll hopefully come up with some level of abstraction to make it a bit easier. And then you're writing peripherals and bits and pieces like that. And there's this one joyous moment when for the very first time you load up a ROM and you execute it and something recognizably associated with what you're emulating happens,
Ben Rady: Something game like happens?
Matt Godbolt: In the case of this computer, when you first done it on, this is sort of very familiar to certainly people of my age and from Britain, um, sound that comes out and then a particular sort of text comes up. Like you think of a DOS mode, black screen with just ASCII text on the screen saying BBC computer 32K acorn MOSS basic, and then a prompt. And then that's it right now because the hardware, at least for this particular default video mode is essentially ASCII mapped. That meant that all I could do is I could plumb a routine in JavaScript that just runs through the memory area associated with the screen and dump it out to console dot log. And so I was able to see BBC computer and all that stuff appear in the console before I had anything else going.
Matt Godbolt: So it's a marvelous, marvelous thing to do. And it's so rewarding. And as you add more bits and pieces, you get more functionality. Eventually you get bitmap graphics, and then, then you can get the joy that you mentioned, which is playing games. And that's fun. That's super fun. But what you've kind of start doing is realizing that the, the manuals that you've got, or the memory of how computers work you've got, is incomplete and wrong. And some of the video games of yesteryear, maybe less so today with more, uh, random hardware available. But you know, back in the day, video game, programmers would discover
Matt Godbolt: weird things that the graphics chips did or the CPU did, or the interrupt system did. And they would incorporate that into their game, or they would incorporate it into the copy protection for their game. So that, It was really, really hard for you to either copy or hack.
Matt Godbolt: Or they got extra performance because they were relying on some extra weird thing, but of course everyone had the same physical hardware, so they could genuinely rely on these kinds of things. But you didn't know that as an emulator writer. So you just emulate it, like it says in the, you know, the 6502 manual says, yeah, well, this thing, this is, this sequence of bytes is meaningless. It's not used. And you, so you don't do anything with it. And then you discover games and you've, you've hit your debug trap that says, no, no, some, some game used that opcode. And you're like, whaaaa? The first time I hit this, it was,
Matt Godbolt: I think it was the first time I hit it was in a game called Repton, which is like a Balderdash kind of game. And as part of the code actually jumps to an ASCII string, right. It starts interpreting an ASCII string as bytes now, as it happens, it's that doesn't have any meaningful side effect, but they're all these undefined opcodes in there. It doesn't do anything. And eventually it falls through, into the real code and carries on whether that was intentional, whether it was a mistake, I don't know, but it was like a first wake-up call. So like, hey, well, it doesn't like stop these computers, didn't have the facility to like throw an illegal instruction, exception at the hardware level, they just plowed on. And of course it's all logic gates at the end of the day. And so those bit patterns still meant something to the hardware and it did something and so the game designers would discover that, Hey, if I use this opcode, it's, it's the store instruction, but with one bit different now, normally there's, there's the store a, the A register, store x, store the X register, store y, the Y register.
Matt Godbolt: And then in the fourth, if you were to literally just looking down the list of opcodes, there was like, No, don't use that one. And then it went on to the next instruction. Yeah. And of course, you know, you're thinking to yourself, wait a minute, if I, any normal human being
Matt Godbolt: Who was designing this would see that essentially the only thing that was changing from one instruction to the next are the bottom two bits, you know, in the store A, It's 00. Store X is 01, 10, and then 11 you're like, well, what the heck is this doing? And so it turns out that through whatever means people realize that what's happening at that point, is it storing a, and'ed with the X register. And now I have literally no idea why that would be useful in a game context, maybe for masking actually. Now I think about it. I can't believe I've never taught this before, but if you had like a sprite mask in the X register, you've saved yourself an opcode, right? You don't have to and with the, a register and then store the a register, use this, this SAS sax command, which isn't, again, it's not an official thing, but it just comes out because that's the way the hardware physically work.
Matt Godbolt: And so you start finding out more and more of these things, you start realizing that the systems are not as well understood as the manuals would have you believe they're not as well as understood, understood even as the designers of the original hardware necessarily understood them. And so to sort of give some context to this when I was about 14, 15, a good friend of mine, and I were trying to take a game that was stored on cassette tape back in the day, when that was a thing, you know, you plug in a cassette tape into the port on the back of your computer. And the beeps and boops were like really low, um, modem speed sounds. And it would eventually load, it would take, you know, four or five minutes and it would load, but we had disk drives, right? We were lucky that we had disk drives.
Matt Godbolt: We wanted to take the game that we had on tape and put it onto disk, but that meant hacking it. Now whether we actually legally owned the cassette tape that we were hacking it from is one for the pub. Although in fairness, I have now met and spoken to the, the author and I've kind of made my peace with him over this particular, um, thing. But anyway, this was a particularly, um, interesting hardware protection that was put on the cassette tape version. So effectively the code was, was loading the beeps and boops from the, the, the cassette tape. And then it would jump to the, the beginning of the, uh, the, the code. Now I could write a piece of code that listened to the beeps and boops and gets this big block of, of, uh, of code. And then I could probably just save it to the disk and then just run it, but that it didn't work.
Matt Godbolt: And I forget for the reasons and my friend rich, who would maybe listen to this, will perhaps remember why, but that wasn't possible. What you needed to be able to do is essentially decrypt. And just pull out the bit of the code that you needed for the game. Now, the author didn't want you to copy it. The author didn't want you to put it on this. The author didn't need to look at his code, right? His, his admittedly compiled or assembled code. So he would write an encryption routine right at the beginning of where the program started executing. And that decryption routine would decrypt the game that was immediately after the decryption routine. So the code literally follows off of the last bite of the loop of the, for all the bites in the remainder, E OR with FF worst possible protection system. And then it would fall into the middle of the game.
Matt Godbolt: And there was no point in between those two where you could put a break point or similar. And the reason for that there were no hardware break points. And you know, the first thing you might want to do is you might want to modify the first bite after the end of the decryption routine to, to say, no, no return here. But of course you don't know what to write there because it's going to be decrypted. So you have to write the encrypted return, and you could probably guess there's only 255 combinations, so you could try all of them. But then if you actually looked at the decrypt routine, you see that things that go into the key, you know, I just said exclusive or with FF, but the things that go into the key that decrypts the rest of the code are the bytes of the decrypted code, the bytes of the decryption routine itself, the decryption routine also modifies itself while it's running.
Matt Godbolt: So the key is mutating as it's going, and there's no way for you to poke at it, um, to stop you from doing like time-based attacks where you kind of stop it, jump out of that, and then patch up the routine instrumented in some way. And then jump back to the beginning of that part of the routine that you patched out. Um, it would use hardware timers. So they were timers that were VAT changing variably as the thing was going on, it also would configure interrupts, and it wrote its own interrupt handler that deliberately did not like leave the state of the machine alone. And it came back from the interrupt routine, having modified another part of the key, so where this interrupt was triggered and all that stuff, all factored into it, which made it basically impossible for us to.
Ben Rady: So the guy built the Hellraiser cube into a cassette tape?
Matt Godbolt: Effectively that's exactly what he did. Yes, that's a perfect way of putting it. So,
Matt Godbolt: Richard and I, back in the mid nineties, we wrote, uh, an emulator for the very computer that we were writing an emulator on in the basic routine, basic sort of language of the computer that we had in an attempt to emulate the protection system perfectly or emulate the BBC perfectly, such that we could get to a point and then stop it and then save. And we spent a couple of weeks on this and we gave up because it was just so complicated. All these things I just described to you, we're in this code, we'd have to be perfect and they would have to be cycle perfect. Like literally every tick of the clock, all these numbers are changing. Hardware states changing, interrupts are being scheduled or descheduled or canceled or not canceled. And the whole thing was in there and it was brilliant and we'd like, well, we can't crack it. So we gave up on it, but fast forward, 30 years, Oh gosh, really is. Fast forward several decades. And I'm writing an emulator for that same machine, but for recreational purposes, not for nefarious purposes. And I came across this game again, and I'm like, no, this time, this time game, I'm getting you, I'm having you
Matt Godbolt: So, one of the evil. I mean, in addition to all the things we just talked about, one of the things that this particular, uh, protection system does is it used a rotate instruction. So rotate is like a shift where the top bit goes back into the bottom part. So a bit seven goes off and comes back into bit zero. And if you think, um, uh, and this particular rotate instruction was a rotate, a memory address, which was unusual on the 6502. Normally you'd have to load something into a register, do something with it and then store it back. But in this particular case, you could rotate directly a memory address. So if you think about what's happening inside that instruction it's going to take several CPU cycles, it's going to have to read from that memory location, it's going to have to do something, and it's going to write it back out again.
Matt Godbolt: And this is all within a single instruction. Now what Kevin's protection system did is he actually rotated the address of a hardware timer. So the hardware, this is a piece of hardware that pretended to be memory, but actually was, was not memory was a counting timer. And it was, it was configured on the system such that reading would give you the current value of the timer. Writing would reset the timer and start it again from that value or, you know, something equivalent to that. So that's interesting now, suddenly in order to modify, uh, to, to emulate this instruction perfectly, I have to emulate within the instruction because as the instruction is executing, that timer is ticking down. So does it read from the timer on the third cycle or the fourth cycle does it write to the time on the fifth cycle or the seventh cycle or what what's going on?
Matt Godbolt: And so you're there now you're assigned to answer, this was never put in the, in the, in any of the, the data sheets to my knowledge, right? This is just like, no, a rotate takes seven cycles. So how do you answer these questions? Right? I didn't, unfortunately I didn't have a 6502, well I did, but it wasn't in the BBC micro. Uh, it was in a, on a breadboard in front of me right here, but, um, how do you answer these questions? And so this is when I went digging and I discovered a whole world of folks who are taking chips, decapping them that is taking the protective plastic or ceramic layer off of the top of the chip using extremely distressing chemicals to etch way at the, the plastic and the surface, really horrible, horrible stuff. I mean, there's terror stories out there, uh, for, for what happens if you don't, if you don't look after yourself.
Matt Godbolt: So please, if anyone's listening to this, do not attempt any of these things,
Ben Rady: Do not try this at home, kids.
Matt Godbolt: Do not try exactly, but these people are doing this. And then they're using high resolution metallurgical microscopes to take lots and lots and lots of extremely close up photographs of the die. That is the actual integrated circuit that's underneath the plastic, like chip that you would see if you sort of, you know, computer chip, right. And then they're stitching those images together. And then using a combination of just brute force of human, like line drawing, a bit of very primitive AI. And then a lot of head-scratching, they are piecing together how these older chips worked. And what I'm talking about here is specifically enthusiasts that are doing this as opposed to like the professional laboratories that will do this for like, um, reverse engineering of competitors, project type things, which is definitely not in the scope of what I want to talk about.
Matt Godbolt: But these are like folks who are just more, um, uh, what's the word, more relaxed around scary chemicals than most people who are interested in this kind of stuff. And, and in particular, the 6502, because it was such an important processor, a they, they, they targeted that first. So there's a website. You can go to visual six, five Oh two.org, which shows you the high resolution photographs and the polygonal data that they've extracted from those high definition photographs showing the different layers of the chip. And like these, these chips are old enough that the manufacturing process, A is visible under a high quality,
Matt Godbolt: But affordable, um, light microscope rather than anything kind of scanning tunneling, electron nonsense, but also that there are, so there are a few enough layers that it's practical for you to extract the layers one by one, by like washing it and another, and counting to 10, and then quickly washing off the solution and then looking at it again and like taking off these layers, you know, there's a metal layer at the top, and then there's various level layers of polysilicon and there's various doped regions, which sometimes you can see cause they look a bit different. Other times you just have to infer that they're there, but the, the whole, uh, premises that they have now, a gate level, like I was going to say transistor level, really. It is what it is. It's a transistor level simulation of a 6502 that you can go and run in a web browser.
Matt Godbolt: And they've more over, you know, as well as just emulating it and all the pads that is the big connections that come up around the outside of the chip. There's been enough work that they can tag areas of the die and say, well, this is the, A register, right? These, these flip flops hold the, A register. This is the X register. Here's all the other internal state. Here's how the PLL works. Here's how the, um, the, the the instruction decode works. So with this information, they were able to explain comprehensively things like how the store AX, uh, instruction works. You know, that SAX that we found, they could trace the two low bits that we, I was talking about before the O the O one and say, well, what happens when the one and the one, well, this just activates the, A register and the X register at the same time now, because of the way the physics works.
Matt Godbolt: And this is in the particular case of the, the 6502, it's an NMOS, um, type of part, the, the sort of pulling low of a zero wins out. So if there is a one being pushed onto the, onto, like the output bus and a zero being pushed onto the output bus, because both the, A register and the X registers kind of like enable, has been turned on at the same time, then the zero wins. Cause it pulls the one down to zero. So that's the, and that you're getting, that's why it's store A and X. If you put them both on, at the same time, now, different fabrication would have come up with different other, uh, sort of side effects. But that's like one example of like, that's what's happening in that particular instruction. So I was able to go to their site and do essentially a gate level, transistor level simulation of the rotate instruction and see exactly what happens and what cycles the read happens and what cycle the write happens.
Matt Godbolt: And I was able to discover, and this was actually well-known before this is, I'm not discovering anything new here, but the, in between the read and write, there is a one cycle delay while this chip is like thinking it's like reads the value into a temporary part of the chip, does the rotate, the next clock tick, and then writes it out. But there aren't enough pins on the outside of the 6502 chip itself to sort of say, I don't need to read or write to memory this clock cycle. So it's always reading and writing from whatever the pins are.
Ben Rady: Okay.
Matt Godbolt: Right. So there are address pins and, there are data pins. And there's 16 address pins. Eight data pins and one read, not write. Which means that if it's, if it's, if it's high, I'm reading it. If it's low I'm writing. So in this one,
Matt Godbolt: In between step the chip, isn't doing anything, but these pins are still on the outside. And the RAM doesn't know that you're not talking to it or you don't care. And so normally it wouldn't matter, right? If it's no harm, no foul, it's like I'm reading or writing to a piece of memory that, okay. It doesn't, it doesn't matter. Or hopefully as long as you're not writing zero to something, but it turns out that what's actually happening is it's. It is already updated. The address pins
Matt Godbolt: To, sorry, the address pins remain the address of where it just read from. Okay. And the databus still has the old unshifted value on it. And it's, but it's already told to, write. So it's done a read.
Matt Godbolt: Read the address, got a value 12 out. And then while it's doing its rotate the microcode inside. It's not even microcode, but it's not going into that. But like the, the, the, the design of the chip inside, it said, well, we might as well toggle to a write. Ready to go, because in a minute, we're about to write out whatever we've just, we're calculating. But of course what's really happening is a write of the old value has happened to the memory location it just read from.
Matt Godbolt: Okay. So, so far, no harm, no foul. It doesn't really matter, right? If you've just read from a memory location and you mainly wrote back to it, and then now you're going to update it with the shifted value. That's not a problem. Except of course, if that hardware was hardware, it wasn't memory because writing to that address changes the state of the hardware you're writing to. And so this was a really significant find because the timer that is actually on the end of this rotate gets two writes. And each time it resets itself. Now you are about
Matt Godbolt: to write a new value over the top of it. But what happens if in that one clock cycle, the value you wrote to triggered the timer to go off, now an interrupt scheduled. And so this side effecting right, actually was important to model. And so that was the first surprise. That was like, my gosh, okay, there's this redundant write that happens. That's normally no problem. But now is, is, uh, is important to model. And it was, it was an important thing and getting this correct. Then there were some other things to do with the way site, the, the bus specifically to the BBC micro worked in terms of slowing down. Um, did you ever go through the apple? I know I'm frothing away here, but what did you have an Apple, um, Apple two or anything?
Ben Rady: I had a commodore, commodore 64.
Matt Godbolt: Got it. Do you remember if that had a one megahertz, um, peripheral bus versus like a faster, main CPU? I forget.
Ben Rady: I do not remember.
Matt Godbolt: And I know people will be shouting at their no, because the, I I'm aware that the Apple had, uh, like a one megahertz 6502 and the 6502 can go faster than that.
Matt Godbolt: But the peripherals, which are all memory mapped were all one megahertz. So they kind of just took the lowest common denominator and said, well, we're run the CPU at the same speed as the peripherals. And then there's no problem. And later on, there were like aftermarket boards that you could plug into the Apple that sort of sped up the CPU. And then they detected when it was accessing peripheral memory and they went, Whoa. And they slowed the clock cycle down to match. Yeah. And that's what the BBC micro did out of the gate. But that exact synchronization, when you imagine you've got like something running twice as fast and it can be an even or odd cycle with respect to the one megahertz. And now, also this is happening every time it accesses, um, these peripherals. So in that rotate instruction, not only is it having a side effect of, of writing to the, uh, the one megahertz bus speed peripheral, that's the timer that slows the chip down as well, because it's like, well, I read once that slowed me down, I wrote once redundantly, that slowed me down again.
Matt Godbolt: And then finally I wrote the actual value that I wanted to write. And I got that through, and that was also split out. So like finding this out was amazing and a godsend and, and piqued my interest entirely because like staring at a bunch of seemingly hugely cryptical, cryptical seemingly hugely cryptic, rectangular regions that don't appear to be connected in any way was somehow something the wizards could read and say, Oh, that's an and gate. And I'm like, what? No, it isn't. It's like you highlighted that one green and that one red, why did you do that? Oh, well, this one's clearly doped this way, because it's an even number of things from the outside. And I know that, and I'm like, whaaaa?
Matt Godbolt: How are you doing this? What is this witchcraft?
Matt Godbolt: And as it happens, you know, the 6502 is, and I'm probably definitely belittling the work that went into it. Cause it was a huge amount of work. But as I've learned a little bit more, the NMOS fabrication technology that was available to them in like the early seventies was, is easier to understand coming in from the outside. And you can sort of see, Oh, that's a really big squiggly bit. That's probably a resistor. Um, this bit over here is like a power transport, a transistor this thing here with like, you can sort of see it if you squint. But again, it's one of those things where I, I don't know how often you have this experience when something you're being explained makes perfect sense while you're listening to the engaging, present presenter.
Ben Rady: I mean, that's happening right now.
Matt Godbolt: But then you're going to walk away afterwards and be like, that was really good, but I still, I don't get it.
Ben Rady: As long as there's not a quiz later I'm completely fine.
Matt Godbolt: All right. No, no, that's it. That's it. No, as it's, I've had that experience with a whole bunch of folks. And so there's a, there's a number of people on, on, on YouTube that have like cataloged their reverse engineering of, of chips. Uh there's uh, um, Robert Baruch, if that's how you pronounce his name, there's a fantastic person. Um, uh, Ken Shirriff, Ken Shirriff has got a blog and he does probably one or two a where he tears down a, a, an IC or similar from, from like pre-history or like something from the seventies or eighties, like he's on one bit, computers has done the 8088, 8086. And it's just super interesting and exciting and it's written engagingly. And again, it's like a bit like you can let it flow over you and you just appreciate somebody else understands this well enough to be able to do it and explain it well enough for you to at least understand how cool it is.
Matt Godbolt: And so I've really loved that it's been a really fun experience. Um, and so, you know, obviously it's touched me in terms of my emulation work. And then more recently, some friends of mine have started looking at a disk drive controller, which was something that came alongside the BBC micro to control the disk drive. I mean, I know on the Commodore, for example, they actually just throw another 6502 inside the disk drive and say, well, that can just drive the disk and do the, all of the, the, the magic to understand how to, you know, shift, uh, the stepper motors in and out and how to turn on the sense detection and all those kinds of bits and pieces, how to do the MFM and FM decoding, which made it really a, an amazing target for like, well, here's a co-processor if it's not running the disk drive, surely we can do something else, which is exactly the kind of cool thing that was done back in the day.
Matt Godbolt: Because again, you could guarantee it on the end of a wire, I just loaded it off desk. Um, I got a spare 6502 brilliant. Now the, the disk controller that, um, that, uh, my friend took apart is an Intel disk controller. And it's long had this again, a big list of here's the commands you can run. And you're like, well, there's a gap somewhere in this list of couple, I wonder what it does. Right. But yeah, through, through decapping it and getting a friend to photograph it and then go over it. They were able to isolate an area which looked like ROM. And so it became clear that it was basically a CPU, an embedded CPU, probably with a very strange, um, uh, opcode list. But they were able to isolate where the ROM was and then using some real tricks of like known bytes that were written to the disk drive when you like format it, for example, they were able to like find regions of the ROM that corresponded to that, and then theorize and later on, essentially reverse engineer, the opcodes for this CPU.
Matt Godbolt: That was the drive controller that lived inside the, uh, the, the, the, the BBC micros, uh, disk drive system. And I mean, that's just amazing. And I love that kind of stuff. It's just so interesting that there's always a layer lower than where you are. Right? You can be you can be writing in, in a high-level language and be vaguely aware you're interacting with an operating system. You can be writing an operating system and you could be using C++, or you can be using C and you'd be thinking like, yeah, I'm laying memory out. I know exactly what I'm doing and then you realize well, there's a DMA controller that's could also read and write to memory, or the network card can also read and write to memory, and there are interrupts and, you know, you kind of think, well, okay, I know where I with with interrupts, okay.
Matt Godbolt: I've written an interrupt handler. I've written games before, and I've kind of had to feed the graphics unit, by every time it ran out of work, I would, you know, all these cool things. And you think, I I've, I'm talking about myself here. Like, I think I've reached the bottom of the stack. Now I've written an interrupt handler. I'm that's it I've, I've, we've done now. It was like, you know, when the Lord Kelvin said, there's no more physics we're done here, but then he goes, no, no, no, no, there's, there's more, you know, you can dig down and on Intel processes, you can go where there's microcode. Um, you know, what you put into the processor is not even remotely what it actually executes you like, Oh, really? Well, that's fascinating to me. How do we know? How can we observe it? You know, there are all these clever tricks and you kind of start learning down further and further and optimizations. And you're like, wow, that's amazing. And then somebody else comes along and blows your mind by saying, you know, you can take the lid off that chip. Right. See how it works inside. And like, Oh my gosh. Yes. It's so cool. It's so cool. And I've, yeah, I've really enjoyed the journey. And I'm still learning a lot, a lot about this, but it's just a lot of fun.
Ben Rady: So how do you think that, uh, going back to the game that you were, you were trying to crack, how do you think Kevin was his name? You said you talked to him, right?
Matt Godbolt: Yes. Kevin...
Ben Rady: How did he figure out all these things? Cause if you didn't figure it out for 30 years later, by stripping the top off the chips, how in the heck did he do it?
Matt Godbolt: That is a splendid question. And one that we thought for a long, long time about, but my friend rich was able to theorize how he did it and got him to reply on a forum post. That indeed, that was the case. And through that, we've become, I wouldn't say buddies, but you know, we were acquainted now, which is great. Right. Um, they say, don't meet your heroes, but I've had some great experiences with the people who are writing video games in the, the late eighties, early nineties, probably cause they're only a couple of years older than me even. Right. You know, it's not like that, that far ahead of me in age terms at least. But, but yeah. So the way that he did it, so the way that it actually worked, I simplified it a little bit, a little bit early, obviously for, for the purposes of explanation. But what really happened is that the code ran the decrypt code ran and it landed not in the game, but in a CRC check and that CRC checks checked the CRC of the decoded game. And if it didn't match the expected known plain text, CRC, it jumped to the ROM routine that wiped all memory and reset the computer.
Matt Godbolt: Okay. So there was still no way you couldn't go in and say, let's take out the jump to the reset and kind of get in there because again, the CRC code itself that the bytes that implemented the CRC were part of the key. So by modifying it to say, Hey, don't do that. You would destroy the key and it wouldn't decrypt correctly and all that stuff. Right. So essentially any minor perturbation to the code anywhere would either crash it, or it would get to the end of the decode routine, the CRC check would fail and it would wipe the memory of the computer. And it would have to start again, loading off tape for another four or five of your life. Right.
Matt Godbolt: And so Richard theorized that Kevin had desoldered the ROM on his motherboard and replaced it with a ROM that did not wipe memory when you jumped to the, it failed the CRC routine. So far. So, okay. How do you use that to, to help, right? Because you run it and if you've got, if the CRC was wrong, well, it would just reset the computer, but the memory would be okay. Still. The second thing is that Richard realized is that the way that the cipher worked essentially is cyclical. It's like a random number generator style, like co uh, encryption key generator, right. Dynamic, but, but cyclical. And so it would repeat after a certain amount of time, certain number of iterations. So what the process was is Kevin loaded up this decrypt and CRC put the known CRC of the un-encrypted game into the CRC code, because that's what, you know, that's how it should, that's how it knew it got the decode right.
Matt Godbolt: And then he put the un-encrypted game after it. And then he put in his desoldered ROM with the don't wipe memory. And he executed it once. So what of course happened is that the decrypt code ran on the un-encrypted game and caused it to change in some totally unpredictable way, but critically the same way every time. And in a way that if you were to continue doing it, it would eventually cycle back to where it started. Ah, okay. So then it would get to the CRC code, the CRC code would fail and it would reset the machine, but keep everything intact. Kevin would save that image to disk and then he would run it again and then he would run it again and keep running and saving and running and saving. And every time it reset, he knew that he was another iteration along.
Ben Rady: Is the idea here, it eventually it cycles back around and then you get something. Yeah. Okay.
Matt Godbolt: And then, yeah, and the last iteration, it would cycle back to the beginning of the CRC check would, would complete, and it would jump into the game. And of course at that point he was locked out. You couldn't do anything, but that's fine because he kept the previous iterations disk image. And that was the one that was sent off to be fabricated. So he himself doesn't understand how his own code works. Right. He still doesn't understand, I mean, he does cause we've talked about it, but like at the time that he made the, um, at the time that he made the code, all he knew was that it was deterministic and that the the, it would be, um, enough repetitions of that decrypt code would eventually cause it to repeat back to where it started from. And I think that's just genius. So it was both an incredible piece of engineering. In the first point to think of all the things
Matt Godbolt: that a hacker would use timers and counters, and, you know, self-modifying code and all this kind of nonsense to stop them from hacking it, it was, um, understanding enough about the way that this particular type of encryption works, that it would cycle. And it was at the end of the day, desoldering something off the motherboard of your machine and replacing it, which like nobody thought of. Wow, it's so cool. It's so cool. And so interesting. And, and yeah, I'm, I'm glad to say that, um, jsbeeb will run the alien eight protection system. And in fact, uh, Kevin gave me permission to make it a test so we can get back to testing. Uh, one of the tests is to actually run all of that, that I just described and make sure that it gets to the end and executes the first bite after the CRC code, which incidentally I don't, um, I don't even have right.
Matt Godbolt: The rest of that code is like, it stops it there's, there's, there's not enough for it to be the game. So it's just the code, but it's so fun. It's so cool. And there's so much to learn from it. And this is again, 40, 50, 45 year old computer now. Quite. I mean, not 45. Gosh, now I'm 45, 35 year old computer. That's slightly better. Yeah. Wow. So do you have any experiences like that? I mean, we, I mean, we we've taught that like at, uh, you know, late teens, early twenties, we, we, our paths diverged.
Ben Rady: Yeah. No, I, I was not into hardware at all as a kid. I mean, I, I was very, uh, very much into, uh, you know, I, I learned how computers work mostly to play games because it was not exactly a straight forward thing back then to, you know.
Matt Godbolt: You mean himem.sys
Ben Rady: yeah, yeah. Figuring out what the heck is wrong with my sound card and all that sort of stuff. That was my first real,
Matt Godbolt: Gosh, moving jumpers.
Ben Rady: Yeah, exactly. That was my first real attraction into it. And so, you know, for me, it was about, you know, when I was a kid, my, my dad owned this software company. It's kind of how I got into it. Um, that did audio, video processing and so I.
Matt Godbolt: I believe I never knew that. That's so cool. We're going to have to talk about that sometime.
Ben Rady: Yeah. Oh, that was, it was, I mean, it was amazing for me as a kid, cause I got access to computers and high-end video cards to do like video capture. He'd, he'd built a system for digital editing. I mean, now you can do it on your phone, but you know, back in the early nineties, late eighties, it was, it was something unique.
Matt Godbolt: gen lock boards and things that were yeah. That kind of nonsense. I remember like the time TV shows that we were watching, you know, like the, the music TV shows would be using the same kind of technology that you're talking about. Did I do their onscreen graphics and silly? Yeah. Yeah. And you had that like in the basement or wherever or your dad's office.
Ben Rady: Let me tell you I had some really amazing video presentations is a grade schooler when I was a kid, I was just blowing the other kids away. It was like, kind of unfair actually. Um, but, um, but yeah, so I mean, I never really got too much into hardware until very much later in life. I mean, it was really like 10 years ago and it was like, you know, Ardunios and, you know, I built this little, um, like house temperature environment monitoring system with ZigBee radios and, um, you know, stuff like that. But it was all just like a hobby thing for me as an adult. It's never, never something that I really did as a kid. I mean, I had like little like electronics kits and stuff like that, you know, make the buzzer go off, make the light go off, but nothing really serious. So, you know, this is, this has been sort of more of a, um, for, for something to me later in life early was all about, you know, software and to, I guess to some extent some of this video hardware, but it was all super specialized. I never really got down to the bits and bytes. It was all about, you know, if I take this video stream and I blit myself on top of it, that sort of stuff.
Matt Godbolt: Right, right. But I mean, there's just so much interesting stuff going on with, with, with hardware. I mean, outside of the box, and that's what I love the most about the, both the emulation and the sort of hacking side of the, uh, of, of, of that was like, you really have to think how, how could I achieve what I need, maybe not by not using the manuals, maybe by using some other observable things, you know, just, it was just forced a lot of problem solving, which I think ultimately ultimately if you strip away everything about what I love about our job and what I do as a hobby and what I do in my evenings when I'm not doing my hobby hobby, um, it's problem solving in a, in a sort of constrained environment as well. I think that's what makes it more interesting. Like if, if, if you've got like no real constraint, then you just kind of do the next obvious thing. But if you're like, no, I have a, I have a two megahertz, eight mega, uh, eight bit computer and I want to do ray tracing. You kind of have to pull something magical out to make it work. And that's, that's kind of it.
Ben Rady: I was going to say, it's sort of like, there's two kinds of people in the world. People who watch Apollo 13 and are compelled by the story and people watch Apollo 13 and they're like, man, I really want to figure out how to get that air filter into that square hole. Right. Like, you're like really, like, I want to know how did you figure it out? You had the things in the thing. Um, and, and yeah, that, that sort of mentality is, is kind of unique. And it's super fun when you have those sorts of problems that are like, all right, guys, we have 90 minutes to figure this out. These are the materials you have, it has to fit into this box go. Right. Yeah.
Matt Godbolt: And I mean sort of just to try and sort of give something out that our listeners can play along with, rather than just listening to be babble on this kind of answers, this things that you mentioned, actually like, if, so, if, if any of this is interesting to people, I'm sure you can Google and find a lot of this stuff, reverse engineering chips. Um, Ken Shirriff's blog, um, is great as well, I think which is righto.com, but you mentioned Arduinos, you know, there's raspberry pis, there's Arduinos. And in fact, like the raspberry pie very specifically is modeled on the BBC B because the founder of the company was like, I don't have a way to hand to my children, the same experience that I had when I was, was hacking around on my BBC micro was at school. I had one at home. It has all these vague, weird ports hanging out the back, which was another sort of like relatively unusual thing about the beeb.
Matt Godbolt: You know, you can put a ribbon cable on the back of it, and then you could like plug it into turtles that drove draw lines on the, on the ground or hardware, things. And magazines would even have like printable stuff. I remember having like a, a, an eight, uh, element led, um, wand that you could build. And there was inside of, it was a, uh, like a motion detector or actually it was a mercury switch. And so it could tell how quickly you were waving it backwards and forwards. And for those listening on the podcast, they're not looking at me in the video, which is only Ben I'm wiggling my hand back and forth, like I'm batting a bat. And so, sorry. And then allowed you to like, write a message in the air by serializing out and turning on the LEDs. And that was so cool because I was building with a regular computer.
Matt Godbolt: So anyway, that exposure to hardware and the low level, and the fact that just hackability was what inspired the raspberry PI. And so I feel like there's a nice warm glow. I get from like, realizing that now on my desk here, as you can vauges see just out of shot, I have a whole bunch of raspberry PI related gear. Cause I'm hacking on this kind of stuff again, and it's brilliant. And so people can get raspberry pis, they can get Arduinos. The Arduino IDE in particular is just really, really user-friendly. So you can go write an emulator. You can hack around with this hardware. And there's just a lot of cool things that people can still do this nowadays, even though we, it feels as we're staring at these, these huge supercomputers in front of us. Well, even the ones that we have in our pockets are super computers by anyone's, um, yardstick. These are all there. There's still an accessible route to playing around with interrupt handlers and assembly and, and, and just understanding at a deeper level how computers actually really work.
Ben Rady: Yep. Yeah. And it's super fun. And I love it when you can kind of solve a problem that you have with some of these devices. It's, it's really amazing. Uh, and if you can, if you know, you can get your kids interested too. That's a huge bonus, but.
Matt Godbolt: Well, for, for us, that's a big deal, right? I, if you can get them to see that life, isn't all Minecraft and it isn't all Roblox it's.. There. You can do a real thing with a relatively small amount of code and get something interesting happen. Yeah. Yeah. And as I'm saying this, my son has just popped his head around the corner was looking at me quizzically. Like, no, I won't. It's like,
Ben Rady: Yeah. Right.
Matt Godbolt: I missed that boat.
Ben Rady: Well, Redstone is the gateway drug. So if your kids are into Minecraft and they're playing around with Redstone, You can be like, how would you like some real life Redstone?
Matt Godbolt: Well, that's the funny thing. Uh, I, I did a sort of, um, passive aggressive parenting technique of buying some sort of textbooks on CPU design and on physics and on, uh, I can't remember the other, but they were like manga style ones. And I left them around the house. And then my eldest did pick it up and kind of read enough of the CPU stuff for two things. One thing was for him to go, Hey, this is just like Redstone, which is like the biggest compliment we could get. He's like, well, you understand enough about how Redstone works. And you're able to see it out of context, in like a truth table in a, in a, uh, admittedly in a manga style, but a relatively dry text. And then the other thing is he found, um, how numbering systems work in binary. And he said, pointing at the thing he goes for, for, for signed numbers going like, Oh, I see where you got the name of your podcast now. So like the circle is now
Ben Rady: The circle is complete
Matt Godbolt: Alright my friend. Well, it's been so much fun. Thank you for letting me froth and talk excitedly at you.
Ben Rady: Oh no, this is great.
Matt Godbolt: And I mean it would be fascinating if our listeners or listener
Matt Godbolt: However many we've got found this interesting too, if we haven't just alienated everyone, then I'd be surprised. But I mean, I'm just hoping that the, the, the enthusiasm, which I feel for this subject comes across and some, you know, at least one person is inspired to go and have a look or hack on some hardware or dig down through the chip layers or read up how their CPU isn't really doing, what they're telling it to do is a deep, deep subject. But yeah, I guess we can talk about other things another time, right?
Ben Rady: Yeah. Different topics every week. That's what we should do. I dig it.
Matt Godbolt: You reckon? Okay.
Ben Rady: Yes.
Matt Godbolt: It's a deal, my friend. Talk to you next time.