Syntax Error #8: What we can learn from hardware debugging?

Happy & spooky October everyone! Welcome to the 8th issue of Syntax Error newsletter. This month I wanted to take a tiny detour away from software debugging and talk about hardware debugging and what we can learn from it.

Even if you're not planning to ever take out an oscilloscope and start poking at circuit boards and chips, these learnings can help you become better at your software debugging processes.

Lesson #1: Everything is a suspect

One of my favorite Youtube channels on retro hardware fixing and debugging is Adrian's Digital Basement. He has a great catalogue of videos where he fixes retro computers, giving them new life. Fixing those machines requires a lot of debugging to figure out where the problems lie.

One of my key takeaways from watching Adrian find where the issues are is that with old hardware, everything is a suspect. That means you cannot rely on assumptions on where the issues are but you need to intentionally and deliberately go through all the possible cases. It could be broken capacitors, malfunctioning chips or disconnected traces and it's not usually visually clear which one is the culprit.

Personally, my aim in becoming better at debugging is to become better at following a process, step by step, with zero assumptions or guesses. Sometimes a guess can lead to success really fast but often it just leads to a wild goose hunt and since you didn't start with a process and checklist, you may end up poking at things semi-randomly. Once you become really good at that, you'll be able to debug nearly anything – even if you don't know the language or have domain knowledge of the software.

Lesson #2: Proper tooling is a key

Another takeaway is using proper tooling. Oscilloscopes, voltage meters, chip testers and loads of other tools are all used when trying to narrow down where the issue is. In software world we have debuggers, loggers and other tools that help us similarly well, as long as we know how to use them and incorporate the best tools to our daily utility belt.

It's worth the time to dive deep into understanding how these tools work and how to use them effectively in your current project. Sometimes it might even require making changes to the project code and infrastructure to make sure that you can run your debugging tools effectively.

If you work on a team, ask your colleagues how they debug when they run into issues. Host learning sessions regularly where you come together to discuss the tools that you use and share with each other what you've learned. Modern tools are often so complex that we only end up using a small portion of them and might not follow constantly their changelogs for new features. Sometimes it can be as simple as sharing keyboard shortcuts that can increase the productivity and quality of life for developers a lot.

Lesson #3: Make the smallest changes possible

When you build something physical, you need materials that cost money. The bigger and more specialized your build is, the more likely it is to rack up the bill.

Because of this, if something doesn't work, it's even more important in hardware than software to make small experiments and narrow down the problem and aim to solve it with the smallest changes possible. Once you've found and confirmed the issue and you've found a cheap fix that works, you can then be confident to invest more money and resources to build a more robust solution.

While the direct cost of added lines of codes is essentially zero in software projects, the complexity grows with every change. And once you find a solution, it becomes harder to simplify it down to the essentials because you have more candidates for what actually fixed the problem. So you'll have to do another round or two of figuring out the real problem, this time by removing changes and seeing if the problem reappears.

I find that this requires discipline or at least active conscious thinking. It's easy to go ahead and try a few different things without walking old changes back between them. This becomes more true when the problems themselves become more complex and difficult to figure out.

Lesson #4: If all else fails

Like the wonderful Marius Hornberger puts it well on his latest video:

To summarize it: "Why? Just move." [- -] And when you're sitting on the floor like this, you know it's bad. Eventually though, you run out of ways to do it wrong.

Story Time

Oona Räsänen has a great story in her blog that was shared with me by Daniel on Mastodon.

This story, too, begins with noise. I was browsing the radio waves with a software radio, looking for mysteries to accompany my ginger tea. I had started to notice a wide-band spiky signal on a number of frequencies that only seemed to appear indoors. Some sort of interference from electronic devices, probably.

In this very interesting and well written exploration, she eventually figures out the display she's using, is leaking the image as radio waves (or something, I'm honestly not an expert or even a beginner on any of that).

I found the exploration, experimentation and eventual results of that story fascinating. When the starting point is essentially noticing random noise and the outcome is transmitting a webcam image through that noise to another computer, it's nothing short of marvelous.

I recommend checking out the full post and the accompanying video.

PS. I will be speaking about Debugging Python in PyCon Sweden, 9th-10th of November in Stockholm, If you're planning to be around Stockholm then, there are still tickets available for the conference. If you will attend, come say hi - I'll have Syntax Error stickers with me to share with you.

Syntax Error is created with love by Juhis. If you liked it, why not share it with a friend? Or if you have any feedback or just want to say hi, hit reply. I'm always happy to hear from my readers and learn about what you do and how you debug your issues.