HN Reader

I don’t mind using a debugger, but one of the advantages to printed debugging is that it is universal. I work on a project with code in five languages (6 if you count the build system, 7 if you add shell scripts). Two of them I know the debugger very well. One of them I might be able to get by. One of the others I might have use the debugger once and the last I’ve never touched.

printf debugging works in all of them, even the build system and shell scripts.

A debugger can be great, no question about it, even for remote debugging. In my experience, I’ve seen fewer people effectively debug unfamiliar systems quickly.

Maybe someone can give me idea, how can I debug this particular rust app, which is extremely annoying. It's a one of Rustdesk.

It won't run if I compile with debug info. I think it's due to a 3rd party proprietary library. So, to run the app I have to use release profile, with debug info stripped.

So, when I fire up gdb, I can't see any function information or anything, and it has so many system calls it's really difficult to follow through blindly.

So, what is the best way to handle this?

Author missed one of the best features: easy access to hardware breakpoints. Breaking on a memory read or write, either a raw address or via a symbol, is one of the most time saving debugging tools I know.

It's not a silver bullet, but Visual Studio is leaps and bounds ahead of gdb et. al. for debugging C/C++ code. "Attach to process" and being able to just click a window is so easy when debugging a large Windows app.

IME console-based debuggers work great for single-threaded code without a lot of console output. They don't work that well otherwise. GUI-based debuggers can probably fix both of those issues. I just haven't really tried them as much.

pdb is great for python, though.

This is refreshing. I get triggered by people writing "I don't use a debugger because I'm too smart to need one".

Some other things I'd add:

Some debuggers allow you to add actions. For example logging at the breakpoint is great if I can't modify the source, plus there's nothing to revert afterward. This just scratches the surface. Some debuggers allow you to see entire GPU workloads, view textures etc.

Debuggers are extremely useful for exploring and helping edit code. I can't be the only person that sprinkles breakpoints during development which helps me visualise code flow and quickly jump between source locations.

They're not just for debugging.

Honestly, I feel like the print vs. debugger debate isn't about the tool, it's about the mindset. Print statements feel like you're just trying to patch a leak, while the debugger is about understanding the plumbing. I’m starting to think relying only on print is a symptom of not truly wanting to understand the system you're working in.

things I can do with print statements but not a debugger: trace the flow of several values across a program, seeing their values at several different times and execution points in a single screen.

There are two kinds of bugs: the rare, tricky race conditions and the everyday “oh shucks” ones. The rare ones show up maybe 1% of the time—they demand a debugger, careful tracing, and detective work. The “oh shucks” kind where I am half sure what it is when I see the shape of the exception message from across the room - that is all the rest of the time. A simple print statement usually does the trick for this kind.

Leave us be. We know what we’re doing.

In my experience, conditional breakpoints are so unreliable, that I just dont bother trying to use them. When I need one I add code like

    if (condition) 
        print("");

Then add a breakpoint on the print. Calling print ensures the line with the breakpoint won't be optimised out (rarely matters as I'm normally debugging in... debug mode, but it's just a reflex at this point, and I need to put something in there)

I would add to that list the important point that in a large codebase rebuilding after changing a line of code can take a very long time. In fact this is one of the most important reasons to get familiar with your debugger.

Every engineer should understand how to use a debugger and a time profiler (one that gives a call tree). Knowing how to do memory profiling is incredibly valuable too.

So many problems can be solved with these.

And then there's some more specialized tooling depending on what you're doing that can be a huge help.

For SQL, the query planner and index hit/miss / full table scan.

And things like valgrind or similar for cache hit/miss.

Proper observability (spans/ traces) for APIs...

Knowing that the tools exist and how to use them can be the difference between software and great software.

Though system design / architecture is very important as well.

I think the obvious benefit of a debugger is the ability to introspect when you have the misfortune of investigating the behavior of a binary rather than source code. In the vast, vast majority other instances, it is more desirable (to me) to encode evidence of investigation in the source itself. This has all the other benefits of source code—you can persist it, share it, let ai play with it, fork it, commit it to source control, use git bisect, etc.

There are a few other instances where the interaction offers notable benefits—bugs in the compiler, debugging assembly, access to registers, a half-completed runtime or standard library that occludes access to state so that you might print it. If you have the misfortune of working with C or C++, you have the benefit of breaking on memory access—but I tend to file this in the "half-completed runtime" category. There are also a few "heisenbugs" that may actually prevent the bug from occurring by using print itself; but I've only run into this I think twice. This is also possible with the debugger, but I've only run into that once. The only way out of that mess is careful reasoning, and i recommend printing the code out and using a pen.

I also strongly suspect that preference for print debugging vs interactive debuggers comes down to internal conception of the runtime and aesthetic preference. I abhor debuggers—especially thosr in IDEs. I think they tend to reimplement the runtime of a language a second time, except with more bugs and a less intuitive interface. But I have the wherewithal to realize that this is ultimately a preference.

While I enjoy using debuggers and find them valuable tools, the reality is that many environments have network or security constraints that make debugging significantly more challenging than simply using logs.

The really hard bugs are those that disappear if you make changes to the binary, like adding a print.

Two of the benefits listed (call stack and catch exceptions at the source) are available in logging as well. A good logging framework lets you add the method name, source file and line number for the logging call-after a few debugging sessions you will construct the call stack quite easily. And C# at least lets you print the exception call stack from where it was thrown.

I agree that adhoc dynamic expression evaluation at run time is very useful and can only be done in a debugger.

And that's why I never learned Elixir, despite being an interesting languge with an awesome web Framework, Phoenix.

The fact that there ist No Debugger is super unfortunate

Meh. None of these sway me. I'm a die hard printf() debugger and always will be. But I do use debuggers regularly, for circumstances where printf() isn't quite up to the task. And there really are only two such categories (neither of which appear in the linked article!):

1. Code where the granularity of state change is smaller than a function call. Sometimes you actually have to step through things one instruction at a time, and I'm lucky enough to have such problems to solve. You can't debug your assembly with printf(), basically[1a].

2. State changes that can't be easily isolated. Sometimes you want to log when something change but can't for the life of you figure out when it's changing. Debuggers have watchpoints.

But... that's really it. If I'm not hitting one of those I'm not reaching for the debugger. Logging is just faster, because you type it in right at the code you're already reading.

[1a] Though there's a caveat: sometimes you need to write assembly and don't even have anything like a printk. Bootstrap code for a new device is a blast. You just try stuff like writing one byte to a UART address or setting one GPIO pin as the first instructions and hope it works, then use that one bit of output to pull the rest up.

Print debugging is, checking patient's life signs, eye color, blood pressure, skin inflammation and so on. However using debuggers are like putting the patient through an MRI machine. It can provide you very advanced diagnostic information, but it's expensive, time consuming, requires specialized hardware and education. Alike medicinal doctors it's easier and logical to use the basics until absolutely necessary.

Most languages let you print the stack, so you can easily see the stack using print debugging.

Anecdotally, dynamic expressions are impossibly slow in the cases I’ve tried them.

As the author mentions, there are also a number of cases where debuggers don’t work. Personally, I’m going to reach for the tool that always works vs. sometimes works.

"Thinking before debugging" ... advice from Rob Pike about Ken Thompson's approach.

https://www.informit.com/articles/article.aspx?p=1941206

Things you can do with print debugging but not with most debuggers.

Watch expressions with history of values. Especially aggregate history of multiple watch expressions.

I am surprised all the time in this industry how many software engineers still debug with printf. It's entirely baffling how senior / staff folks in FAANG can get there without this essential skill.

Don’t tell Primeagen. Although he’s right about debugging sprawling systems in Prod. I’d argue the stateful architecture of these apps is the root cause.

Something I haven't seen discussed here that is another type of debugging that can be very useful is historical / offline debugging.

Kind of a hybrid of logging and standard debugging. "everything" is logged and you can go spelunk.

For example:

https://rr-project.org/

I don't really get the hate that debuggers sometimes get from old hands. "Who needs screwdrivers if we always used knives?" - You can still use your knife, but screwdriver is a useful tool.

It seems to me that this is one of the many phenomena where people want to judge and belittle their peers over something completely trivial.

Personally, I get the appeal of printing out debugging information, especially if some bug is rare and happens in unpredictable times (such as when you are sleeping). But the amount of info you get this way is necessarily lower than what can be gleaned from a debugger.

"you can’t use them when your application is running on remote environments"

This isn't always the case. Maybe it's really hard in a lot of cases, but it's always not impossible.

It isn't either/or. Good programmers know how to use both and know how to choose the appropriate tool for the job.

It may sound obvious to folks who already use a debugger, but in my experience a decent chunk of people don't use them because they just don't know about them.

Spread the good word!

debuggers are hard to use outside of userland.

For really hairy bugs in programs that can't be stopped (kernel/drivers/realtime, etc) logging works.

And when it doesn't, like when you can't do I/O or switching of any kind, log non-blocking to a buffer that is dumped elsewhere.

also, related. It is harder than it should be to debug the linux kernel. Just getting a symboled stack trace is ridiculously hard.

While a debugger is of high value, having access to a REPL also covers the major use cases.

In particular, REPL tools will work on remote session, on pre-production servers etc. _if_ the code base is organized in a somewhat modular way, it can be more pleasant than a debugger at times.

Makes me wonder if the state of debugging improved in PHP land. It was mostly unusable for batch process debugging, or when the server memory wasn't infinite, which is kinda the case most of the time for us mere mortals.

Cases where I absolutely need to do these things to solve a bug: 0%

Maybe 0.1%

Don't show the discussion to John Carmack. He's baffled why people are so allergic to debuggers: https://youtu.be/tzr7hRXcwkw?si=beXGdoePRkbgfTtL

I have counter-points to several of these... But this one is my favorite (This didn't go very far, but I loved the idea of it...):

I once wrote a program that opened up all of my code, and at every single code curly brace, it added a macro call, and a guid.

  void main() { DEBUGVIKINGCODER("72111b10c07b4a959510562a295cb2ac");
    ...
  }

I had to avoid doing that inside other macros, or inside Struct or Class definitions, enums, etc. But it wasn't hard, and it was a pretty sizeable codebase.

The DEBUGVIKINGCODER macro, or whatever I called it, was a no-op in release. But in Debug or testing builds, would do something like:

  DebugVikingCoder coder##__LINE__("72111b10c07b4a959510562a295cb2ac");

(Using the right macros to append __LINE__ to the variable, so there's no collisions.)

The constructor for DebugVikingCoder used a thread-local variable to write to a file (named after the thread id). It would write, essentially,

  Enter 72111b10c07b4a959510562a295cb2ac (epoch time)

The destructor, when that scope was exited, would write to the same file:

  Exit 72111b10c07b4a959510562a295cb2ac (epoch time)

So when I'd run the program, I'd get a directory full of files, one per thread.

Then I wrote another program that would read those all up, and would also read the code, and learn the File Name, Line Number of every GUID...

And, in Visual Studio, this tool program would print to the Output window, the File Name and Line Number, of every call and return.

And, in Visual Studio, you can step forward AND BACK in this Output window, and if you format it correctly, it'll open the file at that point, too.

So I could step forwards and backwards, through the code, to see who called where, etc. I could search in this Output window to jump to the function call I was looking for, and then walk backwards...

Then I added some code that would compare one run to another, and argued we could use that to figure out which of our automated tests formed a "basis set" to execute all of our code...

And to recommend which automated tests we should run, based on past analysis.

In addition to being able to time calls to functions, of course.

So then I added printing out some variables... And printing out lines in the middle of functions, when I wanted to time a section...

And if people respected the GUIDs, making a new one when they forked code, and leaving it alone if they moved code, we could have tracked how unit tests and other automation changed over time.

That got me really wishing that every new call scope really did have a GUID, in all the code we write... And I wished that it was essentially hidden from the developers, because who wants to see that? But, wow, it'd be nice if it was there.

I know there are debuggers that can go backwards and forwards in time... But I feel like being able to compare runs, over weeks and months, as the code is changing, is an under-appreciated objective.

> Some would’ve also heard about time travel debuggers (TTD) which let you step back in time. But most languages do not have a mature TTD implementation. So I am not writing about that.

Shame as that's likely the only option with significant universal UX advantage vs. sprinkling prints...

There was a time when I built games entirely using Visual Studio 6 Edit and Continue. These were the days when debuggers were reliable. Nowadays, I treat the debugger’s output like a best guess: it’s probably right about local variable values and the call stack, but it sometimes has nothing useful to say, and very occasionally is actively misleading.

Didn't expect this to blow up, and now I realize it's bit of a flame bait topic, haha.

What’s a good debugger for bash?