As a followup to my recent post on debugging, when confronted with an extremely difficult bug, I find there are essentially two mental processes you mix:

Deductive: Spend enough time to fully work through all of the code paths to try and find an intuitive solution to the problem. In some cases, you can eliminate the debugging step entirely, particularly if you sleep on it and take the time to unleash your subconscious problem solving abilities. How awesome when you wake up or get out of the shower with the fix ready to apply?

Inductive: Figure out how to get as much context for the problem as quickly as possible in the form of either log files or a targeted debugging session before spending too much time on difficult thought processes. Try to break the debugger in the affected code path when it is in as close a state as possible to the problematic code path and step around. You may trip across the bug or at least guide your deductive thinking with more information.

Back in college, my professors taught us to focus on the deductive process. You should understand your problem domain, ensure it is well designed from top to bottom. Avoid relying on the narrow insights you gain by observation to guide code changes. Leaping into the debugger may help you quickly patch the problem but lead to an overall less stable system. Maybe the fix you find in the debugger only addresses one case of many or a category of similar bugs could be fixed at a higher level with a different change.

But to counter that wisdom, it’s often much faster to understand the problem with a more targeted approach. Time spent visualizing the code paths in your head may be more productive with a debug session in front of you so you can more quickly trace code paths, and avoid drawing out data structures.

In debugging, either approach may waste time. For the inductive process, you can waste time writing unnecessary logging code or throwaway code to catch a complex code condition with a breakpoint, or stepping through a morass of code and learning nothing. Your deductive approach may lead you to waste time relearning a piece of code at a level of detail you’ll quickly forget again when the problem is something simple or unrelated to your initial assumptions. We programmers tend to find it hard to account for debugging time so learning how to debug more efficiently is key to making deadlines. So let me reveal the key to debugging productivity: use induction to focus deducation and deduction to focus induction. Make sense?

Some more related tips. When your system’s turnaround time between code change and test is low, your ability to use inductive processes increases, leading to less frustrating time on thinking which derails other creative thought, like the next new feature. But don’t let ideal conditions lead you to lazy thinking. You’d be a fool to eliminate deductive reasoning during debugging entirely. When your system’s turnaround time is high, use deductive thinking while waiting to plan each test to make most use of your next session.

I’ve found the best state is to let your intuition guide you and mix approaches at least enough to avoid wasting too much time on the other one. A debugging session to learn more or writing simple logging or conditional breakpoint code can be a mental break from too much deductive thinking. Some deductive thinking can lead to inspiration when you are mentally taxed by stepping through too much code.

Now go make your stuff work so we have less crappy software to infuriate our lives 🙂

The complexity of the solutions you can build is limited by the complexity you can debug. When code gets complex, debugging gets harder. More data is involved, setup is more difficult, problems harder to reproduce. Code paths are more complex. Data structures become unwieldy nested graphs. Stack traces many hundreds of frames deep. Multi-threaded timing problems, deadlocks, and intermittment errors. Do you yield at this point or do you dig in and find and fix the problems? If you walk away in fear, your system faces failure or at least may live as a buggy, hated thing people want to replace, instead of a solid, stable system that runs for years.

One of the reasons I’ve been successful in my career is that I’m good at debugging the hard problems. I’m not afraid to tackle a more complex design as I am confident I can solve the more complex problems that will arise. When working with teams, I can save a lot of time by helping others find those problems that can suck up days and weeks often in much less time.

I’ve wanted to write a post about debugging for a while but it is a dauntingly complex problem, worthy of a book. I learned most of my tricks by sitting over the shoulders of great programmers while they debug problems. I’ve learned over the years that there’s no substitute for taking an intuitive approach in the time savings involved. Even with a list of the many things you to find a problem, the secret is applying them in the right way at the right time.

Despite the difficulty, I took some time to write down some approaches I’ve found useful over the years:

1. Have the right attitude. You will find the bug, it’s only a matter of time. The more frequent a bug occurs, the easier it is to find because of all the data gathering opportunities. The longer between occurrences, the more time you have to prepare for the next occurrence so you can catch it.
2. Familiarize yourself with all of the processes, threads, data structures involved.
3. Even if you can’t easily reproduce the bug in the lab, use the debugger to understand the affected code. Judiciously stepping into or over functions based on your level of “need to know” about that code path. Examine live data and stack traces to augment your knowledge of the code paths.
4. When you can’t reproduce the bug, you may need to instrument the code with additional logging. Make sure the skeleton of major operations has adequate logging to understand what’s happening in the system. Investing some effort in improving the targeted quality, readability of these logs will go far in the overall lifecycle of a complex system. Too much logging swamps performance and hurts readability. But with time-stamps, user-ids, user-agent strings, session-id, basic operations, you learn a lot about the running system and why it might have failed for one particular user. Logging is crucial for multi-threaded interactions.
5. Generate theories as to what might be causing the problem and test those theories. Keep an open mind. Generate as many theories as possible before you start the longer process of testing those theories. You may decide to test more than one at the same time.
6. If you have no theories, you need to learn more about the system, particularly information relevant to the code paths causing the bug. Adding additional logging is a good way to do that when sporadic errors cannot be reproduced.
7. Familiarize yourself with all of the layers of the system, at least at an intuitive level, from the hardware on up. This will help you visualize what’s going on in in your mind’s eye so your intuition can help steer you towards the most likely source of the problem.
8. For certain types of complex code, I will write debugging code, which I put in temporarily just to isolate a specific code path where a simple breakpoint won’t do. I’ve found using the debugger’s conditional breakpoints is usually too slow when the code path you are testing is complicated. You may hit a specific method 1000’s of times before the one that causes the failure. The only way to stop in the right iteration is to add specific code to test for values of input parameters. Aways do a System.out.println or log some visible, unique consistent token.   This makes it easy to find and remove these code snippets when debugging is complete. Once you stop at the interesting point, you can examine all of the relevant state and use that to understand more about the program.
9. Some people start out by drawing pictures, flow charts, entity-relationship diagrams of their data structures, and detailed state tables. I will do this only as a last resort or for documenting the project as it is time consuming. Examining a real instance of that data structures in the debugger is much faster, more accurate and more informative than any diagram. The JavaDoc or structured code browser are enough for me to understand the entities and relationships. I try to visualize data structures in my head and only resort to a drawings, or state tables when necessary. I think that over time, this has made me faster at visualizing and building systems.
10. If you get stuck, take a break. Sleep on it, approach things fresh the next day. You may not have enough information and may need more to get the next piece of the puzzle. Too much frustration impedes your motivation and ability to focus. For the best debugging approach, you need to research all relevant aspects of a system, simulate that all in your head, use your intuition to flush out ways things could be going wrong. Instead of trying to find the problem, perhaps you need to learn more about the failure.  Under what conditions does it happen?   What’s unique about those cases?  How can you learn more about those unique code paths?
11. Do not spend too much time on minor bugs but do keep in mind the value of true reliability in a system.  Your pride is not relevant.  Your customers’ experience in using the software is all that matters.

If you become good at debugging complex problems, your confidence as a programmer will grow, letting you tackle bigger, more relevant problems. When things go wrong, you’ll be able to step up and make things right again.

Did I miss any of your favorite debugging tips?  Continue the discussion in the comments!