Debugging a kernel you cannot attach a debugger to

The nicest kernel bugs announce themselves: a panic, a bug check, a stack trace on the console. The ones that cost weeks do not. The board resets with nothing on the serial line. The system hangs with interrupts apparently working and no thread making progress. A fault appears only when the fourth core is online, only after twenty minutes, only with the display driver loaded. There is no JTAG header, or there is one and the vendor’s debug tools do not understand your kernel.

This is the toolkit we reach for in that situation, roughly in the order we reach for it. None of it is exotic. The point is that every tool below turns an opinion (“I think it’s the scheduler”) into something the machine can confirm or deny.

1. Make the console tell the truth

Before anything else: a serial console with timestamps, written by a path that does not take locks, does not allocate, and works with the MMU on or off and from any core. Most “silent” failures are silent because the logging path itself needs something that is broken, a lock held by the hung core, a buffer that is not mapped yet, an interrupt that never fires to drain a ring. A raw polled UART write, used only for debugging, bypasses all of that. Put the core number and a monotonic timestamp in front of every line so that interleaved output from four cores can be untangled afterwards.

2. Build a model you can attach to

If the bug reproduces in QEMU, you have won most of the battle, because QEMU gives you what the board does not: a gdb stub (-s -S) that stops every core at once and lets you inspect any register, any page table and any memory location; a monitor (QMP) that can dump guest memory, take a screenshot of the framebuffer and inject events on demand; and deterministic restarts. We keep a QEMU model of every board we work on, even when the model is imperfect, because “reproduces in the model” and “does not reproduce in the model” are both informative.

Two refinements make this far more useful than raw gdb:

• Route the kernel’s own debugger through the model. Most kernels have a native debugging protocol (a kernel debugger that speaks over serial). Expose a QEMU character device as that serial port, and you can use the kernel’s own symbolic debugger, with its understanding of threads, IRQ levels and kernel data structures, against the virtual machine, while gdb still sits underneath for the moments when the kernel debugger itself is what broke.
• Choose the accelerator deliberately. QEMU’s software emulator on an x86 host hides ARM64 memory-ordering bugs, because the host executes with stronger ordering than the guest architecture. For that class of bug run the model on an ARM host with hardware virtualisation, where the guest reorders for real. We wrote about this in ARM64 SMP bugs are memory-ordering bugs until proven otherwise.

3. A crash ring that survives the reset

For the board that resets silently, the console is useless: whatever was about to be printed is gone. What survives a warm reset on most SoCs is DRAM. Reserve a small region that the kernel never allocates from, put a ring buffer with a magic header and a checksum in it, and write every significant event there, cheaply, before it happens: entering an interrupt handler, taking a lock that is known to be involved, starting a DMA, switching threads on a core. On the next boot, before anything else, check the magic and the checksum, and if the ring is intact, print it.

This is the same idea as Linux’s pstore and ramoops, and it is worth building even in kernels that do not have it, because it converts “the board reset” into “the board reset 40 microseconds after core 2 started a page-table update while core 0 was in the display interrupt”. That sentence is a bug report; “the board reset” is not.

The reserved region must be excluded from cache maintenance that would wipe it, must not be zeroed by the bootloader (check; many do), and must be written with a barrier after each entry so the entry is in DRAM and not in a write buffer when the reset comes.

4. A watchdog with a pre-timeout

Hangs are the mirror image of resets: the machine is alive but nothing progresses. A hardware watchdog turns a hang into a reset, which combined with the crash ring turns it into a report. Better still is a watchdog with a pre-timeout interrupt: it fires a few seconds before the reset, giving you a chance to dump every core’s program counter, the current thread on each core, the lock owners and the last entries of each per-core event ring to the polled console. Many SoCs’ watchdogs have this; where they do not, a spare timer interrupt at a high priority that is only ever reset by the scheduler tick does the same job.

The dump routine must itself work from interrupt context on a system that may have every lock held. This is the polled-console rule again: no locks, no allocation, no dependencies.

5. Counters that make hypotheses falsifiable

Every debugging session generates hypotheses. “Interrupts are being lost.” “The DMA completes but the completion is not seen.” “Core 3 never gets the IPI.” Each of these is a claim about a count, and each can be settled by adding the count. We keep a block of per-CPU counters, readable from the console at any time, for exactly this: interrupts received per source per core, IPIs sent and received, DMA started and completed, locks contended, TLB shootdowns issued and acknowledged. When a hypothesis survives contact with its counters, it is probably right. Most do not.

Counters are cheap enough to leave in production builds. A field failure that comes with a counter dump attached is a field failure you can fix.

6. Bisection with freeze points

When the failure is probabilistic and late (“about 40 percent of boots, some time after the desktop appears”), the aim is to shrink the window. A freeze point is a place in the boot sequence where the kernel can be told to stop starting new things: no further drivers, no further cores, no display initialisation. Boot fifty times with the freeze point at stage N; if the failure never appears, move it later; if it does, move it earlier. Each round halves the code that can be responsible. It is slow and mechanical, and it works on problems where thinking does not.

Automate the fifty boots. A script that power-cycles the board (a relay, a USB-controlled power switch, or QEMU restarts), captures the console, and classifies each run as pass or fail is a day of work that pays for itself the first time you need to bisect anything.

7. When you can get real debug access

None of the above replaces hardware debug where it exists. Cortex-M targets have SWD; most Cortex-A boards expose JTAG on a header or on multiplexed GPIOs (the Raspberry Pi family among them), and an inexpensive probe with OpenOCD will halt all cores and read anything. The reason we lead with the software tools is that the hardest bugs, ordering races and reset storms, often disappear or change when a probe halts the system, and because the software tools keep working on the units in the field.

The habits that matter more than the tools

• Keep a bug journal. Every hypothesis, every experiment, every result, with timestamps. Long investigations loop without it.
• Change one thing at a time, and record the base commit of every run. The failure that “went away” after three simultaneous changes is still there.
• Save the rig. The QEMU command line, the console scripts, the counter dump parser, the power-cycle loop: check them in next to the kernel. The next bug will need them, and so will the regression test for this one.
• Write the root cause down before writing the fix. If you cannot explain, in one paragraph, why the failure happened and why the fix prevents it, you have a workaround, not a fix. Workarounds come back.