ARM64 SMP bugs are memory-ordering bugs until proven otherwise

Here is a pattern we see on almost every ARM64 port of code that grew up on x86. The system boots and runs for hours with one core. With four cores it fails roughly one boot in fifty: a wait that never completes, a pointer that is briefly null, a device that stops responding. Nothing in the logs. The failure moves when you add a print statement. The team suspects the hardware, then the compiler, then each other.

It is almost always memory ordering. x86 implements a strong model (total store order) in which the only reordering a programmer can observe is a store being delayed past a later load. ARM64 is weakly ordered: loads and stores to different addresses may become visible to other cores in almost any order unless you say otherwise. Code that was accidentally correct on x86 for a decade is simply incorrect on a Cortex-A76, and the A76 is aggressive enough to prove it.

This article is the taxonomy we use when we are handed one of these bugs. Six patterns cover nearly everything we have chased.

Pattern 1: publish before signal

A producer fills in a structure, then sets a flag, increments a sequence number, or writes a pointer into a table so a consumer can find it. The consumer sees the flag and reads the structure. On x86 this works because the two stores are observed in program order. On ARM64 the consumer can observe the flag before the structure’s contents.

We hit exactly this in a USB host controller driver: a slot table entry was written, then the waiter was signalled, and one time in a few hundred the woken thread read a slot marked “in use” whose fields were still zero. The fix is a release store on the producer side and an acquire load on the consumer side, or a full barrier if the primitives do not exist in your kernel’s toolkit.

/* producer */
slot->dev   = dev;
slot->state = SLOT_READY;
__atomic_store_n(&table[i].ready, 1, __ATOMIC_RELEASE);   /* STLR */

/* consumer */
if (__atomic_load_n(&table[i].ready, __ATOMIC_ACQUIRE)) { /* LDAR */
    use(table[i].slot->dev);                             /* now guaranteed visible */
}

The rule: any time one core hands data to another through a flag, a pointer or a queue index, the hand-off needs release/acquire semantics. If you find yourself writing “it’s just a bool” you have found the bug.

Pattern 2: per-CPU state read with interrupts enabled

Kernels keep per-CPU structures: the current thread, the current interrupt level, scheduler queues. Reading “my CPU’s structure” is two steps, find the pointer, then dereference it. If an interrupt can arrive between the two, and the interrupt handler or a rescheduling on return migrates the thread to another CPU, the second step operates on the wrong CPU’s data.

On x86 kernels this is often masked by the way per-CPU access is implemented (a segment-relative load is a single instruction). On ARM64 the per-CPU base lives in a system register such as TPIDR_EL1, the access is several instructions, and the window is real. We had a set of per-CPU accessors used from paths that could be interrupted, and the resulting corruption of the interrupt-level bookkeeping looked like random priority inversions. The fix was to mask interrupts (msr daifset, #2) around every read-modify-write of per-CPU state that is not already running with interrupts off, and to assert on entry to those accessors that DAIF.I is set.

Pattern 3: page tables published without barriers

Writing a page-table entry is a store to memory like any other. The MMU’s table walker on another core, or on this core after a TLB miss, will only see it after the store is visible at the point of coherency for table walks. The sequence for a change that must be seen everywhere is:

1. Write the entry (for a change to a valid mapping, first write an invalid entry, invalidate, then write the new one: break-before-make).
2. DSB ISHST so the store is complete before the invalidate.
3. TLBI VAE1IS, Xn (or the appropriate variant) to invalidate the old translation on all cores in the inner shareable domain.
4. DSB ISH to wait for the invalidate to complete everywhere.
5. ISB if the mapping affects instruction fetch on this core.

Skipping the first DSB is a classic: the TLB invalidate can complete before the new entry is visible, another core walks the table, caches the stale translation, and you get a fault on a page that “is definitely mapped”. The symptom is a spurious page fault or a null-pointer dereference at an address that is valid by the time you look at it in the debugger.

Pattern 4: locks that are not actually locks

Home-grown spinlocks

A spinlock acquire must have acquire semantics and a release must have release semantics; otherwise the critical section’s memory accesses can leak outside the lock. The ARM64 idiom is LDAXR/STXR in the acquire loop (the load-acquire is what orders the critical section after the lock) and STLR for the release. A lock built from plain LDXR/STXR with no barriers mutually excludes two cores from the lock word and nothing else. Use the atomics your compiler provides with explicit memory orders and stop writing these by hand.

Lock hierarchy across scheduler paths

The other lock bug that only appears with SMP is not ordering but coverage: a structure that is protected by a lock in the common path and modified without it in an uncommon one. We found a thread-affinity change path that rewrote a thread’s scheduling state while the scheduler on another core was reading it under the thread’s lock. One core would then wait on a timer that had been re-queued elsewhere and never fire. The uncommon path had been written for a single-core system where “nobody else can be looking” was true.

Pattern 5: memory-mapped I/O ordered against normal memory

Device registers are mapped as Device memory, which has strong ordering among accesses to the same device. That does not order them against stores to Normal memory. The classic sequence is: fill in a DMA descriptor in RAM, then write the doorbell register. Without a barrier between them, the doorbell write can become visible to the device before the descriptor store, and the device reads a stale descriptor. The fix is a DMB OSHST (or DSB if the descriptor memory is non-cacheable and you want to be certain the write has reached it) between the two, wrapped in whatever your kernel calls its write-memory-barrier for I/O.

The same applies in reverse for reading a status register and then reading the DMA’d data: a read barrier between them, or the data read can be satisfied from a stale line before the status read confirms it is there.

Pattern 6: signalling a core that is not listening

Cross-core work (inter-processor interrupts for rescheduling, TLB shootdowns, “freeze all cores” for a debugger) is a protocol with two sides, and each side’s state must be published with the same care as Pattern 1. Our worst example was a freeze/thaw protocol in which a core could acknowledge a freeze, be thawed, and be frozen again before the coordinator had consumed the first acknowledgement; the second freeze then waited for an acknowledgement that had already been counted. On one core none of this can happen. On four it happens under load about once an hour.

Design these protocols with explicit sequence numbers, make every transition a compare-and-swap, and write a state diagram before code.

Why QEMU on your laptop will not show you any of this

QEMU’s software emulator translates ARM64 instructions into host instructions. On an x86 host, the host executes those with x86’s strong ordering, so most weak-memory behaviour never occurs: the emulated system is far more forgiving than the real chip. Ordering bugs that fail hourly on a Cortex-A76 can run for weeks in TCG on an x86 laptop.

To reproduce them you need either real hardware or an ARM host with hardware virtualisation (KVM on an ARM server, or Apple’s Hypervisor framework on Apple silicon), where the guest’s loads and stores execute natively and reorder for real. Our standard rig for this class of bug is a four-core ARM64 virtual machine on an ARM host, with the kernel’s own debugger attached over a serial channel, running a stress mix of thread migration, timer churn and device I/O. Bugs that took a week to see on a board usually appear within minutes there.

Instrumentation that finds ordering bugs

• Spin timeouts with backtraces. Any spin that exceeds a generous bound (say 100 ms) dumps the spinning core’s backtrace and the owner’s. Hangs become stack traces.
• Lock owner and level tracking. Every lock records which CPU and thread hold it and at what interrupt level; an acquire at the wrong level asserts immediately instead of deadlocking later.
• Invariant assertions on interrupt entry. The OS interrupt level and the hardware priority mask must agree; the current thread’s CPU field must equal the CPU it is running on. Checked on every entry, these catch Pattern 2 in seconds.
• Per-CPU event rings. A fixed-size ring of (timestamp, event, argument) per core that survives the crash. When a wait never completes, the last 64 events on each core tell you who signalled whom and when.