Bringing an OS up on Raspberry Pi 5: what the RP1, PCIe and the GIC demand from your kernel

The Raspberry Pi 5 looks like a bigger Raspberry Pi 4. From the kernel’s point of view it is a different machine. The BCM2712 SoC keeps the four Cortex-A76 cores, the GIC and the memory controller close, and pushes almost everything a user cares about, USB 3, Gigabit Ethernet, GPIO, UART, SPI and I²C, into a separate chip called RP1 that hangs off a PCIe Gen 2 x4 link. A kernel that “supports Raspberry Pi” in the Pi 4 sense boots to a serial console on the Pi 5 and then discovers that it cannot see a single peripheral until it can enumerate PCIe.

These are our notes from bringing an ARM64 operating system kernel up on that board. They are not a tutorial; they are the list of things that cost us days, in the order we hit them, so that the next team spends hours instead.

1. The boot path: UEFI, PSCI and cores that arrive at the wrong exception level

We boot through the EDK2-based UEFI firmware for the Pi rather than the vendor’s Linux-only loader chain. UEFI hands the kernel a framebuffer through the Graphics Output Protocol, a memory map, and either ACPI tables or a device tree describing the SoC. That is enough to get a console up with no driver work at all, which is exactly what you want on day one.

Secondary cores are started with PSCI CPU_ON calls into the firmware. The surprise is where they show up. On this firmware stack the application processors enter the kernel’s secondary entry point at EL2, the hypervisor exception level, not at EL1 where the boot core is running by the time it starts them. If your secondary entry code assumes EL1 it will fault on the first system-register access that is not valid at EL2, and it will do so with the MMU off and no console, which reads as “the other three cores never come up”.

The fix is a small trampoline executed before anything else on the secondary path: detect the current exception level from CurrentEL, and if it is EL2, configure HCR_EL2 for a non-VHE EL1 guest, give EL1 a sane SCTLR_EL1, set SPSR_EL2 to EL1h with interrupts masked, load ELR_EL2 with the real entry point and eret. Ten instructions, one afternoon to find.

mrs   x0, CurrentEL
cmp   x0, #(2 << 2)
b.ne  1f                      // already EL1
mov   x0, #(1 << 31)          // HCR_EL2.RW: EL1 is AArch64
msr   hcr_el2, x0
mov   x0, #0x30d00800         // SCTLR_EL1 reset value, MMU off
msr   sctlr_el1, x0
mov   x0, #0x3c5              // SPSR: EL1h, DAIF masked
msr   spsr_el2, x0
adr   x0, 1f
msr   elr_el2, x0
eret
1:  // EL1 secondary entry continues here

2. PCIe first, everything else second

The RP1 appears as an ordinary PCIe endpoint behind the BCM2712 root complex. Its peripheral register blocks are exposed through a BAR, so the order of operations is fixed: bring the link up, enumerate, assign BARs, map the window, and only then start talking to “the GPIO block” or “the xHCI controller” at their offsets inside it. Any driver that hard-codes the Pi 4 physical addresses for those peripherals is simply reading unmapped memory on the Pi 5.

The part that cost us the most time was DMA. Devices inside RP1 (the xHCI controllers, the Ethernet MAC) are bus masters that read descriptors and write completion records into system memory across the PCIe link. In our bring-up those inbound writes were not coherent with the A76 data caches. Concretely: the xHCI controller would write a transfer event into the event ring, and the CPU would keep reading the stale cache line that said “nothing here yet”. In the other direction, the CPU would write a transfer request block, and the controller would fetch the old contents from DRAM because the new bytes were still sitting dirty in L1.

There are two honest ways to handle this and one dishonest one. The dishonest one is to sprinkle cache flushes until the symptom goes away. The honest options are:

• Map DMA memory as non-cacheable. Descriptor rings, event rings and small control structures are read once and written once; mapping them Normal Non-cacheable removes the problem for that memory entirely, at the cost of slower CPU access. For rings this cost is negligible.
• Bounce and maintain for bulk data. Large data buffers stay cacheable for the CPU’s benefit; before handing a buffer to the device you clean the cache lines it covers (DC CVAC per line, then DSB SY), and after the device has written you invalidate them (DC IVAC) before the CPU reads. The buffer must be cache-line aligned and padded, or the invalidate will destroy an unrelated variable sharing the last line. We learned that one the usual way.

We chose non-cacheable rings plus bounce buffers with explicit maintenance for xHCI payloads. Whatever you choose, write it down in the DMA layer’s header comment, because the next engineer will assume coherent DMA. Everyone does.

3. Interrupts: MSIs, the ITS and one invariant that must never break

Devices behind PCIe signal interrupts as MSI or MSI-X writes, and on a GICv3 system those land in the Interrupt Translation Service, which turns a (device ID, event ID) pair into a Locality-specific Peripheral Interrupt delivered to one redistributor. Getting LPIs flowing end to end means:

1. Allocating the LPI configuration and pending tables, programming GICR_PROPBASER and GICR_PENDBASER on every redistributor with cacheability and shareability attributes that match how you actually access the tables, and enabling LPIs in GICR_CTLR.
2. Allocating the ITS device table, a command queue, and per-device interrupt translation tables.
3. Issuing the command sequence MAPC (collection to redistributor), MAPD (device to ITT), MAPTI (event to LPI and collection), then INV/INVALL after any change to the property table, and SYNC before you expect anything to be delivered.
4. Programming the endpoint’s MSI capability with the ITS translation register address and the event ID as data.

We found six distinct bugs in our path through that list, and every one of them presented as the same symptom: the device asserts an interrupt, the pending bit never shows, nothing happens. The most useful debugging aid was a tiny telemetry block per redistributor (LPIs received, acknowledged, EOI’d, spurious) that we could dump from the serial console. Without it, “no interrupt” is one symptom hiding six causes.

The invariant worth engraving somewhere: the GIC priority mask must always agree with the operating system’s notion of the current interrupt level. If your kernel raises its internal level without writing ICC_PMR_EL1, an interrupt that should have been held off arrives inside a critical section. If it lowers the level without restoring the mask, interrupts stay silently blocked and the machine looks hung. Every path that changes one must change the other, atomically with respect to interrupts, and an assertion that checks the pair on every interrupt entry catches violations within seconds.

4. xHCI: the flag that decides whether bulk transfers ever finish

With PCIe, DMA and MSIs working, the xHCI bring-up is by the book: reset the controller, program the device context base array, the command ring and the event ring segment table, allocate the scratchpad buffers the controller asks for, then reset ports and enumerate. Control transfers worked on the first try. Bulk-IN transfers did not: a device would answer with data and our completion logic would wait forever.

The cause was one bit. A bulk-IN request is usually posted as a transfer descriptor spanning several TRBs, with Interrupt-On-Completion set only on the last one. When the device returns a short packet, which is normal for anything that is not a multiple of the max packet size, the controller ends the transfer early. Whether it then tells you depends on the Interrupt-on-Short-Packet flag on the TRB where the short packet landed. We had not set it, so our driver saw nothing and waited for an IOC that could never come. Setting ISP on every TRB of an IN transfer, which is what production drivers do, fixed bulk-IN outright.

A smaller lesson from the same layer: our first enumeration path reset every port twice, once in the hub logic and once in the device-attach logic. That is harmless and costs about 160 milliseconds per device. On a board where the keyboard, the mouse and the Ethernet controller all enumerate at boot, removing the second reset took first-USB-device time from roughly 620 to 460 milliseconds. Bring-up code accumulates this kind of redundancy; measure it before shipping.

5. Timers, and where the next bugs live

The ARM generic timer is the easy part: read CNTFRQ_EL0 for the frequency, program per-core timers through CNTP_CVAL_EL0, take the PPI. Do not hard-code the frequency; firmware sets it, and it differs between boards and between firmware versions.

Once four cores are running with real interrupts and real DMA, the bugs change character. They stop being “the hardware is not configured” and become “this works 49 boots out of 50”. Almost all of those are memory-ordering problems, and they deserve their own article: ARM64 SMP bugs are memory-ordering bugs until proven otherwise.