Where the packets per second go: SDIO Wi-Fi throughput on a Broadcom CYW43455

The Broadcom CYW43455 is the Wi-Fi and Bluetooth combo chip on the Raspberry Pi 3B+, Pi 4, Compute Module 4 and Pi 5, and on a long list of other boards. The host talks to it over SDIO: firmware is downloaded through backplane windows on function 1, and control and data frames move over function 2 using Broadcom’s SDPCM framing. The datasheet says 802.11ac; a good link should carry tens of megabits without effort.

The driver we were handed carried about one megabit per second, with the CPU idle and a strong signal. This is a common shape of problem, and it is almost never the radio. It is the host side spending its time on bus overhead instead of payload. Here is where the packets per second went, in the order we found them, and the measurements that pointed at each one.

First, count transactions, not megabits

Before touching the driver we added four counters: SDIO commands issued, bytes moved on the bus, frames delivered to the network stack, and interrupts taken. Dividing them gave the two numbers that explain everything below: SDIO transactions per frame and bytes per transaction. A healthy driver moves a few kilobytes per transaction and needs about one transaction per frame or less. Ours was doing three transactions per frame, each carrying a single frame, and the receive path was capped at roughly 460 frames per second regardless of frame size. At 1500-byte frames that is not far from the one megabit we were seeing.

Each SDIO CMD53 has a fixed cost: command, response, data block handshake, and the interrupt or poll that tells you it finished. On this host controller that fixed cost was around two milliseconds end to end with the way the driver waited for completion. 460 frames per second is simply one divided by that. Every fix below is a way to do fewer transactions or make each one carry more.

Cause 1: receive glomming was never enabled

The CYW43455 firmware can aggregate several received frames into one “superframe” so the host fetches them in a single transfer. This is the bus:rxglom setting, exposed as an iovar the host writes during initialisation, with a matching bus:txglom for the transmit direction. Our driver never set it. The firmware, sensibly, delivered one frame per transfer.

Enabling receive glomming, and teaching the receive path to walk the sub-frames inside a glom (each carries its own SDPCM header with a length; the glom descriptor at the front lists them), was the single largest improvement. Frames per transaction went from one to a variable number that under load averaged around six. Nothing else changed and throughput more than quadrupled.

Cause 2: the bus clock was left at the enumeration rate

SDIO cards enumerate at a low clock and the host is expected to switch to high speed, and on hosts and cards that support it, to the faster UHS modes. The driver had switched to 50 MHz high-speed mode and stopped there; the host controller and the chip both supported a 100 MHz mode. Doubling the clock did more than double the receive rate, because the fixed per-transaction overhead (which includes clocked command and response phases) shrinks as well. This is the second-cheapest fix on the list and it belongs in the initialisation code, not a tuning guide.

Two cautions. The faster modes need the signalling voltage switch and, on some hosts, tuning; if the host controller driver does not implement tuning correctly you will see CRC errors under load rather than a clean failure. And the SDIO clock is often shared with, or constrained by, the board’s other SD interfaces; check the board’s clock tree before assuming the datasheet maximum is reachable.

Cause 3: byte-mode transfers longer than 512 bytes

CMD53 has two modes. Byte mode transfers up to 512 bytes in a single unit. Block mode transfers a count of fixed-size blocks and is how anything larger should move. The driver used byte mode everywhere and computed the count field modulo 512 for large transfers, which meant that glommed superframes bigger than 512 bytes were silently truncated on the bus. The symptom was maddening: enabling glomming worked for small frames and corrupted large ones, which looked like a firmware bug in the glom feature.

The fix is boring: set the function’s block size once (512 is conventional for these chips), use block mode for any transfer larger than one block, and pad the transfer to a whole number of blocks. Once that was in place, glomming with large frames worked exactly as documented.

Cause 4: an interrupt mask that absorbed interrupts

The card interrupt that stopped arriving

SDIO delivers a card interrupt on a dedicated line (or in-band on DAT1) when the chip has frames for the host. The usual handling is: interrupt fires, the host masks it, drains everything the chip has, unmasks it. Our interrupt handler masked the line in one place and unmasked it in another, and there was a path, taken when the drain loop found nothing because the chip was mid-update, that returned without unmasking. From then on the card could raise the interrupt as much as it liked; nobody was listening. The driver fell back to a slow poll it happened to have, which is why the link limped along instead of dying, and why the problem looked like “sometimes throughput drops for a few seconds”.

Structure interrupt handling so that mask and unmask are in one function with one exit, and count masks and unmasks. If the two counters ever differ by more than one, you have this bug.

The interrupt storm that never stopped

The opposite failure appeared once the mask was fixed. The SDPCM protocol includes a credit and sequence handshake in every frame header; the host acknowledges what it has consumed by echoing sequence numbers back in the headers of frames it sends, and the firmware keeps a “frame indication” asserted until it sees that acknowledgement. If the host consumes frames but never sends anything, the indication stays asserted and the interrupt line stays high. A receive-only workload (a download) therefore produced an interrupt storm: tens of thousands of interrupts a second, each finding nothing new. The fix was to send a zero-length control frame carrying the updated sequence numbers whenever we had consumed data and had nothing else to send, which is what the firmware expects.

Cause 5: two transactions per frame on the host side

The receive path read a frame in two steps: first the SDPCM header to learn the frame length, then the body. Two transactions, two fixed costs. The chip already tells you the length of the next frame in a “lookahead” field of the current frame’s header, precisely so the host can issue a single correctly sized read. Caching the lookahead and using it, falling back to the two-step read only when the lookahead is absent or inconsistent, halved the transactions per frame on top of the glomming gain.

The same pass fixed a copy problem. Received data landed in a DMA bounce buffer mapped non-cacheable, and was then parsed in place, header field by header field, with every access going to uncached memory. Copying the whole superframe into cacheable memory once, then parsing, was several times faster than parsing in place. Non-cacheable memory is for transferring data, not for working on it.

Cause 6: TCP was bound by round-trip time, not by the link

With the bus fixed, raw UDP throughput was where it should be and TCP still lagged. The network stack’s default send buffer was small enough that a single TCP connection could not keep more than a few kilobytes in flight; over a Wi-Fi link with 30 to 50 milliseconds of round-trip time that caps throughput at a few hundred kilobytes per second regardless of link rate. Raising the default socket buffers, and making sure delayed acknowledgements were not adding a further 200 milliseconds to every window, brought TCP up to the UDP number. This is not a driver fix, but nobody thanks you for a fast driver under a slow stack.

What the numbers looked like

We give ratios rather than absolute megabits deliberately: the absolute number depends on the access point, the channel and the board, and quoting it would invite the wrong comparison. The ratios reproduce.