genet MTU for upstream

[nbuchwitz]

Based on @6by9's #7614, with the goal of upstreaming the MTU support.

I've tested the original patch on CM4 and discovered some issues. So I've created a slightly different patch (series) which I intend to send to netdev. It also contains some fixes Sashiko would have flagged any way...

1. TBUF_PKT_RDY_THLD (TBUF + 0x10) is never programmed and it stays at 0x80. At MTU 3824 TX iperf3 is stuck at 0.00 Mbit/s while ping works and the link is up. tx_pkts rises, but tx_good_pkts doesn't. Kudos to @wtschueller who discovered this Jumbo frame support on Pi4 ethernet (Genet) #5561
2. UMAC_MAX_FRAME_LEN gets the MTU value, but it's a frame length and counts the FCS. Frames from 3824 up result in rx_length_errors (at least in my testing), so the real limit seems to be MTU 3806.
3. Wire budget is THLD*16-2 = 3838, so 3824 + VLAN = 3842 breaks setups with VLANs configured. Therefore I used 3820.
4. RX_BUF_LENGTH 10240 costs no throughput (936/941 at MTU 1500, same as unpatched) but is above KMALLOC_MAX_CACHE_SIZE on arm64, thus it cant hurt to derive it from the MTU instead.

0xf0 seems to be the real limit: 0xfb receives fine but resulted in TX hard-hung on my setup.

Happy to add @6by9 as Co-developed-by since it's based on your findings. But this requires a Signed-off, which I wouldn't add without consent.

[nbuchwitz]

39dfaf2 contains a brutal approach to make MTU 9000 work (without any offloading). Performs quite ok, but needs more testing

[6by9]

I'm still waiting on documentation from Broadcom to read the official word on how jumbo frames with offload was meant to work (if it was).

Seeing as it was the offloading headers that seemed to cause issues, I did wonder if disabling offloading would allow it to work with bigger buffers. I only had a very quick read through the patches, but wonder if we can "dynamically" disable offload when the mtu is increased above the magic threshold. Possibly not based on the comment of losing the queues as well.

I had considered VLAN headers, but didn't know the answer off the top of my head, and wasn't in a position to set up VLANs to test. Thanks for taking care of it.

I'm not fussed over Co-developed-by:. I'm very grateful that someone else is having a look at the patches, particularly when they're looking to upstream it too.

[nbuchwitz]

@ffainelli and @Ryceancurry if you can spare some time, your thoughts on this would be really appreciated (as always). Thanks!

[starchivore]

https://lore.kernel.org/netdev/20260406-devel-autonomous-eee-v1-1-b335e7143711@tipi-net.de/t/

Other BCM54xx PHYs likely have the same AutogrEEEn register layout, but I only have access to the BCM54210PE/BCM54213PE datasheets.

---

https://datasheets.raspberrypi.com/cm4/cm4-datasheet.pdf#page=7

The CM4 has an on-board Gigabit Ethernet PHY — the Broadcom BCM54210PE

https://www.broadcom.com/products/ethernet-connectivity/phy-and-poe/copper/gigabit/bcm54210

• Supports jumbo packets up to 18 KB

---

https://magazine.raspberrypi.com/articles/raspberry-pi-4-in-detail

The BCM54213PE chip connects the Ethernet to a high-speed interface to the CPU.

https://www.broadcom.com/products/ethernet-connectivity/phy-and-poe/copper/gigabit/bcm54213

• Support for jumbo packets up to 10 KB

---

While we do understand the importance of taking one step at a time, it would be great to test whether 10K (BCM54213PE) and 18K (BCM54210PE) are genuinely supported by the hardware or otherwise. Thanks.

[Ryceancurry]

With the status blocks off jumbo frames seem to come through.

Can you give me more color on the failure? Do we see fragmented packets? Is the packet corrupted? Or do we not receive a RX descriptor at all?

Full disclosure, I threw AI at the RTL(I'm a SW guy), it suggests a RTL bug where the RSB is reserved at every packet ready threshold. So I wonder if we are seeing a 64B hole between each 3820B chunk within the jumbo packet. At least that is the running theory right now. I will continue to dig.

[herisson-88]

Status blocks off, what's the true max frame length the MAC can handle ?

[nbuchwitz]

Thanks for looking into this too!

With the status blocks off jumbo frames seem to come through.

Can you give me more color on the failure? Do we see fragmented packets? Is the packet corrupted? Or do we not receive a RX descriptor at all?

With status block enabled and threshold at 0xf0 I get a descriptor (one per oversized frame):

desc len=3904 status=0x0f402000 SOP=1 EOP=0

3904 = 64 (RSB) + 2 (align) + 3838. Payload is fine and matches my test pattern. I also don't see any holes, just a hart cut off.

I've also tested with 3840, 5000 and 9014 B frames and all of them produce the same descriptor.

Some things I've noticed and might be worth mentioning:

• the MAC MIB counts the frame correctly (9014 B increments rx_4096_9216_oct), so the MAC gets the complete frame and I suspect the loss somewhere in RBUF to RDMA handoff
• no corruption to follow-up traffic, sending 9014 and 1514 B one after the other, every 1514 one is OK

For contrast, with RBUF_64B_EN and TBUF_64B_EN cleared, MTU 9000 works at line speed with byte exact payloads and the threshold still at 0xf0.

[nbuchwitz]

Seeing as it was the offloading headers that seemed to cause issues, I did wonder if disabling offloading would allow it to work with bigger buffers. I only had a very quick read through the patches, but wonder if we can "dynamically" disable offload when the mtu is increased above the magic threshold. Possibly not based on the comment of losing the queues as well.

I've tested further and came up with a solution which allows to switch to higher MTU on a live interface (tested 1514, 4096, 8192, 9014 B frames with threshold at 0xf0). Anyway, blocks needs to be disabled for anything higher. If I keep the TSB to preserve TX checksum offload, RX is still fine at 986 Mbit/s but TX drops to 0.

A while ago I proposed to get rid of the TX queues in genet [1]. Florian and Justin reviewed and tested it, but the reasoning was not good enough. Even though the queues are not absolutely blocking it,the TSB has no queue selection role anymore and it would simplify the jumbo patch. Might be worth a v2.

[1] https://lore.kernel.org/netdev/20260612205915.3156127-1-nb@tipi-net.de/

[herisson-88]

Patch tested on Audiolinux.
Many hours of Diretta streaming at MTU 9000 without issue.

[Ryceancurry]

Status blocks off, what's the true max frame length the MAC can handle ?

As far as I can see the only limitation is the MAC's 14 bit frame len field. so 16383B

I reproduced the 9000B frames with RSB enabled. I printed out the entire 9000B packet and see corruption at each PKT RDY THRESHOLD. 3838B and ~7700B. This confirms my suspicion. The HW puts a 64B header per PKT RDY THRESHOLD. I think the correct way to do this is to set rx_buf_size to PKT RDY THRESHOLD. Then use rx scatter gather with multiple descriptors. We need to strip 64B off of each fragment. Unfortunately this means a big rework on the RX side.

[herisson-88]

@nbuchwitz we can test 16K with the current patch, it is just a question of ENET_MAX_JUMBO_MTU ?

[nbuchwitz]

I reproduced the 9000B frames with RSB enabled. I printed out the entire 9000B packet and see corruption at each PKT RDY THRESHOLD. 3838B and ~7700B. This confirms my suspicion. The HW puts a 64B header per PKT RDY THRESHOLD. I think the correct way to do this is to set rx_buf_size to PKT RDY THRESHOLD. Then use rx scatter gather with multiple descriptors. We need to strip 64B off of each fragment. Unfortunately this means a big rework on the RX side.

That helped a lot, thanks. I swapped the MTU 9000 patch for your approach and it works well. The TSB even can stay on with a little quirk. I also bumped max_mtu to what the 14 bit UMAC_MAX_FRAME_LEN allows, 16347.

Do you now if the status block bug is in all GENET (non v1) versions? I only have v5 here to test.

[herisson-88]

@nbuchwitz work great in 1G (I get 9184 the limitation is on other side) but configured in 100M there are packet loss with mtu > 9080

[antonellocaroli]

I reproduced the 9000B frames with RSB enabled. I printed out the entire 9000B packet and see corruption at each PKT RDY THRESHOLD. 3838B and ~7700B. This confirms my suspicion. The HW puts a 64B header per PKT RDY THRESHOLD. I think the correct way to do this is to set rx_buf_size to PKT RDY THRESHOLD. Then use rx scatter gather with multiple descriptors. We need to strip 64B off of each fragment. Unfortunately this means a big rework on the RX side.

That helped a lot, thanks. I swapped the MTU 9000 patch for your approach and it works well. The TSB even can stay on with a little quirk. I also bumped max_mtu to what the 14 bit UMAC_MAX_FRAME_LEN allows, 16347.

Do you now if the status block bug is in all GENET (non v1) versions? I only have v5 here to test.

Could you show me the little quirk you used to keep the TSB enabled? On my Raspberry Pi 4 / GENET v5, MTU 13500 works, but around 13505 it becomes unstable and MTU 14000 fails with RX CRC errors. I noticed that a 14000 MTU results in a 14014-byte skb becoming a 14078-byte DMA buffer after the 64-byte TSB is added.

[nbuchwitz]

Could you show me the little quirk you used to keep the TSB enabled? On my Raspberry Pi 4 / GENET v5, MTU 13500 works, but around 13505 it becomes unstable and MTU 14000 fails with RX CRC errors. I noticed that a 14000 MTU results in a 14014-byte skb becoming a 14078-byte DMA buffer after the 64-byte TSB is added.

The "quirk" is to dynamically switch of TX checksum based on the mtu (threshold is the previous 3820). See the last patch for details.

Pi4 is afaik limited by the phy around 10k (see comment above). Cm4 should (theoretically) something around 18k

[nbuchwitz]

@nbuchwitz work great in 1G (I get 9184 the limitation is on other side) but configured in 100M there are packet loss with mtu > 9080

Haven't tested it yet with fast ethernet. If the time permits I will do some measurements with different mtu and speed. I want to measure the cpu impact of sw checksum. For jumbo frames I assume not much of a penalty

[antonellocaroli]

@nbuchwitz work great in 1G (I get 9184 the limitation is on other side) but configured in 100M there are packet loss with mtu > 9080

Haven't tested it yet with fast ethernet. If the time permits I will do some measurements with different mtu and speed. I want to measure the cpu impact of sw checksum. For jumbo frames I assume not much of a penalty

Thanks, that clarifies the TSB quirk. Interestingly, with two Pi4 Model B (Rev 1.1 and Rev 1.5) directly connected, I can get MTU 13500 working reliably in one direction (10/10 pings), while the opposite direction fails. Around 13503–13507 it becomes unstable/fails. So the Pi4 PHY seems capable of going significantly beyond 10k in at least some cases. Do you know what exactly imposes the ~10k PHY limit you mentioned (PHY register/buffer/specification), and whether it differs between Pi4 board revisions?

[nbuchwitz]

Do you really see such big payload or is this already capped by phy and it just works "magically" with the 10k limit?

Limit is stated in the datasheet. So I'd assume it's related to the buffer / state machine

[antonellocaroli]

Do you really see such big payload or is this already capped by phy and it just works "magically" with the 10k limit?

Limit is stated in the datasheet. So I'd assume it's related to the buffer / state machine

Yes, at least at the GENET MAC/driver level I really see the full size. For 10 successful MTU 13500 pings, txq3_packets increases by 10 and txq3_bytes by 135140, i.e. exactly 13514 bytes per packet. tx_oversize also increases by 10 on TX and rx_oversize by 10 on RX, with no additional CRC errors. I'm using ping -M do, so there is no IP fragmentation.

However, I haven't verified on the wire between MAC and PHY, so you're right that this doesn't prove the PHY actually handles the full ~13.5K frame as such. Interestingly, Rev 1.1 -> Rev 1.5 works at MTU 13500, while Rev 1.5 -> Rev 1.1 fails, even though the receiving side counts the request and generates a 13514-byte reply.

Which PHY datasheet/section states the ~10K limit? I'd like to check exactly what that limit refers to.

[nbuchwitz]

https://www.broadcom.com/products/ethernet-connectivity/phy-and-poe/copper/gigabit/bcm54213pe

Support for jumbo packets up to 10 KB

[antonellocaroli]

https://www.broadcom.com/products/ethernet-connectivity/phy-and-poe/copper/gigabit/bcm54213pe

Support for jumbo packets up to 10 KB

Thanks!

yes, I confirmed that both of my Pi4s (Rev 1.1 and Rev 1.5) are using the BCM54213PE PHY (phy_id 0x600d84a2).

So the 10 KB limit you mentioned is indeed the one stated in the BCM54213PE datasheet.

However, MTU 13500 is really passing end-to-end in my tests: with ping -M do -s 13472, I can get 10/10 replies with no fragmentation. So it looks like the 10 KB figure is a guaranteed/specification limit rather than a strict hardware cutoff.

Above that it becomes unreliable very quickly (around 13503–13507 in my tests), and at MTU 14000 it fails. I also see RX CRC errors when operating around this boundary, so this is clearly outside the PHY's guaranteed operating range.

Interestingly, both Pi4 revisions use exactly the same BCM54213PE, so the different behaviour I saw between the two boards isn't explained by a different PHY model.

I agree that this could be related to an internal PHY buffer/state-machine limit rather than a simple hard packet-size check.

[herisson-88]

I will be able to test 16k CM4 tomorrow

[nbuchwitz]

I've created a second draft PR so it can be A/B tested. The other PR contains the patch set + 3 prep patches and is based on the series for net-next, as the driver upstream has been converted to page pool.

#7623

[nbuchwitz]

@nbuchwitz work great in 1G (I get 9184 the limitation is on other side) but configured in 100M there are packet loss with mtu > 9080

Haven't tested it yet with fast ethernet. If the time permits I will do some measurements with different mtu and speed. I want to measure the cpu impact of sw checksum. For jumbo frames I assume not much of a penalty

I can reproduce the issue and also found the likely culprint. The phy driver needs to call bcm_phy_enable_jumbo() like it's already done in bcm7xxx.c. I will add a patch for this.