I'm working on my carrier board, and my goals are to have NVME and USB3 ports and of course the USB2 OTG native to the BCM2711 all along side each other. I want fast disk io, which is done with the NVME, multiple USB host ports for HID devices and mass storage, and also a USB device port for PC connectivity. Since the native BCM2711 OTG interface doesn't allow OTG functionality through a hub, my only choice is to use a PCIe-USB adapter, just like on the Pi4B, for instance. But then the PCIe x1 interface is occupied completely. This means the only way to go is to use a PCIe switch.
I've got the PCIe-Packet-Switch-4P from Waveshare here https://www.waveshare.com/wiki/PCIe-Packet-Switch-4P. And I've got their PCIe to USB 3.2 Gen1 (B) PCB as well https://www.waveshare.com/pcie-to-usb-3.2-gen1-b.htm, as well as a nice PCIe-NVME adapter https://www.waveshare.com/pcie-to-m.2-a.htm.
The NVME adapter works fine on its own. The CM4 can boot from it, and it's all fast, large, and everything. The USB adapter works too. But using both of them at the same time through the bridge results in a nasty kernel panic. Especially when I go:
hdparm -t /dev/nvme0n1
The thing crashes 100% guaranteed.
The backtrace is on my pi's hdmi monitor only, so I used text extraction on my phone:
1379.3285481 SError Interrupt on CPU3, code 0x00000000bf000002-- SError
1379.3285631 CFU: 3 PID: 116 Comm: usb-storage
1379.328656 Kernel panic-not syncing: asynchronous SError Interrupt
1379.3286601 CPU: 3 PID: 116 Comm: usb-storage Tainted: G..C 6.1.21-u8+ 81642
1379.3286661 Call trace
1379.3286681 dump_backtrace+0x120/0x130
1379.3286731 show_stack+0x20/0x30
1379.3286771 dump_stack_Iul-ox8c/oxb8
1379.3286831 dump_stack-0x18/0x34
1379.3286871 panic+@x1a4/0x37c
1379.3286921 nmi_panic+@xbc/oxco
1379.326971 arm64_serror_panic+0x6с/0x80
1379.3287011 do_serror+0x3c/0x88
1379.3287041 el1h_64_error_handler+0x38/0x50
1379.3287091 el1h_64_error+0x64/0x68
1379.3287131 xhci_ring_ep_doorbell+0x78/0x108
1379.3287171 xhci_queue_bulk_tx+0x420/0x9a0
1379.3287211 xhci_urb_enqueue+0x44c/0x500
1379.3287271.usb_hcd_submit_urb+0xd4/0x9d0
1379.3287321 usb_submit_urb+0x248/0x5c8
1379.328735]usb_stor_msg_common+@x9c/@x1d0
1379.328741] usb_stor_bulk_transfer_buf+0x58-0x98
1379.3287471 usb_stor_Bulk_transport+@xe0/0x370
1379.3287521usb_stor_inuoke_transport+0x44/0x5e8
1379.3287571usb_stor_transparent_scsi_command+0x18/0x28
1379.3287621usb_stor_control_thread+@x1e4/0x2b0
1379.3287671 kthread+@xfc/ox110
1379.3287731 ret_from_fork+0x10/0x20
1379.3282791 SMP: stopping secondary CPUs
1379.3287861 Kernel Offset: @x115c400000 from Oxffffrfc008000000
1379.3287881 PHYS_OFFSET: OxO
1379.3287891 CPU features: 0x20000,2013c080,0000421b
1379.3287931 Memory Limit: none
Obviously something goes wrong in the XHCI driver's interrupt. I've tried some device tree configurations, but the problem stays. I've read from a Broadcom developer that the problem lies with PCIe adapters or switches that don't use the CLKREQ# signal. So does this mean I should just hook that signal up and everything will work?
Here's a photo of my off-the-shelf breadboard system:
