3G-4G/LTE Base hat cycling USB

richard.ayers · November 3, 2021, 9:43pm

Hi
I’m running some 3G/4G/LTE base hats with Telit modules onboard. I’m getting issues with my external SSDs getting disconnected and reconnected while running, which is causing issues as I’m streaming video to them. I stopped both the core_agent and core_manager services but that didn’t stop the problem so I powered down the base hat using GPIO 37 and the problem went away.

It is not power related, the external SSDs are powered separately and the power lines on their USB connectors are cut to ensure they are not pulling power off the USB bus, their earth/ground lines are joined with those for the Pi as their power is coming directly from the same DC-DC convertor that powers the Pi, the DC-DC is more than capable of supplying the power needed.

I know the agent and manager services do some kind of reboot of the unit if they aren’t getting a connection (my devices are often out of range) so I only enable those services when my software wants to talk to the world but I can’t power the unit down the same way as I need the GNSS position out of it.

The Telit modules have the latest firmware (at least those that would update) and the Pis are running the latest version of Rasbian. The SSDs are Sandisk so pretty reliable and standard.

My investigation so far points quite firmly at the base hat and/or the Telit module and/or their interactions with the OS.

richard.ayers · November 5, 2021, 10:29am

For some reason, I can’t edit my own post.

This has now become a project killer as the knock-on effects are causing so much work-around coding that it simply won’t work as a production system anymore.

Further information regarding the USB dis-connect re-connect issue:

Setup:

Pi 4 8Gb, Latest Rasbian, booting from 32Gb USB with another USB stick inserted.

Created a shell script that waited 15 seconds and then cleared dmesg so we got a short, clean record.

While the script was waiting, unplugged keyboard and mouse (again so we get as clean a record as possible)

Waited until I got the ‘Disk removed without ejecting’ complaint, plugged keyboard and mouse back in and grabbed the dmesg log :

From the log we can see the GSM modem disconnecting and reconnecting successfully a few times and then at time-stamp 3480.755405 it fails to reconnect, goes through some reset hoops and then at 4180.150761 the unmount complaint appears.

Output from dmesg:
[ 3321.042622] option1 ttyUSB0: GSM modem (1-port) converter now disconnected from ttyUSB0

[ 3321.042742] option 1-1.1:1.0: device disconnected

[ 3321.043073] cdc_ether 1-1.1:1.2 wwan0: unregister ‘cdc_ether’ usb-0000:01:00.0-1.1, Mobile Broadband Network Device

[ 3321.119182] option1 ttyUSB1: GSM modem (1-port) converter now disconnected from ttyUSB1

[ 3321.119302] option 1-1.1:1.4: device disconnected

[ 3321.119806] option1 ttyUSB2: GSM modem (1-port) converter now disconnected from ttyUSB2

[ 3321.119925] option 1-1.1:1.5: device disconnected

[ 3321.120431] option1 ttyUSB3: GSM modem (1-port) converter now disconnected from ttyUSB3

[ 3321.120542] option 1-1.1:1.6: device disconnected

[ 3321.121047] option1 ttyUSB4: GSM modem (1-port) converter now disconnected from ttyUSB4

[ 3321.121161] option 1-1.1:1.7: device disconnected

[ 3321.226128] usb 1-1.1: reset high-speed USB device number 50 using xhci_hcd

[ 3321.372764] option 1-1.1:1.0: GSM modem (1-port) converter detected

[ 3321.372997] usb 1-1.1: GSM modem (1-port) converter now attached to ttyUSB0

[ 3321.377627] cdc_ether 1-1.1:1.2 wwan0: register ‘cdc_ether’ at usb-0000:01:00.0-1.1, Mobile Broadband Network Device, ee:34:13:ec:5f:e9

[ 3321.377814] option 1-1.1:1.4: GSM modem (1-port) converter detected

[ 3321.378388] usb 1-1.1: GSM modem (1-port) converter now attached to ttyUSB1

[ 3321.378589] option 1-1.1:1.5: GSM modem (1-port) converter detected

[ 3321.382035] usb 1-1.1: GSM modem (1-port) converter now attached to ttyUSB2

[ 3321.382270] option 1-1.1:1.6: GSM modem (1-port) converter detected

[ 3321.382505] usb 1-1.1: GSM modem (1-port) converter now attached to ttyUSB3

[ 3321.382699] option 1-1.1:1.7: GSM modem (1-port) converter detected

[ 3321.382949] usb 1-1.1: GSM modem (1-port) converter now attached to ttyUSB4

[ 3334.467324] usb 1-1.1: USB disconnect, device number 50

[ 3334.468120] option1 ttyUSB0: GSM modem (1-port) converter now disconnected from ttyUSB0

[ 3334.468253] option 1-1.1:1.0: device disconnected

[ 3334.469041] cdc_ether 1-1.1:1.2 wwan0: unregister ‘cdc_ether’ usb-0000:01:00.0-1.1, Mobile Broadband Network Device

[ 3334.539714] option1 ttyUSB1: GSM modem (1-port) converter now disconnected from ttyUSB1

[ 3334.539857] option 1-1.1:1.4: device disconnected

[ 3334.541836] option1 ttyUSB2: GSM modem (1-port) converter now disconnected from ttyUSB2

[ 3334.541981] option 1-1.1:1.5: device disconnected

[ 3334.545407] option1 ttyUSB3: GSM modem (1-port) converter now disconnected from ttyUSB3

[ 3334.545931] option 1-1.1:1.6: device disconnected

[ 3334.549914] option1 ttyUSB4: GSM modem (1-port) converter now disconnected from ttyUSB4

[ 3334.550069] option 1-1.1:1.7: device disconnected

[ 3348.075097] usb 1-1.1: new high-speed USB device number 60 using xhci_hcd

[ 3348.218173] usb 1-1.1: New USB device found, idVendor=1bc7, idProduct=1206, bcdDevice= 3.18

[ 3348.218192] usb 1-1.1: New USB device strings: Mfr=1, Product=2, SerialNumber=3

[ 3348.218208] usb 1-1.1: Product: LE910C1-EU

[ 3348.218222] usb 1-1.1: Manufacturer: Android

[ 3348.218237] usb 1-1.1: SerialNumber: 0123456789ABCDEF

[ 3348.230797] option 1-1.1:1.0: GSM modem (1-port) converter detected

[ 3348.232173] usb 1-1.1: GSM modem (1-port) converter now attached to ttyUSB0

[ 3348.242426] cdc_ether 1-1.1:1.2 wwan0: register ‘cdc_ether’ at usb-0000:01:00.0-1.1, Mobile Broadband Network Device, de:44:22:4f:4a:a8

[ 3348.245112] option 1-1.1:1.4: GSM modem (1-port) converter detected

[ 3348.245587] usb 1-1.1: GSM modem (1-port) converter now attached to ttyUSB1

[ 3348.246173] option 1-1.1:1.5: GSM modem (1-port) converter detected

[ 3348.246488] usb 1-1.1: GSM modem (1-port) converter now attached to ttyUSB2

[ 3348.247049] option 1-1.1:1.6: GSM modem (1-port) converter detected

[ 3348.247353] usb 1-1.1: GSM modem (1-port) converter now attached to ttyUSB3

[ 3348.247905] option 1-1.1:1.7: GSM modem (1-port) converter detected

[ 3348.248199] usb 1-1.1: GSM modem (1-port) converter now attached to ttyUSB4

[ 3480.655491] usb 1-1.3: reset high-speed USB device number 59 using xhci_hcd

[ 3480.755405] usb 1-1.3: device descriptor read/64, error -71

[ 3480.975238] usb 1-1.3: device descriptor read/64, error -71

[ 3481.195250] usb 1-1.3: reset high-speed USB device number 59 using xhci_hcd

[ 3481.295225] usb 1-1.3: device descriptor read/64, error -71

[ 3481.515262] usb 1-1.3: device descriptor read/64, error -71

[ 3481.735239] usb 1-1.3: reset high-speed USB device number 59 using xhci_hcd

[ 3481.735446] usb 1-1.3: Device not responding to setup address.

[ 3481.955245] usb 1-1.3: Device not responding to setup address.

[ 3482.175047] usb 1-1.3: device not accepting address 59, error -71

[ 3482.275119] usb 1-1.3: reset high-speed USB device number 59 using xhci_hcd

[ 3482.275357] usb 1-1.3: Device not responding to setup address.

[ 3482.495154] usb 1-1.3: Device not responding to setup address.

[ 3482.714988] usb 1-1.3: device not accepting address 59, error -71

[ 3482.716730] usb 1-1.3: USB disconnect, device number 59

[ 3483.025014] usb 1-1.3: new high-speed USB device number 61 using xhci_hcd

[ 3483.125274] usb 1-1.3: device descriptor read/64, error -71

[ 3483.345333] usb 1-1.3: device descriptor read/64, error -71

[ 3483.565056] usb 1-1.3: new high-speed USB device number 62 using xhci_hcd

[ 3483.675271] usb 1-1.3: device descriptor read/64, error -71

[ 3483.895264] usb 1-1.3: device descriptor read/64, error -71

[ 3484.016144] usb 1-1-port3: attempt power cycle

[ 3484.675044] usb 1-1.3: new high-speed USB device number 63 using xhci_hcd

[ 3484.675252] usb 1-1.3: Device not responding to setup address.

[ 3484.895249] usb 1-1.3: Device not responding to setup address.

[ 3485.115027] usb 1-1.3: device not accepting address 63, error -71

[ 3485.215030] usb 1-1.3: new high-speed USB device number 64 using xhci_hcd

[ 3485.215268] usb 1-1.3: Device not responding to setup address.

[ 3485.435204] usb 1-1.3: Device not responding to setup address.

[ 3485.655027] usb 1-1.3: device not accepting address 64, error -71

[ 3485.656169] usb 1-1-port3: unable to enumerate USB device

[ 4151.980480] option1 ttyUSB0: GSM modem (1-port) converter now disconnected from ttyUSB0

[ 4151.980637] option 1-1.1:1.0: device disconnected

[ 4151.981277] cdc_ether 1-1.1:1.2 wwan0: unregister ‘cdc_ether’ usb-0000:01:00.0-1.1, Mobile Broadband Network Device

[ 4152.049009] option1 ttyUSB1: GSM modem (1-port) converter now disconnected from ttyUSB1

[ 4152.049167] option 1-1.1:1.4: device disconnected

[ 4152.049875] option1 ttyUSB2: GSM modem (1-port) converter now disconnected from ttyUSB2

[ 4152.050032] option 1-1.1:1.5: device disconnected

[ 4152.055844] option1 ttyUSB3: GSM modem (1-port) converter now disconnected from ttyUSB3

[ 4152.056018] option 1-1.1:1.6: device disconnected

[ 4152.059611] option1 ttyUSB4: GSM modem (1-port) converter now disconnected from ttyUSB4

[ 4152.059697] option 1-1.1:1.7: device disconnected

[ 4152.156158] usb 1-1.1: reset high-speed USB device number 60 using xhci_hcd

[ 4152.302742] option 1-1.1:1.0: GSM modem (1-port) converter detected

[ 4152.305142] usb 1-1.1: GSM modem (1-port) converter now attached to ttyUSB0

[ 4152.308318] cdc_ether 1-1.1:1.2 wwan0: register ‘cdc_ether’ at usb-0000:01:00.0-1.1, Mobile Broadband Network Device, de:44:22:4f:4a:a8

[ 4152.308500] option 1-1.1:1.4: GSM modem (1-port) converter detected

[ 4152.308825] usb 1-1.1: GSM modem (1-port) converter now attached to ttyUSB1

[ 4152.309012] option 1-1.1:1.5: GSM modem (1-port) converter detected

[ 4152.309208] usb 1-1.1: GSM modem (1-port) converter now attached to ttyUSB2

[ 4152.309387] option 1-1.1:1.6: GSM modem (1-port) converter detected

[ 4152.309566] usb 1-1.1: GSM modem (1-port) converter now attached to ttyUSB3

[ 4152.309743] option 1-1.1:1.7: GSM modem (1-port) converter detected

[ 4152.309918] usb 1-1.1: GSM modem (1-port) converter now attached to ttyUSB4

[ 4164.536016] usb 1-1.1: USB disconnect, device number 60

[ 4164.536904] option1 ttyUSB0: GSM modem (1-port) converter now disconnected from ttyUSB0

[ 4164.537056] option 1-1.1:1.0: device disconnected

[ 4164.537933] cdc_ether 1-1.1:1.2 wwan0: unregister ‘cdc_ether’ usb-0000:01:00.0-1.1, Mobile Broadband Network Device

[ 4164.586375] option1 ttyUSB1: GSM modem (1-port) converter now disconnected from ttyUSB1

[ 4164.586488] option 1-1.1:1.4: device disconnected

[ 4164.588504] option1 ttyUSB2: GSM modem (1-port) converter now disconnected from ttyUSB2

[ 4164.588620] option 1-1.1:1.5: device disconnected

[ 4164.589654] option1 ttyUSB3: GSM modem (1-port) converter now disconnected from ttyUSB3

[ 4164.589758] option 1-1.1:1.6: device disconnected

[ 4164.590456] option1 ttyUSB4: GSM modem (1-port) converter now disconnected from ttyUSB4

[ 4164.590561] option 1-1.1:1.7: device disconnected

[ 4178.035046] usb 1-1.1: new high-speed USB device number 65 using xhci_hcd

[ 4178.178118] usb 1-1.1: New USB device found, idVendor=1bc7, idProduct=1206, bcdDevice= 3.18

[ 4178.178139] usb 1-1.1: New USB device strings: Mfr=1, Product=2, SerialNumber=3

[ 4178.178158] usb 1-1.1: Product: LE910C1-EU

[ 4178.178175] usb 1-1.1: Manufacturer: Android

[ 4178.178193] usb 1-1.1: SerialNumber: 0123456789ABCDEF

[ 4178.189639] option 1-1.1:1.0: GSM modem (1-port) converter detected

[ 4178.190037] usb 1-1.1: GSM modem (1-port) converter now attached to ttyUSB0

[ 4178.199013] cdc_ether 1-1.1:1.2 wwan0: register ‘cdc_ether’ at usb-0000:01:00.0-1.1, Mobile Broadband Network Device, 1e:6e:6a:21:1d:71

[ 4178.201441] option 1-1.1:1.4: GSM modem (1-port) converter detected

[ 4178.201823] usb 1-1.1: GSM modem (1-port) converter now attached to ttyUSB1

[ 4178.202486] option 1-1.1:1.5: GSM modem (1-port) converter detected

[ 4178.202837] usb 1-1.1: GSM modem (1-port) converter now attached to ttyUSB2

[ 4178.203474] option 1-1.1:1.6: GSM modem (1-port) converter detected

[ 4178.203803] usb 1-1.1: GSM modem (1-port) converter now attached to ttyUSB3

[ 4178.204416] option 1-1.1:1.7: GSM modem (1-port) converter detected

[ 4178.204753] usb 1-1.1: GSM modem (1-port) converter now attached to ttyUSB4

[ 4178.305011] usb 1-1.3: new high-speed USB device number 66 using xhci_hcd

[ 4178.446381] usb 1-1.3: New USB device found, idVendor=090c, idProduct=1000, bcdDevice=11.00

[ 4178.446398] usb 1-1.3: New USB device strings: Mfr=1, Product=2, SerialNumber=3

[ 4178.446412] usb 1-1.3: Product: Flash Drive

[ 4178.446425] usb 1-1.3: Manufacturer: Samsung

[ 4178.446438] usb 1-1.3: SerialNumber: 0374818070007119

[ 4178.448755] usb-storage 1-1.3:1.0: USB Mass Storage device detected

[ 4178.449344] usb-storage 1-1.3:1.0: Quirks match for vid 090c pid 1000: 400

[ 4178.449578] scsi host1: usb-storage 1-1.3:1.0

[ 4179.777527] scsi 1:0:0:0: Direct-Access Samsung Flash Drive 1100 PQ: 0 ANSI: 6

[ 4179.777861] sd 1:0:0:0: Attached scsi generic sg1 type 0

[ 4179.778783] sd 1:0:0:0: [sdb] 62656641 512-byte logical blocks: (32.1 GB/29.9 GiB)

[ 4179.778992] sd 1:0:0:0: [sdb] Write Protect is off

[ 4179.779001] sd 1:0:0:0: [sdb] Mode Sense: 43 00 00 00

[ 4179.779161] sd 1:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn’t support DPO or FUA

[ 4179.837881] sdb: sdb1

[ 4179.839389] sd 1:0:0:0: [sdb] Attached SCSI removable disk

[ 4180.150761] FAT-fs (sdb1): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.

[ 4260.935031] usb 1-1.4: new high-speed USB device number 67 using xhci_hcd

[ 4261.065418] usb 1-1.4: New USB device found, idVendor=0409, idProduct=0059, bcdDevice= 1.00

[ 4261.065441] usb 1-1.4: New USB device strings: Mfr=0, Product=0, SerialNumber=0

[ 4261.067120] hub 1-1.4:1.0: USB hub found

[ 4261.067261] hub 1-1.4:1.0: 4 ports detected

[ 4261.385060] usb 1-1.4.1: new low-speed USB device number 68 using xhci_hcd

[ 4261.532789] usb 1-1.4.1: New USB device found, idVendor=413c, idProduct=2005, bcdDevice= 1.05

[ 4261.532811] usb 1-1.4.1: New USB device strings: Mfr=1, Product=2, SerialNumber=0

[ 4261.532829] usb 1-1.4.1: Product: DELL USB Keyboard

[ 4261.532846] usb 1-1.4.1: Manufacturer: DELL

[ 4261.542806] input: DELL DELL USB Keyboard as /devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb1/1-1/1-1.4/1-1.4.1/1-1.4.1:1.0/0003:413C:2005.0014/input/input19

[ 4261.605845] hid-generic 0003:413C:2005.0014: input,hidraw0: USB HID v1.10 Keyboard [DELL DELL USB Keyboard] on usb-0000:01:00.0-1.4.1/input0

[ 4261.715011] usb 1-1.4.2: new low-speed USB device number 69 using xhci_hcd

[ 4261.854549] usb 1-1.4.2: New USB device found, idVendor=047d, idProduct=2048, bcdDevice= 6.00

[ 4261.854560] usb 1-1.4.2: New USB device strings: Mfr=1, Product=2, SerialNumber=0

[ 4261.854568] usb 1-1.4.2: Product: Kensington Eagle Trackball

[ 4261.854575] usb 1-1.4.2: Manufacturer: Primax

[ 4261.864255] input: Primax Kensington Eagle Trackball as /devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb1/1-1/1-1.4/1-1.4.2/1-1.4.2:1.0/0003:047D:2048.0015/input/input20

[ 4261.864505] hid-generic 0003:047D:2048.0015: input,hidraw1: USB HID v1.11 Mouse [Primax Kensington Eagle Trackball] on usb-0000:01:00.0-1.4.2/input0

AND

A further gremlin in the works, I have a twin USB-serial convertor running which is hooked into 2 RFID heads, when the system first boots the 2 ports are allocated to USB 0 and 1 and I can happily read data off them whilst the Base hat and Telit are running and I’m getting GNSS off USB3, however when the mystery USB reset happens, the 2 RFID ports get shunted to USB 5 and 6, I still get data coming into the RFID reader software but it is now the NMEA GNSS feed and not my RFID readers.

ensar · November 6, 2021, 2:44pm

Hi,

Have you had an LTE connection problem while not using an SSD? If possible, can you share the models and pictures of the products you used in the hardware setup?