RPi locks up when Cellular HAT powers off

cecil · February 2, 2021, 9:13pm

Hi there,

I have a Cellular IoT HAT. I have installed the PPP service and everything works, except when I lose cell signal in my basement. The PI seems to lock up, my SSH connection disconnects and I can’t reconnect over WiFi. I will be using this HAT in the field where cell service is abundant but if I lose signal for any reason it would seem the Pi won’t recover.

If I remove the Cellular HAT, or disconnect it from USB everything is fine. So this must be the culprit.

This is the syslog output just before it becomes unresponsive. Any idea what could be going on?

raspberrypi bash[430]: #033[0;32m2021/02/02->#033[1;34m21:03:00 #033[0m ppp0 is added as       default interface manually.
raspberrypi pppd[1957]: Script /etc/ppp/ip-up finished (pid 1958), status = 0x7
raspberrypi dhcpcd[427]: wwan0: using IPv4LL address 169.254.229.30
raspberrypi dhcpcd[427]: wwan0: adding route to 169.254.0.0/16
raspberrypi avahi-daemon[265]: Joining mDNS multicast group on interface wwan0.IPv4 with address 169.254.229.30.
raspberrypi avahi-daemon[265]: New relevant interface wwan0.IPv4 for mDNS.
raspberrypi avahi-daemon[265]: Registering new address record for 169.254.229.30 on wwan0.IPv4.
raspberrypi dhcpcd[427]: wwan0: no IPv6 Routers available
raspberrypi pppd[1563]: Device ttyUSB3 is locked by pid 1957
raspberrypi pppd[935]: Device ttyUSB3 is locked by pid 1957
raspberrypi dhcpcd[427]: wlan0: Router Advertisement from fe80::9691:7fff:fedb:6555
raspberrypi pppd[1563]: Device ttyUSB3 is locked by pid 1957
raspberrypi pppd[935]: Device ttyUSB3 is locked by pid 1957
raspberrypi dhcpcd[427]: wlan0: Router Advertisement from fe80::9691:7fff:fedb:6555
raspberrypi pppd[1563]: Device ttyUSB3 is locked by pid 1957
raspberrypi pppd[935]: Device ttyUSB3 is locked by pid 1957
raspberrypi dhcpcd[427]: wlan0: Router Advertisement from fe80::9691:7fff:fedb:6555
raspberrypi pppd[1563]: Device ttyUSB3 is locked by pid 1957
raspberrypi pppd[935]: Device ttyUSB3 is locked by pid 1957
raspberrypi dhcpcd[427]: wlan0: Router Advertisement from fe80::9691:7fff:fedb:6555
raspberrypi env[428]: fatal: not a git repository (or any of the parent directories): .git
raspberrypi pppd[1563]: Device ttyUSB3 is locked by pid 1957
raspberrypi pppd[935]: Device ttyUSB3 is locked by pid 1957
raspberrypi bash[430]: ./#033[0;32m2021/02/02->#033[1;34m21:05:29 #033[0m Connection is down, reconnecting...
raspberrypi bash[430]: #033[0;32m2021/02/02->#033[1;34m21:05:29 #033[0m Power of the module is restarting...

cecil · February 3, 2021, 4:24pm

The concerning lines here are:

Connection is down, reconnecting...
Power of the module is restarting...

That is what I get right before the Pi becomes unresponsive over SSH.

Any suggestion would be appreciated.

cecil · February 5, 2021, 2:46pm

More research!

I think I have found the issue. The RPi has issues with GPIO, I’ve seen it in my own code just turning pins on and off at the “wrong” time. The reconnection script for PPP pulls the Cellular HAT power pin (17?) HIGH and then LOW to reboot the module. Periodically this will hang the CPU.

I believe that the reconnection script should utilize the power down AT commands of the BG96 module rather than just “pulling the plug” so to speak.

This is is the restart code which logs to syslog just before the CPU hangs… Sixfab_PPP_Installer/reconnect_cellulariot at master · sixfab/Sixfab_PPP_Installer · GitHub

Disabling PPP completely “solves” this issue but then you are with out an internet connection. @saeed is this something you have seen before?

Happy to supply any logs or additional information!