Question
Remote Rpi's freeze from time to time. How to wake them up?
Answer
Update 2019jul27hkt1406
I recently upgraded my Rpi3B+ stretch to Rpi4B buster and again I followed @berto's tutorial to set the watch dog timer. I found everything works as smoothly as before. In other words, no changes need to make to @berto's tutorial when upgrading to Rpi4.
Last time I knew nothing about the watchdog timer thing. So it took me more than 3 hours to google to understand everything inside out (well, almost inside out). This time I know what is going on, and all the linux tricks, so it took me only a couple of minutes to complete @berto's tutorial.
2019jun18 Updates
After more thoughts, I concluded that my answer is coming to an end.
My conclusion it that @berto's watchdog tutorial and experiment
suggestion is good, and his answer is the real answer for the OP's
question.
I did his suggested experiment successfully, verified results by the
forkbomb program, and after a lot of googling and reading for more
than 10 hours, I think I finally understood thoroughly the idea of
watchdog timer.
Earlier I wrongly thought that I still needed to learn how to set the
timer to 10 seconds or more. But as @berto says, 10 seconds is all
that to be set. I also read that I can set timer to as long as 16
seconds, and linux watchdog default is even one minute. But that is
not critical.
I have removed all the long winded reading notes in the appendices, to
make the answer shorter. I would suggest newbies not to try to
understand all the details of watchdog, not to mention the much more
complicated daemon SystemD, because our life is short, and those
system things are too complicated for non professionals.
I would like to add two points to end my answer.
(1) There are many reasons for an Rpi to hang in a couple of days
(but usually not months). Often it is not the application program's
fault, but because of the drivers or library functions creating too
much garbage, eg. sockets created, used but not properly disposed. If
it is the application program itself making garbage, the program can
do "garbage collection" and problem solved. But it is hard to remove
garbage sockets which are not generated by the application program.
So a watchdog timer is useful here.
(2) Other ways to avoid too much garbage using up resources include
rebooting every now and then by software or hardware. I do think
rebooting every morning and also use software switchable power supply
to do the system resetting adds another layer of protection. And
using only one Rpi is not very safe. Using two Rpi's as each other's
watchdog (using URT for message passing, eg) add one more layer of
protection. Another method I have not explored is using ESP8266 Wifi
sockets. I hope I can try that later.
This the the end of my answer. Cheers.
2019jun17 Updates
So I tried the fork bomb. The system rebooted after executing the program, in about 15 seconds.
2019jun16 Updates
I found @berto's fork bomb program is a bit newbie scary. So I am learning Bash to find out what that fork bomb is doing. Basically it is just a function named ":", which is defined as a function calling itself two times, thus forking indefinitely, as fast as rabbits growing exponentially, using up all the resources, and crashing linux.
I have also found the following interesting version of forkbomb using Unicode symbols:
( ) { | & } ;
2019jun14/15 Updates
@thesnow suggests a very nice layered approach using a smart plug. I
think the smart plug or smart IoT stuff is the way to go. However, I
am a not so smart newbie in smart stuffm though I am keen to learn.
So I am going to buy a smart plug, do some research, and improve my
answer afterwards. For now, I have added some related learning
resources in the reference section below.
I found @berto's suggestion of using Rpi's hardware watchdog timer also very good. I have not played with any watchdoog stuff before. So I am going to try it now. @berto's instructions are very detailed, but still a bit hard for me, because I don't know very well the meaning of the commands "grep" and "dmseg". So I googled and made some reading notes in the appendices below. Then I followed @berto's suggestion, and strugged a bit to complete part 1. I have not yet reboot, because I need to take a break to digest things. Anyway, here is the screen capture.
I rebooted and got the following dmesg:
I think I am going too fast and now need to take a break to first study more linux things, like systemd, before coming back to carry on the test on watchdog.
/ to continue, ...
The Answer
I have the same problem. I am building a rooftop garden with a couple of Rpi's each of which connects to various wireless stuff (BlueTooth, Wifi) sensors, relays, and solenoids. There are two huge motors near by, controlling big water tanks and lifts. The motors generate EMI and from time to time freeze nearby electronics things.
My plan is to use software switchable PSUs (Power Supply Units) to power switch off/on frozen Rpi's and other devices (Bluetooth devices freeze most often. The BlueTooth and other little devices do not have any software reset command or hardware reset pin, so powering off/on their 5V Vcc is a quick and dirty, but still safe get around). In short, The Rpi's regularly watch each other and their devices and POR (Power On Reset) any guy fallen to sleep.
Of course I can also use a GPIO pin to trigger the Rpi hardware on board reset pin. But I am too lazy to do extra wiring, and too poor a hobbyist to afford professional/industrial grade non stop system devices such as the SwitchDoc Labs Dual WatchDog Timer (see reference below)
I modify ordinary DC-DC (12V to 5V) PSUs' so that any Rpi or MCP23x17 GPIO pins can power on/off the LM2956/LM2947 voltage regulator chip of the PSU. (LM2941 can be used for 1A current switches, LM2596 for 5V 3A PSU. The on/off pin is also connected to a push button, for manual power on/off testing.)
Actually each of my 7 Rpi3B+'s is connected to a cheapy DS3231 Real Time Clock Module which has a hardware interrupt pin to reset PSU, Rpi, or other devices.
Whenever possible and practical I tie up all the devices' reset pins together (removing some of the pull up resistors, so not to overload the GPIO pin).
Now the external DS3231 RTC wakes up everybody in the morning, and switches off lights at midnight, so everybody goes to bed.
References
1. LM2596/LM2941 Based Software Resettable PSU / Current Switches - Rpi StkEx Discussion
Rpi Hardware watchdog Discussion
SwitchDoc Labs Dual WatchDog Timer
ATXRaspi R3 - LowPowerLab US$14.95
A hackable ESP8266 inside a smart plug Want to play with ESP8266 without worrying about the hardware? - Mat 2017aug06
Reverse Engineering 101 of the Xiaomi IoT ecosystem HITCON Community 2018 – Dennis Giese
Xiaomi WiFi socket + MiHome app 21,307 views
espHome [ESP8266/ESP32]
AliExpress WiFi Smart Plug
Smart device -Wikipedia
WiFi Garage Door Opener using ESP8266 - Ray Wang 2016may13 56,335 views
Appendices
Appendix A - WatchDog Timer Reading Notes
Watchdog timer -Wikipedia
Linux WatchDog Man Page
Linux Watchdog - General Tests
Appendix B - Linux commands grep and dmesg reading notes
Appendix C - systemd references
systemd System and Service Manager - FreeDeskTop
systemd - Wikipedia
Appendix D - Fork and Fork Bomb References
Fork (system call) Wikipedia
Appendix E - Bash Learning Notes
