Reef-pi/OS Instance rebooting issue, troubleshooting....

bishoptf · Jul 31, 2023

I will preface this with that I am a linux guy and usually I am pretty comfortable on the cli front with the OS but had something happen yesterday to one of my newer units that has me a little stumped and was hoping that I could get some others to see if they had any suggestions. I have a rref-pi unit running one of the @Michael Lane hats with a rpi zero w2. Been up and running for about 4 months and yesterday I went to connect to it from my phone and it wouldn't respond from the web page which was odd. I then I grab my PC and couldn't ping the address. Everything appeared to be working, at least all my outlets and lights etc were still in operation but it wasn't responding, first thing I tried was to pull power and reboot, which I hate but with no monitor its hard to see what is going on. I re-applied power and waited and nothing came back online, so thinking the worse that my SD card was corrupted etc, I disconnected everything and pulled the SD card out and took it to the bench where I could insert it into another rpi and boot with a monitor and to my surprise it booted right up, now that was odd. I did not see any issues, so I put it back in and powered up and it to booted back up, what the heck is going on.

So once I had everything connected (which I did find one of my 12v power inputs to be loose, maybe this was an issue?) and appeared to be stable I started poking around in the log to see maybe I could see what was going on. I have logs back through June (which I thought I should be rolling them monthly, need to look into that) but what I came across is something I have never seen before. A few times in June and july the box appears to be rebooting, when looking at the logs I do not see anything but a Boot statement like this:

-- Boot 0a45a896c2c345e5b53e90e8f5ade4d8 --

Nothing before that indicates any issues just reef-pi doing its thing and then more messages like its booting. But I see a lot of these messages, sometimes over and over. But the messages actually do not look like the box is rebooting, at least they do not show up in the reboot log:

bishop@atollpi:~ $ last reboot
reboot system boot 6.1.21-v7+ Wed Dec 31 18:00 still running
reboot system boot 6.1.21-v7+ Wed Dec 31 18:00 - 17:35 (19568+22:35)
reboot system boot 6.1.21-v7+ Wed Dec 31 18:00 - 17:35 (19568+22:35)
reboot system boot 6.1.21-v7+ Wed Dec 31 18:00 - 17:35 (19568+22:35)
reboot system boot 6.1.21-v7+ Wed Dec 31 18:00 - 17:35 (19568+22:35)
reboot system boot 6.1.21-v7+ Wed Dec 31 18:00 - 17:35 (19568+22:35)
reboot system boot 6.1.21-v7+ Wed Dec 31 18:00 - 17:35 (19568+22:35)

Not sure this is recording what I think its recording but, yesterday it was rebooting or appearing to reboot over and over and over. I've attached the journalctl output for all boot statements with the previous line for context but right now I have no idea what is causing this or how to address since I am not sure its really rebooting but something is going on. I am running an up to date Bullseye installation, since booting it back up yesterday and correcting the loose wire no reboot messages but now I am pretty paranoid, lol. I've attached 2 files one just pulls out all the boot statements with a line before and the other was a segment from yesterday when it was constantly rebooting, you can see it appears to be rebooting and then another boot statement but doesnt appear to be rebooting, I dunno this is pretty odd stuff but thought I would post to see if anyone has any ideas....

Thanks

theatrus · Jul 31, 2023

Power supply?

bishoptf · Jul 31, 2023

theatrus said:
Power supply?

Yeah possibly but its using an oem apple power supply and maybe that is part of the issue but I think there is more going on. I think those only supplied 1a of power but keep in mind that its a rpi zero2 and how much could it be drawing but the boot statements are odd since there are no low voltage etc it just drops off and i've seen the boot statements one after the other, but it appears to be booting...

According to this article - https://www.cnx-software.com/2021/12/09/raspberry-pi-zero-2-w-power-consumption/ looks like under stress it may draw 1.5 amps so maybe its something related to that but I still think there is something else going on since it's not doing it constantly, hasn't done it since I rebooted it yesterday, but will swap out with a higher usb charger, that is easiest enough to do.

To me the boot statements are odd since I've not seen them that way with the string after it...

bishoptf · Jul 31, 2023

Looking at the logging I am not sure what the --Boot statement signifies, since I see them over and over. I did another grep of the logs and looking for actual booting and got these which seems more in line, so not sure what the "--Boot" statement signifies.

$ sudo journalctl | grep Booting
Jun 12 12:17:01 atollpi kernel: Booting Linux on physical CPU 0x0
Jun 26 19:17:01 atollpi kernel: Booting Linux on physical CPU 0x0
Jul 02 09:17:01 atollpi kernel: Booting Linux on physical CPU 0x0
Jul 07 13:17:01 atollpi kernel: Booting Linux on physical CPU 0x0
Jul 30 17:17:01 atollpi kernel: Booting Linux on physical CPU 0x0
Jul 30 17:17:01 atollpi kernel: Booting Linux on physical CPU 0x0
Jul 30 17:35:26 atollpi kernel: Booting Linux on physical CPU 0x0

I also find it interesting that it happen on the same minute and second for all the reboots now that is not concidence...maybe I will look at the logs above that and see what its doing..

theatrus · Jul 31, 2023

bishoptf said:
Looking at the logging I am not sure what the --Boot statement signifies, since I see them over and over. I did another grep of the logs and looking for actual booting and got these which seems more in line, so not sure what the "--Boot" statement signifies.

$ sudo journalctl | grep Booting
Jun 12 12:17:01 atollpi kernel: Booting Linux on physical CPU 0x0
Jun 26 19:17:01 atollpi kernel: Booting Linux on physical CPU 0x0
Jul 02 09:17:01 atollpi kernel: Booting Linux on physical CPU 0x0
Jul 07 13:17:01 atollpi kernel: Booting Linux on physical CPU 0x0
Jul 30 17:17:01 atollpi kernel: Booting Linux on physical CPU 0x0
Jul 30 17:17:01 atollpi kernel: Booting Linux on physical CPU 0x0
Jul 30 17:35:26 atollpi kernel: Booting Linux on physical CPU 0x0

I also find it interesting that it happen on the same minute and second for all the reboots now that is not concidence...maybe I will look at the logs above that and see what its doing..

That is curious, and implies the RTC is running fine across the reboots.

bishoptf · Jul 31, 2023

theatrus said:
That is curious, and implies the RTC is running fine across the reboots.

I grep out those booting statements and 3 line before...to me the fact that all of the boots have happened at X:17:01 is not a conincedence, if it was power or some other thing it would not happen at the same minute and second across ALL the reboots...keep in mind the last boot was one that I did, lol.

I get all the fun ones, prolly something dumb etc...but its odd for sure.

Sral · Aug 1, 2023

bishoptf said:
Yeah possibly but its using an oem apple power supply and maybe that is part of the issue but I think there is more going on. I think those only supplied 1a of power but keep in mind that its a rpi zero2 and how much could it be drawing but the boot statements are odd since there are no low voltage etc it just drops off and i've seen the boot statements one after the other, but it appears to be booting...

According to this article - https://www.cnx-software.com/2021/12/09/raspberry-pi-zero-2-w-power-consumption/ looks like under stress it may draw 1.5 amps so maybe its something related to that but I still think there is something else going on since it's not doing it constantly, hasn't done it since I rebooted it yesterday, but will swap out with a higher usb charger, that is easiest enough to do.

To me the boot statements are odd since I've not seen them that way with the string after it...

I‘d caution against using an Apple 1A phone charger (if that’s the one you’re using).

Tried it with my Pi Zero (not W!) and an Ethernet USB-Hub (max draw 280 mA) but this combo only booted if I booted without the Ethernet Hub and only connect the Hub after the Pi had booted. Together the power dropped too much for the Boot process. When I used the larger 2.5A Apple charger everything booted fine.

I remember the Pi Zero (not 2) doesn’t have a low voltage warning. How is that with the Pi Zero 2 ?

bishoptf · Aug 1, 2023

Sral said:
I‘d caution against using an Apple 1A phone charger (if that’s the one you’re using).

Tried it with my Pi Zero (not W!) and an Ethernet USB-Hub (max draw 280 mA) but this combo only booted if I booted without the Ethernet Hub and only connect the Hub after the Pi had booted. Together the power dropped too much for the Boot process. When I used the larger 2.5A Apple charger everything booted fine.

I remember the Pi Zero (not 2) doesn’t have a low voltage warning. How is that with the Pi Zero 2 ?

Yeah I am going to swap out the power supply but something else is going on, nothing reboots to the minute and second multiple times. If it was a power issue then I would expect to see different times, the only difference is the hour but the minute and second are the same for all the reboots. To me that is not random that is something else but still looking and trying to figure it out but I am going to replace the power supply just for good measure.

Sral · Aug 1, 2023

bishoptf said:
Yeah I am going to swap out the power supply but something else is going on, nothing reboots to the minute and second multiple times. If it was a power issue then I would expect to see different times, the only difference is the hour but the minute and second are the same for all the reboots. To me that is not random that is something else but still looking and trying to figure it out but I am going to replace the power supply just for good measure.

Okay, I didn't look into your logs, but if you say it like that it reminds me of the erroneous "LowVoltage" warnings of somebody else with a Pi 3B+ here in the forum. Those were also regular by the hour, with only one second difference.

bishoptf · Aug 1, 2023

Well I shutdown and swapped out the power supply, I do not think that is causing any issues but this will be better than the 1a that I was using. So I shutdown gracefully but applying power it seems to be really slow to boot, I get all these boot statements in the log and I do not understand what they are or signify. Looks like when it initially tried to bring up reef-pi service it had a panic and exited, re-started after a period of time but I dunno something doesnt appear to be happy with this installation. I may re-image a card and swap it out but something is wonky, attached are the some of the log output...

@Ranjib not sure if you have seen that Go panic called out in the logs, if you need more information let me know, seems to be happening on this instance when coming online from a boot.