reef-pi :: An opensource reef tank controller based on Raspberry Pi.

[pickupman66]

I wouldn't know what to even look for or when. Is there a way I can just pull the log as a file for the last 5 or 6 days and scan it?

 

[bishoptf]

Did u have a power blink possibly where the reef pi got shutdown hard and then booted back up?

I made my power relay NC on my heaters so that if the reef pi goes wonky i still have heat...

I've thought about heaters a good bit, but for freshwater its better to be cold then hot. I am more concerned about a heater getting stuck on and killing everything in the tank, I am using reef-pi as a fail safe. It does have me thinking of have a backup of some sort to at least notify me if the reef-pi goes sideways but will have to think of what kind of check would be best.

 

[Ranjib]

@Ranjib MAJOR ISSUE today. as you know I had issues with my system alerting last week on the ATO. I have been traveling all week away from home and now that i am home, something was amis. my Clam did not look good. Plus one of the BTA walked around. I went to move a frag away from it and thought WOW, that water is COLD> I went to check Reef-pi and could not login. I put my cooking thermometer in the tank and it read 68°! HOLY MOLY! This is not good. I could not access reef-pi so I power cycled it. as soon as it came up, it started blowing up my phone with ATO alerts and also the LOW temperature alerts. I checked and I am about 4 gallons low. This means the system has not been working properly for at least 4 days. I dont know where to go to diagnose this. It appears to be operational at this moment as I can access and heater and ATO is working.

ALSO, since the reboot today, I have no historical graph data. all of that is missing.

Tank is at 70° but i am sure it got as low as 65° during the night.

I am on version 2.2 and I show no errors in the web GUI. All equipment was powered as my lights were on and all water movement pumps had power.

Oh my god :-(
How are things now? Corals ?
Let me think through this. The loss of usage data is something I am working to fix (it’s expected currently , under hard power reset condition ) . Was reef-pi not working or the entire raspberry pi ?
Can you check the log of reef-pi , particularly around the time when the t went unresponsive ? If nothing it should give you some idea since when it became unresponsive (absence of logs).
I’ll share some details on how to configure watchdog (hardware induced reboot when pi is unresponsive ) once I’m home.
Again I’m very sorry , happy to help anyway I can , and let’s hope livestock impact is limited

 

[bishoptf]

I wouldn't know what to even look for or when. Is there a way I can just pull the log as a file for the last 5 or 6 days and scan it?

I would go looking at the syslog files here :

bishop@tankpi:~ $ ls -l /var/log/syslog*
-rw-r----- 1 root adm 3193068 Feb  9 21:44 /var/log/syslog
-rw-r----- 1 root adm 5174563 Feb  9 06:25 /var/log/syslog.1
-rw-r----- 1 root adm  238499 Feb  8 06:25 /var/log/syslog.2.gz
-rw-r----- 1 root adm  238902 Feb  7 06:25 /var/log/syslog.3.gz
-rw-r----- 1 root adm  237523 Feb  6 06:25 /var/log/syslog.4.gz
-rw-r----- 1 root adm  238900 Feb  5 06:25 /var/log/syslog.5.gz
-rw-r----- 1 root adm  238923 Feb  4 06:25 /var/log/syslog.6.gz
-rw-r----- 1 root adm  237620 Feb  3 06:25 /var/log/syslog.7.gz

Rasbian only keeps the daily log uncompressed the previous days are compressed (.gz files) to unzip them do this:

sudo gunzip /var/log/syslog.2.gz

Then to look at the contents of each the file(unzip them first as above):

more /var/log/syslog

Substitute each syslog.1-8 in the command to look at each file, let us know what you find and if you have any questions let me know and I can help with the linux stuff.

:)

 

[pickupman66]

Ok. I'll try to do this tomorrow as I hope to have time. I'm good with Excel. Sure wish it was a simple CSV.

 

[bishoptf]

Ok. I'll try to do this tomorrow as I hope to have time. I'm good with Excel. Sure wish it was a simple CSV.

I will be glad to help anyway that I can just let me know, in fact I will send you a PM with my email address and you can reach me that way.

:)

 

[pickupman66]

Hmmm...

 

[bishoptf]

Hmmm...

Yeah thats not good, you want to look at the first one,

more /var/log/syslog.1

Go page by page until you come to something that looks not correct, something happened that day and some information should be in the log.

 

[pickupman66]

is there a way to advance page by page vs just holding enter and running thru it?

 

[bishoptf]

is there a way to advance page by page vs just holding enter and running thru it?

oops sorry, space bar will take you through page by page.

 

[pickupman66]

Thank you. I have a feeling my answer will be at the end of the log here.

 

[pickupman66]

and that is the end....

Feb 3 06:24:38 raspberrypi reef-pi[25088]: 2019/02/03 06:24:38 health check: Used memory: 15.89 Load5: 0.1
Feb 3 06:24:39 raspberrypi reef-pi[25088]: 2019/02/03 06:24:39 temperature sub-system: sensor Sump-Temp value: 77.45
Feb 3 06:24:39 raspberrypi reef-pi[25088]: 2019/02/03 06:24:39 temperature sub-system: sensor Tank-Temp value: 77.3366
Feb 3 06:24:39 raspberrypi reef-pi[25088]: 2019/02/03 06:24:39 temperature subsystem: Current temperature is below minimum threshold. Executing warm up routine
Feb 3 06:24:39 raspberrypi reef-pi[25088]: 2019/02/03 06:24:39 Setting GPIO Pin: 6 State: 1
Feb 3 06:24:41 raspberrypi reef-pi[25088]: 2019/02/03 06:24:41 ato sub-system: sensor ATO value: 1
Feb 3 06:24:41 raspberrypi reef-pi[25088]: 2019/02/03 06:24:41 Setting GPIO Pin: 5 State: 0
Feb 3 06:24:46 raspberrypi reef-pi[25088]: 2019/02/03 06:24:46 Reading temperature from device: 28-021316a280aa
Feb 3 06:24:46 raspberrypi reef-pi[25088]: 2019/02/03 06:24:46 ato sub-system: sensor ATO value: 1
Feb 3 06:24:46 raspberrypi reef-pi[25088]: 2019/02/03 06:24:46 Setting GPIO Pin: 5 State: 0
Feb 3 06:24:46 raspberrypi reef-pi[25088]: 2019/02/03 06:24:46 temperature sub-system: sensor Room-Temp value: 67.775
Feb 3 06:24:51 raspberrypi reef-pi[25088]: 2019/02/03 06:24:51 ato sub-system: sensor ATO value: 1
Feb 3 06:24:51 raspberrypi reef-pi[25088]: 2019/02/03 06:24:51 Setting GPIO Pin: 5 State: 0
Feb 3 06:24:56 raspberrypi reef-pi[25088]: 2019/02/03 06:24:56 ato sub-system: sensor ATO value: 1
Feb 3 06:24:56 raspberrypi reef-pi[25088]: 2019/02/03 06:24:56 Setting GPIO Pin: 5 State: 0
Feb 3 06:25:01 raspberrypi reef-pi[25088]: 2019/02/03 06:25:01 ato sub-system: sensor ATO value: 1
Feb 3 06:25:01 raspberrypi reef-pi[25088]: 2019/02/03 06:25:01 Setting GPIO Pin: 5 State: 0
Feb 3 06:25:01 raspberrypi CRON[21992]: (root) CMD (test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ))

 

[bishoptf]

Thank you. I have a feeling my answer will be at the end of the log here.

You can start from the end also, depends on how you want to go through it:

tail -n 1000 /var/log/syslog | more

That will display the last 1000 lines, you can then page through them that way, lots of options. :)

 

[bishoptf]

and that is the end....

Feb 3 06:24:38 raspberrypi reef-pi[25088]: 2019/02/03 06:24:38 health check: Used memory: 15.89 Load5: 0.1
Feb 3 06:24:39 raspberrypi reef-pi[25088]: 2019/02/03 06:24:39 temperature sub-system: sensor Sump-Temp value: 77.45
Feb 3 06:24:39 raspberrypi reef-pi[25088]: 2019/02/03 06:24:39 temperature sub-system: sensor Tank-Temp value: 77.3366
Feb 3 06:24:39 raspberrypi reef-pi[25088]: 2019/02/03 06:24:39 temperature subsystem: Current temperature is below minimum threshold. Executing warm up routine
Feb 3 06:24:39 raspberrypi reef-pi[25088]: 2019/02/03 06:24:39 Setting GPIO Pin: 6 State: 1
Feb 3 06:24:41 raspberrypi reef-pi[25088]: 2019/02/03 06:24:41 ato sub-system: sensor ATO value: 1
Feb 3 06:24:41 raspberrypi reef-pi[25088]: 2019/02/03 06:24:41 Setting GPIO Pin: 5 State: 0
Feb 3 06:24:46 raspberrypi reef-pi[25088]: 2019/02/03 06:24:46 Reading temperature from device: 28-021316a280aa
Feb 3 06:24:46 raspberrypi reef-pi[25088]: 2019/02/03 06:24:46 ato sub-system: sensor ATO value: 1
Feb 3 06:24:46 raspberrypi reef-pi[25088]: 2019/02/03 06:24:46 Setting GPIO Pin: 5 State: 0
Feb 3 06:24:46 raspberrypi reef-pi[25088]: 2019/02/03 06:24:46 temperature sub-system: sensor Room-Temp value: 67.775
Feb 3 06:24:51 raspberrypi reef-pi[25088]: 2019/02/03 06:24:51 ato sub-system: sensor ATO value: 1
Feb 3 06:24:51 raspberrypi reef-pi[25088]: 2019/02/03 06:24:51 Setting GPIO Pin: 5 State: 0
Feb 3 06:24:56 raspberrypi reef-pi[25088]: 2019/02/03 06:24:56 ato sub-system: sensor ATO value: 1
Feb 3 06:24:56 raspberrypi reef-pi[25088]: 2019/02/03 06:24:56 Setting GPIO Pin: 5 State: 0
Feb 3 06:25:01 raspberrypi reef-pi[25088]: 2019/02/03 06:25:01 ato sub-system: sensor ATO value: 1
Feb 3 06:25:01 raspberrypi reef-pi[25088]: 2019/02/03 06:25:01 Setting GPIO Pin: 5 State: 0
Feb 3 06:25:01 raspberrypi CRON[21992]: (root) CMD (test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ))

Well thats certainly odd, looks like it just locked up, that last line is running cron jobs....

 

[pickupman66]

Tom, I found it. went to just the syslog. it happened at 17:38

@Ranjib amazingly I didnt even notice it until I put my hands into the water. Corals looked good. clam was not open which initially concerned me.

 

[bishoptf]

and that is the end....

Feb 3 06:24:38 raspberrypi reef-pi[25088]: 2019/02/03 06:24:38 health check: Used memory: 15.89 Load5: 0.1
Feb 3 06:24:39 raspberrypi reef-pi[25088]: 2019/02/03 06:24:39 temperature sub-system: sensor Sump-Temp value: 77.45
Feb 3 06:24:39 raspberrypi reef-pi[25088]: 2019/02/03 06:24:39 temperature sub-system: sensor Tank-Temp value: 77.3366
Feb 3 06:24:39 raspberrypi reef-pi[25088]: 2019/02/03 06:24:39 temperature subsystem: Current temperature is below minimum threshold. Executing warm up routine
Feb 3 06:24:39 raspberrypi reef-pi[25088]: 2019/02/03 06:24:39 Setting GPIO Pin: 6 State: 1
Feb 3 06:24:41 raspberrypi reef-pi[25088]: 2019/02/03 06:24:41 ato sub-system: sensor ATO value: 1
Feb 3 06:24:41 raspberrypi reef-pi[25088]: 2019/02/03 06:24:41 Setting GPIO Pin: 5 State: 0
Feb 3 06:24:46 raspberrypi reef-pi[25088]: 2019/02/03 06:24:46 Reading temperature from device: 28-021316a280aa
Feb 3 06:24:46 raspberrypi reef-pi[25088]: 2019/02/03 06:24:46 ato sub-system: sensor ATO value: 1
Feb 3 06:24:46 raspberrypi reef-pi[25088]: 2019/02/03 06:24:46 Setting GPIO Pin: 5 State: 0
Feb 3 06:24:46 raspberrypi reef-pi[25088]: 2019/02/03 06:24:46 temperature sub-system: sensor Room-Temp value: 67.775
Feb 3 06:24:51 raspberrypi reef-pi[25088]: 2019/02/03 06:24:51 ato sub-system: sensor ATO value: 1
Feb 3 06:24:51 raspberrypi reef-pi[25088]: 2019/02/03 06:24:51 Setting GPIO Pin: 5 State: 0
Feb 3 06:24:56 raspberrypi reef-pi[25088]: 2019/02/03 06:24:56 ato sub-system: sensor ATO value: 1
Feb 3 06:24:56 raspberrypi reef-pi[25088]: 2019/02/03 06:24:56 Setting GPIO Pin: 5 State: 0
Feb 3 06:25:01 raspberrypi reef-pi[25088]: 2019/02/03 06:25:01 ato sub-system: sensor ATO value: 1
Feb 3 06:25:01 raspberrypi reef-pi[25088]: 2019/02/03 06:25:01 Setting GPIO Pin: 5 State: 0
Feb 3 06:25:01 raspberrypi CRON[21992]: (root) CMD (test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ))

Can you do this and see what your disk space looks like:

df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root       7.3G  1.2G  5.8G  17% /
devtmpfs        212M     0  212M   0% /dev
tmpfs           216M     0  216M   0% /dev/shm
tmpfs           216M   22M  195M  11% /run
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           216M     0  216M   0% /sys/fs/cgroup
/dev/mmcblk0p1   44M   23M   22M  51% /boot
tmpfs            44M     0   44M   0% /run/user/1001

 

[bishoptf]

Tom, I found it. went to just the syslog. it happened at 17:38

@Ranjib amazingly I didnt even notice it until I put my hands into the water. Corals looked good. clam was not open which initially concerned me.

Looks like it rebooted for some reason, but it should have started back up fine, ATO looks out of range but I'm not sure why it didn't pick right back up where it left, off. will Look at this more in the morning..

 

[Ranjib]

It looks like the nightly cron job that rotates log etc caused the system to hung up on 3rd. This was not reef-pi related , but Linux /raspbian related (does not matter really ). I’m glad livestock is ok (finger crossed). I’ll dig up and share the watchdog thing soon

 

[Ranjib]

@Tom Bishop awesome diagnosis man :-) , love it , very methodical , I would do the exact same step. We are lucky to have you with us. Really appreciate it.

 

[theatrus]

It looks like the nightly cron job that rotates log etc caused the system to hung up on 3rd. This was not reef-pi related , but Linux /raspbian related (does not matter really ). I’m glad livestock is ok (finger crossed). I’ll dig up and share the watchdog thing soon

Do you have the watchdog based on reef-pi running or are you clearing it in a systemd timer? It may be useful to fully integrate this into reef-pi. Happy to noodle through this one, since I also want to ship the hard power off watchdog that’s on the Pico board.