Reef-pi doser stuck on... Bad Day

NeonRabbit221B

2500 Club Member
View Badges
Joined
Jun 21, 2019
Messages
3,037
Reaction score
5,610
Location
Richmond, Va
Rating - 0%
0   0   0
Dumped about 3/4 gallon of bicarb in my 13.5 gallon reef. First issue I have had with equipment failure so not sure how to approach this. Finally got alkalinity down to around 20 but I am thinking about when I eventually turn the system back online what should I do to prevent this happening again? I ran a calibration and the doser seemed to turn off just fine so not sure what happened...
 

HuduVudu

2500 Club Member
View Badges
Joined
Jun 2, 2020
Messages
3,241
Reaction score
3,665
Location
Houston
Rating - 0%
0   0   0
Dumped about 3/4 gallon of bicarb in my 13.5 gallon reef. First issue I have had with equipment failure so not sure how to approach this. Finally got alkalinity down to around 20 but I am thinking about when I eventually turn the system back online what should I do to prevent this happening again? I ran a calibration and the doser seemed to turn off just fine so not sure what happened...
Are you running a closed loop?
 

ReeferLou

Active Member
View Badges
Joined
Dec 24, 2020
Messages
193
Reaction score
151
Location
Michigan-Oakland County
Rating - 0%
0   0   0
Had a similar experience with my Reef-Pi but it was the calcium and I run a diluted solution and small reservoir so not much damage done. I searched a bit and there is at least 1 other instance in the build thread. I have only had this once and I was around when it was happening. I tried to SSH into the module at the time but it would not respond. I still run v4.1 I reboote the reef-pi and it has been fine since. I did notice my CPU and MEM use and pi CPU Temp creeping up prior. What ver are you running?

 
Last edited:

Lavey29

10K Club member
View Badges
Joined
Apr 29, 2021
Messages
11,383
Reaction score
12,059
Location
United States
Rating - 0%
0   0   0
Sorry for your unexpected bad situation but curious why you even have to dose a nano tank. Doesn't the small weekly water change suffice in small set ups?
 
OP
OP
NeonRabbit221B

NeonRabbit221B

2500 Club Member
View Badges
Joined
Jun 21, 2019
Messages
3,037
Reaction score
5,610
Location
Richmond, Va
Rating - 0%
0   0   0
Had a similar experience with my Reef-Pi but it was the calcium and I run a diluted solution and small reservoir so not much damage done. I searched a bit and there is at least 1 other instance in the build thread. I have only had this once and I was around when it was happening. I tried to SSH into the module at the time but it would not respond. I still run v4.1 I reboote the reef-pi and it has been fine since. I did notice my CPU and MEM use and pi CPU Temp creeping up prior. What ver are you running?

Looking at the log CPU usage spiked right around the same time frame but didnt really climb or seem abnormal prior until I look at historical. Maybe I will upgrade the pi and do regular reboots... Running 4.1 btw. Will do some digging into logs tomorrow morning.

Sorry for your unexpected bad situation but curious why you even have to dose a nano tank. Doesn't the small weekly water change suffice in small set ups?
Tons of coralline and pretty high alk demand so I do have to dose atleast for alk/calcium. I have tried water change only but everything looked ticked off the last few days before a change and if I have a busy schedule then I am in real trouble.
 

ReeferLou

Active Member
View Badges
Joined
Dec 24, 2020
Messages
193
Reaction score
151
Location
Michigan-Oakland County
Rating - 0%
0   0   0
Looking at the log CPU usage spiked right around the same time frame but didnt really climb or seem abnormal prior until I look at historical. Maybe I will upgrade the pi and do regular reboots... Running 4.1 btw. Will do some digging into logs tomorrow morning.


Tons of coralline and pretty high alk demand so I do have to dose atleast for alk/calcium. I have tried water change only but everything looked ticked off the last few days before a change and if I have a busy schedule then I am in real trouble.
Here is what my graph looked like. issue with the journctl and systemctl logs is that they do not persist across power boots by default so you may not seen any logs from that time it happened.

1647757557123.png
 

Ranjib

7500 Club Member
View Badges
Joined
Apr 16, 2016
Messages
9,843
Reaction score
17,058
Location
Pleasant Hill, Concord
Rating - 0%
0   0   0
Dumped about 3/4 gallon of bicarb in my 13.5 gallon reef. First issue I have had with equipment failure so not sure how to approach this. Finally got alkalinity down to around 20 but I am thinking about when I eventually turn the system back online what should I do to prevent this happening again? I ran a calibration and the doser seemed to turn off just fine so not sure what happened...

Sorry to hear this, and thank you for taking the time to report this. Alk 20 is still too high. Do you see any error in logs.
Dosers run on timers. you should be able to use journald to get the logs for the time when the doser supposed to trigger (till the time it should turn off)
Code:
journalctl --since "2015-01-10 12:00" --until "2015-01-11 03:00"
if the logs reflect the pump turn on and off at expected time, its probably after reef-pi, due to hardware (pump didnt switch off). But if the log instructions are missing for that specific pump turn off, while other reef-pi logs are present, will indicate a reef-pi software bug. If you dont see any log, be it reef-pi or anything else, or kernel panic (dmesg), that will likely indicate pi had a hiccup/freeze or reboot that lead to reef-pi being hard restarted while a timer was in progress, which can lead to pump remained on state,
 

ReeferLou

Active Member
View Badges
Joined
Dec 24, 2020
Messages
193
Reaction score
151
Location
Michigan-Oakland County
Rating - 0%
0   0   0
Sorry to hear this, and thank you for taking the time to report this. Alk 20 is still too high. Do you see any error in logs.
Dosers run on timers. you should be able to use journald to get the logs for the time when the doser supposed to trigger (till the time it should turn off)
Code:
journalctl --since "2015-01-10 12:00" --until "2015-01-11 03:00"
if the logs reflect the pump turn on and off at expected time, its probably after reef-pi, due to hardware (pump didnt switch off). But if the log instructions are missing for that specific pump turn off, while other reef-pi logs are present, will indicate a reef-pi software bug. If you dont see any log, be it reef-pi or anything else, or kernel panic (dmesg), that will likely indicate pi had a hiccup/freeze or reboot that lead to reef-pi being hard restarted while a timer was in progress, which can lead to pump remained on state,
Hi - thanks for jumping in.

I was thinking of changing the journal config from "auto" to "persistent" to have the file written to /var/log/journal to fix case of not having logs after a reboot.

DO you seen an issue with that?

Also, here is what my logs show when doser runs - this line repeats but not for the entire duration of the dose. I do not see any "OFF" command, just absence of the lines. My doser ran at 1:10am for 12 secs but these lines stop after 2 secs

Mar 20 01:10:02 LousReef reef-pi[281]: 2022/03/20 01:10:02 doser sub system: scheduled run Calcium-6
Mar 20 01:10:02 LousReef reef-pi[281]: 2022/03/20 01:10:02 doser sub system: scheduled run Calcium-6
Mar 20 01:10:02 LousReef reef-pi[281]: 2022/03/20 01:10:02 doser sub system: scheduled run Calcium-6
Mar 20 01:10:02 LousReef reef-pi[281]: 2022/03/20 01:10:02 doser sub system: scheduled run Calcium-6
Mar 20 01:10:02 LousReef reef-pi[281]: 2022/03/20 01:10:02 doser sub system: scheduled run Calcium-6
Mar 20 01:10:02 LousReef reef-pi[281]: 2022/03/20 01:10:02 doser sub system: scheduled run Calcium-6

Thanks
 
Last edited:

Ranjib

7500 Club Member
View Badges
Joined
Apr 16, 2016
Messages
9,843
Reaction score
17,058
Location
Pleasant Hill, Concord
Rating - 0%
0   0   0
Here is what my graph looked like. issue with the journctl and systemctl logs is that they do not persist across power boots by default so you may not seen any logs from that time it happened.

1647757557123.png
what happened during Mar 8-10. Something happened during that time which resulted into significantly higher cpu utilization (upwards of 2). That thing progressively got worse till it hit cpu limit of pi (4). That would cause it to freeze. Do you have "alert on health check" enabled ?
 

Ranjib

7500 Club Member
View Badges
Joined
Apr 16, 2016
Messages
9,843
Reaction score
17,058
Location
Pleasant Hill, Concord
Rating - 0%
0   0   0
Hi - thanks for jumping in.

I was thinking of changing the journal config from "auto" to "persistent" to have the file written to /var/log/journal to fix case of not having logs after a reboot.

DO you seen an issue with that?

Also, here is what my logs show when doser runs - this line repeats but not for the entire duration of the dose. I do not see any "OFF" command, just absence of the lines. My doser ran at 1:10am for 12 secs but these lines stop after 2 secs

Mar 20 01:10:02 LousReef reef-pi[281]: 2022/03/20 01:10:02 doser sub system: scheduled run Calcium-6
Mar 20 01:10:02 LousReef reef-pi[281]: 2022/03/20 01:10:02 doser sub system: scheduled run Calcium-6
Mar 20 01:10:02 LousReef reef-pi[281]: 2022/03/20 01:10:02 doser sub system: scheduled run Calcium-6
Mar 20 01:10:02 LousReef reef-pi[281]: 2022/03/20 01:10:02 doser sub system: scheduled run Calcium-6
Mar 20 01:10:02 LousReef reef-pi[281]: 2022/03/20 01:10:02 doser sub system: scheduled run Calcium-6
Mar 20 01:10:02 LousReef reef-pi[281]: 2022/03/20 01:10:02 doser sub system: scheduled run Calcium-6

Thanks
thats not a good sign. looks like the doser's schedule run has fired for 6 times. are there any other lines in the log? does logs before this show similar pattern (same time multiple trigger of the same thing)?
 

ReeferLou

Active Member
View Badges
Joined
Dec 24, 2020
Messages
193
Reaction score
151
Location
Michigan-Oakland County
Rating - 0%
0   0   0
what happened during Mar 8-10. Something happened during that time which resulted into significantly higher cpu utilization (upwards of 2). That thing progressively got worse till it hit cpu limit of pi (4). That would cause it to freeze. Do you have "alert on health check" enabled ?
I think I VNC into my PI to look at it as I am going on vacation next week and wanted to check it. I normally do not use that feature but was getting ready. Hmmm...

I do not have alert Health Check enabled. I am guessing I should . I did not know 2 or more was an issue for CPU.
 
Last edited:

ReeferLou

Active Member
View Badges
Joined
Dec 24, 2020
Messages
193
Reaction score
151
Location
Michigan-Oakland County
Rating - 0%
0   0   0
thats not a good sign. looks like the doser's schedule run has fired for 6 times. are there any other lines in the log? does logs before this show similar pattern (same time multiple trigger of the same thing)?
There are more log lines than that repeating -I only copied a few. I guess that explains my graph - note most days it was 12 sec but the they got scaled out due to the high values of 5000. I see same pattern for the ALK doser too but it does not really run that long. I deleted my Calc and re-entered it so that graph is gone but it looked the same. It is triggering multiple times in a short period like you said.

1647779626498.png


1647779818224.png



</code> Mar 20 07:05:02 LousReef reef-pi[281]: 2022/03/20 07:05:02 doser sub system: scheduled run Alk Mar 20 07:05:02 LousReef reef-pi[281]: 2022/03/20 07:05:02 doser sub system: scheduled run Alk Mar 20 07:05:02 LousReef reef-pi[281]: 2022/03/20 07:05:02 doser sub system: scheduled run Alk Mar 20 07:05:02 LousReef reef-pi[281]: 2022/03/20 07:05:02 doser sub system: scheduled run Alk Mar 20 07:05:02 LousReef reef-pi[281]: 2022/03/20 07:05:02 doser sub system: scheduled run Alk Mar 20 07:05:02 LousReef reef-pi[281]: 2022/03/20 07:05:02 doser sub system: scheduled run Alk Mar 20 07:05:02 LousReef reef-pi[281]: 2022/03/20 07:05:02 doser sub system: scheduled run Alk Mar 20 07:05:02 LousReef reef-pi[281]: 2022/03/20 07:05:02 doser sub system: scheduled run Alk Mar 20 07:05:02 LousReef reef-pi[281]: 2022/03/20 07:05:02 doser sub system: scheduled run Alk Mar 20 07:05:02 LousReef reef-pi[281]: 2022/03/20 07:05:02 doser sub system: scheduled run Alk Mar 20 07:05:02 LousReef reef-pi[281]: 2022/03/20 07:05:02 doser sub system: scheduled run Alk Mar 20 07:05:02 LousReef reef-pi[281]: 2022/03/20 07:05:02 doser sub system: scheduled run Alk Mar 20 07:05:02 LousReef reef-pi[281]: 2022/03/20 07:05:02 doser sub system: scheduled run Alk Mar 20 07:05:02 LousReef reef-pi[281]: 2022/03/20 07:05:02 doser sub system: scheduled run Alk Mar 20 07:05:03 LousReef reef-pi[281]: 2022/03/20 07:05:02 doser sub system: scheduled run Alk Mar 20 07:05:03 LousReef reef-pi[281]: 2022/03/20 07:05:02 doser sub system: scheduled run Alk Mar 20 07:05:03 LousReef reef-pi[281]: 2022/03/20 07:05:02 doser sub system: scheduled run Alk Mar 20 07:05:03 LousReef reef-pi[281]: 2022/03/20 07:05:02 doser sub system: scheduled run Alk Mar 20 07:05:03 LousReef reef-pi[281]: 2022/03/20 07:05:02 doser sub system: scheduled run Alk
 
Last edited:

ReeferLou

Active Member
View Badges
Joined
Dec 24, 2020
Messages
193
Reaction score
151
Location
Michigan-Oakland County
Rating - 0%
0   0   0
@Ranjib Quick question if you have time - I seem to have dosers running that are not in my GUI page meaning old ID that I deleted and now the old names shows. How can I manually delete these? If I cant i am going to need to do a fresh install I guess is one way. Looking to avoid that. Thanks
 

ReeferLou

Active Member
View Badges
Joined
Dec 24, 2020
Messages
193
Reaction score
151
Location
Michigan-Oakland County
Rating - 0%
0   0   0
Just to clarify the doser that run are not seen with the db command and are in addition to these:
i@LousReef:~/scripts $ sudo reef-pi db list doser { "5": { "id": "5", "name": "Calcium", "jack": "1", "pin": 0, "regiment": { "enable": true, "schedule": { "day": "*", "hour": "10", "minute": "10", "second": "0", "week": "*", "month": "*" }, "duration": 10, "speed": 100 } }, "6": { "id": "6", "name": "Alkalinity", "jack": "2", "pin": 1, "regiment": { "enable": true, "schedule": { "day": "*", "hour": "8", "minute": "55", "second": "0", "week": "*", "month": "*" }, "duration": 12, "speed": 100 } } }
 

Ranjib

7500 Club Member
View Badges
Joined
Apr 16, 2016
Messages
9,843
Reaction score
17,058
Location
Pleasant Hill, Concord
Rating - 0%
0   0   0
@Ranjib Quick question if you have time - I seem to have dosers running that are not in my GUI page meaning old ID that I deleted and now the old names shows. How can I manually delete these? If I cant i am going to need to do a fresh install I guess is one way. Looking to avoid that. Thanks
That’s odd. Just reload reef-pi , that should do
 

ReeferLou

Active Member
View Badges
Joined
Dec 24, 2020
Messages
193
Reaction score
151
Location
Michigan-Oakland County
Rating - 0%
0   0   0
That’s odd. Just reload reef-pi , that should do
Rebooting PI fixed the multiple dose triggers. Very odd how that happened. I was experimenting with some POST commands to modify the dose duration based on PH (new feature suggest?) so this is the only cause I can think of.
 

Keeping it clean: Have you used a filter roller?

  • I currently use a filter roller.

    Votes: 69 34.7%
  • I don’t currently use a filter roller, but I have in the past.

    Votes: 7 3.5%
  • I have never used a filter roller, but I plan to in the future.

    Votes: 51 25.6%
  • I have never used a filter roller and have no plans to in the future.

    Votes: 64 32.2%
  • Other.

    Votes: 8 4.0%
Back
Top