reef-pi :: An opensource reef tank controller based on Raspberry Pi.

[Ranjib]

Starting to get a little frustrated, grumble, grumble....Looks like my existing reef-pi unit that has been running for a year plus corrupted the SD card, I really don't like SD cards. I took this as an opportunity to rebuild with the latest version, easy enough and I got it back up an working, using 3.4. Either I have run into a bug or I am doing something wrong, but I get a portion of the way through configuring and if I reboot the service will not start up, here are the journalctl lines:

Started raspberry pi based reef tank controller.
May 28 22:04:36 tankpi reef-pi[1147]: 2020/05/28 22:04:36 driver-subsystem: registering driver id: rpi Name: Raspberry Pi
May 28 22:04:36 tankpi reef-pi[1147]: 2020/05/28 22:04:36 INFO: equipment subsystem: Finished syncing all equipment
May 28 22:04:36 tankpi reef-pi[1147]: 2020/05/28 22:04:36 Successfully started subsystem: equipment
May 28 22:04:36 tankpi reef-pi[1147]: 2020/05/28 22:04:36 Successfully started subsystem: temperature
May 28 22:04:36 tankpi reef-pi[1147]: 2020/05/28 22:04:36 Successfully started subsystem: lightings
May 28 22:04:36 tankpi reef-pi[1147]: 2020/05/28 22:04:36 Successfully started subsystem: macro
May 28 22:04:36 tankpi reef-pi[1147]: 2020/05/28 22:04:36 Successfully started subsystem: timers
May 28 22:04:36 tankpi reef-pi[1147]: 2020/05/28 22:04:36 Successfully started subsystem: system
May 28 22:04:36 tankpi reef-pi[1147]: 2020/05/28 22:04:36 reef-pi is up and running
May 28 22:04:36 tankpi reef-pi[1147]: 2020/05/28 22:04:36 Starting https server at: 0.0.0.0:443
May 28 22:04:36 tankpi reef-pi[1147]: 2020/05/28 22:04:36 Starting health checker
May 28 22:04:36 tankpi reef-pi[1147]: Methods: [GET]
May 28 22:04:36 tankpi reef-pi[1147]: 2020/05/28 22:04:36 API Doc path: /api/settings

May 28 22:05:36 tankpi reef-pi[1147]: 2020/05/28 22:05:36 Reading temperature from device: 28-011938443c3c
May 28 22:05:36 tankpi reef-pi[1147]: 2020/05/28 22:05:36 Reading temperature from device: 28-00000a9aeb40
May 28 22:05:36 tankpi reef-pi[1147]: 2020/05/28 22:05:36 health check: Used memory: 11.42  Load5: 0.24
May 28 22:05:37 tankpi reef-pi[1147]: 2020/05/28 22:05:37 temperature sub-system:  sensor 10g-green value: 78.24
May 28 22:05:37 tankpi reef-pi[1147]: panic: descriptor Desc{fqName: "10g_green_reading", help: "Module:10g-green Item:reading", constLabels: {}, variableLabels: []} is invalid: "10g_green_reading" is not a valid metric name
May 28 22:05:37 tankpi reef-pi[1147]: goroutine 44 [running]:
May 28 22:05:37 tankpi reef-pi[1147]: github.com/prometheus/client_golang/prometheus.(*Registry).MustRegister(0x2176380, 0x200fd18, 0x1, 0x1)
May 28 22:05:37 tankpi reef-pi[1147]:         /home/ranjib/gocode/pkg/mod/github.com/prometheus/[email protected]/prometheus/registry.go:400 +0x80
May 28 22:05:37 tankpi reef-pi[1147]: github.com/prometheus/client_golang/prometheus/promauto.Factory.NewGauge(0x767ce0, 0x2176380, 0x0, 0x0, 0x0, 0x0, 0x202bd20, 0x11, 0x202bd40, 0x1d, ...)
May 28 22:05:37 tankpi reef-pi[1147]:         /home/ranjib/gocode/pkg/mod/github.com/prometheus/[email protected]/prometheus/promauto/auto.go:295 +0xc0
May 28 22:05:37 tankpi reef-pi[1147]: github.com/prometheus/client_golang/prometheus/promauto.NewGauge(...)
May 28 22:05:37 tankpi reef-pi[1147]:         /home/ranjib/gocode/pkg/mod/github.com/prometheus/[email protected]/prometheus/promauto/auto.go:190
May 28 22:05:37 tankpi reef-pi[1147]: github.com/reef-pi/reef-pi/controller/telemetry.(*telemetry).EmitMetric(0x21b8180, 0x21a19c0, 0x9, 0x6742ce, 0x7, 0x28f5c28f, 0x40538f5c)
May 28 22:05:37 tankpi reef-pi[1147]:         /home/ranjib/gocode/src/github.com/reef-pi/reef-pi/controller/telemetry/stats.go:175 +0x464
May 28 22:05:37 tankpi reef-pi[1147]: github.com/reef-pi/reef-pi/controller/modules/temperature.(*Controller).Check(0x2198840, 0x2197200)
May 28 22:05:37 tankpi reef-pi[1147]:         /home/ranjib/gocode/src/github.com/reef-pi/reef-pi/controller/modules/temperature/control.go:30 +0x3dc
May 28 22:05:37 tankpi reef-pi[1147]: github.com/reef-pi/reef-pi/controller/modules/temperature.(*Controller).Run(0x2198840, 0x2197200, 0x21acd80)
May 28 22:05:37 tankpi reef-pi[1147]:         /home/ranjib/gocode/src/github.com/reef-pi/reef-pi/controller/modules/temperature/tc.go:191 +0xc8
May 28 22:05:37 tankpi reef-pi[1147]: created by github.com/reef-pi/reef-pi/controller/modules/temperature.(*Controller).Start
May 28 22:05:37 tankpi reef-pi[1147]:         /home/ranjib/gocode/src/github.com/reef-pi/reef-pi/controller/modules/temperature/controller.go:71 +0x1c4
May 28 22:05:37 tankpi systemd[1]: reef-pi.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
May 28 22:05:37 tankpi systemd[1]: reef-pi.service: Failed with result 'exit-code'.
May 28 22:07:07 tankpi systemd[1]: reef-pi.service: Service RestartSec=1min 30s expired, scheduling restart.
May 28 22:07:07 tankpi systemd[1]: reef-pi.service: Scheduled restart job, restart counter is at 9.

I have enable prometheus and haven't tried turning that off but something is amiss here, I can always drop back to 3.3 but I see a panic and an invalid argument error messages. It will not start up, I thought it was my naming convention or something but I think something is amiss in 3.4 that's not very nice.

Let me know if you need additional information, I will delete the database once again and see if I can figure out what it doesn't like.

:)

UPDATE: I've narrowed it down to the temp sensor readings, it crashes right after trying to read the temp value, doesn't make a difference which temp senor is enable it crashes after trying to read the sensor:

2020/05/28 23:59:45 Reading temperature from device: 28-011938443c3c
May 28 23:59:46 tankpi reef-pi[695]: 2020/05/28 23:59:46 temperature sub-system:  sensor 10g value: 78.24
May 28 23:59:46 tankpi reef-pi[695]: panic: descriptor Desc{fqName: "10g_reading", help: "Module:10g Item:reading", constLabels: {}, variableLabels: []} is invalid: "10g_reading" is not a valid metric name

If I disable the temp sensor it stays up...what am I missing? I've enabled 1-wire and I2C in raspi-config, and it sees the sensors but is crashing non the less. If anyone has any idea's let me know.

Thanks

We updated the prometheus library in 3.4. It looks like the new update does not allow metric name starting with number and thus "10g" is causing it to panic (i am assuming you have a temperature controller with name 10g). For the time being changing it to something else that does not starts with a number might fix it (e,g. TenG or IOg) or just disable prometheus from the preferences. Meanwhile I'll check if we can prevent this in reef-pi side , i.e check if the metric name is valid or not, and error instead of panic if its invalid.

 

[Ranjib]

Hi,
Did you manage to find anything related to my issue? I have re-installed Reef-Pi twice now, the last time on a new card with fresh install of Raspbian but I still get the same effect, no red bar for "Off" but green bar for "On" as normal. Also, I have a temperature graph on my dashboard, it no longer updates until I hit Reload so data is there but no updating on the graph.
Thanks.

I tried to reproduce this but was not able to. My equipment chart looks correct. Can you share you equipment tab and dashboard screenshot? If you think this is always happening, then please file a github bug with all the details.

 

[oreo54]

Well really it only experiences "wear" when it is turned off and on really. When it is left running, it is all in the RAM. A lot of major companies rely on flash storage for the OS of virtual machine hypervisors. Myself included. You could always set up a SSD or external drive and use that instead

you may be right but some people don't feel quite that secure w/ sd..

"When I was scrounging in the guts of Swift itself, I was noticing how many folks were burning out SD cards doing these big expensive builds (with swap files) instead of using a USB drive for the kind for stuff that thrashes the SD card. Faster too.

My dev board has a pocket USB drive hooked up to it for that reason. "

The fragility of SD cards is the weak link in the Raspberry Pi ecosystem. Most of us seem to have at least one Pi tucked away somewhere, running a Magic Mirror, driving security cameras, or even taking care of a media library. But chances are, that Pi is writing lots and lots of log files. Logging is good — it helps when tracking down issues — but uncontrolled logging can lead to problems down the road with the Pi’s SD card.

Give Your Raspberry Pi SD Card A Break: Log To RAM

The fragility of SD cards is the weak link in the Raspberry Pi ecosystem. Most of us seem to have at least one Pi tucked away somewhere, running a Magic Mirror, driving security cameras, or even ta…

hackaday.com

Guess it depends...

Consider it just an FYI atm..

Log2Ram: Extending SD Card Lifetime for Raspberry Pi LoRaWAN Gateway

My LoRaWAN gateway (“Contributing an IoT LoRaWAN Raspberry Pi RAK831 Gateway to The Things Network” is running and working great now for more than a month and it already has transmitted…

mcuoneclipse.com

 

[Ranjib]

Well really it only experiences "wear" when it is turned off and on really. When it is left running, it is all in the RAM. A lot of major companies rely on flash storage for the OS of virtual machine hypervisors. Myself included. You could always set up a SSD or external drive and use that instead

All write operations in reef-pi is written on disk. This include all types of creation, deletion, update (equipment, connector, ato , ph etc). Everytime an equipment is switched on or off, that too is written on disk (so that reef-pi can restore the equipment in expected state upon startup). All usage data (temperature readings, heater/cooler usage etc) are written on disk once every hour.
We did some optimizations (the usage data is stored in-memory ring buffer) to reduce the disk write scenario.. but still we have some disk write operations.

 

[Ranjib]

So i have another question... In a Paludarium i would need the humidity to be 70-80 during the day and 100 during the night...

So during the day i would set the humidity control with the PH settings : Upper treshold would activate a fan when i reach 80. Lower treshold wil deactivate the fan or wil activate a humidifier when it reach 70. hysteresis 2-3.. (i have to experiment with this)

But when the night fals i need to get other settings. to reach 100% humidity. How can i approach this? do i look at Macro's ?

Yes, create two separate ph controller with specific setting and use a macro to turn one off, wait for few second and turn the other on. Make the two macros to turn them on/off at specific time. Then use two timers with those two target macros.
Let me know how it goes,,,

 

[Badfish2too]

All write operations in reef-pi is written on disk. This include all types of creation, deletion, update (equipment, connector, ato , ph etc). Everytime an equipment is switched on or off, that too is written on disk (so that reef-pi can restore the equipment in expected state upon startup). All usage data (temperature readings, heater/cooler usage etc) are written on disk once every hour.
We did some optimizations (the usage data is stored in-memory ring buffer) to reduce the disk write scenario.. but still we have some disk write operations.

But like you said minimal writes. I have seen cheap SD cards fail. But in my experience, its pretty rare for one to fail unless it is constantly reading and writing large files. I have never had one fail on me, but they are mostly relegated to cell phones, reefpi, and my servers for the hypervisor. Which really never writes

 

[Ranjib]

To be perfectly honest, I turned them off. I've enabled them again and will monitor.

that interrupt error is rare but a possible scenario that can happen, when linux kernel intervenes the sensor file reading operations. The kernel does so due to the reading process taking longer than expected. Its rare, and we can add logic to check this and retry (which is recommended). But this is the first time im learning about this error. We never really write such failure handling logic (even in profession coding work), because most often it indicates something is not working correct, i.e. the kernel is getting busy in something else, or the sensor /device management system itself is encountering some issues. Keep us posted, im curious,

 

[Ranjib]

But like you said minimal writes. I have seen cheap SD cards fail. But in my experience, its pretty rare for one to fail unless it is constantly reading and writing large files. I have never had one fail on me, but they are mostly relegated to cell phones, reefpi, and my servers for the hypervisor. Which really never writes

I would agree to that. Im using samsung evo, and most of them running fine for more than two years. Although almost all my cheaper sandisk cards did failed soon after first year of their usage. After that i aggressively updated all builds with samsung cards. The whole in-memroy ring buffer optimization came out of that experience. I also keep the reef-pi db copy in google drive for backup.

 

[Yov]

Yes, create two separate ph controller with specific setting and use a macro to turn one off, wait for few second and turn the other on. Make the two macros to turn them on/off at specific time. Then use two timers with those two target macros.
Let me know how it goes,,,

so something like this:

Sorry for the Dutch in there XD Dag = Day nacht : night.

 

[Des Westcott]

So i have another question... In a Paludarium i would need the humidity to be 70-80 during the day and 100 during the night...

So during the day i would set the humidity control with the PH settings : Upper treshold would activate a fan when i reach 80. Lower treshold wil deactivate the fan or wil activate a humidifier when it reach 70. hysteresis 2-3.. (i have to experiment with this)

But when the night fals i need to get other settings. to reach 100% humidity. How can i approach this? do i look at Macro's ?

*Edit* And by Temperature Control I mean your humidity control.

My first guess would be to use a Timer to trigger Macros which control set the desired Temperature Controls.

1. Set up a "Day temp" Temperature Control
2. Set up a "Night temp" Temperature Control using your settings above
3. Create Macro "Day On" that fistly disables "Night Temp" and then enables "Day Temp"
4. Create a Macro "Night On" that disables "Day Temp" and then enables "Night Temp"
5. Create Timers to run "Night ON" and "Day On" Macros

Remember for Macros to be triggered by Timers, they must be written in reverse and "run in reverse" must be enabled.

Let mem know if that works - I think it should.

 

[Des Westcott]

that interrupt error is rare but a possible scenario that can happen, when linux kernel intervenes the sensor file reading operations. The kernel does so due to the reading process taking longer than expected. Its rare, and we can add logic to check this and retry (which is recommended). But this is the first time im learning about this error. We never really write such failure handling logic (even in profession coding work), because most often it indicates something is not working correct, i.e. the kernel is getting busy in something else, or the sensor /device management system itself is encountering some issues. Keep us posted, im curious,

There definitely seems to be sometning going on, but with no dates, it's difficult to tie int the alerts, but I will keep a watch on it.

 

[Ranjib]

so something like this:

Sorry for the Dutch in there XD Dag = Day nacht : night.

Add few seconds of sleep in between turn on/off of the two controller.

 

[Ranjib]

I tried to reproduce this but was not able to. My equipment chart looks correct. Can you share you equipment tab and dashboard screenshot? If you think this is always happening, then please file a github bug with all the details.

I take it back. I just tested this again, and hit this bug. As i suspected earlier , this was due to the chart library upgrade. I have patched the code, it will not be exactly same as before . I am not able to get the exact old chart using the new version of the library, but its fairly similar. Thank you for letting us know, and apologies for the earlier false verification


https://github.com/reef-pi/reef-pi/pull/1157

 

[Yov]

*Edit* And by Temperature Control I mean your humidity control.

My first guess would be to use a Timer to trigger Macros which control set the desired Temperature Controls.

1. Set up a "Day temp" Temperature Control
2. Set up a "Night temp" Temperature Control using your settings above
3. Create Macro "Day On" that fistly disables "Night Temp" and then enables "Day Temp"
4. Create a Macro "Night On" that disables "Day Temp" and then enables "Night Temp"
5. Create Timers to run "Night ON" and "Day On" Macros

Remember for Macros to be triggered by Timers, they must be written in reverse and "run in reverse" must be enabled.

Let mem know if that works - I think it should.

like this Humidity day night control:

MAcro's:

Timers set for day start at 8 AM night start at 6pm (18:00):

 

[Yov]

Add few seconds of sleep in between turn on/off of the two controller.

So not the solution like @Des Westcott ? If i need some time in betweezn i need 4 macro's and 4 timers :
1 macro : turn off day + 1 timer turn off day.
1 macro turn on night + 1 timer turn on day

1 macro turn off night + 1 timer turn off night;
1 macro turn on day + 1 timer turn on day.

The solution now includes on/off in 1 macro.

 

[mr.salty]

So I have made a couple of 8 channel power Controllers using these relays.

8 Channel 5V Relay Module

DESCRIPTION RETIRED PRODUCT - We've stopped stocking this product or it has been retired by the manufacturer and is no longer for sale. This page remains available for its resources and as a source of potential interest. This is a simple 8-channel relay module. It can be used to control...

coolcomponents.co.uk

I am using a buck converter to deliver the 5v to the VCC and gnd on the relays

The raspberry pi will turn on the outlets individually and may go upto three outlets but anything more then that turns off every outlet of the relay and the lights stop illuminating on the relay.

Do I need a logic converter to get 3.3 to 5v?

I wouldn't of thought it would of switched one relay if voltage was an issue?

Are there particular Gpio on the raspberry pi 4b that I can't use beside the usual power and grounds?

 

[Des Westcott]

So not the solution like @Des Westcott ? If i need some time in betweezn i need 4 macro's and 4 timers :
1 macro : turn off day + 1 timer turn off day.
1 macro turn on night + 1 timer turn on day

1 macro turn off night + 1 timer turn off night;
1 macro turn on day + 1 timer turn on day.

The solution now includes on/off in 1 macro.

You could quite easily add a minute wait in the "Day on" and "Night On" Macros

 

[Des Westcott]

like this Humidity day night control:

MAcro's:

Timers set for day start at 8 AM night start at 6pm (18:00):

That's the right idea, you just need to reverse the steps in your "Day On" and "Night On" Macros for it to work right. It needs to be written backwards in reverse state for it to work.

So "Day On" should read:

1. Humidity Day -> Off
2. Wait 60s (Or whatever value you want if you want that)
3. Humidity Night -> On

So when the Timer triggers this macro, it will turn Humidity Night off, wait 1 minute then Humidity Day On.

It's a bit Bass Ackwards I know, but that 's how it works at the moment. The Dev's are working on sorting it out though from what I know

 

[Des Westcott]

So I have made a couple of 8 channel power Controllers using these relays.

8 Channel 5V Relay Module

DESCRIPTION RETIRED PRODUCT - We've stopped stocking this product or it has been retired by the manufacturer and is no longer for sale. This page remains available for its resources and as a source of potential interest. This is a simple 8-channel relay module. It can be used to control...

coolcomponents.co.uk

I am using a buck converter to deliver the 5v to the VCC and gnd on the relays

The raspberry pi will turn on the outlets individually and may go upto three outlets but anything more then that turns off every outlet of the relay and the lights stop illuminating on the relay.

Do I need a logic converter to get 3.3 to 5v?

I wouldn't of thought it would of switched one relay if voltage was an issue?

Are there particular Gpio on the raspberry pi 4b that I can't use beside the usual power and grounds?

I think you have run out of power to control the relays. Have a look at using a ULN2308 with external power supply to power the relays.

reef-pi :: An opensource reef tank controller based on Raspberry Pi.

Hi again Q if i drive the 8 ch relay board from external 5v how to give vcc from raspberry to relay board.What Ranjib send me gpio pin out ther is vcc on #1 pin but that is Temp VCC I run my relay on a external power supply. The only thing from the relay connected to the pi are the GPIO Pins...

www.reef2reef.com

Post 4803

 

[Des Westcott]

Sorry for the Dutch in there XD Dag = Day nacht : night.

Just spotted this comment now.

Being from South Africa, despite being English, I do speak fluent Afrikaans which is very similar to Dutch - Day = Dag and Night = Nag

Ek kan jou so redelik bietjie verstaan! ;)