Guide: Telemetry server

Ranjib

7500 Club Member
View Badges
Joined
Apr 16, 2016
Messages
9,825
Reaction score
17,041
Location
Pleasant Hill, Concord
Rating - 0%
0   0   0
This guide will walk through the installation and configuration of a long term, robust and featureful telemetry stack. With this you can observe your reef tank ato usage, long term temperature trends, build custom dashboard and more. To do this we install a trifecta of three awesome opensource software, namely: prometheus, grafana and node exporter. It will be extra awesome If you have an existing reef-pi installation that you can use that to integrate with this telemetry server. But this is not necessary, and we'll cover a minimal all contained in one mini pc, including reef-pi setup.


telemetry-guide-reefer300xl.png




Background

reef-pi 6.0 is just out with esp32 integration and x86 builds, this means we can now run reef-pi on non pi computers and offload the hadware interfacing to esp32 and raspberry pi pico like microcontroller. With the ongoaing shortage of chip supply and consequently raspberry pi 3/4/zeros, reef-pi 6.0 provides an alternative to raspberry pi based installation. But tangential to this(bypassing supply chain issue) running on x86 systems mean reef-pi controller can now have significantly more computing power. There are multitude of functionalities in reef-pi that are purpose built to work on low computing power environment (like metric retention, dashboard etc) and with low dependencies. While this gives a slim working controller runnable to raspberry pi, it deprives whole bunch of awesomeness opensource can offer. Running on x86, with all the firepower available now removes that limitation. In this guide i want to walk through one such awesome capabilities "telemetry" that was very hard to setup on raspberry pi, but can be easily run on mini pc, laptop or desktop based setup (x86 /intel systems).

Hardware

We'll start with installing reef-pi in a common non raspberry pi computer (x86). Any laptop, PC, mini-pc should do.

For this guide i am using a cheaper intel celeron based mini pc with ssd (A genuine, 250$+ Intel NUC will be awesome).

At minimum, consider a somewhat beefy, 2-4 core (2.4GHz+celeron), 8 GB ram, 128Gb SSD (no emmc ) computer. It is expected to be capable of running a full dekstop environment, with peripherals attached, (often powered through it) .

As of the time of writing this guide such systems cost aroun 120-170$ USD. I am experimenting with a amd a90 based 87$ system, but its not well tested yet for me to vouch for it,

I am not covering how to install linux on mini pc, there are ample tutorials and youtube videos on this topic, please refer to one of those. I am also assuming users are familiar with terminal or command line. You will be using it to install and configure a few things.

Software Installation & configuration

- Prometheus, at the heart of this stack is the metrics storage software. The metrics emitted from reef-pi (such as temperature, ato usage, ph ) are scraped by prometheus and stored for longer duration. Anything can programmaticaly query obtain those metrics. Prometheus is a cloud native metric store that is lingua franca for all things metrics storage in web technology, you are riding on the shoulder of a giant. Hence we start with prometheus installation.

Download prometheus from github release page or their homepage.
Code:
wget -c https://github.com/prometheus/prometheus/releases/download/v2.40.4/prometheus-2.40.4.linux-amd64.tar.gz
Decompress to obtain the prometheus binary and move it in /usr/bin directory
Code:
tar -zxvf prometheus-2.40.4.linux-amd64.tar.gz
sudo mv prometheus-2.40.4.linux-amd64/prometheus /usr/bin/

Copying over a starter example configuration file for prometheus
Code:
sudo cp prometheus-2.40.6.linux-amd64/prometheus.yml /etc/prometheus.yml


Create a systemd unit file to be able to run and supervise promethus on this system across reboots.

Code:
[Unit]
Description=prometheus

[Service]
TimeoutStartSec=0
ExecStart=/usr/bin/prometheus --config.file=/etc/prometheus.yml --storage.tsdb.path=/var/lib/prometheus/ --storage.tsdb.retention.time=5y --web.listen-address=:8888

[Install]
WantedBy=multi-user.target

Next lets ask systemd to read the new unit file (at /etc/systemd/system/prometheus.service) and start prometheus
Code:
systemctl daemon-reload
sudo systemctl start prometheus.service
sudo systemctl enable prometheus.service
sudo systemctl status prometheus.service


Once prometheus is up and running we can move on to installing grafana, it is a software

- Grafana - the dashboard software. installation grafana by following their official installation guide. and integration with prometheus. Once grafana is installed configure prometheus as a datasource in grafana. With this now you can visualize and see any metric from prometheus in grafana.

- This is an option step. If you have an existing reef-pi (running on pi ) then skip this. If you are using this system itself as reef-pi controller (with esp32) then proceed with installing reef-pi. start with downloading reef-pi package for x86 systems
Code:
wget -c https://github.com/reef-pi/reef-pi/releases/download/6.0/reef-pi-6.0-x86.deb

followed by installing
Code:
sudo dpkg -i reef-pi-6.0-x86.deb

- Next configure reef-pi to emit prometheus metric by enabling the integration under Configuration->Settings
prometheus.png


and reload reef-pi (Configuration -> Admin->Reload)
- Next configure prometheus to scrape reef-pi metrics.
Open and edit the "/etc/prometheus.yaml" file and add a job name "pico1" with appropriate ip and port. specify the "metrics_path" option as shown and set it to "/x/metrics". reef-pi provide prometheus metrics at "http://<reef-pi-ip>/x/metrics" path. you can type it in your browser (replace with you reef-pi controller ip) to get a glimpse of the raw metrics.
Code:
global:
  scrape_interval: 15s
  evaluation_interval: 15s
scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]
  - job_name: "reef-pi"
    metrics_path: '/x/metrics'
    static_configs:
      - targets: ["localhost:80"]

- Lets create a dashboards with reef-pi metrics
- Optionally: Node exporter - host metrics gathering software
"/etc/systemd/system/node_exporter.service"

Code:
[Unit]
Description=Prometheus Node Exporter
Documentation=https://github.com/prometheus/node_exporter
After=network-online.target

[Service]
User=root
ExecStart=/usr/bin/node_exporter
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

Start and enable node exporter as a service.
Code:
sudo systemctl daemon-reload
sudo systemctl start node_exporter.service
sudo systemctl status node_exporter.service
sudo systemctl enable node_exporter.service

Configure prometheus to scrape from node exporter , for host metrics by editing the /etc/prometheus.yaml file.

Restart prometheus. Go to your grafana dashboard and import the node exporter community dashboard template. Specify prometheus data source that you have configured with grafana in previous step.

Node exporter-grafana-import.png


Once imported, you should be able to see the localhost metrics within next 10 seconds. Here is how mine looks

node-exporter-telemetry-server-guide.png



Now off to building a reef-pi specific dashboard. The key thing to remeber is this, that
a) reef-pi emits metrics that can be observed in /x/metrics path in the app to get their name,
b) then these metrics are then scraped and stored by prometheus, and then
c) we configure the dashboard in grafan to get those metrics from prometheus and display it in any fancy way our heart desire.

Since our current reef-pi does not have any sensor attached we'll use a controller health metrics , system load to build a graph. the emitted metrics look like this (from /x/metics page)
system-load.png



Which means prometheus will scrape and store these metrics with exactly those names under the job "reef-pi" (because of the configuration in prometheus.yaml file). And hence we can use that in grafana, by creating a new dashboard -> and then a new panel under the dashboard, followed by selecting the system load metric

system-load-dashboard.png



Now, because we dont have anything for controlling this setup in itself does not have any data to show. But in reality a controller will have many sesnors, controlling equipment attached to it. And to build a comprehensive dashboard we have to know the names of the corresponding metrics. reef-pi follows certain convention for converting the data in controller to metric names,. For example the temperature module emits three metric "<name>_reading" (current temperature), <name>_heater (heater usage of that hour) and <name>_cooler (chiller usage of that hour). "name" here represent the name of the temperature controller. The screenshot below shows my pico 1 controller where the temperature probe is named as "Tank"
temperature-telemetry-guide.png


corresponding reef-pi metrics page (/x/metrics)


now, i can configure prometheus to scrape my pico1 controller metrics and then use those metrics to build a temperature graph.
Pico-1 -temp.png


Here is more elaborate my pico tank metrics, notice there is ato metyric, health metri and more.
x-metrics-telemetry-guide.png
,

and powered by those, i have a comprehensive dashboard for my pico1 tank that provide, ato, temperature and system health status
pico-dashboard.png

I have also added ambient temperature (my workshop where the tank is located) from another sensor in this dashboard.

In the beginning of the guide we talked about durability and robustness of this setup. Now we can actualize those capabilities. We can retain data for years (my oldest setup has 4+ years of telemetry data now) and from a number of different controllers in one place. Here is an example of a dashboard that covers all my 7 setups (1 reefer 300xl, 1 biocube 29, and 5 picos of different configuration) all in one place
sensors.png


Sky is the limit from here. You are now running the same tech that powers some of the largest and world renowned tech organizations. Its is been vetted by their use and it is here by the larger opensource communities hard work. You are riding one the shoulder of giants, this setup can work for multiple dozens of controller. This same stack could be run on beefire rack mounted servers to support thousands of controller and more.

This completes the telemetry guide for reef-pi 6.0 with x86 setup. Please let me know if you have any feedback and I will try my best to incorporate it as and when i receive it.
 
Last edited:
OP
OP
Ranjib

Ranjib

7500 Club Member
View Badges
Joined
Apr 16, 2016
Messages
9,825
Reaction score
17,041
Location
Pleasant Hill, Concord
Rating - 0%
0   0   0
You forgot to link to the guide right? :) Please :)... @Ranjib I have hacked away at alot of the Grafana/Prometheus stuff, but would love to see all that I missed.
this is the guide, its just im planning to update it iteratively like code
 

Sral

Valuable Member
View Badges
Joined
May 2, 2022
Messages
1,005
Reaction score
971
Location
Germany
Rating - 0%
0   0   0
Quick question, since I don't remember this mentioned anywhere:
- How much Computation power does a Reef-Pi setup and a Prometheus-Server with a Grafana interface need ?
- e.g.: Could one run a Prometheus Server on a dedicated Pi and what version would this need ?
- In light of the new Pi Zero 2, which iirc has a dual core, would this be able to run both ?
 

ColoredRock

Well-Known Member
View Badges
Joined
Aug 9, 2021
Messages
849
Reaction score
1,021
Location
Murrieta, CA (North of San Diego)
Rating - 0%
0   0   0
Quick question, since I don't remember this mentioned anywhere:
- How much Computation power does a Reef-Pi setup and a Prometheus-Server with a Grafana interface need ?
- e.g.: Could one run a Prometheus Server on a dedicated Pi and what version would this need ?
- In light of the new Pi Zero 2, which iirc has a dual core, would this be able to run both ?
I ran it on a RP 4b... got pretty hot... also a mcrosd card wont cut it.. to much IO so went with an external HD before my tank blew a seam... going to run a dedicated ubuntu box now when I get the 280 gallon wet. Sanding sucks... but its coming along.

Not sure on the new Pi Zero 2 have not looked at it yet,
 
OP
OP
Ranjib

Ranjib

7500 Club Member
View Badges
Joined
Apr 16, 2016
Messages
9,825
Reaction score
17,041
Location
Pleasant Hill, Concord
Rating - 0%
0   0   0
Quick question, since I don't remember this mentioned anywhere:
- How much Computation power does a Reef-Pi setup and a Prometheus-Server with a Grafana interface need ?
- e.g.: Could one run a Prometheus Server on a dedicated Pi and what version would this need ?
- In light of the new Pi Zero 2, which iirc has a dual core, would this be able to run both ?
it is possible but i'll not recommend. The required computation power and auxillary infrastructure (ssd, multiple cores), power of individual cores, ability to run ubuntu /debian all are key factor. All guides will expect a somewhat beefy, 2-4 core (2.4GHz+celeron), 8 GB ram, 128Gb SSD (no emmc ) computer. It is expected to be capable of running a full dekstop environment, with peripherals attached, (often powered through it) .
As of the time of writing this guide such systems cost aroun 120-170$ USD. I am experimenting with a amd a90 based 87$ system, but its not well tested yet for me to vouch for it,
hope this help. i'll copy the details in the guide .
 

cmitcham

New Member
View Badges
Joined
Feb 25, 2019
Messages
13
Reaction score
17
Rating - 0%
0   0   0
excited about reef-pi with full pc power :)

can the guide be dumbed down just a little please? like :


1670337659060.png


the mv to /usr/bin/ needs sudo i think...

the copy starter example command is incomplete?

and where am i creating the systemd unit file?

i'm comfortable at the unix command line, but i'm no expert :)

thanks!
 

ColoredRock

Well-Known Member
View Badges
Joined
Aug 9, 2021
Messages
849
Reaction score
1,021
Location
Murrieta, CA (North of San Diego)
Rating - 0%
0   0   0
excited about reef-pi with full pc power :)

can the guide be dumbed down just a little please? ...
For me there is a fine line of to much information.. and not enough... I would rather sift thru a little to much information and learn the how and why it works. But that is just me....

Thanks @Ranjib for all the work you do on this!
 

ColoredRock

Well-Known Member
View Badges
Joined
Aug 9, 2021
Messages
849
Reaction score
1,021
Location
Murrieta, CA (North of San Diego)
Rating - 0%
0   0   0
May I suggest instead of giving general instructions on how to install Prometheus in this guide, giving links, for example, I have an Ubuntu box up and I used:
https://www.rosehosting.com/blog/how-to-install-prometheus-on-ubuntu-20-04/
Like you did with Grafana.
Im sure its next :winking-face-with-tongue: but some API examples! like you have on your dashboard for the Under voltage and Temperature Control chart.

1670586870498.png
 
Last edited:
OP
OP
Ranjib

Ranjib

7500 Club Member
View Badges
Joined
Apr 16, 2016
Messages
9,825
Reaction score
17,041
Location
Pleasant Hill, Concord
Rating - 0%
0   0   0
May I suggest instead of giving general instructions on how to install Prometheus in this guide, giving links, for example, I have an Ubuntu box up and I used:
https://www.rosehosting.com/blog/how-to-install-prometheus-on-ubuntu-20-04/
Like you did with Grafana.
Im sure its next :winking-face-with-tongue: but some API examples! like you have on your dashboard for the Under voltage and Temperature Control chart.

1670586870498.png
temperature and under voltage are both emitted as metric from reef-pi and should be available in prometheus/grafana to build dashboard . no api required. I have updated the guide to explain how to understand the metric name and build the dashboard. let me know if this does not explain things,. we'll start from there
 
OP
OP
Ranjib

Ranjib

7500 Club Member
View Badges
Joined
Apr 16, 2016
Messages
9,825
Reaction score
17,041
Location
Pleasant Hill, Concord
Rating - 0%
0   0   0
excited about reef-pi with full pc power :)

can the guide be dumbed down just a little please? like :


1670337659060.png


the mv to /usr/bin/ needs sudo i think...

the copy starter example command is incomplete?

and where am i creating the systemd unit file?

i'm comfortable at the unix command line, but i'm no expert :)

thanks!
yes :) .. thank you for proof reading. updating now. much appreciate the kind words and patience,
 

bishoptf

Valuable Member
View Badges
Joined
Jan 1, 2019
Messages
1,344
Reaction score
1,720
Location
Missouri
Rating - 0%
0   0   0
Awesome work I have been running Grafana and Prom for sometime and this is a good guide to let folks see how to do it in a more step by step method. I think if you had a pi4 you could run it and shouldnt be an issue, I haven't tried it but should be able to do it no problem.
 
OP
OP
Ranjib

Ranjib

7500 Club Member
View Badges
Joined
Apr 16, 2016
Messages
9,825
Reaction score
17,041
Location
Pleasant Hill, Concord
Rating - 0%
0   0   0
Awesome work I have been running Grafana and Prom for sometime and this is a good guide to let folks see how to do it in a more step by step method. I think if you had a pi4 you could run it and shouldnt be an issue, I haven't tried it but should be able to do it no problem.
Thank you for the kind words :)
 

A worm with high fashion and practical utility: Have you ever kept feather dusters in your reef aquarium?

  • I currently have feather dusters in my tank.

    Votes: 66 37.5%
  • Not currently, but I have had feather dusters in my tank in the past.

    Votes: 59 33.5%
  • I have not had feather dusters, but I hope to in the future.

    Votes: 25 14.2%
  • I have no plans to have feather dusters in my tank.

    Votes: 26 14.8%
  • Other.

    Votes: 0 0.0%
Back
Top