Community based MQTT standard

[Dennis Cartier]

I use a bunch of home brew development currently. So not a Reef-Pi user. What I have found though is that MQTT is very effective to solicit communication and cooperation between disparate systems.

This got me to wondering of any thought had been given to setting up a MQTT standard schema that could be implemented by the various community efforts (Reef-Pi, Robotank, Levianthan, etc.) to share operational data?

If the schema were comprehensive enough and flexible, then perhaps it could become ubiquitous in the community efforts, and provide a path for the proprietery controller and device manufacturers to implement. Lack of openness and the walled garden of the proprietary controllers is their biggest failing point in my view. So by having the community based controllers take the initiative and define a public schema may be enough to get the manufacturers to open up. It would only take 1 manufacturer to decide to support it, to make it difficult for the resistant manufacturers to not follow. E.g if Hydros decides to add support, then Apex becomes even less attractive.

Even if the manufacturers don't add it, the community efforts will still gain a competitive advantage by supporting it.

However I suspect that many manufacturers would be interested. Take the Alkatronic from Focustronic. It contains an RPI Zero W that doesn't do a whole lot, other than communicate with the App and cloud. Their current offering uses a custom BNC port to send dKH readings to other devices through a pH interface. This takes up hardware resources and could be supported in a much more flexible way by having the Alkatronic publish the latest dKH reading to MQTT for any other device or system that requires knowledge of the current alkalinity. All that is required on Focustronic's part is the addition of 1 package and a couple of scripts. Having this support, would make the Alkatronic a more useful device and help it to maintain it's market position.

Thoughts?

 

[Ranjib]

i would love and support such initiative. I think even if we dont get a consensus, we should get started. And this can be applied to not just MQTT but also API and auth.
Let me know how i can be of any help. Right now we have mqtt based metric emission, but no control yet(should not be impossible, juts some major work). I think we can start the discussion with at least the metric emission side, to see whats the community apetitie and if we converge on a reasonable spec, and take it from there.

This can be a protracted amount of work due to the sheer surface area involved. Hence im emphasizing on ietf style thematic rfcs and take it from there. reefing community itself is significantly smaller, the intersection of reef keeping and home automation is even smaller, thats why its important to strategize such effort with broader iot community specs.

 

[Dennis Cartier]

i would love and support such initiative. I think even if we dont get a consensus, we should get started. And this can be applied to not just MQTT but also API and auth.
Let me know how i can be of any help. Right now we have mqtt based metric emission, but no control yet(should not be impossible, juts some major work). I think we can start the discussion with at least the metric emission side, to see whats the community apetitie and if we converge on a reasonable spec, and take it from there.

This can be a protracted amount of work due to the sheer surface area involved. Hence im emphasizing on ietf style thematic rfcs and take it from there. reefing community itself is significantly smaller, the intersection of reef keeping and home automation is even smaller, thats why its important to strategize such effort with broader iot community specs.

Just to be clear, you are suggesting we use the IETF style for RFCs for the spec, but not necessarily attempt to publish it as an official RFC?

I will try to read up on the latest MQTT versions to see what new features we may want to take into account.

 

[Ranjib]

Just to be clear, you are suggesting we use the IETF style for RFCs for the spec, but not necessarily attempt to publish it as an official RFC?

I will try to read up on the latest MQTT versions to see what new features we may want to take into account.

i am suggesting the we can make similar effort in spirit and publish it as rfc from reef-pi side (not ietf) or some independant place. reef-pi of course will need to implement/support that. But its best if its a multi vendor effort (think of not have the john deer problem[right to repair]). But i doubt very much that commercial vendors have apetite, due to the small nature of this community (no RoI). Hence i feel its realsitic to make the collaboration on DIY community first(and take the rfc spirit in articulating, specify.. all technical rigor to our best extent) and go for it

 

[Dennis Cartier]

Ok, I am interested in undertaking this. Let me get my local broker upgraded to support 5.0 as a first step.

I have been looking around on the IoT side of things for information on best practices for suggested hierarchical structuring, but have not found anything too useful thus far. If you know of any, please point me in the right direction.

I have found the W3C Web of Things (WoT) standard. It might be helpful in this regard, but I am still researching it.

In the meantime, I am going to upgrade my existing code to MQTT 5.0 to become familiar with the new features. One addition that I was pleased to see is official support for the Request/Response pattern as part of the protocol. That will make adding control more straightforward than having to layer it on top (as required in the past).

 

[theatrus]

Just subscribing. I've generally ignored MQTT so I don't have much to add right now, but happy to pretend like I know what I'm talking about

 

[Wolfw28]

MQTT would be awesome especially bi directional work great with esp32 and esp8266

 

[Dennis Cartier]

I am still in the info collection phase, but I want to provide my thoughts to solicit the community's views.

After some initial investigations, I am leaning towards using the Home Assistant MQTT implementation as the basis for our Reef Community standard. My reasoning for this is that it is already an opensource project and I expect that the Devs from the HA community will be open to answering questions about the specifics of their implementation.

The HA MQTT standard is well documented for an opensource project, but not at the level of an RFC, so some effort will need to be undertaken to fill in the blanks where things are fuzzy.

The HA version already includes support for auto discovery, which I see as a must have for our standard. Auto discovery will be the glue that allows easy seamless integration between IoT systems, in this case HA, and reef controllers. It will also be the base for allowing homogeneous and heterogeneous reef controller ecosystems to seamlessly integrate. So using an auto discovery system that is based on, or at the very least, compatible with the HA implementation, will benefit us (in my view).

By being able to integrate to HA and the IoT device world, we will gain the benefit of a broad range of cheap consumer targeted devices. As an example, think of a leak sensor. An opensource controller may have support for a single type of leak sensor, or perhaps a couple from community contributions. In the IoT world, many vendors have a leak sensor offering. By being able to either directly or indirectly use the IoT leak sensors, we can choose the best tool for the job, without worry about if there is support for it already.

I have spoken to Rob ( @robsworld78 ) from Robo-Tank, and he is on board. Though his intention to add support for MQTT is aspirational at this point.

The proprietary manufacturers will be another story. They will be a tough nut to crack. I expect that they will see a community supported MQTT standard as more of a threat than an opportunity. The walled garden approach that they follow gives them an initial benefit of helping to grow a market by forcing their user base to use them for all their automation needs. This is a real benefit for the vendor, but a horrible situation for users. I have seen a lot of products offered by the vendors that I deem to be questionable, but the user base has no choice but to use them if they want that functionality. Kind of like the leak sensor example from above. If vendor A's leak sensor is sub standard, having the option of using a different vendor's leak sensor (if it is smart), or using one from the IoT world, rather than suffering through with whatever your vendor deemed to be "good enough".

Once a vendor has market share, then the crutch of the closed off, locked in approach starts to become an impediment for them leading to mediocre product offerings, and a drag on innovation. However they can gain some tangible benefits by being more open.

Let's take Coravue's Hydros and Focustronic's Alkatronic. They offer integration so your alk readings will appear in the Hydros applications. I am assuming this is mutually agreed upon, and that Hydro's is not just scrapping the Alkatronic API to achieve this. The most obvious point of integration would be at the cloud level. This would be the easiest, least effort method of gaining integration (in my view), and what I suspect they are doing. If we dig a bit deeper, having your alk graph appear in your Hydro's application is great, but the real value lies in being able to base your Hydros logic on the data you are receiving from the Alkatronic, the alk reading. This greatly increases the utility of the integration between these two vendors.

What happens when your internet connection goes down and you lose connectivity to the cloud? Your alk reading that your code is using is no longer current. Is the reading zeroed out, or is there a method of informing dependent code that the reading is stale? At the very least, you lose the benefit of the integration, while your cloud connection is interrupted.

Let's consider now if the Hydro's and Alkatronic both had support for our proposed MQTT standard. The Alkatronic device, through auto discovery, would appear amongst the devices and sensors listed in the Hydro's configuration. Adding alk as a parameter to include in your code would be as simple as toggling the Alkatronic on in the Hydros config. If your internet connection goes down, the Hydros and Alkatronic integration would be unaffected as it is happening at the local network level. We remove one layer of failure points by not depending on cloud accessibility in this case.

Some pain points for adoption by the vendors will revolve around expanding the customer support footprint, and such rudimentary points, like where is the broker and who is running the broker? I am hoping that we can address some of these as part of our standard.

 

[theatrus]

After some initial investigations, I am leaning towards using the Home Assistant MQTT implementation as the basis for our Reef Community standard. My reasoning for this is that it is already an opensource project and I expect that the Devs from the HA community will be open to answering questions about the specifics of their implementation.

The HA MQTT standard is well documented for an opensource project, but not at the level of an RFC, so some effort will need to be undertaken to fill in the blanks where things are fuzzy.

The HA version already includes support for auto discovery, which I see as a must have for our standard. Auto discovery will be the glue that allows easy seamless integration between IoT systems, in this case HA, and reef controllers. It will also be the base for allowing homogeneous and heterogeneous reef controller ecosystems to seamlessly integrate. So using an auto discovery system that is based on, or at the very least, compatible with the HA implementation, will benefit us (in my view).

By being able to integrate to HA and the IoT device world, we will gain the benefit of a broad range of cheap consumer targeted devices. As an example, think of a leak sensor. An opensource controller may have support for a single type of leak sensor, or perhaps a couple from community contributions. In the IoT world, many vendors have a leak sensor offering. By being able to either directly or indirectly use the IoT leak sensors, we can choose the best tool for the job, without worry about if there is support for it already.

Agree with this. You also gain the pretty decent visualization and dashboard abilities of HA.

What happens when your internet connection goes down and you lose connectivity to the cloud? Your alk reading that your code is using is no longer current. Is the reading zeroed out, or is there a method of informing dependent code that the reading is stale? At the very least, you lose the benefit of the integration, while your cloud connection is interrupted.

Yes, this a risk, and a lot of control systems aren't built to understand "sensor is now missing or not reporting" as one of the conditions. This is something to make sure recency of data is available when dealing with the overarching control stuff.

Some pain points for adoption by the vendors will revolve around expanding the customer support footprint, and such rudimentary points, like where is the broker and who is running the broker? I am hoping that we can address some of these as part of our standard.

Ironically MQTT-over-internet is a not great system. If you're dealing with a future vendor cloud API, assume you might need to federate messages back to a local control queue. I don't see most vendor boxes, even Apex, running the MQTT broker locally. Maybe they can offer multiple publish support to write to a local broker as well. Something like the Hydros is basically an ESP32 module, and maybe even already uses MQTT on the backend, but I have no idea (could be HTTP long polling, etc).

I wouldn't too much about vendors right now. Build something compelling and integrated, and then you'll get interest in maybe interoperating with that. In the mean time, plan on needing to community source adapters (e.g., an APEX XML local scrape over HTTP to MQTT adapter, etc) to glue things together.

 

[Ranjib]

Those concerns are ok. lets not get detered by those. we do what we feel is the right thing for this hobby and from an opensource /right to repair perspective. Lets start with HA MQTT standard and do a gap analysis with reef-pi API to understand what we can cover, what we can not. And from the bits (of reef-pi api) that can be covered with MQTT, how much effort it will take for me to make this happen?

 

[Inamo]

Sad this never went anywhere. That is my issue now, so many systems I wish I could integrate into one place.

 

[ffiren]

I use mqtt, home assistant, node-red and a load of ESP32s for everything tank related. In fairness, I bought an Apex just for the HA integration.

I can automate everything except ReafBeat and I know that’s mqtt. If I could sniff and relay that locally I’d be set. Irritates the hell out of me.

 

[theatrus]

I use mqtt, home assistant, node-red and a load of ESP32s for everything tank related. In fairness, I bought an Apex just for the HA integration.

I can automate everything except ReafBeat and I know that’s mqtt. If I could sniff and relay that locally I’d be set. Irritates the hell out of me.

I've been using HA as the basic collection hub for lots of things - so much active development, even if it's a rickety pile of Python. Are you using ESPHome or some other stock firmware for bridging to HA/MQTT?

I haven't tried to look at ReefBeat communication yet.

 

[apista]

What happens when your internet connection goes down and you lose connectivity to the cloud? Your alk reading that your code is using is no longer current. Is the reading zeroed out, or is there a method of informing dependent code that the reading is stale? At the very least, you lose the benefit of the integration, while your cloud connection is interrupted.

Who would design a control system that requires the cloud to operate, there is a big difference between control and indication.

 

[theatrus]

Who would design a control system that requires the cloud to operate, there is a big difference between control and indication.

Yes, you shouldn't. I fear that this is more common than you think though.

At some point you do have process control - outside of the small control loops or heater controls or the like, how do you orchestrate bigger actions? This is what a lot of people end up using HA or similar for. Hopefully not in the cloud.

 

[apista]

Not in any industry i have worked in.

 

[theatrus]

Not in any industry i have worked in.

Industrial control? Yeah, not going to happen.

Home? Like half or more of the things

 

[yury88]

I use mqtt, home assistant, node-red and a load of ESP32s for everything tank related. In fairness, I bought an Apex just for the HA integration.

I can automate everything except ReafBeat and I know that’s mqtt. If I could sniff and relay that locally I’d be set. Irritates the hell out of me.

Hi, sorry, I'm not from USA, so not familiar with Apex system
Can you tell me how MQTT looks like in Apex?

I'm going to add MQTT support to my smart doser project:

ReefRhythm - Advance Stepper Motor Controller with OpenSource

Would it be possible to use something like a btt octopus controller board instead of the mks 42c boards

www.reef2reef.com

I'm not sure if Apex is able to send commands like "dose 60 ml" to the controller, but it will be nice to get access to all sensors in Apex.

For example, the doser can get the PH sensor value from the apex to adjust the dosing of limewater.
Or adjust the flow through the calcium reactor depending on its PH sensor.

///
I Just found that apex has REST API for read sensors values:
'http://<Apex host>/cgi-bin/status.json'
It can be an option. The doser can fetch values and turn on or off some sensors to reach some limit.

 

[theatrus]

Hi, sorry, I'm not from USA, so not familiar with Apex system
Can you tell me how MQTT looks like in Apex?

I'm going to add MQTT support to my smart doser project:

ReefRhythm - Advance Stepper Motor Controller with OpenSource

Would it be possible to use something like a btt octopus controller board instead of the mks 42c boards

www.reef2reef.com

I'm not sure if Apex is able to send commands like "dose 60 ml" to the controller, but it will be nice to get access to all sensors in Apex.

For example, the doser can get the PH sensor value from the apex to adjust the dosing of limewater.
Or adjust the flow through the calcium reactor depending on its PH sensor.

///
I Just found that apex has REST API for read sensors values:
'http://<Apex host>/cgi-bin/status.json'
It can be an option. The doser can fetch values and turn on or off some sensors to reach some limit.

Yup, this local APEX API has been a standard for awhile. The JSON version is a bit newer as the Classic units (IIRC) export XML only (because it's like 2008 era).

Adding MQTT is great - it will let a lot of things integrate including with HomeAssistant out of the box. I do control a Ca reactor through a Trident measurement this way.