theatrus' "Perfect Temperature Controller"

[theatrus]

I've been keeping aquariums of all flavors for a very long time, and have seen and first hand experienced what happens when our lovable basic heaters (and chillers) run amok and fail to do their jobs correctly. Even worse, we often decide to go whole hog into the aquarium controller systems, which promise us web graphs and e-mails, while also being in charge of critical functions such as temperature and top-off, mixing software and control domains that should never be mixed (e.g., https://www.reef2reef.com/threads/fixing-a-broken-apex-classic-main-unit.366496/ - I've had units just fail, and when your entire tank runs on it, you're SOL). Even worse, a lot of hobby gear is designed down to a price (especially with imports, or to improve margins) because, well, a lot of us end up tossing the equipment in the trash after a year either because we found a new hobby, or hated the equipment we did have. Some equipment is of course a "wear item", but the number of stuck mechanical thermostats, or miscalibrated heaters, or heaters with bad hystersis so your aquarium swings by several degrees, or broken controllers, or heaters which explode in your tank never seems to go down.

So, how do we improve the situation for establishing the critical temperature control functions? By using a robust, intrinsically safe, and well designed control scheme for heating elements. I'm not going to focus on the heating elements here, but assume there are > 1 element of a good quality, and that those elements are dumb - if you apply energy, they heat, and if you don't, they won't. I'm also assuming these elements may fail in such a way they stop providing heat.

What makes a good temperature controller then? Well, here is the deign principle I'm working on:

- Low tech. No Linux, no fancy communications, no wireless, no Wifi, no clouds, no multi-color displays or touch. A single stand-alone controller which does one job and one job only. Use parts that are robust and simple or well understood. Rate parts so they are well over designed and not driven purely by cost.
- Accurate, calibrated temperature sensing of the water. Preferably at two points, or with two independent circuits and probes to compensate and allow for a probe to fail.
- Smart heater control using an active PID loop, instead of a single hammer on/off approach at on/off setpoints. Smoothly and accurately holding a temperature in the tank instead of having large stair steps or sawtooth temperatures.
- An indicator system to warn of failure without having to check on it ("the buzzer"). Don't rely on e-mail which may break at any moment. An audible alarm is the way to go for this critical function. If you do want an e-mail or a text, a full aquarium controller in a monitoring (not control) function would be ideal here.
- Heater control failsafes: if the control element (say, a solid state relay) were to fail in an on position, there is a backup power control mechanism which can remove power from the heater circuits (say, a relay)
- PID controller fail safes: by using a linear temperature sensor element, the high and low thresholds can be built into the hardware in such a way that no code or software/firmware is needed to control the extreme edges (say, above 83 or below 75),; the system by design is extremely unlikely to exceed critical temperatures which can rapidly cook a tank.

I'd love to share the ideas here, as well as release an open hardware schematic, board, and firmware to go along with it for those interested. Stay tuned for more details as the project develops

 

[gabrieltackitt]

I will definitely be watching this one! I was seriously starting to think about doing something similar on my own, so glad to see someone else had the same idea!

 

[Galvano]

You're absolutely right by addressing a key problem, that extends far beyond stabilizing water temperature. Controlling terminal equipment like heaters, chillers, lamps and pumps should always include the interpretation of feedback coming from these devices.

And when it's about securing mission critical tasks, whether in space or in our tanks, two principles rule: KISS and redundancy.

But when taking that seriously you need at least two, better three completely independent units, which have to control and, if necessary, put each other back on track to cure the system. A buzzer to call deus ex machina can only be seen as a last resort. In case your budget doesn't allow to invest in that kind of all-embracing redundancy and your sensors indicate unrecoverable discrepancies, e.g. with attached devices no longer reacting, even a complete auto-reboot may be helpful to regain a well-defined state.

It's wise to start with temperature, as it's only about a single easily quantifiable parameter. For me, thinking about that part of my AquariOS temperature/light/pump controller, KISS means separate sensors guarding the upper and lower limit of the allowed temperature range and activating simple switches, either mechanical or SSRs, when a malfunction of the actual controller unit gets detected. But which reading to trust? Decision by mayority? And, different from a thermometer or an LDR, it's getting much more complex with pumps, where flow rate has to be estimated.

So I'm eagerly interested in the outcome of this thread.

 

[mnl]

I agree with you completely. What worries me about most controllers is that they are heavy on user interface at the expense of testable control loops.

I have been think of something like this as well, and you seem to be a bit father ahead. Are you thinking of something like the old TRIAC based PID controllers that would modulate each cycle of the AC?

Here are few of my other thoughts on such a system.

1) Three temperature sensors, I would probably just use thermistors and read them with a muxed ADC. That way I could also add a reference voltage to the mix to read each cycle to verify proper operation of the ADC. I would then vote the three thermistors, and if they are within range average them for the control. If one failed then that would throw an error and the remaining two would be averaged.

2) A very simple polled telemetry interface. The host would request status, temps, etc which would be echoed back. The unit would not speak unless spoken to.

3) Any configuration would be handled then same way. Probably the only commands would be set temp and enable/disable. The latter could also just be handled with a switch on the front panel.

4) Fail safe would be a fourth thermistor into a comparator driving a relay large enough to disconnect with no concerns. Since the heater is a resistive load that shouldn’t present much of a derating issue. The comparator would be set with a voltage divider or other reliable circuit.

5) Unit would store default values and cold start with no interaction from host.

6) I might consider bonding a thermistor to the heater in case it was accidentally removed from the system to prevent thermal runaway. That would also feed a comparator tot he disconnect relay.

The question is not am I paranoid, but I am paranoid enough.

 

[theatrus]

The point of 2, or 3+ temperature senors is something that I'm very much looking at, in accordance with Segal's Law https://en.wikipedia.org/wiki/Segal's_law

> "A man with a watch knows what time it is. A man with two watches is never sure."[1]

Are you thinking of something like the old TRIAC based PID controllers that would modulate each cycle of the AC?

Yes, using an SSR integrated circuit or monolithic block.

4) Fail safe would be a fourth thermistor into a comparator driving a relay large enough to disconnect with no concerns. Since the heater is a resistive load that shouldn’t present much of a derating issue. The comparator would be set with a voltage divider or other reliable circuit.

Absolutely, and planning on doing just this, but the debate on the 4th sensor also failing in a "no heat" scenario comes to mind. Granted, colder is better than hotter, but its still a point to take into consideration.

But which reading to trust? Decision by mayority?

This is the question at hand - how do you fail? We can probably agree that there should be a fail safe to not latch a heater on, but is the failure mode to ask for intervention and stop heat cycles?

 

[Bigtrout]

I've been keeping aquariums of all flavors for a very long time, and have seen and first hand experienced what happens when our lovable basic heaters (and chillers) run amok and fail to do their jobs correctly. Even worse, we often decide to go whole hog into the aquarium controller systems, which promise us web graphs and e-mails, while also being in charge of critical functions such as temperature and top-off, mixing software and control domains that should never be mixed (e.g., https://www.reef2reef.com/threads/fixing-a-broken-apex-classic-main-unit.366496/ - I've had units just fail, and when your entire tank runs on it, you're SOL). Even worse, a lot of hobby gear is designed down to a price (especially with imports, or to improve margins) because, well, a lot of us end up tossing the equipment in the trash after a year either because we found a new hobby, or hated the equipment we did have. Some equipment is of course a "wear item", but the number of stuck mechanical thermostats, or miscalibrated heaters, or heaters with bad hystersis so your aquarium swings by several degrees, or broken controllers, or heaters which explode in your tank never seems to go down.

So, how do we improve the situation for establishing the critical temperature control functions? By using a robust, intrinsically safe, and well designed control scheme for heating elements. I'm not going to focus on the heating elements here, but assume there are > 1 element of a good quality, and that those elements are dumb - if you apply energy, they heat, and if you don't, they won't. I'm also assuming these elements may fail in such a way they stop providing heat.

What makes a good temperature controller then? Well, here is the deign principle I'm working on:

- Low tech. No Linux, no fancy communications, no wireless, no Wifi, no clouds, no multi-color displays or touch. A single stand-alone controller which does one job and one job only. Use parts that are robust and simple or well understood. Rate parts so they are well over designed and not driven purely by cost.
- Accurate, calibrated temperature sensing of the water. Preferably at two points, or with two independent circuits and probes to compensate and allow for a probe to fail.
- Smart heater control using an active PID loop, instead of a single hammer on/off approach at on/off setpoints. Smoothly and accurately holding a temperature in the tank instead of having large stair steps or sawtooth temperatures.
- An indicator system to warn of failure without having to check on it ("the buzzer"). Don't rely on e-mail which may break at any moment. An audible alarm is the way to go for this critical function. If you do want an e-mail or a text, a full aquarium controller in a monitoring (not control) function would be ideal here.
- Heater control failsafes: if the control element (say, a solid state relay) were to fail in an on position, there is a backup power control mechanism which can remove power from the heater circuits (say, a relay)
- PID controller fail safes: by using a linear temperature sensor element, the high and low thresholds can be built into the hardware in such a way that no code or software/firmware is needed to control the extreme edges (say, above 83 or below 75),; the system by design is extremely unlikely to exceed critical temperatures which can rapidly cook a tank.

I'd love to share the ideas here, as well as release an open hardware schematic, board, and firmware to go along with it for those interested. Stay tuned for more details as the project develops

The key item in the perfect heater controller is an adequately rated relay with longevity, whether you use solid state or mechanical relays. Unfortunately good relays can get expensive, so maybe the design should keep this in mind. I like the idea of a solid state relay running the heaters with a mechanical failsafe. Maybe design the board with a socket for real industrial rated relays instead of the songle type hobby relays. @theatrus if you want to share ideas here feel free to PM me.

 

[theatrus]

Let's talk temperature sensors, at least let's talk about the guts of a temperature sensor and less about the housing, cables, etc.

There are three main varieties you'll find in your typical "probe" style sensor, and a fourth I'll mention because its common for integrated circuits.

I'm considering "relevant reefing temperatures" to be about 75F to 83F - sorry, I haven't switched to metric on temperature just yet, even though I reference them below. Its a hard habit.

Thermocouple

Thermocouples are a sensor made of two dissimilar metals welded at one point. The dissimilar junction produces a small voltage due to the thermoelectric effect. However, you won't see thermocouples used for temperatures for reefing jobs, as the sensors are hard to make precise under a few degrees F. Thermocouples excel at high temperature measurements (1000C or higher) - and since they are actually a voltage source can be used in stack arrangements called a thermopile, commonly used as a safety valve sensor for natural gas appliances with pilot lights. We won't discuss them further, as they're not an ideal sensor type.

Diode sensors and digital sensors

Most digital sensors, such as the DS18B20 type or even the sensors on your CPU are of the diode band-gap kind. These are effectively voltage references which means the voltage drop (not resistance, its a diode) which has been selected for temperature sensitivity. They're common on digital electronics for the sole reason that they are effectively free to fabricate on a silicon die and require no special extra processes to do so. With careful calibration, they can be accurate, but are designed around digital circuits.

I also suspect a large number of Chinese probes with the DS18B20 sensors are actually using a compatible but non-Maxim part, and in my experience do not meet the stated accuracy.

As its a digital sensor and requires logic and code to read, I'm going to rule out using this sensor family.

RTD / Platinum RTD

This is one of the two main classes of resistive sensors, in that the resistance of the sensor will change in response to the temperature change. Unlike the other class of sensor (the thermistor), an RTD is mostly linear - a 1 ohm resistance change is nearly the same across all of the same sensor type (Platinum) and nearly across the entire temperature range. RTDs also have a wide operating range from -50C to 500C.

RTDs are usually considered the more expensive choice, and if you get a Class A sensor, the reference accuracy choice due to its stability - the element is an encapsulated platinum wire and there isn't much that can go wrong with it, except for mechanical damage (or melting).

The Achilles heel of a platinum RTD sensor is the usual low resistance - at 25C, its a mere 100 ohms for a typical sensor. This means system inaccuracies such as the wire and the connections can matter for accuracy, and long wire lengths are usually ruled out. There are compensation schemes available for this, including three and four wire RTD cabling, where one or both legs of the sensor has an extra wire run in parallel, and the measuring circuit also measures the resistance of this wire to compensate for the cable resistance. This provides an added benefit of being able to detect an open or shorted RTD (or more likely to) by checking for conductivity on the 3rd/4th wire pair.

Thermistor / NTC Thermistor

Thermistors are the other kind of resistive temperature sensor, made up of a proprietary mix of a various metal oxides, ceramics and polymers in a powder form, baked or sintered together to produce a material with large resistance swings over temperature. Thermistors have the lowest temperature range (up to about 130C), but for our application that is more than adequate. Thermistors also come in two flavors, PTC (Positive Temperature Coefficient) and NTC (Negative Temperature Coefficient). Most temperature sensors are made of the NTC material: as temperature goes up, resistance goes down.

Thermistors have a few problems of course. Every thermistor material is different, so they are actually hard to interchange as the curves the temperature follows are different from one type to the next. There are some common values, but its hard to know which you have without further information. The response is also not linear, and can change from over 15% per-degree resistance change to about 5%, depending on how far away from the center of the curve you are. One saving grace thermistors have for our application is that their curves are usually much flatter around 20-25C than near freezing, so circuitry does not need as extensive of compensation in the small span we consider the sweet spot.

What to choose?

So, thats a lot of words. Which to choose? Due to the differences in the sensors (a thermistor is nominally 10k ohm, an RTD 100 ohm), its often not feasible to simply support both.

My initial circuit design I'm going to focus on using platinum RTDs. Why?
- Linear response (though thermistors would be well suited since we only require operation in a very narrow band of temperatures)
- Grade A RTDs are considered a reference standard once calibrated, and are very stable over time.
- Common availability of three/four wire RTDs opens an interesting possibility of determining sensor health and connectivity.

PRTDs are going to be more expensive, and I have no ruled out a thermistor design, or simply making a board with both analog front ends if the cost is not prohibitive.

Using PRTDs does require some special measurement setups, which I will get into next. If you want to do some "light reading" this guide is very well done over the nuances of the subject:

http://www.ti.com/lit/an/sbaa275/sbaa275.pdf

 

[theatrus]

The key item in the perfect heater controller is an adequately rated relay with longevity, whether you use solid state or mechanical relays. Unfortunately good relays can get expensive, so maybe the design should keep this in mind. I like the idea of a solid state relay running the heaters with a mechanical failsafe. Maybe design the board with a socket for real industrial rated relays instead of the songle type hobby relays. @theatrus if you want to share ideas here feel free to PM me.

While we're jumping a little ahead, the goal is to use multiple SSRs as the active switching element, and a failsafe of a NO large relay from a reputable brand (Omron, etc) in case the SSRs fail in the on position. Happy to take recommendations of relay sizes

 

[Bigtrout]

While we're jumping a little ahead, the goal is to use multiple SSRs as the active switching element, and a failsafe of a NO large relay from a reputable brand (Omron, etc) in case the SSRs fail in the on position. Happy to take recommendations of relay sizes

The idea here is to decide what size heater is the maximum to be supported.
Relays have a mechanical and electrical life expectancy listed on the datasheet in a graph. Good relays have lives in the tens of millions of operations(omrons etc) , where the cheaper songle types have a lives in the 100,000 range. The more current they switch the shorter the life of the relay. A general rule for designing control boards is to use a relay of at least double the size of the current being controlled. So for a max heater watts of 600watts at 120v, you would be switching 5 amps of current so a well branded mini ice cube rated 10 amps would be a good start.

 

[gabrieltackitt]

Have you seen the Atlas Scientific Temperature probes? They are platinum rated for full submersion, and their resistance at 0ºC is 1000 Ω. And they are relatively inexpensive, $20 a piece.

PT-1000 Temperature Probe | Atlas Scientific

The Atlas Scientific PT-1000 probe is a Class-A high purity platinum RTD temperature probe with a thick silicone rubber cable.

www.atlas-scientific.com

 

[Bigtrout]

@theatrus
Since you are going the SSR route are you controlling with a simple adjustable hysteresis. Or are you thinking about a PID loop?

 

[theatrus]

It

Have you seen the Atlas Scientific Temperature probes? They are platinum rated for full submersion, and their resistance at 0ºC is 1000 Ω. And they are relatively inexpensive, $20 a piece.

PT-1000 Temperature Probe | Atlas Scientific

The Atlas Scientific PT-1000 probe is a Class-A high purity platinum RTD temperature probe with a thick silicone rubber cable.

www.atlas-scientific.com

Good reference. I have a handful of Pt100 Omega probes on hand which I’m starting with.

 

[theatrus]

@theatrus
Since you are going the SSR route are you controlling with a simple adjustable hysteresis. Or are you thinking about a PID loop?

Targeting PID.

 

[theatrus]

Have you seen the Atlas Scientific Temperature probes? They are platinum rated for full submersion, and their resistance at 0ºC is 1000 Ω. And they are relatively inexpensive, $20 a piece.

PT-1000 Temperature Probe | Atlas Scientific

The Atlas Scientific PT-1000 probe is a Class-A high purity platinum RTD temperature probe with a thick silicone rubber cable.

www.atlas-scientific.com

Looking at it a second time:

only two wire
Also looks like the standard import market jacket and double crimp. I’ll still grab one but compare it to the AliExpress look alikes.

The three wire Omega probes are a solid 316 body and IP67 connector at the end, but a Class A there runs $45.

 

[Soulreefer]

whether this type is the same,
cable length of 4 meters
only about 4 dollars


Jerman JUMO Jiu Mao PT100

 

[mnl]

The point of 2, or 3+ temperature senors is something that I'm very much looking at, in accordance with Segal's Law https://en.wikipedia.org/wiki/Segal's_law

> "A man with a watch knows what time it is. A man with two watches is never sure."[1]

My assumption was for single fault tolerance. Reading three and voting them is sufficient. If one fails the two that agree must be the right answer. The use of a reference on the same mux validates the mux and ADC every measurement.

Maybe design the board with a socket for real industrial rated relays instead of the songle type hobby relays.

Why use anything else? Then it is easy to get a high cycle count relay you can derate 5x - 10x.

Pt vs thermistor: Both are reasonable choices. I’ve used thermistors more recently than Pt thermometers. That was a very nice HP unit, also had a quartz thermometer, counted cycles. I’ve always run the control loop in raw ADC counts, allows integer math which suites most microcontrollers better and saves cycle counts. The command and telemetry systems can do the conversions.

I will likely roll my own probes by potting bead sensors in a small tube. That way I can source nice sensors. I don’t think I have seen a decent quality thermistor fail in circuit, and that is without resorting to Class B or Class S.

The easiest and most reliable solution is to have a warm spare, controller, heater, sensors, the whole none yards. Each unit can take an inhibit from another box. As long as box A is working Inhibit is high and Box B is quiet. If Box A has a problem it shuts down dropping Inhibit and box B takes over.

 

[theatrus]

Pt vs thermistor: Both are reasonable choices. I’ve used thermistors more recently than Pt thermometers. That was a very nice HP unit, also had a quartz thermometer, counted cycles. I’ve always run the control loop in raw ADC counts, allows integer math which suites most microcontrollers better and saves cycle counts. The command and telemetry systems can do the conversions.

I will likely roll my own probes by potting bead sensors in a small tube. That way I can source nice sensors. I don’t think I have seen a decent quality thermistor fail in circuit, and that is without resorting to Class B or Class S.

The easiest and most reliable solution is to have a warm spare, controller, heater, sensors, the whole none yards. Each unit can take an inhibit from another box. As long as box A is working Inhibit is high and Box B is quiet. If Box A has a problem it shuts down dropping Inhibit and box B takes over.

Agree with you here. I used to (we're talking like up until 2012) deal with thousands of thermistors in airflow for measuring computer room datacenter temperatures as part of a wireless sensor network. Most of the logic on the nodes (on a tiny MSP430) was all done in ADC counts for exception reporting.

In the past I've had great luck with US Sensor beads and encapsulated probes - very accurate right out of the gate requiring no post calibration step or pairing of sensors for the application. I think you can get the beads from Digikey.

As for the active/passive, this is a great design idea. I had considered it, but I think I want to enable it from the get go. The question I have is from testing: is it better to run active/active, or is the intent to reduce cycles on the backup to not have wear related failure?

My assumption was for single fault tolerance. Reading three and voting them is sufficient. If one fails the two that agree must be the right answer. The use of a reference on the same mux validates the mux and ADC every measurement.

With your failover model, is would be ideal. If any of these sanity checks fail, declare the unit degraded and move to the backup.

Good line of thinking!

 

[mnl]

I used the high rel YSIs, very nice, but pricey for this application.

Unless you need immediate failover you are better off with as cold a spare as possible. Things generally fail on power cycle so the fewer cycles the better.There is enough heat capacity that the failover time is really not critical. I would include a manual switch over button so that the sides could be swapped every month or so to check that the other side was still operational.

The problem with redundancy is usually not the nominal failure case, but the special cases that may not be failures. For example, what if there is a power failure and the controllers go down. Coming back up is interesting, ideally there should be something to hold off switchover until the system boots and can figure out if it is OK. The problem is that if there is a hardware problem that keeps it rebooting it will always be held off and switchover will never happen. The other problem is that if the water has had a chance to cool a few degrees the system will recognize it has had a problem and switchover even though it doesn’t need to. You can have the same problem if the ambient temperature is too warm, but this could be fixed by declaring that too warm AND no heater power is not a fault condition.

The first case above would be most easily addressed by saying that the system is single fault tolerant and the fault was the power failure, accept the switchover , and let the operator switch it back at a later time.

What ever the design is the key is to design for testability And make it simple enough that you can test every path.

I also had the following idea for a slightly different system that should have a lower parts count and probably be simpler to build. It would have two independent temperature controls, each of which could be completely single string, a single sensor, control loop, and heater and the third element is a monitor processor with it’s own sensors. The system starts off running on A side and if the monitor detects the temperature out of range it switches to the B side. Again it is single fault tolerant. If the A side fails the monitor will switch to B side and things will be fine. If the monitor fails either it will switch to the B side, A side is still fine but that is not a problem because B side also works. Or the monitor will not switch in which case the system will be running happily on the still good A side. The monitor can also be completely single string as any failure in it will still leave the system on one of the two working sides.

It would look like A side — Monitor — B side. The two sides only control the heaters. The monitor can only switch between sides. This should be very testable.

 

[Daniel@R2R]

Very cool thread! Following!

 

[theatrus]

The problem with redundancy is usually not the nominal failure case, but the special cases that may not be failures. For example, what if there is a power failure and the controllers go down. Coming back up is interesting, ideally there should be something to hold off switchover until the system boots and can figure out if it is OK. The problem is that if there is a hardware problem that keeps it rebooting it will always be held off and switchover will never happen. The other problem is that if the water has had a chance to cool a few degrees the system will recognize it has had a problem and switchover even though it doesn’t need to. You can have the same problem if the ambient temperature is too warm, but this could be fixed by declaring that too warm AND no heater power is not a fault condition.

You hit on a number of interesting edge conditions here. This is something in the state machine we may need to determine what happens after a power on or reset event, and if its safe to make different assumptions in this event. It would be interesting (and infuriating) to have a heater refuse to heat because it thought the temperature being somewhat too low was a fault

I think its worthwhile to sketech out a matrix of failure conditions and expected actions, since we are hitting on a number of fun ones. I'll whip up a quick GitHub repo for dumping documentation. https://github.com/theatrus/pertempco

What ever the design is the key is to design for testability And make it simple enough that you can test every path.

Absolutely.

I also had the following idea for a slightly different system that should have a lower parts count and probably be simpler to build. It would have two independent temperature controls, each of which could be completely single string, a single sensor, control loop, and heater and the third element is a monitor processor with it’s own sensors. The system starts off running on A side and if the monitor detects the temperature out of range it switches to the B side. Again it is single fault tolerant. If the A side fails the monitor will switch to B side and things will be fine. If the monitor fails either it will switch to the B side, A side is still fine but that is not a problem because B side also works. Or the monitor will not switch in which case the system will be running happily on the still good A side. The monitor can also be completely single string as any failure in it will still leave the system on one of the two working sides.

It would look like A side — Monitor — B side. The two sides only control the heaters. The monitor can only switch between sides. This should be very testable.

This is pretty intriguing, as long as the monitor doesn't end up sending an inhibit signal to both heater control loops simultaneously, which requires coordination between the heater elements as well. With simple logic, this could be avoided, but if we are using a simple logic signaling all it takes is a stray salt crystal to pull the inhibit or run line up or down and ruin our day.

Initially I was planning on using a utility CAN interface to provide communications; if we want something with more intelligence in the path this is an option to use as its ready to use multi-master bus.

I think the approach here would be develop some of the functional blocks (temp sensing+logic, heater control), and then combine as needed at a macro-module level.