Disaster Tank Overflow: Apex + ATK

[ariellemermaid]

Out of town for about a week and everything on Apex seemed fine. Then today I started getting various errors (the main one was ATO res low but other errors may have been masked). When I got to the tank (at our old house) it had overflowed with water on the floor and in the electronics compartment, and the ATO Brute was basically completely empty with tank PPT at 29 down from 33-35 (and was filled not long before leaving).

Where I'm at currently is almost every module is offline, even my Trident which is directly connected to the Apex controller (this is after shutting off all breakers to everything to reboot it all). The EBB832's show as offline as well but for some reason I can still control them just not see power usage, etc. The EBB's, controller, 1-link's, etc. did not seem to get wet (on a controller board). My light bricks and bottom of the Trident did, but the lights are still working. If you have any suggestions I'd love to hear them, but that's not even my biggest concern at the moment; we need to break it down and move it anyway.

Main question is: how did this happen?! I've been very careful with my programming. Here is my ATK output (manually typed, ignore caps and syntax):

Fallback off
Set off
If ATK_Lo open then on #sump low sensor
If ATK_High closed then off #sump high sensor like 1-2" above low
If Output Maintenance = on then off #manual maintenance mode
If Leak1 closed then off
When on > 005:00 then off #hard off- if runs >5min it can't turn back on without manual intervention and raises "ATK error" alarm
Defer 005:00 then on #wait at least 5 min before any "on state"
Defer 000:30 then off #wait 30 second before off
Min Time 060:00 Then off #don't run more than once ever hour

In addition to that, the ATO line goes into a float valve. As best as I can tell, it seems like my 1-link module(s) lost contact in an "on" state because when I got to the tank I noticed the ATO pump was on even though it had been in a hard "off" state for at least 24-48 hours. And, clearly my code is Fallback off, Set off, only turn on for a max of 5 minutes an hour under certain circumstances. The 5 minute hard-off has triggered many times over the years.

My understanding was that Fallback off means if contact is lost, then it's off by default. Does my experience suggest that this logic can only occur when the controller is connected? Or is this just a 1-link failure (of note- the 1-link controlling the pump is in a completely dry fish room). Even if the tank-side ATO sensor 1-link failed in an "open" state then the 5 minute hard-off limit should still have kicked in.

I really thought I had engineered this situation out with programming and sensors, but apparently not. Any idea what went wrong?

 

[ariellemermaid]

Update: apparently the system just needed overnight to reconnect all the modules. Still begs the question of how the out of control ATO happened in the first place.

 

[n2585722]

Update: apparently the system just needed overnight to reconnect all the modules. Still begs the question of how the out of control ATO happened in the first place.

Not sure on Apex but the Hydros has a maximum on time you can set in the output and set the run past max on to to off and if the ATO water level sensor does not go wet before the maximum on time has been reached it will turn the output off and send an alert that the maximum on time has been exceeded. It also has a maximum off time you can set. If it has not done a top off in that amount of time it will send an alert but it will not turn on the output. There may be a way of setting these on the Apex.

 

[ariellemermaid]

I can confirm that when the pump is plugged into a 1-link on the ACC outlet with no controller attached that it will just run continuously. I’m actually using that to my advantage at the moment filling a QT tank. But in a polished system…yikes!