Apex issue that took out 1/2 my tank

areefer01

2500 Club Member
View Badges
Joined
Jun 28, 2021
Messages
3,475
Reaction score
3,612
Location
Ca
Rating - 0%
0   0   0
Apex seems to be working fine currently. But my reef maintenance guy is guessing that Apex could have pushed a software patch as they are normally sent at night. Patch could have temporarily caused my heaters to not turn off once the water temperature reached 79.0 as per the setting I have it on.


Neptune does not "auto" push firmware updates. It has to be done manually by the hobbyist. If that is what your maintenance person believes then I would recommend finding another one.
 

KTTX

Active Member
View Badges
Joined
Nov 7, 2015
Messages
230
Reaction score
85
Rating - 0%
0   0   0
Sorry to hear that. I would make sure to set the heater temp cut off too so if apex fails, heaters will not go over 81 or so.
 

KStatefan

2500 Club Member
View Badges
Joined
Jun 24, 2015
Messages
4,508
Reaction score
4,325
Location
MHK
Rating - 0%
0   0   0
Will absolutely do that. Kicking myself for not thinking of that myself.

Still do not think the root cause has been determined though. If it was due to the filter sock the heater should have shut off when she changed the sock and not required you to turn the socket off.
 

Lionfish hunter

Valuable Member
View Badges
Joined
Jan 18, 2021
Messages
1,072
Reaction score
680
Location
Houston
Rating - 0%
0   0   0
Evening fellow Reefers,


I wanted to ask a question about my Apex controller and a recent event where my temperature dropped heater stayed on till it reached 90 degrees.

I have a Red Sea G2. 65 gallon display and 20 gallon sump. I changed out my glass heater 2 months ago as it was approaching 12 months old. I replaced it with a BRS Titanium 350w heater. Everything was working great until last Thursday.

I have the Apex programmed to keep the temperature between 77.5 and 79.0 degrees. Heater is plugged into the Apex Power bar and controller turns the heater on and off as needed. I haven’t had any issues with the Apex system for over 10 months.

I travel a lot for work and last week I was in Europe. I came out of a meeting and saw I had a major temperature swing in the last 12 hours. The swing was 73.5 to 90.2 degrees. I turned the heater off using the Apex app on my phone. One challenge was it was 2 Am in the States so didn’t call my wife that early.

Over the next few hours I watched the temperature slowly drop down and at 6 am I called my wife. She went and looked at the tank, two fish were already dead along with a fire shrimp. Tank was cloudy. LPS were stressed.

My wife called our local Reef store who maintains my tank and he was able to come over that afternoon between servicing other customers tanks.

He put a new large bag of carbon in the sock as he could tell all the corals were very stressed. He recommended waiting 24-48 hours before doing a water change as everything was already stressed enough. And changing the carbon bag every 24 hours for the next few days.

I returned home Saturday afternoon and did a 15 gallon water change. Changed out the carbon.

I lost 2 large SPS colonies, 7 SPS frags, large frogspun , 2 hammers and all the remaining corals both SPS and LPS show signs of stress and now STN.

Apex seems to be working fine currently. But my reef maintenance guy is guessing that Apex could have pushed a software patch as they are normally sent at night. Patch could have temporarily caused my heaters to not turn off once the water temperature reached 79.0 as per the setting I have it on.

Two most frustrating things about this is, I had just turned the corner and with my reef shops help, my tank was doing fantastic for the last 4-5 months. I finally had the Nitrate / phos ratio where we wanted it and the doser was dialed in, corals were going great. Second frustration is I lost about $750 in corals and fish. I don’t know exactly why.

I have already submitted a trouble ticket to Apex. I would like to know what they can see in their system and if they know what the happened.

Has anyone else had this issue?

Thanks

Brian

IMG_0201.png
My dos qd drained my tank(set to auto water change) causing 20 gallons of freshwater to dump in from the full to the top ato. I took a video of the pump running full speed with the fusion showing the pump off and my virtual outlet to shut it off tripped. Apex acknowledged there was an issue and it was eventually sorted out. I asked for power supply’s for the dos’s so I wouldn’t have to worry about this issue by plugging it in and have the power bar shut it off in a disaster. They told me that I could buy my own. Their support is always good but having something happen like this(that is completely unacceptable and should never happen) They should have given me a couple power supplies.
 

CHSUB

Active Member
View Badges
Joined
May 4, 2015
Messages
105
Reaction score
58
Location
Miami
Rating - 0%
0   0   0
If you Apex was controlling your heater and your Apex reads 90 degrees. Your Apex should shut off your heater. Did the power bar fail?
 

BeanAnimal

2500 Club Member
View Badges
Joined
Jul 16, 2009
Messages
4,782
Reaction score
7,558
Rating - 0%
0   0   0
If you Apex was controlling your heater and your Apex reads 90 degrees. Your Apex should shut off your heater. Did the power bar fail?
The temp probe was reading ambient in the house, as it became uncovered.

Which brings me to the next point, the probe and heater should always be in the same compartment with the probe just upstream of the heater and situated so that it can never run dry (probe or heater).
 

CHSUB

Active Member
View Badges
Joined
May 4, 2015
Messages
105
Reaction score
58
Location
Miami
Rating - 0%
0   0   0
Still do not think the root cause has been determined though. If it was due to the filter sock the heater should have shut off when she changed the sock and not required you to turn the socket off.

The temp probe was reading ambient in the house, as it became uncovered.

Which brings me to the next point, the probe and heater should always be in the same compartment with the probe just upstream of the heater and situated so that it can never run dry (probe or heater).
As Kstatefan says. The probe at some point is reading reef temp, because his wife cleaned socks and at this point Apex should turn heat off but he did when noticing high temp.
 

BeanAnimal

2500 Club Member
View Badges
Joined
Jul 16, 2009
Messages
4,782
Reaction score
7,558
Rating - 0%
0   0   0
1727742042464.png


Look at the graph forensically please.

1727742447959.png


With the probe out of the water in in ambient air the tank heated runaway, to 90F because the probe was reading ambient of 65. As it dried and the air around it raised in temp due to the heater overheating the tank, you can see it gradually rise to a little over 70 (90F water near it). At some point it becomes submerged again and INSTANTLY reads 90F. OP gets alarm and shuts off the power outlet assigned to it, but it was ALREADY off, as the single peak shows that the moment the controller saw 90F it started to cool...
 

CHSUB

Active Member
View Badges
Joined
May 4, 2015
Messages
105
Reaction score
58
Location
Miami
Rating - 0%
0   0   0
1727742042464.png


Look at the graph forensically please.

1727742447959.png


With the probe out of the water in in ambient air the tank heated runaway, to 90F because the probe was reading ambient of 65. As it dried and the air around it raised in temp due to the heater overheating the tank, you can see it gradually rise to a little over 70 (90F water near it). At some point it becomes submerged again and INSTANTLY reads 90F. OP gets alarm and shuts off the power outlet assigned to it, but it was ALREADY off, as the single peak shows that the moment the controller saw 90F it started to cool...
Yes, however op seems to think he turned heater off with his phone manually, but your explanation seems more plausible.
 

Tamberav

10K Club member
View Badges
Joined
Jul 4, 2014
Messages
10,579
Reaction score
15,931
Location
Duluth, MN
Rating - 0%
0   0   0
Seems like user error but the way to fix this is to run a heater that has its own thermostat so that both would have to fail in the ON.

An Inkbird wouldn't help if the probe was also not submerged (assuming the compartment got low).
 

BeanAnimal

2500 Club Member
View Badges
Joined
Jul 16, 2009
Messages
4,782
Reaction score
7,558
Rating - 0%
0   0   0
Yes, however op seems to think he turned heater off with his phone manually, but your explanation seems more plausible.
Yes, he saw the temp and likely manually set the control to off or disabled even though it was already shut down, not understanding what was happening until days later.

Data can be very revealing if you look at it with an open mind and try to understand how it was created.
 

apeshot

Community Member
View Badges
Joined
Aug 4, 2024
Messages
44
Reaction score
24
Location
Sioux Falls, SD
Rating - 0%
0   0   0
This has been a good reminder for me to spend this weekend looking at how I have my heater thermostat, probes, inkbirds, and apex setup. I haven’t done all I can to prevent this situation.
 

bakbay

Valuable Member
View Badges
Joined
Sep 1, 2022
Messages
1,788
Reaction score
2,103
Location
Orange County
Rating - 0%
0   0   0
I have Apex on all my 3 systems, recently playing with Hydros. Regardless of aquarium controllers, I use them for monitoring and fail-safe mechanism but never for controlling on/off operations. My heaters have their own controllers and similarly, the calcium reactors will not overdose since that is separately controlled. However, if Alk gets above a certain threshold (unlikely) the Apex will kill the power to the CO2 solenoid. Basically, I use the Apex as a backup solution.

Having said that — I’ve observed 2 separate occasions where the Apex went down completely due to a dirty WAV (drawing too much power) and failed Trident. Both devices were connected to the 24v ports. It’s a shame — I would expect the Apex to let me know or disable the port. Instead, it died completely and I had to unplug all devices and identified the issue via process of elimination. Luckily for me, I had an MP40, not on Apex, that kept on running. The MP40 is also on UPS.

In short, always have redundancy for critical components.
 

iReefer12

Well-Known Member
View Badges
Joined
Apr 22, 2012
Messages
885
Reaction score
783
Location
Boulder, CO
Rating - 100%
1   0   0
Will absolutely do that. Kicking myself for not thinking of that myself.

Brian, add another line of code into the heater, so if the temp drops below 75, the heater shuts off, I have this just in case the probe was to ever come out of the water.

(The temp should never be below 75 really, if it got there, the heater is either dead, or the probes out of the water)

If Tmp < 75.0 Then OFF
 

aquadise

Community Member
View Badges
Joined
Dec 14, 2020
Messages
63
Reaction score
33
Location
Vietnam
Rating - 0%
0   0   0
Temperature is the most important factor. I never let one devices to control the temp of my tank. I usually have 1 back up for each other.
In my tank, i set up a separated controller to control only the temp. I set my Apex to monitor the temp, to alarm, and not control. I have another temp controller, which is a minor function of my Kamoer X5S.
 

IF YOU HAD A CAREER IN REEF-KEEPING, WHAT WOULD YOU BE DOING? AND WHY?

  • Selling and distributing livestock!

    Votes: 53 32.9%
  • Selling and distributing equipment!

    Votes: 15 9.3%
  • Breeding and Aquaculture!

    Votes: 77 47.8%
  • Livestock Disease and Treatment!

    Votes: 7 4.3%
  • Chemistry!

    Votes: 16 9.9%
  • Designing and Maintenance of tanks!

    Votes: 32 19.9%
  • Research and Education!

    Votes: 39 24.2%
  • Reefing Equipment and Dry Goods Creation and Manufacturing!

    Votes: 19 11.8%
  • Conservation!

    Votes: 40 24.8%
  • Other (please explain)!

    Votes: 6 3.7%
Back
Top