Polo reef disaster . . . OMG

[areefer01]

If you haven't heard of POLO REEF or Andrew Sandler, then I'm not sure how your in this hobby. He has one of the best reef set-ups in world inside his house. His large tank that had the mishap of leaking out 10,000 gallons is actually 17,000 gallons reef tank heavy in Acro's, plus he has many more tanks as well.

This makes no sense. You do understand that not every hobbyist partakes in social media or is new? Plenty of members here and outside predate this system.

 

[vetteguy53081]

message from Andrew at polo reef:
The Support And Reach Out From All Of You And The Whole Reefing Community Has Been Nothing Short Of Incredible! I Thank You All From The Bottom Of My Heart !!
Today We Removed Most Of The Dead Corals Some The Size Of 3 Basketball Balls !!
We Will Probably Use This Time To Rebuild Old Equipment Including Engines, Valves And Lighting Fixtures, Wifi, And A Siren In My Bedroom Lol
La Videos Will Be Coming Soon!
My Late Father Used To Say You Flushing Down The Drain !! He Was A Die Hard Investment Guy !! But I Did Not Expect It To Be Literally Down The Drains Lol !
Bless You All !!

 

[Gregg @ ADP]

There is alot of theories and possibilities. I'm hoping we can keep this civil and learn from this experience and for me has opened eyes on the risks of automation. This is one hobby that invites risk

100% this.

 

[Mebbid]

If you haven't heard of POLO REEF or Andrew Sandler, then I'm not sure how your in this hobby. He has one of the best reef set-ups in world inside his house. His large tank that had the mishap of leaking out 10,000 gallons is actually 17,000 gallons reef tank heavy in Acro's, plus he has many more tanks as well.

Sorry I guess. This accident is the first i've heard of his reef.

 

[jsmkmavity]

Sad to hear this. The loss of livestock is painful for us to hear/see and devastating for Andrew i am sure. I hope they figure out everything that led to this disaster and are able to correct it and prevent it from happening again or to someone else.
This is not an "empty fake" response or without honest sympathy for what has happened!!! Hopefully we can re"gain" some of our humanity and actually care about someone we have never met.

 

[SeeFu]

andrew sandler made a statement about what possibly went wrong.

 

[maroun.c]

Im not a huge fan, so I only skimmed through the video, but didnt he say several times that there were two valves on that line? One electronically control, but one was manual, that had to be opened by a wrench? Perhaps I misunderstood or misheard.

Yes, that’s what he said.
My concern with valves that need a wrench is that while they’re much less likely to be opened or closed accidentally, they can still loosen over time from vibration, or fail to close fully because of a mistake or debris stuck in the valve, and if they do this won't be visible. In a case where the valve is rarely or never operated, it could still loosen after years of vibration without being noticed, especially if the electronic valve along it is closed. At that point, a default open command or a controller error could potentially open it. Many controllers apply default states when connectivity is lost, and a glitch could mess this default.
Of course, this is just one possible explanation among many.
I now make sure any semi-automatic water change or drain on my system is:

• turned off at the controller side, with the default state set to closed if connection is lost
• physically unplugged, so I only connect the pumps manually when I’m doing a water change and then control that during the water change, then unplug.

 

[Largeangels]

He mentioned Ozone as possibly degrading rubber seals out. Once they take the valves out and inspect them they will know more.

 

[Luke Schnabel]

It's very sad to see this happen. I'm actually going to be installing a float valve on my sump because of this. Make it send me an alert if my sump level is low.

 

[jmaneyapanda]

He mentioned Ozone as possibly degrading rubber seals out. Once they take the valves out and inspect them they will know more.

I thought I mentioned this elsewhere, but I have some questions about that. I find it hard to believe that ozone would degarde TWO ballvalve to the point they go from holding water to leaking 10,000 gallons very quickly. Its possible, but I would imagine a gradual degradation, not a dam busting. But also, if there is that degree of reactive ozone free floating in the water column and oxidizing things, then that would be causing big time health issues in the tank and the ORP wouldve been 600+. I have a hard time believing that they would run that system so recklessly.

But again, Im curious to see the transparency and findings. From what Ive read and seen from his accounts thus far, not a lot is adding up.

 

[jmaneyapanda]

Yes, that’s what he said.
My concern with valves that need a wrench is that while they’re much less likely to be opened or closed accidentally, they can still loosen over time from vibration, or fail to close fully because of a mistake or debris stuck in the valve, and if they do this won't be visible. In a case where the valve is rarely or never operated, it could still loosen after years of vibration without being noticed, especially if the electronic valve along it is closed. At that point, a default open command or a controller error could potentially open it. Many controllers apply default states when connectivity is lost, and a glitch could mess this default.
Of course, this is just one possible explanation among many.
I now make sure any semi-automatic water change or drain on my system is:

• turned off at the controller side, with the default state set to closed if connection is lost
• physically unplugged, so I only connect the pumps manually when I’m doing a water change and then control that during the water change, then unplug.

Agreed. And if this is true, then there were some massive oversights in their day to day operations. I get it, **** happens, people get complacent, etc. But per his account, this was attaching a filter on the aquarium directly to a open RO source with an unmonitored drain nearby. Thats likely *why* they designed the valve redundancy, but to never check or look at it to where a controller program could do this is pretty neglectful.

 

[Gregg @ ADP]

The lesson to be learned here is, alarms are cheap.

In the grand scheme of the cost and value of his tank, the cost of having a system to send out alerts any time something is not working optimally is probably about the same as their food cost for a week. I don’t understand how that system looses 100 gal of water without somebody knowing about it, much less 10,000 gal.

I was once consulting for a zoo that lost and entire shark and ray petting tank due to a power failure. They didn’t want to restart the tank until they had a state-of-the-art life support system in place. I told them they didn’t need that…they could have avoided the entire tragedy with $500 worth of sensors and back-up air pumps. Or $1000 for a back-up generator. Point being, these safety measures are very cheap.

 

[W31Olds]

It's very sad to see this happen. I'm actually going to be installing a float valve on my sump because of this. Make it send me an alert if my sump level is low.

Float Valves are very inexpensive and reliable (as long as you clean them). I use one instead of an Automated ATO. Reef Tanks also use an overflow type of design. If something fails, you lose an inch or two water from your DT as they are self-limiting. Not half the Water in your Tank like Polo Reef. Anytime you have Filtration and Equipment below the water level of your Tank, it becomes risky. It may have been valves that caused Polo to fail but could have just as easily been a burst PVC Pipe or a UV Housing. It sounds like they really need to modify how the water flows from the Tank to the Filters.

 

[areefer01]

But again, Im curious to see the transparency and findings. From what Ive read and seen from his accounts thus far, not a lot is adding up.

I have noticed a couple of replies like this here. What is it that you want in transparency and what is not adding up to you? Or better yet why it matters. I do not mean this disrespectful just curious.

Insurance company, builders, owners I can fully understand.

 

[jmaneyapanda]

I have noticed a couple of replies like this here. What is it that you want in transparency and what is not adding up to you? Or better yet why it matters. I do not mean this disrespectful just curious.

Insurance company, builders, owners I can fully understand.

Because it's learnable (?) And its publicly discussed, even by the owner?

I dont understand why anyone would be villianized for wanting to know what caused it. Its directly in the wheelhouse of interest in this forum.

 

[areefer01]

Because it's learnable (?) And its publicly discussed, even by the owner?

I dont understand why anyone would be villianized for wanting to know what caused it. Its directly in the wheelhouse of interest in this forum.

I read it as you are second guessing what they are sharing or not believing it. I did not call you a villain - merely asked to you clarify what you mean when you said 'not a lot is adding up'

 

[CHSUB]

I think Andrew outlined what happened well. Closed loop connected to sand filter drained tank, not the overflow to sump. Alarm triggered however Wi-Fi failed to send signal. Closed loops are a risk for this exact reason, tank can drain.

 

[ayef12]

Hard to fire someone for equipment failure..

I dissagree. Andrew is paying a team to make sure he has triple redundancy on his system. They should have been alerted way before this became catasrofic. A simple Neptune Apex system would send out alarms for low water levels. Someone was asleep at the wheel.

 

[jmaneyapanda]

I read it as you are second guessing what they are sharing or not believing it. I did not call you a villain - merely asked to you clarify what you mean when you said 'not a lot is adding up'

I am questioning whats been said, because its not adding up. So far, what weve been told is that 2 independent valves spontaneously malfunctioned. That doesnt add up. I dont understand why questioning it is so wrong? Is Andrew above questions? Criticisms? Could everyoen else here learn from it? Could not being transparent preserve his prestige and fame? Could not being transparent protect someones reputation? All these things might be true or not. Andrew publicly posting and illuminating this issue doesnt prohibit these questions from being asked. I *assume* you're implying Im being negative by asking and by stating the thinsg said dont add up. Believe me, Im not. I could say far worse things, which I wont. All Im saying is, if this is going to be discussed AT ALL, which it is, then we shoudl really discuss it. If you think its not proper, and that we should all just give our performative "I wish the best" and sing kumbayah, we'll disagree. Because then this forum has lost its educational value.

Also, for the record, I didnt accuse you, or anyone else of calling me a villian. I simply said (and still will argue), that villianizing asking "why" is vapid. Im not aiming that at you, or anyone else specifically. No offense intended. Im just saying calling foul for asking why isn't where it's at.

I think Andrew outlined what happened well. Closed loop connected to sand filter drained tank, not the overflow to sump. Alarm triggered however Wi-Fi failed to send signal. Closed loops are a risk for this exact reason, tank can drain.

He, himself, even said they are still looking into it. Any your simplifying it a bit much, from what can gather. Two independent valves simultaneously spontaneously failed. And closed loop weas connected to sand filter that was connect to RODI reservoir that overflowed. Anyway, Im just saying Im curious to see what comes of it.

 

[CHSUB]

He, himself, even said they are still looking into it

I agree, I’m highlighting what I heard from Andrew’s messages. Stuff happens even at NASA…