THANK YOU APEX FOR LETTING ME QUIT THE HOBBY

[ZaneTer]

GHL is not just "hobby grade" equipment. It is used in commercial applications, including research facilities.
https://www.aquariumcomputer.com/category/application-examples/

I assure you that it is still only hobby grade equipment. For your information I work for Siemens as a senior automation engineer.

My comments don’t come from someone that may have done some light internet reading and then formed an uniformed opinion. This is my profession.

Their equipment is really no superior to anything available 30 years ago. Trying to say that APEX and GHL are above hobby grade equipment is like saying that a trabant and a Bugatti veyron are on equal terms.

For your information I am not knocking GHL without experience. I have had the P4 and the KH Director. The first director failed outright, nice and safe. The second failed and dumped in 600ml of triton core7. The dosing pumps were inaccessible because of their horrendous software. I found the problem quickly but had to watch as it dumped in enough to kill almost every single coral I own.

If GHL want to come on here and dare start to kick up a fuss I still have every photograph and correspondence that was sent along with pictures from the event detailing to complete and utter failure of their equipment. In the end I threw everything of theirs away. I couldn’t even sell it in good conscience. I purchased equipment from Siemens and have built a significantly better and more robust system for the same price.

@d2mini
If you want to have a good professional discussion on advanced automation and their applications please feel free to contact me. I take pleasure and pride in my work and am more than willing to share my knowledge.

 

[d2mini]

You two are completely missing my point. I probably wasn't clear enough.
I never said it was actually commercial grade... whatever you feel that is. I'm saying it's used in professional/commercial applications.
If research institutes, aquariums, universities, etc use and trust the equipment, that says something to me.
And it's not a knock against Apex. They may be able to provide plenty of examples of that as well. I just haven't seen it but did know of GHL's success in this area.

I also have no experience or interest in the new KH Director. Sorry to hear you've had a problem. Calcium Reactor is the way to go IMO on anything bigger than a nano.
But the P3 and P4 have always been rock solid for me. 6 years and running.

 

[ZaneTer]

@d2mini
I dare say you are missing the point yourself. When you really read into the installation examples they have used in the link you provided you will see that really there is actually very little control. A humble reef-pi can kick the **** out of what they have done on those sites.

I don’t hate GHL or APEX products but they are certainly not robust or reliable enough to warrant their cost.

My view is heavily skewed because I know what modern control systems should be capable of. I know the hardware inside out, user interfaces, communication protocols, redundancy operation and advanced applications.....the list goes on. In this day and age their equipment is not where it should be.

I do blame GHL for the disaster I experienced. I place the blame squarely at their feet. If I were to produce such shoddy equipment and software I would be sued without a moments hesitation. A fair portion of my work is in the pharmaceutical industry. We test everything to death with the EMA and CHMP to ensure that single points of failure cannot be catastrophic to a system.

The American side of GHL were fantastic, truly great customer service. The German side was atrocious. I will never deal with them again. Unfortunately I bought my equipment directly from Germany. I could only get a response from Germany when I begged the American side for assistance.

You are completely correct about the calcium reactor. It’s fantastic. It is a wonderful piece of kit. I wrote a block of code for the system to learn my alk requirements so that every hour the ph set point adjusts itself to maintain 8.5 because of the consumption curve throughout the day. Of course I would love to see GHL or APEX implement that.......despite it only took half an hour to write.

 

[jda]

Those places are using them because there are not better alternatives, save for a Ranco and a lab-grade multi-probe PH controller.

I do agree that a Pi based controller can crush an Apex, but this takes real coding skill and not what apex calls "coding." Most people with these skills get enough of this at work to be fully committed at home - at least in my experience.

I was a software engineer for a long time and most recently at Google. I wrote a J2EE controller to run on a MacMini using USB controlled outlets and probes. The abilities that I had dwarf what an Apex or GHL can do with a MySQL database to store unlimited data points, full web hosting, static IP, etc. I could tail the log to my phone, query the database from anywhere and even ssh into the thing for FULL control. What I concluded is that even though it never did have a failure, trusting the thing to perform any action was just dumb. I eventually got sick of writing code, building and deploying and I quit using it - I did this all day at work and did not need any more. The most complicated thing that I have on my tanks right now is a Medusa or Ranco. Everything is redundant - never a crash - the redundancy is what saves the tank, not a controller alone (unless it is part of a redundancy).

What makes a hobby grade piece of equipment work is how the user implements it. To me, they are good for nothing outside of being timers to unimportant things (lights) and for alerting.

 

[jda]

Let me outline a few things that enterprise grade equipment would have:

• A self montioring and testing system that continuously tests your "program," every probe and every peripheral and logs results and alerts of any suspicious activity
• A redundant controller/brain
• Two probes of each kind, or even three
• Actually be able to alert you still if it failed (needs a mini, self-contained UPS, or the like). All enterprise systems will send a last SOS before they shut down.
• Every good enterprise system will tell you why it failed in acute detail
• Never stick on - always fail off so that you can depend on it (I have seen Apex fail "on," no idea about the GHL)
• In today's world, a way to remotely offload the brain in case of a failure - maybe to a data center in Africa, Asia, Europe or at least another part of the US power grid. You interface all of your "stuff" through a BUS and then have that BUS connect to the local brain while always testing the connection - then it can connect to a remote brain if your local one goes out. Then, you need a redundant BUS.
• When you get done, you have the smartest people try and crash it both by hand, script and code... for about 100 million cycles, or more.
• ZaneTer can probably add more

I know that this all seems extreme, but this is not a Die Hard movie from a decade ago... kids in High School can do all of this now with free tools available from AWS, RackSpace, MicroSoft and Google and free, easy to use code from GitHub.

 

[ZaneTer]

Let me outline a few things that enterprise grade equipment would have:

• A self montioring and testing system that continuously tests your "program," every probe and every peripheral and logs results and alerts of any suspicious activity
• A redundant controller/brain
• Two probes of each kind, or even three
• Actually be able to alert you still if it failed (needs a mini, self-contained UPS, or the like). All enterprise systems will send a last SOS before they shut down.
• Every good enterprise system will tell you why it failed in acute detail
• Never stick on - always fail off so that you can depend on it (I have seen Apex fail "on," no idea about the GHL)
• In today's world, a way to remotely offload the brain in case of a failure - maybe to a data center in Africa, Asia, Europe or at least another part of the US power grid. You interface all of your "stuff" through a BUS and then have that BUS connect to the local brain while always testing the connection - then it can connect to a remote brain if your local one goes out. Then, you need a redundant BUS.
• When you get done, you have the smartest people try and crash it both by hand, script and code... for about 100 million cycles, or more.
• ZaneTer can probably add more

I know that this all seems extreme, but this is not a Die Hard movie from a decade ago... kids in High School can do all of this now with free tools available from AWS, RackSpace, MicroSoft and Google and free, easy to use code from GitHub.

@jda
You have hit the nail square on the head.

On the software side we add full keystroke and event logging to redundant historian servers. All equipment is hot-swappable.

I’m glad you brought up using multiple probes for 2 out of 3 voting. This allows you to find your failed probe with a very high degree of certainty when you factor in MTBF on components.

We also write self diagnostic and testing routines.

I had no idea you had a software background, I am impressed.

 

[tiggs]

Let me outline a few things that enterprise grade equipment would have:

• A self montioring and testing system that continuously tests your "program," every probe and every peripheral and logs results and alerts of any suspicious activity
• A redundant controller/brain
• Two probes of each kind, or even three
• Actually be able to alert you still if it failed (needs a mini, self-contained UPS, or the like). All enterprise systems will send a last SOS before they shut down.
• Every good enterprise system will tell you why it failed in acute detail
• Never stick on - always fail off so that you can depend on it (I have seen Apex fail "on," no idea about the GHL)
• In today's world, a way to remotely offload the brain in case of a failure - maybe to a data center in Africa, Asia, Europe or at least another part of the US power grid. You interface all of your "stuff" through a BUS and then have that BUS connect to the local brain while always testing the connection - then it can connect to a remote brain if your local one goes out. Then, you need a redundant BUS.
• When you get done, you have the smartest people try and crash it both by hand, script and code... for about 100 million cycles, or more.
• ZaneTer can probably add more

I know that this all seems extreme, but this is not a Die Hard movie from a decade ago... kids in High School can do all of this now with free tools available from AWS, RackSpace, MicroSoft and Google and free, easy to use code from GitHub.

EXACTLY. This is what most people skim over. I'm the first to admit that the controllers available are fine for our use, but they should never be trusted alone and you should have redundancy/failsafes in place in both your code and your equipment setup. Everything fails eventually, so if you're relying on one piece of equipment/software for the survival of your tank, you've already lost the battle. I'm going on my 19th year in the IT industry and I was a systems engineer for the better part of that in Fintech before going into management. It seems like the common trend is the folks that have experience working with enterprise grade stuff understand the difference and realize there simply isn't an enterprise grade aquarium controller on the market because the target demo is too small.

I've considered making my own controller, but came to the conclusion that it simply isn't worth the time and effort for the end result. It's great as a fun project, but it would take years to get to the point where Apex and GHL are already. The stuff that can be improved on really isn't worth the extra leg work to me, so I've taken the current controllers' possible shortcomings into consideration when designing my setup.

Lastly, I just want to say (again) that I really like both GHL and Apex and nothing I'm saying is meant to dog them or bring attention to anything that's insufficient. I think they're both great in the reef keeping industry and admittedly, I'm one of the bigger Apex fanboys you'll find. I just hate seeing people end up with a false sense of security and trust a single piece of equipment with their entire reef's well being. If everyone understands exactly what could happen with these devices during a failure and designs their systems accordingly to prevent a total loss, we'd see less of these posts.

 

[jda]

Most people have never seen "tech" like this and are incredibly proud of their "coding" ability to turn a light on or off with an apex. I think that this can kind of give a false sense of security. To again quote one of the best cautionary tales of our lifetime about relying too much on automation, the great and somewhat prophetic Jurassic Park:

Ian Malcolm: I’ll tell you the problem with the scientific power that you’re using here: it didn’t require any discipline to attain it. You read what others had done and you took the next step. You didn’t earn the knowledge for yourselves, so you don’t take any responsibility for it. You stood on the shoulders of geniuses to accomplish something as fast as you could and before you even knew what you had you patented it and packaged it and slapped it on a plastic lunchbox, and now you’re selling it, you want to sell it.

Being able to write pseudo-code like on an APEX has been around since people programmed computers with punch cards. ...only the people who did this in the 1950s and 1960s learned that you could not rely on them, but they actually learned it for themselves. People who do not know are just figuring it out today.

The great thing is that nobody needs to know too much about this if they follow two simple rules... just use them to monitor & alert, and do not plug anything of critical importance into them. If you want to do more than this, then be redundant.

 

[BeelzeBob]

Your wifi router needs a good backup power supply, too. I learned that the hard way

 

[ZaneTer]

@akma

I am very sorry for your loss. Have you decided to continue or to give up the hobby? Many of us have been through the same as you. I very recently had a similar situation.

Wish you all the best of luck no matter which route you took.

 

[Lowell Lemon]

So I think it is important to have technology work for you if it is too complex you just add possible failure points. I would rather use ink bird heater controllers, an independent wave control, light control and possibly a kh control in the future. But integration can also lead to unintended failure across systems. This seems to compound the negative outcomes many experience with integrated systems. The rule of kiss may benefit us and the natural systems we try to replicate.

 

[ZaneTer]

I liken automation to building a house....it’s fine if you know what you are doing but a disaster waiting to happen for those that don’t. This goes for equipment manufacturers and end users alike.

 

[Genomecop]

@ZaneTer do you use a controller for dosing?

 

[ZaneTer]

@ZaneTer do you use a controller for dosing?

Yes, I do. Do you want to know more about it? It will be quite a lengthy post but I will describe the setup and component selections as well as operation and accuracy.......it’s awesome. I’ll outlay the costs for the route I took and how much it would cost to add another 32 dosing heads (overkill but easily done).

 

[Mr. Brooks]

My apex failed two weeks ago during a firmware update. They told me it was a hardware issue, but for some reason the hardware issue made its way onto my backup data file.

I had a backup brain on hand, but it was useless because the backup data file bricked the backup brain.

I should’ve configured the backup brain ahead of time, but I never had time to get around to it. It’s a pretty in-depth process if you’ve ever looked into it. You can’t just swap brains and upload the backup file. You have to plug each module in in the exact same order you did when you first set it up.

Luckily the failure happened in front of me so I was able to move things around and keep things going.

Even though I really didn’t have time to rebuild my 22 module Apex system from scratch due to the addition of my baby daughter, I still turned the situation around into a positive. I am now only using the Apex for light duty tasks and monitoring. I’m making things as modulated as possible so the next Apex failure won’t shut my entire system down.

I did a much needed clean up and labeling job on all of the wires etc. All-in-all I feel much more confident in my system today than I did before the failure. Mainly due to re-familiarizing myself with all the details of my system and programming as many failsafes and backups as I can.

No more auto top off on the Apex. I can tell you that. Tunze osmolator for the win.

I’m concerned for the guys that are buying into the Apex world with dosing and return pumps etc. It’s so intoxicating to have your entire system connected and accessible on your phone. But so dangerous to be so dependent on a single piece of hardware that I have seen fail with my own eyes.

The really amazing thing for me, looking back on it, are the mistakes I made while under stress, dealing with something that I never thought would happen to me, woefully unprepared with a false sense of security (my tank is in my office and I had to deal with the failure during business hours). I bought the backup brain just in case, but I never thought I would actually need it. It’s difficult to plan for the mistakes you may make while under stress. A failure of imagination on my part. I’m sure many others are guilty of the same. As I spent many hours contemplating what could go wrong. You can plan for equipment failure. It’s more difficult to plan for human brain failure.

 

[Monkeytank]

I warn people not to get a false sense of security just because they have alerts set for e-mail or text. You absolutely should have those set, but it's only helpful when you are awake and checking your texts and e-mails. It doesn't help when you are asleep and it happens at midnight. It won't help when you are half way around the world and your sleep cycle is opposite the people you would ask to go check on the system. I've had both of my EB832's start cutting off all power to everything plugged in multiple times over the past 2 weeks. All the sudden this just started happening with no changes made by me to the system or components. When I come down to check after finally seeing the text that came in the middle of the night warning this and that and my temp has dropped, the bar has every light on orange like normal, but not a single bit of juice is flowing to anything I have plugged in. At first I thought it was just one bar, so I moved everything to another one. That one did it too. Then I plugged only a few completely different items into the first bar that was failing, and it seemed to work.... only to fail again the next day. I've had to unplug them from the power outlet and the Aquabus cables and reset the circuit breaker on the back multiple times to get them back on. Sometimes just pulling them from the wall and resetting while plugged in works. Software is all up to date, and has been the whole time. I have a heavy SPS system and the fluctuations that have been happening because of this are really stressing me out, hopefully not about to kill everything. I'm talking no circulation, no off cycle lighting, and no heating. Accordingly I've had big PH and temp swings when it was several hours before I saw the alerts and could go reset things. I'm about to be gone for 10 days, so I'm sweating this right now.

True, the Apex is consumer grade, but with all the parts included, the price is a little above the prosumer level in my wallet's view. They could do better for the price. Not everyone can afford to have multiple $200 and up components lying around in stock ready for a failure at a moment's notice. I work ALL the friggin time, so I need automation or I simply can't be in this hobby, which is my relaxing reset from work. I've chosen Apex, and I don't see a better option for me at this point. I've invested a lot of money in the system, and I replace the parts as they die, often just months after the warranty expires. I'm not very happy right now.

I'm tired of seeing product bashers and fan boys argue back and forth over this. I wish as much time was spent by people developing a product designed with more redundancy. There is a huge market for that.

 

[Monkeytank]

Oh, by the way. 3 minutes after posting that last comment, both bars cut off power to EVERYTHING again. Whoo Hooo!

 

[Monkeytank]

Yes, I do. Do you want to know more about it? It will be quite a lengthy post but I will describe the setup and component selections as well as operation and accuracy.......it’s awesome. I’ll outlay the costs for the route I took and how much it would cost to add another 32 dosing heads (overkill but easily done).

Please post it.

 

[BZOFIQ]

Please post it.

I second that. I'd love to see other (more robust) options.

 

[BZOFIQ]

I too am an APEX owner and while I dont trust the thing a bit I still use it; due to lack of viable options.

That said, I'm surprised nobody in their suggestions for redundancy mentioned the extra power supply that would supply the AUX port. I have that on an APC UPS and so the "half-a-brain" module never loses power and keeps the bus powered through any short flicks and not too long power outages.

That AUX P/S should have been included in every box Neptune sells.