Apex Modules change address after power interuptions

[Enderg60]

For the second time in a month we have had some electrical storms pass through. The last time we had massive power surge, power dips and outages. When that happened my EB8s and a few other modules decided to change address. This caused my current programing to assume the modules were disconnected and new units were added at a different address.

I had to track down what sensor was on what module, what EB8s where what, reprogram all the outlets and redo my dashboard.

Today we had an electrical storm with a few power dips, I noticed my wavebox acting odd and the apex was no longer responding. I had to remove the USB cables and reconnect them to get it to reboot. Now the same issue has occurred where the EB8s and modules have changed addresses.

It has taken me 2 hours to reconfigure it all again. Ive been using Apex gear for nearly 20 years and Ive never had this issue until this past month.

Can anyone offer advice for how to stop this from happening again?

 

[blaxsun]

When you say "change address", so you mean the #ID each module was assigned to? I just had a 1-hour power outage and everything came back exactly as before.

Do you have any before/after screenshots we could look at?

 

[A_Blind_Reefer]

Electrical issues like you mention can be problematic. I forgot if you already have some sort of surge protection or backup. That would be a first step. Save your configuration report and keep current copies after any changes. It makes it a little easier for reprogramming if needed if you’re good at coping and pasting. Is this an old unit? You mentioned almost 20 years, and nothing lasts forever. Either case, old or newer, I like to use the accessory power port on the head unit and have that plugged into a computer battery backup surge protector. Bummer, once is enough….twice….ugh

 

[_AV]

Neptune couldn't get their state machine done right for module reconnect events. So they've applied multiple updates to the code to fix several edge cases and it's still effectively broken.
The only way I know how to fix this is to provide a 12v battery back up to the head unit which would have enough juice to run the head unit even when everything else is off until the power is restored. There are small and cheap backups available for network routers, you can use one of those:

 

[Enderg60]

I just got it all redone so I dont have screen shots but lets say I start with the following:

EB8-1 Address #9
EB8-2 Address #7
PM-1 Address #11

After it went wonky I had

EB8-1 Missing
EB8-2 Missing
PM-1 Missing

EB8-3 Address #6 - This was #9
EB8-4 Address #12 - This was #7
PM-1 Address #8 - This was #11

The "new" modules have "new" outlets and probe readings that show up in the lock for the dash. The "old" outlets still show but are not functional. The EB8s revert back to old settings and random crap turns on.

I have to place all the "new" outlets on the dash and turn them off. Then figure out what modules are what and rename everything to what it used to be. Then I can copy the "old" outlet programs one by one to the "new" ones. Even though its not starting from scratch it still takes hours because I have a bunch of stuff on here.

First time around my whole house surge protector tripped and was offline when a few more surges came through. Those fried my halide ballasts, and made the apex freak out.

This time the whole house surge protector did not trip but the apex freaked out.

There are only 2 ways to reboot the apex at this point. Unplug the two USB cables to cut power and signal or turn off all 5 breakers to the fish room.

I opted to pull the USB wires, wait 60 sec and plug them back in. I think this is where everything readdresses. After the fact I considered if I should have just cut the breakers to reset it all at once.

My current plan is to get a UPS to maintain clean power to the EB832, and get a Apex power cable which I will plug into a "not backed up" power source to monitor if the power is out. Im also adding surge protectors to everything since apparently the whole house one is only good for one surge and has to be manually reset.

 

[blaxsun]

Just absolutely bizarre! Faulty SD card in the Apex...?

 

[_AV]

Just absolutely bizarre! Faulty SD card in the Apex...?

If that was the case, the head unit wouldn't boot at all.

One of the two things happened here: (a) all modules somehow lost the memory of the head unit serial number and forced a new module connect event or (b) the head unit failed to negotiate a reconnect with each module and did a new module connect routine instead.

"a" is highly unlikely to happen to all modules at once. What remains is "b". Again, the logic for module connect/reconnect is very very broken in the head unit code and it becomes more and more apparent the more modules you have connected. If you put the head unit on a UPS, at least it prevents a whole system reconnect every time your lights flicker.

 

[_AV]

My current plan is to get a UPS to maintain clean power to the EB832, and get a Apex power cable which I will plug into a "not backed up" power source to monitor if the power is out. Im also adding surge protectors to everything since apparently the whole house one is only good for one surge and has to be manually reset.

I wouldn't put EB832s on a UPS. Too much load plus pumps and UPS don't really play nicely together.
Also, in your config there is no way to know if the power goes out anyway because the backed up EB832 would power the head unit in the event of an outage. But if you only back up the head unit, you can still tell if the power goes out when your EB832s get zero amps on all outlets.

 

[A_Blind_Reefer]

I just got it all redone so I dont have screen shots but lets say I start with the following:

EB8-1 Address #9
EB8-2 Address #7
PM-1 Address #11

After it went wonky I had

EB8-1 Missing
EB8-2 Missing
PM-1 Missing

EB8-3 Address #6 - This was #9
EB8-4 Address #12 - This was #7
PM-1 Address #8 - This was #11

The "new" modules have "new" outlets and probe readings that show up in the lock for the dash. The "old" outlets still show but are not functional. The EB8s revert back to old settings and random crap turns on.

I have to place all the "new" outlets on the dash and turn them off. Then figure out what modules are what and rename everything to what it used to be. Then I can copy the "old" outlet programs one by one to the "new" ones. Even though its not starting from scratch it still takes hours because I have a bunch of stuff on here.

First time around my whole house surge protector tripped and was offline when a few more surges came through. Those fried my halide ballasts, and made the apex freak out.

This time the whole house surge protector did not trip but the apex freaked out.

There are only 2 ways to reboot the apex at this point. Unplug the two USB cables to cut power and signal or turn off all 5 breakers to the fish room.

I opted to pull the USB wires, wait 60 sec and plug them back in. I think this is where everything readdresses. After the fact I considered if I should have just cut the breakers to reset it all at once.

My current plan is to get a UPS to maintain clean power to the EB832, and get a Apex power cable which I will plug into a "not backed up" power source to monitor if the power is out. Im also adding surge protectors to everything since apparently the whole house one is only good for one surge and has to be manually reset.

Unplugging and replugging the aquabus cables while power was on is most likely what caused the new addresses. A faulty sd card as mentioned above could cause issues like missing data, graphs, unresponsive outlets, etc and still be able to boot up after a power cycle, at least that was my case when I rebooted several times troubleshooting mine. The battery backup surge device if large enough, like a fancy diy or blue yeti type, could work for the entire system but is pricey. A small unit is really only good for the head unit as stated above.

 

[Enderg60]

The UPS I have coming has plugs that are battery backed up, and plugs that are only surge protection without backup. The EB832 will be plugged into a backed up socket. The power adapter for the apex will be plugged into a no backup socket. When the power goes out the apex adapter will lose power. This even can be used to code pumps and other high amp devices to turn off when the power is out, which would reduce the load on the UPS.

BUT I dont care about the load on the UPS because it only needs to last 60 seconds (I actually did math to make sure it will run for at least 60 seconds). I have a whole house backup generator that will start running after the power is out for a minute. It keeps going until the power is back on for 3 minutes and it cuts off with no interruption in power.

Its the power surges and that 1 minute of no power I need to deal with apparently.

Im not sure if the SD card is an issue, the modules that didnt get readdressed maintained their history. It was only the "new" items that only recorded since they reconnected.

Hopefully the UPS will keep things running but if it happens again ill leave it all connected and try doing a "flip all the breakers" reboot.

 

[blaxsun]

Best of luck in any event (what an ordeal). You'll want to make sure your UPS can handle the "burst" (watts) of everything starting up as well.

 

[_AV]

The UPS I have coming has plugs that are battery backed up, and plugs that are only surge protection without backup. The EB832 will be plugged into a backed up socket. The power adapter for the apex will be plugged into a no backup socket. When the power goes out the apex adapter will lose power. This even can be used to code pumps and other high amp devices to turn off when the power is out, which would reduce the load on the UPS.

How would you detect a power loss on the apex adapter though? On the UPS I assume? Because the backed up EB832 would provide power to the head unit over the aquabus... Apex wouldn't know the difference.

 

[A_Blind_Reefer]

The UPS I have coming has plugs that are battery backed up, and plugs that are only surge protection without backup. The EB832 will be plugged into a backed up socket. The power adapter for the apex will be plugged into a no backup socket. When the power goes out the apex adapter will lose power. This even can be used to code pumps and other high amp devices to turn off when the power is out, which would reduce the load on the UPS.

BUT I dont care about the load on the UPS because it only needs to last 60 seconds (I actually did math to make sure it will run for at least 60 seconds). I have a whole house backup generator that will start running after the power is out for a minute. It keeps going until the power is back on for 3 minutes and it cuts off with no interruption in power.

Its the power surges and that 1 minute of no power I need to deal with apparently.

Im not sure if the SD card is an issue, the modules that didnt get readdressed maintained their history. It was only the "new" items that only recorded since they reconnected.

Hopefully the UPS will keep things running but if it happens again ill leave it all connected and try doing a "flip all the breakers" reboot.

Sounds like it’s all figured out. House electrical sounds great, whole house surge, backup Jenny, etc. it doesn’t sound like you should be having events that would stun the apex. Maybe something isn’t working as properly as it should and needs attention. Just a thought as a simple power cycle shouldn’t affect the apex at all. The ups will solve the apex issue, but not fix whatever transient knocked it out originally.

 

[A_Blind_Reefer]

How would you detect a power loss on the apex adapter though? On the UPS I assume? Because the backed up EB832 would provide power to the head unit over the aquabus... Apex wouldn't know the difference.

The apex monitors the accessory voltage. When still powered by the aquabus and no voltage is present on the acc it triggers a ac out alarm that can be programmed to do whatever.

 

[_AV]

The apex monitors the accessory voltage. When still powered by the aquabus and no voltage is present on the acc it triggers a ac out alarm that can be programmed to do whatever.

Exactly. Except the plan is to back up the EB832 which has the accessory ports.

 

[A_Blind_Reefer]

Exactly. Except the plan is to back up the EB832 which has the accessory ports.

The apex can get power through the aquabus from the eb, and separately monitor power with an ac adaptor plugged into the head unit acc power input. Eb can be backed up and running, head unit running off the aquabus, and power monitored on the head unit acc power in. If only the acc power in to the head is lost a power fail alarm is generated.

 

[_AV]

Ah, so the head unit can tell if the power on the 12v jack is present? what's the code for that?

 

[blaxsun]

Ah, so the head unit can tell if the power on the 12v jack is present? what's the code for that?

It's under Misc Setup (where you enable power monitoring).

 

[A_Blind_Reefer]

Ah, so the head unit can tell if the power on the 12v jack is present? what's the code for that?

It’s not in the beginning, you have to scroll through it but it has a lot of detail if I remember right

Neptune Apex Programming Tutorials, Part 4: Power Monitoring

Seawitch submitted a new Article: Neptune Apex Programming Tutorials, Part 4: Power Monitoring Note From the Editor: This article is Part 4 of a series. Feel free to go back and read Part 1 or Part 2 or Part 3. This article and several future ones by the same author were originally part of...

www.reef2reef.com

 

[_AV]

Ha, in all these years I had no idea this was possible.