Hydros Maven is Delayed

[BZOFIQ]

Generally speaking, I’m not disputing any of these points. I’m only wondering whether the recurring profit margins for a testing device that not all of their existing clients will even purchase makes a dent in the overall profitability of CV.

I also wouldn’t expect their cloud costs to be all that high. If all they are storing is device configurations then I’d venture a guess that a couple of VMs and some storage would probably be enough to serve thousands of devices. Unless I’m missing something obvious, I can’t imagine the configs that need to be stored in the cloud for these devices would be more than a few megabytes… so maybe a few gigabytes of storage.

EDIT: also this isn’t CVs only product line. If they derive profitability elsewhere then it’s entirely possible they are concerned less about that which is derived from Maven reagents

I too don't think this require much computing and storage and agree that the running cost is probably not that high. No worries, monthly subscriptions are coming next!

.....that's why you gotta love local access.

 

[BeanAnimal]

Generally speaking, I’m not disputing any of these points. I’m only wondering whether the recurring profit margins for a testing device that not all of their existing clients will even purchase makes a dent in the overall profitability of CV.

Look at this way - lost revenue is lost revenue. Why bleed money if you don't have to?
You will find that many of these vendors are going to have to move to subscription and feature paywalls if they can't lock in consumables.

Reef Factory already bit the dust and I think if you start looking around you can identify others that may soon follow.

I also wouldn’t expect their cloud costs to be all that high. If all they are storing is device configurations then I’d venture a guess that a couple of VMs and some storage would probably be enough to serve thousands of devices. Unless I’m missing something obvious, I can’t imagine the configs that need to be stored in the cloud for these devices would be more than a few megabytes… so maybe a few gigabytes of storage.

I have never looked at their architecture but in general:

The cloud isn’t just holding static config files. It’s handling persistent telemetry from sensor readings, alerts and the historical logs.

It is handling real-time device communication (likely via MQTT) via high availability brokers.

It serves the entire user-facing config UI (web/app) and user auth mechanisms.

I am not sure where the push notifications originate, etc.

That adds up to rather significant ongoing compute, storage, and bandwidth needs.

So it is not just storing a few megabytes and the occasional session.

There are likely several databases at play, user data (billions of records) that must return queries fast enough to render telemetry, databases holding user configs, databases holding SaaS configs, etc.

This service will likely hold the DRM logic and functionality for reagents as well.

The SaaS is managing basically live streamed data for each device and user session, 24/7.

There should be live redundancy spinning, as well is geographic replicas.

The entire thing needs to be mirrored to a DEV environment (same components, smaller scale)

Don't forget THEIR telemetry for system health, server load, watchdogs, alerts and everything else that goes into managing a real customer facing service that must provide snappy, resilient and reliable uptime.

Cost per month if done on a shoestring with band-aids and prayer $5,000. Reasonable real world small scale cost $10,000 - $20,000 month -- and easily $50K or more at scale.

 

[BZOFIQ]

Cost per month if done on a shoestring with band-aids and prayer $5,000. Reasonable real world small scale cost $10,000 month -- and easily $50K or more at scale.

I very much doubt that but we are entitled to our own opinions.

One can have lots of fully dedicated servers hosted for the amounts of money you're listing.

 

[BeanAnimal]

I very much doubt that but we are entitled to our own opinions.

One can have lots of fully dedicated servers hosted for the amounts of money you're listing.

It is not my opinion. It happens to be a very large part of what I do for a living

For a scaled IoT SaaS they are either paying an IoT platform vendor for white label services or they are building it and hosting it at AWS, Google, or Azure... anyplace else would be silly, but who knows.

One way or the other, for a scalable, fast and redundant system, a whole lot more is involved than a "dedicated" server or two.

Compute instances (the "servers")
Load Balancers
Application Firewalls
API gateways
MQTT brokers
Data pipelines
Database instances
Authentication gateways
And and a half dozen other "instances" all tied together.

User telemetry is rather "big" data. The ability for hundreds, if not a few thousands users, to query datasets that large, somewhat simultaneously and return reasonably realtime charts and graphs is not trivial in terms of resources. A simple SINGLE SMALL AWS redshift instance can easily run $3,000 per month, before bandwidth cost or the "server" costs used to run the queries against it, let alone the telemetry, and the rest of the "app".


Did they roll their own, or are they paying Tuya, Lobidots, Blynk or some other IoT vendor for white label hosting. I have no idea. If they are on Tuya's platform (cringe - china hosting) then the costs could be much lower and in the $5,000 or less per month neighborhood -- even as low as $2K per month (Lets' assume scale to 10K devices). Tuya is a GIANT IoT platform hosted in China -- there may be US edge IPs. but as far as I know, most of it is tunneled back one way or the other.

 

[BZOFIQ]

It is not my opinion. It happens to be a very large part of what I do for a living

For a scaled IoT SaaS they are either paying an IoT platform vendor for white label services or they are building it and hosting it at AWS, Google, or Azure... anyplace else would be silly, but who knows.

One way or the other, for a scalable, fast and redundant system, a whole lot more is involved than a "dedicated" server or two.

Compute instances (the "servers")
Load Balancers
Application Firewalls
API gateways
MQTT brokers
Data pipelines
Database instances
Authentication gateways
And and a half dozen other "instances" all tied together.

User telemetry is rather "big" data. The ability for hundreds, if not a few thousands users, to query datasets that large, somewhat simultaneously and return reasonably realtime charts and graphs is not trivial in terms of resources. A simple SINGLE SMALL AWS redshift instance can easily run $3,000 per month, before bandwidth cost or the "server" costs used to run the queries against it.


Did they roll their own, or are they paying Tuya, Lobidots, Blynk or some other IoT vendor for white label hosting. I have no idea.

On one hand you say its a very limited market then you make it sound like they're hosting chase banking with 40 million customers across 7 continents

Don't know about Hydros but Apex Fusion has few months worth of telemetry at your disposal, yeah not much data at all, couldn't be much, perhaps few dedicated servers at most.

Russ used to host people's apexes data (with much longer retention) on his own personal host for fun before Apex Fusion became a thing.

 

[BeanAnimal]

On one hand you say its a very limited market then you make it sound like they're hosting chase banking with 40 million customers across 7 continents

Please don't be obtuse. It serves no purpose.
I don't know how many devices or customers, but it would be safe to assume several thousands devices to be a reasonable number for conversation. I used 10K devices for the Tuya IoT cloud example, based on what I know of their white label pricing.

Each of those devices reports telemetry in near real time for multiple sensors and output states.

Don't know about Hydros but Apex Fusion has few months worth of telemetry at your disposal, yeah not much data at all, couldn't be much, perhaps few dedicated servers at most.

In the kindest way, if you are using the term "dedicated servers" to describe how any of this works, then you don't understand how any of it works beyond very basic idea that data goes to some type of computer. To that end, applications like this don't run on "servers" be they "dedicated" or "shared". They run on connected suites of services like compute instances, database instances, load balancer instances, gateway instances, data pipelines, etc, that run on "server" hardware.

If you are imagining a server or two in a rack, dedicated to one client then you are miles off -- and if this vendor IS using a few such "servers" in a rack with all of the various services piled onto them, then I would run away as fast as I could. That is not how we do things today.

Russ used to host people's apexes data (with much longer retention) on his own personal host for fun before Apex Fusion became a thing.

Hosting Apex data “for fun” on a personal server isn’t in the same universe as managing a production-grade, secure, scalable platform.

You are literally arguing with somebody who builds cloud service instances, dashboards and data warehouses for a living.

I know roughly (first hand) what one vendor in this space pays for their backend SaaS and it is jaw dropping, even to me - my guess was at least 20% low.

 

[BZOFIQ]

Please don't be obtuse. It serves no purpose.
I don't know how many devices or customers, but it would be safe to assume several thousands devices to be a reasonable number for conversation. I used 10K devices for the Tuya IoT cloud example, based on what I know if their white label pricing.

Each of those devices reports telemetry in near real time for multiple sensors and output states.


In the kindest way, if you are using the term "dedicated servers" to describe how any of this works, then you don't understand how any of it works beyond very basic terminology. To that end, applications like this don't run on "servers" be they "dedicated" or "shared". They run on connected suites of services like compute instances, database instances, load balancer instances, gateway instances, data pipelines, etc.

If you are imagining a server or two in a rack, then you are miles off -- and if this vendor IS using a few "servers" in a rack, then I would run away as fast as I could before 1980 comes and reclaims them.


Hosting Apex data “for fun” on a personal server isn’t in the same universe as managing a production-grade, secure, scalable platform.

You are literally arguing with somebody who builds cloud service instances, dashboards and data warehouses for a living.

I know roughly (first hand) what one vendor in this space pays for their backend SaaS and it is jaw dropping, even to me - my guess was at least 20% low.

I'm going to keep it short, makes no sense to argue with you.

Hydros and Neptune would be losing money (lots of!) on every controller sold with a cloud solution you propose and yet they are in business making money.

You are literally arguing with somebody who builds cloud service instances, dashboards and data warehouses for a living.

 

[Dom_P]

Look at this way - lost revenue is lost revenue. Why bleed money if you don't have to?
You will find that many of these vendors are going to have to move to subscription and feature paywalls if they can't lock in consumables.

Reef Factory already bit the dust and I think if you start looking around you can identify others that may soon follow.


I have never looked at their architecture but in general:

The cloud isn’t just holding static config files. It’s handling persistent telemetry from sensor readings, alerts and the historical logs.

It is handling real-time device communication (likely via MQTT) via high availability brokers.

It serves the entire user-facing config UI (web/app) and user auth mechanisms.

I am not sure where the push notifications originate, etc.

That adds up to rather significant ongoing compute, storage, and bandwidth needs.

So it is not just storing a few megabytes and the occasional session.

There are likely several databases at play, user data (billions of records) that must return queries fast enough to render telemetry, databases holding user configs, databases holding SaaS configs, etc.

This service will likely hold the DRM logic and functionality for reagents as well.

The SaaS is managing basically live streamed data for each device and user session, 24/7.

There should be live redundancy spinning, as well is geographic replicas.

The entire thing needs to be mirrored to a DEV environment (same components, smaller scale)

Don't forget THEIR telemetry for system health, server load, watchdogs, alerts and everything else that goes into managing a real customer facing service that must provide snappy, resilient and reliable uptime.

Cost per month if done on a shoestring with band-aids and prayer $5,000. Reasonable real world small scale cost $10,000 - $20,000 month -- and easily $50K or more at scale.

From what I can tell, there is quite a bit of functionality that is still available when I connect to the devices via Bluetooth. As such it’s possible that some of what is described above isn’t necessarily required. But that is conjecture.

No end user UI is offered from hydros devices themselves best I can tell. That is to say that the phone app is the presentation layer and it seems to access both the hydros devices themselves as well as some cloud services directly (As discussed previously, when the apps is in Bluetooth mode, cloud access is not available)

Logs/telemetry do look to be stored in the cloud as they are not available in app Bluetooth mode. I assume the hydros devices themselves send their log data to cloud periodically over whatever protocol used and the phone app accesses those logs (when cloud services are available) when logs are viewed or when needing to perform graphing etc. and then likely caches whatever it needs.

Note that logging is only performed every 10 minutes so I assume the storage cost of those logs isn’t horrible. That log data may be stored in a database as you describe, flat files, or some other data structure. No idea, but it likely compacts, compresses and/or dedups well.

The graphs themselves look to be rendered locally on the phone app given how they render and given the delay it takes to draw

There’s many details of the architecture that aren’t known, but it would surprise me to find out they were spending $600k/yr on cloud services given the amount of storage and compute my org uses and how much I spend on it.

 

[BeanAnimal]

Hydros and Neptune would be losing money (lots of!) on every controller sold with a cloud solution you propose and yet they are in business making money.

The paradox and how to overcome it is precisely part of the discussion -- not just in this space, but in many verticals where SaaS is not subscription based and there is no RMR o



When I did spec, buy and manage infrastructure for SMB clients (for 20 years) -- most of my work came because I was able to bid jobs with the the appropriate hardware for the scale needed. I did not advertise, customers came to me by referral -- often with somebody else's overblown SPEC in hand, looking for a second opinion.

I treated my customers money like it was my money when I was spending it, and billed them as if I were the one being billed. I shuttered my IT business because I did not want to be in the MSP space. It is slimy. It forced me from billing for consulting and fair labor to reliance on dozens of "resold" subscriptions to be profitable. I went from being the "good guy" with an honest invoice to the guy responsible for dozens of overpriced monthly subscriptions for everything under the sun. Lower margin, higher invoices and dismayed clients who didn't blame me (at all) but loathed technology due to the cost.

I don't do that type of work anymore. For the last 10 years I have designed built and managed internal infrastructure -- where every penny counts and I am accountable for it.



If your frame of reference for business computing is a QNAP NAS sitting on a filing cabinet, then we’re not even talking about the same class of infrastructure, reliability, or service -- especially when discussing SaaS workloads that support thousands of live devices and paying customers. That analogy might make sense for home labs or small office backups, but it has zero resemblance to the kind of scalable, distributed computing required for modern IoT platforms.

 

[BeanAnimal]

Logs/telemetry do look to be stored in the cloud as they are not available in app Bluetooth mode. I assume the hydros devices themselves send their log data to cloud periodically over whatever protocol used and the phone app accesses those logs (when cloud services are available) when logs are viewed or when needing to perform graphing etc. and then likely caches whatever it needs.

Correct: Devices handle polling, setpoints, and operational logic. Telemetry is likely MQTT. Config and user inputs come from the cloud.

The app likely acts as an MQTT client when open, or just hits a cloud UI (web app) over HTTPS/WebSockets. Either way, the cloud is the control hub, not just storage.

Note that logging is only performed every 10 minutes so I assume the storage cost of those logs isn’t horrible. That log data may be stored in a database as you describe, flat files, or some other data structure. No idea, but it likely compacts, compresses and/or dedups well.

No chance of flat files.

Likely uses multiple databases. Telemetry should would use a time-series DB for efficient storage and fast queries. Config/state data would be ideal in Mongo, though the actual stack could vary. Mongo is still low adoption compared to Maria, Postgres or RedShift, etc.

The real isn't data storage size or date volume. The issue is compute resources (IOPs, CPU, and memory) needed to ingest, query, and serve that data live to thousands of concurrent users and devices

The graphs themselves look to be rendered locally on the phone app given how they render and given the delay it takes to draw

Two basic models

1 – Client-side rollup
SaaS runs a large query, dumps raw data to the phone, and the app does local rollups.
- Pros: SaaS load is bursty but lower overall. Some offline capability.
- Cons: High bandwidth use. Slower during heavy query periods.

2 – Server-side rollup
SaaS runs each chart query, sends only the pre processed data.
- Pros: Low bandwidth, faster if SaaS is sized properly.
- Cons: Higher, steadier SaaS compute load. No offline data.

Each has trade-offs. Real SaaS platforms are usually built around#2

There’s many details of the architecture that aren’t known, but it would surprise me to find out they were spending $600k/yr on cloud services given the amount of storage and compute my org uses and how much I spend on it.

Correct, a lot is not known. I can't publicly disclose what we spend. I can say that I am not shocked at $100K/month AWS workloads.

I can tell you some real "duct tape and prayers" stories where I have first hand experience, some of them shocking and at large national tech providers you know by name and use.

So who knows what they spend. I know what they "should" be spending if they are serious about speed and reliability, or using a known IoT white label.

Edited for clarity

 

[BZOFIQ]

I think (not know, because I have not seen their cost) that Neptune (or CV/Hydros) would NOT give away their services if there were getting it at the astronomical cost you project. There are plenty of people where Neptune profited 100-200 bucks 16 years ago from a sale of a classic unit who still have full access to the AF component. This would not have been feasible and they would have closed shop long ago.

I feel that when you propose solutions in this thread you think everyone is Chase, Northwell Health or something at similar level. Neptune is miniscule and so are their solutions. You don't need cloud based computing and storage at multiple data centers around the globe with load balancers, failovers, etc, etc, etc to log such basic data of few thousand controllers. Neptune has simple and affordable solution to provide such simple service.

I have monitoring solutions in place that log far, far more data then a well expanded Apex ecosystem. The state of these monitors changes more often that any of the probes or outlets on an APEX unit. To log this data very little resources are needed and I keep years worth of data on hand. Years worth of data that fit on a small SD card.

I worked at a large charity years ago, money was no issue to them. Server room decked out in half a million dollar hardware from top players; perhaps more. The more money they had the more complicated things got. With more complications, more had to be spend to keep things running. Moved away to a private sector where money counts and we had more capabilities on half a rack for less than $30-40K.

 

[Dom_P]

Correct: Devices handle polling, setpoints, and operational logic. Telemetry is likely MQTT. Config and user inputs come from the cloud.

The app likely acts as an MQTT client when open, or just hits a cloud UI (web app) over HTTPS/WebSockets. Either way, the cloud is the control hub, not just storage.


No chance of flat files.

Likely uses multiple databases. Telemetry should would use a time-series DB for efficient storage and fast queries. Config/state data would be ideal in Mongo, though the actual stack could vary. Mongo is still low adoption compared to Maria, Postgres or RedShift, etc.

The real isn't data storage size or date volume. The issue is compute resources (IOPs, CPU, and memory) needed to ingest, query, and serve that data live to thousands of concurrent users and devices


Two basic models

1 – Client-side rollup
SaaS runs a large query, dumps raw data to the phone, and the app does local rollups.
- Pros: SaaS load is bursty but lower overall. Some offline capability.
- Cons: High bandwidth use. Slower during heavy query periods.

2 – Server-side rollup
SaaS runs each chart query, sends only the pre processed data.
- Pros: Low bandwidth, faster if SaaS is sized properly.
- Cons: Higher, steadier SaaS compute load. No offline data.

Each has trade-offs. Real SaaS platforms are usually built around#2


Correct, a lot is not known. I can't publicly disclose what we spend. I can say that I am not shocked at $100K/month AWS workloads.

I can tell you some real "duct tape and prayers" stories where I have first hand experience, some of them shocking and at large national tech providers you know by name and use.

So who knows what they spend. I know what they "should" be spending if they are serious about speed and reliability, or using a known IoT white label.

Edited for clarity

Yes cloud costs can get out of control quickly if designs lend themselves to scale that way.

I don’t know if what you describe as a technical solution is what they use. Maybe someone from CV will chime in and let us know… doubtful I know

If I were to venture a guess however based on the addressable market they serve and the likely revenue they derive I don’t think they spend $600k/yr on hydros cloud services. It might be some white labeled SaaS product or maybe they rolled their own but that kind of cost doesn’t seem in line with what they likely derive in profit on these devices on a yearly basis.

I mean lets say you are correct and they have a total of 10k devices out there that they've sold over the last few years and say people have 3 devices in their collectives on average then they are looking at some 3333 Hydros clients. Let’s say a quarter of them buy a maven. That’s ~800 devices. Is that reasonable, I don’t know, but let’s go with it for now. Putting aside taxes, etc then that means they would need to generate $750 per Maven user per year (or $62.5/month) in profit to pay for a $600k cloud bill if they wanted to pay for it all on the back of reagent profit. Now I know all those numbers are guesses but does that seem a reasonable profit number for these reagents? It doesn’t to me. It seems high, but who knows?

Also I don’t think most people would pay $750 a year in subscriptions to pay for their aquarium controller should they choose to move to a subscription model. So $600k just seem higher than I would expect

But again it’s all conjecture.

 

[METZCOOLS]

Enjoyed reading the last 2 pages but can you guys talk in plain english?

 

[Dom_P]

Enjoyed reading the last 2 pages but can you guys talk in plain english?

Apologies for the technobabble

 

[BeanAnimal]

I think (not know, because I have not seen their cost) that Neptune (or CV/Hydros) would NOT give away their services if there were getting it at the astronomical cost you project

Neptune sells consumables at considerable margin and is part of a much large enterprise. The overall volume and portfolio allow for lower individual margins and even loss leaders.

CV is similar and it is no coincidence that they wish to DRM consumables.

The bigger picture is math. The longer you are a user but not a customer, the more money you cost them, in a literal sense. If they cleared $300 on your purchase and you use it for 80 months, that is 80 months of hosting they are paying for.

There are plenty of people where Neptune profited 100-200 bucks 16 years ago from a sale of a classic unit who still have full access to the AF component. This would not have been feasible and they would have closed shop long ago.

See my first comment. If they were not rolled up, and started selling consumables then they would not likely be in business or at least with the same business model.

I feel that when you propose solutions in this thread you think everyone is Chase, Northwell Health or something at similar level.

Their hosting costs have many more zeros per month, often taking multiple data centers themselves.

Neptune is miniscule and so are their solutions. You don't need cloud based computing and storage at multiple data centers around the globe with load balancers, failovers, etc, etc, etc to log such basic data of few thousand controllers. Neptune has simple and affordable solution to provide such simple service.

You don’t need global servers, but you should have at the bare minimum realtime backups in at least a second geo zone. Anybody serious will be balanced over two geo zones with route failover.

You absolutely need load balancing and failover for a service that handles thousand semi-persistent concurrent data connections.

Again, there are a lot of tied services here even at small scale. These aren’t CPanel web servers running Apache and Wordpress.

I have monitoring solutions in place that log far, far more data then a well expanded Apex ecosystem. The state of these monitors changes more often that any of the probes or outlets on an APEX unit. To log this data very little resources are needed and I keep years worth of data on hand. Years worth of data that fit on a small SD card.

That is simple data storage and logging The difference here is compute power needed to manage and deliver the data and services concurrently. Disk space is irrelevant.

I worked at a large charity years ago, money was no issue to them. Server room decked out in half a million dollar hardware from top players; perhaps more. The more money they had the more complicated things got. With more complications, more had to be spend to keep things running. Moved away to a private sector where money counts and we had more capabilities on half a rack for less than $30-40K.

I see it all of the time. It is not always the IT vendors fault, sometimes people won’t take no for an answer, other times they want to blow money for tax reasons, stature, or business relationships or accreditation.

I was asked to put in $30k worth of Palo-Alto for a client when $5K worth of any other firewall would have been overkill. The $30k made their insurance underwriter and SEC auditor happy… worth every penny.

Reason vary and there are no shortage of bad IT engineers, no different than any other trade.

 

[BeanAnimal]

t they derive I don’t think they spend $600k/yr on hydros cloud services.

I think you missed part of what I posted or missed discussion of ranges of things… but no, that number is not reasonable in the context we are discussing. I think it is likely much lower, but still significant long term.

 

[braaap]

Apex stores your logs/data on the sd card. So that’s a data savings for them.

I don’t think they are spending anywhere near 10k a month for the cloud service. They have the space and ability to spin it up themselves. And I swear someone traced it once and it goes back to their headquarters. But I may be wrong on that.

@JeffB418 might know.

 

[Dom_P]

I think you missed part of what I posted or confused discussion of ranges of things… but no, that number is not reasonable in the context we are discussing. I think it is likely much lower, but still significant long term.

Understood. I’d believe they couldn’t even cover that nut long term as the addressable market is too small and is also somewhat saturated as you have noted previously.

I was looking around and 2 months of branded trident N/P reagents are $60. And alk/ca/mg is $50 for two months. If they want to stay in that range of $55/month for all those tests to remain competitive with Neptune, then even if they derived a profit of $25/mo (which seems unlikely) and there were 1000 mavens in service, then their yearly profit on reagents would be $300k/yr. I still can’t see how that profit supports the cost of a cloud solution as you described even longer term when 1000 units are in service. I have to think whatever they are doing from a cloud perspective is much cheaper.

Also, I only wish one of my Palo firewalls cost $30k . Yearly licensing is even worse. I’m done with Palo moving on to Fortinet.

 

[BeanAnimal]

Also, I only wish one of my Palo firewalls cost $30k . Yearly licensing is even worse. I’m done with Palo moving on to Fortinet.

That was just base, no licenses If you saw the size of the office you would laugh, you can count them without removing socks. It is what the auditors wanted, the law firm recommend and the owner purchased. Happily… along with various other Gartner Magic Quandrant listed expenses.

 

[mmorrison55]

How far back was the last post that was related to the mavens launch? You guys seem to have a lot to talk about as it relates to the network architecture, but it’s way over the heads of most of us who joined this thread to talk about and get information about the mavens release.

Its time for me to unwatch this thread.