"Biodiversity is dead, long live biodiversity" 10 month microbiome data from BRStv.

[areefer01]

I don’t think so.

Me neither. That is why I didn't understand your post but will admit maybe I misunderstood. Thanks for the clarity.

 

[Dan_P]

Me neither. That is why I didn't understand your post but will admit maybe I misunderstood. Thanks for the clarity.

I think you understood correct,upy.

I appreciated the succinct argument for not putting too much time and effort into analyzing data from a flawed, or maybe just incompletely designed and executed experiment.

 

[taricha]

I would say that he saw a potentially interesting but untold story. From the length of the post I’d say his story provided interesting reading and stimulated a good discussion.

I've learned quite a few things anyway.


Just to be a little more clear about my point of view: I don't have to accept most of the conjectures in the BRS videos, nor do I have to think everything measured by aquabiomics is meaningful in order to think that this large set of data is worth looking at.

If I were just aiming to illustrate flaws in the tests and the conclusions, it would be a short and pointless thread.

There is a reason why 99% of research scientists who do this for a living would see these results and say we can't say we have learned anything from them. If they repeated each of the designs over and over and then quantified the results, then we could run stats (can't say which test because we don't have the distribution of the data after repetition) and get some values that would be useful. But one and done runthroughs don't tell us anything. For example, if it rained all day on Tuesday here, and never again for the year, but Tuesday was my only sample, I would think that I live in a rainforest by seeing that 100% of the time it was raining. So each of these results could have been that Tuesday, or maybe two Tuesdays, and so on.

I appreciate the frequent cautions about concluding too much from single tests - really, I do. And I'll continue to avoid doing such a thing, like I said earlier....

Right, the main ideas I want to pull from it are not conclusions about any one tank or start method. I think there's a lot of things we can get from the data that are more general - of the form "all the tanks subject to this effect, followed this path." Those are much more interesting to me than specifically how well a bag of oceandirect live sand did, etc.

Let me explain a bit more how I'm looking at the data. (though it's totally fine to continue to disagree with my perspective.)

Let's say we take 12 otherwise similar test subjects and send them out each one in the rain wearing a different color pair of socks, then we get back the totally shocking data that all of them got wet.

Scientist A says: we can conclude nothing because we have no replicates - we needed two or 3 people in blue socks, two or 3 in red socks, same for green etc.

Scientist B says: Why? We already have 12 replicate results that say sock color provides no protection against getting wet in the rain.

The BRS data I'm highlighting is basically like that. I'm focusing on areas where the tank results tracked in very much the same way across a number of initial conditions. They may have been intended to end up differently - but that ended up as replicates with respect to the data in question.

To be concrete (Idea 3): zero of 12 tanks achieved rapid diversity. They were sampled 4 times over 15 weeks. Only one hit 50th percentile diversity by week 15. How many more replicates of these starting setups do you want to see to be able to conclude that these materials set up in this way are unlikely to rapidly ( ~4weeks) yield a high diversity tank?
I say we already have 12 replicates that tell us this is the case.

 

[sixty_reefer]

I agree that this is being looked into way too much. What I see when I look at their results is that I have learned nothing from the video. Why? Because its not a good experiment by the standards of what is required to be confident in the results. I see it all as events that occurred by chance in unusual environments. The length of this thread is concerning me because again, people in this hobby have little to no experience with properly designing experiments and statistically analyzing results. There is a reason why 99% of research scientists who do this for a living would see these results and say we can't say we have learned anything from them. If they repeated each of the designs over and over and then quantified the results, then we could run stats (can't say which test because we don't have the distribution of the data after repetition) and get some values that would be useful. But one and done runthroughs don't tell us anything. For example, if it rained all day on Tuesday here, and never again for the year, but Tuesday was my only sample, I would think that I live in a rainforest by seeing that 100% of the time it was raining. So each of these results could have been that Tuesday, or maybe two Tuesdays, and so on. These are fun anecdotal observations and we need to stop reading into these as if we have any real results. This whole error is, imo, the main issue when hobbyists attempt to move their hobby thoughts into scientific thought.

Not sure if I understand your analogy here, there were 12 different set ups tested at 4 diferentes time periods meaning a total of 48 tests have been performed (51 if we include the older system tested in brs).

they all had positive and downfalls although all 12 tanks had plenty of positive data that aquarists can use.

 

[Dan_P]

I've learned quite a few things anyway.


Just to be a little more clear about my point of view: I don't have to accept most of the conjectures in the BRS videos, nor do I have to think everything measured by aquabiomics is meaningful in order to think that this large set of data is worth looking at.

If I were just aiming to illustrate flaws in the tests and the conclusions, it would be a short and pointless thread.



I appreciate the frequent cautions about concluding too much from single tests - really, I do. And I'll continue to avoid doing such a thing, like I said earlier....



Let me explain a bit more how I'm looking at the data. (though it's totally fine to continue to disagree with my perspective.)

Let's say we take 12 otherwise similar test subjects and send them out each one in the rain wearing a different color pair of socks, then we get back the totally shocking data that all of them got wet.

Scientist A says: we can conclude nothing because we have no replicates - we needed two or 3 people in blue socks, two or 3 in red socks, same for green etc.

Scientist B says: Why? We already have 12 replicate results that say sock color provides no protection against getting wet in the rain.

The BRS data I'm highlighting is basically like that. I'm focusing on areas where the tank results tracked in very much the same way across a number of initial conditions. They may have been intended to end up differently - but that ended up as replicates with respect to the data in question.

To be concrete (Idea 3): zero of 12 tanks achieved rapid diversity. They were sampled 4 times over 15 weeks. Only one hit 50th percentile diversity by week 15. How many more replicates of these starting setups do you want to see to be able to conclude that these materials set up in this way are unlikely to rapidly ( ~4weeks) yield a high diversity tank?
I say we already have 12 replicates that tell us this is the case.

Dang! You changed my perspective again!! That is the teacher in you. Maybe Taricha should be changed to Socrates

Strictly speaking, there are no replicates in the BRS study and the controls are inadequate to easily conclude much from the study. There are replicates in the sense that repeated car crashes at an intersection with no stop sign is a replicate event. There is data present but the analysis of it takes on the form of an investigation rather than a statistical analysis. The BRS experiments have a messy design, not even qualifying as a highly fractional factorial design. They are car crashes. As a result what conclusion or narrative we tease out from that data will be speculative and potentially contentious. As long as we keep this mind, we can have fun rationalizing the wreckage.

I have participated on teams investigating safety and quality incidents in chemical manufacturing plants and fact finding is key to understand what, why, when, where and how stuff happens. When I look at the work you’ve done, I see excellent fact finding and a clever way of interpreting the facts which may help us gain insight into the BRS Biome Incident.

I am almost finished with a data visualization project to add to your investigation.

Dan

 

[Thales]

I've learned quite a few things anyway.


Just to be a little more clear about my point of view: I don't have to accept most of the conjectures in the BRS videos, nor do I have to think everything measured by aquabiomics is meaningful in order to think that this large set of data is worth looking at.

If I were just aiming to illustrate flaws in the tests and the conclusions, it would be a short and pointless thread.



I appreciate the frequent cautions about concluding too much from single tests - really, I do. And I'll continue to avoid doing such a thing, like I said earlier....



Let me explain a bit more how I'm looking at the data. (though it's totally fine to continue to disagree with my perspective.)

Let's say we take 12 otherwise similar test subjects and send them out each one in the rain wearing a different color pair of socks, then we get back the totally shocking data that all of them got wet.

Scientist A says: we can conclude nothing because we have no replicates - we needed two or 3 people in blue socks, two or 3 in red socks, same for green etc.

Scientist B says: Why? We already have 12 replicate results that say sock color provides no protection against getting wet in the rain.

The BRS data I'm highlighting is basically like that. I'm focusing on areas where the tank results tracked in very much the same way across a number of initial conditions. They may have been intended to end up differently - but that ended up as replicates with respect to the data in question.

To be concrete (Idea 3): zero of 12 tanks achieved rapid diversity. They were sampled 4 times over 15 weeks. Only one hit 50th percentile diversity by week 15. How many more replicates of these starting setups do you want to see to be able to conclude that these materials set up in this way are unlikely to rapidly ( ~4weeks) yield a high diversity tank?
I say we already have 12 replicates that tell us this is the case.

Don't let perfection be the enemy of the good, but be sure to point out what the good is missing.

 

[areefer01]

Don't let perfection be the enemy of the good, but be sure to point out what the good is missing.

Tee-shirt. Just saying. Stay amazing.

 

[taricha]

Strictly speaking, there are no replicates in the BRS study and the controls are inadequate to easily conclude much from the study. There are replicates in the sense that repeated car crashes at an intersection with no stop sign is a replicate event. There is data present but the analysis of it takes on the form of an investigation rather than a statistical analysis. The BRS experiments have a messy design, not even qualifying as a highly fractional factorial design. They are car crashes. As a result what conclusion or narrative we tease out from that data will be speculative and potentially contentious. As long as we keep this mind, we can have fun rationalizing the wreckage.

Fair enough. I can agree that I'm stretching or abusing the term "replicates" in this sense.
And doing so increases the possibility of getting fooled (as I've already done at least once in this thread). Like you say the conclusions drawn will be somewhat speculative and contentious.

To sort of refine my goal in "rationalizing the wreckage", we're hunting for ideas that are worthy of consideration and a more rigorous treatment, vs those that should probably be discarded and are unlikely worth expending much future experimental effort on.
Or to state it the context of a discussion forum community - there's a bunch of ideas floating around on R2R about the bacterial communities / diversity etc, people often make tank decisions based on these ideas. Some of these ideas should be amplified, some should be tweaked, and some should be shot down.

I think the BRS data provides plenty of material for those purposes. (with the understood caveats about fooling ourselves)

 

[Dan_P]

Full disclosure: @taricha provided a huge helping hand pulling this data together and editing my mistakes converting colored bars into numbers for my johnny-come-lately contribution. There has also been a bunch of email discussions about the data.
——————————————

As promised, here is a data visualization of the most populous bacteria families (6) that made up 90% or more of the families found in each BRS experiment at each of the four time points. What is actually plotted is the difference between the population make up in the control at each time point minus the population make up of each treatment. There is a line for each family and a graph for each of the eleven experiments. A horizontal line indicates that the control and treatment are behaving similarly. If they are behaving exactly alike, the horizontal line lies on top of the X-axis. A negative number means the treatment has more of that family than the control. A positive number means the control has more of the family. I use this method to visualize what is happening relative to the control, not so much to explain anything. The explaining comes later after I have better grasp of how the treatments are behaving.

The plots tells us something about the succession of families in each treatment. Some successions seem to track the control closely while others follow a different pattern. I could not invent a reason for the differences but I did not expect to because of the complexity of the bacterial network interactions. Way more complicated than tracking billiard balls after the break. The 15 week data gave me an idea that the communities might be converging, and maybe, it does not matter so much what the starting point is. I am not concluding anything, but rather formulating ideas to test. Another idea is that the bacteria community might not be exerting a strong influence on nuisance algae growth. It probably has a role but the nature of the bacteria-algae interaction remains hidden. Another idea is that without starting the aquarium with nitrifying bacteria, the succession of bacteria in the BRS tanks might be different than what you and I would see. This seems like an easily testable idea. Another idea is that every aquarium probably started with a unique population of algae and would likely influence the bacteria population through the depletion of nitrogen. In turn this means each system’s microorganism heterotrophic/photoautotrophic biomass would vary, affecting how future nutrients are distributed between the trophic levels and how well each bacteria family grew.

I have one more visualization to share which focuses only on the 15 week data where I wonder about the population convergence and data clustering.

 

[ScottB]

Let me expand a little more on Balance Score , Diversity and visual judging of the systems.
From watching the videos I tried to give each tank a subjective "uglies score" based on amount and apparent "health" of the nuisance growth at week 15. I rated each tank zero to 10, with zero being spotless and 10 being the most uglies.

You can see how the diversity doesn't reflect the eyeball assessment of uglies (not that it's meant to, but people wonder if it does). The balance on the other hand does seem to correlate decently with the visual assessment of uglies.

Additionally in terms of correlations - the systems that dropped most notably in balance percentile from weeks 10 to 15 also had an increase in uglies during that time period.

12 - biobrick
9 - gulf wet rock
5 - Coral
and 7 - tank rock/sand
...all had notable increases in nuisance growth from weeks 10-15, which can help to explain one way a system that has better "balance" score can regress on that metric during that time frame.

I am obviously a little late to this party thread, so trying to catch up. While it is an interesting discussion and test, it really only covers the "break in" period where the different surface competitors are battling it out.

Another potential context -- and one I am continually involved in over on the SPS Forum -- is the one where five people per day ask "When will I be able to keep acropora alive in my tank?" I typically answer the question with a question: "Was this a dead rock start?" and then go from there.

I am only on page 5 so maybe this gets covered somewhere later.

 

[taricha]

Another potential context -- and one I am continually involved in over on the SPS Forum -- is the one where five people per day ask "When will I be able to keep acropora alive in my tank?" I typically answer the question with a question: "Was this a dead rock start?" and then go from there.

The below isn't an answer - but it's a comment on this topic.

One thing both views (diversity and balance) show is that there are quantifiable answers to a question that we often pose. Cycling takes 1-2 weeks, but we feel like tank "maturity" takes much longer - on the scale of months. But we don't have good answers to what we think happens between 2 weeks and a few months. One of those answers is bacterial succession is clearly a much slower process than a couple of weeks.
This is a snapshot of all 4 microbiome tests over 15 weeks on all 12 tanks... No tanks are totally settled across any time frame within this 15 week period. Some are in more flux than others.

There's a couple more things I want to put together a chart to illustrate and that's one of them - showing the degree of change in the tank communities from one test point to the next: from 2 to 4, 4 to 10, and 10 to 15 weeks.
This BRS data doesn't even suggest any answers to the question "when does stability happen?" - but it does make me wonder if that the old school practice of putting live rock in big dark tubs circulating in saltwater for months might not have been crazy at all.

 

[areefer01]

but it does make me wonder if that the old school practice of putting live rock in big dark tubs circulating in saltwater for months might not have been crazy at all.

They needed additional tests both of which require time and money. Live rock and sand being one. The second being what you touched here. Dry rock aged like wine. Test at 3, 6, 9, and 12 months.

Still a head scratcher to me though I'll have to admit. Why try and shorten the journey as reefs take years to mature. Unless of course shorter means selling more gear due to failures or trying to keep up with Mr. Jones.

 

[C4ctus99]

I wonder if TBS or a similar outfit would have a large enough interest in these kinds of experiments to either aid in running them or to donate material for the tests.

If the tests turn out beneficial to them, it’s good advertising. If they don’t… I dunno

 

[ReefGeezer]

They needed additional tests both of which require time and money. Live rock and sand being one. The second being what you touched here. Dry rock aged like wine. Test at 3, 6, 9, and 12 months.

Still a head scratcher to me though I'll have to admit. Why try and shorten the journey as reefs take years to mature. Unless of course shorter means selling more gear due to failures or trying to keep up with Mr. Jones.

12 months just gets you to the cheap wine stage for dry rock. LOL.

 

[Koleswrath]

12 months just gets you to the cheap wine stage for dry rock. LOL.

I think this thread has shown that live rock might just be cheap wine in a fancy bottle with a high price tag.

 

[taricha]

As promised, here is a data visualization of the most populous bacteria families (6) that made up 90% or more of the families found in each BRS experiment at each of the four time points. What is actually plotted is the difference between the population make up in the control at each time point minus the population make up of each treatment.

Here's one way to slice Dan's data.

The red circles show the treatments that stayed closest to the Control (Dry Rock & Sand) for the first 2 measurement points - 2 and 4 weeks.
Real Reef (artificial rock cured in seawater)
160 water (got 100% of initial water from the BRS 160 gallon tank)
and to a lesser extent, 900 Brick (the biobrick out of the sump of the BRS 900)

The Blue circles show those 3 treatments that ended the closest to the Control during the last two measurement points - 10 and 15 weeks.
Ocean Direct (bagged live sand)
Dark Rubble (rubble from an established tank placed in the dark overflow)
and the Real Reef (again)
(and others converged nearly as much, but I drew the line at 3 closest.)

One way to think about this is that the treatments that were similar to control in the first 2&4 weeks maybe didn't have a strong initial push in changing the early community from what it looks like when you just put clownfish in a dry rock/sand tank and feed them.
And likewise, those that converged to something very close to the dry rock control at the end may not have had a strong lasting push on how the community developed.

And the bag of ocean direct sand was in both groups.

 

[C4ctus99]

Here's one way to slice Dan's data.

The red circles show the treatments that stayed closest to the Control (Dry Rock & Sand) for the first 2 measurement points - 2 and 4 weeks.
Real Reef (artificial rock cured in seawater)
160 water (got 100% of initial water from the BRS 160 gallon tank)
and to a lesser extent, 900 Brick (the biobrick out of the sump of the BRS 900)

The Blue circles show those 3 treatments that ended the closest to the Control during the last two measurement points - 10 and 15 weeks.
Ocean Direct (bagged live sand)
Dark Rubble (rubble from an established tank placed in the dark overflow)
and the Real Reef (again)
(and others converged nearly as much, but I drew the line at 3 closest.)

One way to think about this is that the treatments that were similar to control in the first 2&4 weeks maybe didn't have a strong initial push in changing the early community from what it looks like when you just put clownfish in a dry rock/sand tank and feed them.
And likewise, those that converged to something very close to the dry rock control at the end may not have had a strong lasting push on how the community developed.

And the bag of ocean direct sand was in both groups.

Interesting how the orange line peaks in every single other tank, before leveling out close to control. With the exception of aqua forest and ocean direct, the yellow line had a much less pronounced inverse

 

[Dan_P]

Here's one way to slice Dan's data.

The red circles show the treatments that stayed closest to the Control (Dry Rock & Sand) for the first 2 measurement points - 2 and 4 weeks.
Real Reef (artificial rock cured in seawater)
160 water (got 100% of initial water from the BRS 160 gallon tank)
and to a lesser extent, 900 Brick (the biobrick out of the sump of the BRS 900)

The Blue circles show those 3 treatments that ended the closest to the Control during the last two measurement points - 10 and 15 weeks.
Ocean Direct (bagged live sand)
Dark Rubble (rubble from an established tank placed in the dark overflow)
and the Real Reef (again)
(and others converged nearly as much, but I drew the line at 3 closest.)

One way to think about this is that the treatments that were similar to control in the first 2&4 weeks maybe didn't have a strong initial push in changing the early community from what it looks like when you just put clownfish in a dry rock/sand tank and feed them.
And likewise, those that converged to something very close to the dry rock control at the end may not have had a strong lasting push on how the community developed.

And the bag of ocean direct sand was in both groups.

Yeah, good call on the trends.

Analogous to your “uglies” score, have you considered a score for the inoculum size? The control might be 0 while the Gulf rock might be a 10, along with the 360 rock and sand. Hard to judge the Indo rock from its appearance. I ask because I have been wondering if community development is inoculum size dependent. Or maybe it has more to do with the level of nitrogen depletion.

 

[sixty_reefer]

Here's one way to slice Dan's data.

The red circles show the treatments that stayed closest to the Control (Dry Rock & Sand) for the first 2 measurement points - 2 and 4 weeks.
Real Reef (artificial rock cured in seawater)
160 water (got 100% of initial water from the BRS 160 gallon tank)
and to a lesser extent, 900 Brick (the biobrick out of the sump of the BRS 900)

The Blue circles show those 3 treatments that ended the closest to the Control during the last two measurement points - 10 and 15 weeks.
Ocean Direct (bagged live sand)
Dark Rubble (rubble from an established tank placed in the dark overflow)
and the Real Reef (again)
(and others converged nearly as much, but I drew the line at 3 closest.)

One way to think about this is that the treatments that were similar to control in the first 2&4 weeks maybe didn't have a strong initial push in changing the early community from what it looks like when you just put clownfish in a dry rock/sand tank and feed them.
And likewise, those that converged to something very close to the dry rock control at the end may not have had a strong lasting push on how the community developed.

And the bag of ocean direct sand was in both groups.

We’re did you guys managed to find the data to fill in the charts? Is there more data anywhere else besides the videos?

 

[taricha]

We’re did you guys managed to find the data to fill in the charts? Is there more data anywhere else besides the videos?

I tricked @Dan_P into thinking it was interesting enough that he took the image of the bar charts from the video and converted the major families into numerical data.