Evaluating the Repeatability and Differences between ICP Tests

[Thales]

On a hobbyist level maybe nothing. If I was trying to land on the moon maybe everything. I guess my question was a bit vague and not my intent. To me it is a variable because one may be able to detect it and the other may not. Or at different levels. The initial tables with the results from vendor A, B, and C we can see that A, and C do not detect Cobalt (for topic of discussion) yet vendor B does. It could be that A and B are MS and C OES. I don't know but there is a difference. It very well could be nothing and just me grasping at the differences in the data.

I can answer this from my sick bed - all vendors that we used were purchased as ICP-OES. If the vendors ran them on a different insturment and didn't let us know...that would make me grumpy.

FWIW, this happened to me once. I sent what was supposed to be and OES test and it was run on and MS. That made me pretty grumpy. I stopped using that service because that seems totally uncool.

 

[sanjay]

I asked a general question on FB on how the ICP vendors measure salinity.

Here is one of the responses I got.

We detect with WTW and use the UNO standard calculation , controlled over a seawater standard

 

[taricha]

Thanks, Sanjay. (I had to look up what that was.) Nice to know that the conductivity measure is giving an answer that's decently correlated to the Chloride.

 

[jda]

They all do at least nitrate, phosphate and salinity with different instruments. One told us that they use Nyos for nitrate. In any case, plasma does not work on these.

Some have said that they do math calculation for total P whereas some have a test kit for that which reduces all types of P to po4 before watching the color change.

I know that these do not use the plasma, but many people order ICP just for these things since they are such hot buttons in the hobby right now. These companies charge for and deliver on these elements too.

 

[taricha]

Gotcha. Thanks. Agree that nitrate, salinity (sometimes), and alk are non ICP tests.
Salinity you could calculate approximately from ICP elements.
Agree to disagree on Phosphate. I think unless they tell you they do photometric, then they are getting P from ICP and multiplying by 3.06. But we can fill another thread on phosphate

 

[taricha]

sidenote on this...

Nice to know that the conductivity measure is giving an answer that's decently correlated to the Chloride.

It's a bit in the weeds on how ICP works, but my interpretation above is wrong.
Salinity and the ICP ions don't seem to be measured by independent methods.

Here's a plot of the reported salinity vs the sum of the major elements reported (S was scaled to give SO4).

The results look so perfectly correlated (especially A and C) that I don't think the ICP values and salinity readings are independently measured.
You can do this two ways: you can take your ICP majors and add them together and report that as salinity. Or you can measure salinity and scale your ICP reported elements to match the salinity measurement. But in either case, one is being calculated based on the other. And A and C are likely doing the exact same thing, maybe B does theirs a hair differently.

 

[Randy Holmes-Farley]

You can see the results in the article for Iodine, Silica and Phosphorus. Three that we have good chemical tests for and ICP is hit or miss on.
What makes you think vendors are doing undeclared chemical tests and passing them off as ICP?

I cannot comment on the "passing them off as ICP" since I have not looked to see, but Christoph at Oceamo says:

concerning the halogens we are measuring fluoride with ion chromatography (not possible with ICP), bromide also with ion chromatography

Flouride low on ICP test

So my ICP test is showing low on fluoride. Corals are mostly good, but thought maybe..... ??? Thought I would try this...

www.reef2reef.com

 

[jda]

Christoph, to his credit, is the only one to have said much:

I do not really understand, please clarify

The methods i posted are the gold standard. We are using ICP (OES or MS), Ion chromatography (cation and anion), photometry for phosphate and titration for alkalinity - salinity is measured using conductometry.

Best regards,
Christoph

I have asked at some shows and get varying answers from Nyos, Salifert, Hannah Eggs, Hach and LaMotte kits. Some of the people in the booths look at me like I told them that a hoard of zombies are headed our way.

 

[Miami Reef]

Some of the people in the booths look at me like I told them that a hoard of zombies are headed our way.

 

[taricha]

Later I'll post a look at the titration elements: Ca, Mg, and K.

Here's the Mg Ca and K, hobby titration elements.

The trend discussed previously holds. The variation within each vendor even across a span of testing on different days is less than the variation from one vendor to another. Very informally, we could say it looks like you can expect to get maybe double the variation if you switch from one vendor to another, vs staying with the same.
If you are looking at Ca or Mg across vendors, +-5% looks pretty typical, and +-10 or 15% on K. In each case you can cut the variation by roughly ~half if you stick with one vendor.

So targeting say 420ppm Ca, you can reasonably expect that values received could be +-5% of the target without you actually being able to tell if it's high or low. So 400 or 440 aren't "low" or "high" they are within expected variation and consistent with the levels actually being perfectly on target.

If you want to elevate K as some do, and want 450 instead of 400. Bad news, +-10 to 15% suggests you probably can't distinguish 450 and 400 with these sorts of tests.

Here's what the next group Br, Sr, B, looks like....

Bromine looks as well-measured as any of the major elements at +-5% across vendors, but Sr and B it looks like +-20% is what the hobbyist needs to accept (well, not all the vendors are doing the same thing on reporting B, but let's not nitpick individual vendors.) In any case, these are probably OK, as long as the hobbyist doesn't try to make noise in the data into something real.

 

[Dan_P]

I love this so much.
especially for this part...
"15 samples were collected for testing within the span of time it took to fill all the sample tubes with water. The samples were shipped arbitrarily, five at a time, on Monday, Wednesday, and Friday of the same week."

We can think of three sorts of variations in our hobby ICP testing (for major elements that are stable).
1. The variation you get from a vendor running replicate samples over and over (back-to-back variability)
2. The variation you get when a vendor runs a replicate sample on a different day, after the machine has gone through a cleaning/recalibration routine. (day to day variability)
3. The variation from choosing vendor A vs B vs C. (vendor to vendor variability)

Lots of people have tested 1 and 3, but the data on 2, day to day variability is super thin. We know 1 - back to back variability - is small. That's been apparent since the Ross & Maupin Triton article years ago, and confirmed many times since.
We know 3 is significantly larger. Sanjay's recent reefbuilders article
gives a great tour of how different results of one sample can look when you send it to different vendors. Many others have noticed (and griped) about the same thing.

What @Rick Mathew , and @Dan_P and I have wondered about is number 2 - day to day variability. What if the larger vendor to vendor variation was really just the day to day variation? I mean if you send replicate samples to the same vendor on different days, does it just look like you sent it to different vendors?
The relevance here is trending elements. We say just stick with one vendor for trending, well what if that vendor looks like a different vendor from one day to the next?

This is a really good data set for answering the question because it includes the back-to-back and day-to-day variability in the vendor data.
Let's look through the first tiers of elements.
This chart is vendors A, B, and C with the means and max/min for each element. All normalized to the combined average across 3 vendors set to 1.

First, you can see that the question of what variations matter the most is clearly answered. Even the day-to-day recalibration/cleaning routines of the vendors don't add enough variation to account for differences between vendors. So systematic variations between vendors is the largest source of variation in data that we see. (The article demonstrates this rigorously by the statistical Tukey's test.)

Thus, the advice to stick to one vendor to trend elements is well-supported.

Secondly, what would people use these particular results in this chart for?
two things: salinity and check major ion balance (Chloride vs Sulfate etc).
The salinity measures look good. The hobbyist can probably live with Chloride +-4% and Salinity +-3% as limits on how well salinity can be measured.
So if your salinity actually was 35ppt, accept that you can get 34 to 36 from ICP, and don't adjust based on expecting tighter than that.
For major ion balance, the Chloride +-4% and Sulfate of +-10% are probably okay too. I don't think anybody would decide to try to adjust ionic balance based on sulfate vs chloride being off by ~10% from an ideal balance.

Later I'll post a look at the titration elements: Ca, Mg, and K.

Great clarification!

Are the percent variations you provide close to one or three standard deviations?

 

[taricha]

Great clarification!

Are the percent variations you provide close to one or three standard deviations?

it looks like the ranges that I gave correspond to about 1.5-2 standard deviations (calculating stdev of an element from all measurements of that element from all 3 vendors).

 

[Dan_P]

it looks like the ranges that I gave correspond to about 1.5-2 standard deviations (calculating stdev of an element from all measurements of that element from all 3 vendors).

OK, thanks.

Maybe a minor point. I would think that we should be roughly doubling your estimates to have a realistic expectation about the variation in results from ICP vendors.

 

[taricha]

Maybe a minor point. I would think that we should be roughly doubling your estimates to have a realistic expectation about the variation in results from ICP vevendors.

Maybe so.
A couple of thoughts Just for conversation. One may be about characterizations of what variations are "typical" (67% in 1 stdev) vs what variations are "possible" (99.5% in 3 stdev).
Another thought here is that most of the variations seen are from the choice of vendor. 3 vendors were chosen. A different choice of 3 vendors or including more vendors may have expanded the variations observed to match your characterization.
Hmmm....

 

[Dan_P]

Maybe so.
A couple of thoughts Just for conversation. One may be about characterizations of what variations are "typical" (67% in 1 stdev) vs what variations are "possible" (99.5% in 3 stdev).
Another thought here is that most of the variations seen are from the choice of vendor. 3 vendors were chosen. A different choice of 3 vendors or including more vendors may have expanded the variations observed to match your characterization.
Hmmm....

HaHa, shows how differently we look at things. I see 1 standard deviation as “good enough for a feel for variation” and 2-3 standard deviations as “what you need to make a decision”.

 

[taricha]

HaHa, shows how differently we look at things. I see 1 standard deviation as “good enough for a feel for variation” and 2-3 standard deviations as “what you need to make a decision”.

Fair enough. I'll go with the widely used two standard deviations, rather than my annoyingly fuzzy statistics of "just eyeball it."

 

[taricha]

Fair enough. I'll go with the widely used two standard deviations, rather than my annoyingly fuzzy statistics of "just eyeball it."

Here's what It looks like using +-2 standard deviations as the marker. All these are normalized so the overall mean of each element is 1.

Here's the major "salinity" elements:
yellow, red and blue are the means and +-2stdev within each vendor - A, B & C.
Black is the +-2 stdev of the measurements across all 3 vendors.
and for additional context/comparison, the gray is +-2stdev calculated from the Sanjay 5-vendor reefbuilders data set.

here's the same exercise for the hobby titration elements: Magnesium, Calcium, Potassium.

For the best measured (Na, Cl, Mg, Ca) among these above groups, you can have +-2stdev within a single vendor of +-3% and across vendors of +-5-10%.
The poorest measured ones in these groups (S & K) 2 standard deviations can look like +-10% within one vendor, and +-15-20% across vendors.


And here's what it looks like for the other >1 ppm elements: Bromine, Strontium, and Boron

Things look less tight here with both variations within the vendors and across vendors increasing.
Note that for Bromine, the +-2stdev across vendors looks small in the current Ross & sanjay data set, but if you expand it to the vendors sampled in the earlier Sanjay 5-vendor 7/12/23 data set, then the differences across vendors increases a lot.
(When we ask how much one vendor typically varies from another, it can matter a lot which vendors you are asking about.)

 

[Dan_P]

Here's what It looks like using +-2 standard deviations as the marker. All these are normalized so the overall mean of each element is 1.

Here's the major "salinity" elements:
yellow, red and blue are the means and +-2stdev within each vendor - A, B & C.
Black is the +-2 stdev of the measurements across all 3 vendors.
and for additional context/comparison, the gray is +-2stdev calculated from the Sanjay 5-vendor reefbuilders data set.

here's the same exercise for the hobby titration elements: Magnesium, Calcium, Potassium.

For the best measured (Na, Cl, Mg, Ca) among these above groups, you can have +-2stdev within a single vendor of +-3% and across vendors of +-5-10%.
The poorest measured ones in these groups (S & K) 2 standard deviations can look like +-10% within one vendor, and +-15-20% across vendors.


And here's what it looks like for the other >1 ppm elements: Bromine, Strontium, and Boron

Things look less tight here with both variations within the vendors and across vendors increasing.
Note that for Bromine, the +-2stdev across vendors looks small in the current Ross & sanjay data set, but if you expand it to the vendors sampled in the earlier Sanjay 5-vendor 7/12/23 data set, then the differences across vendors increases a lot.
(When we ask how much one vendor typically varies from another, it can matter a lot which vendors you are asking about.)

If I use ICP test results to judge how much element to dose or whether to fix an ion imbalance, would you think the uncertainty in the ICP result would cause me to really screw things up or would it be more likely a “so what” event for my fish and coral? Now that I typed this question it looks like a big ask

 

[Thales]

If I use ICP test results to judge how much element to dose or whether to fix an ion imbalance, would you think the uncertainty in the ICP result would cause me to really screw things up or would it be more likely a “so what” event for my fish and coral? Now that I typed this question it looks like a big ask

I don't think we know the answer to that because we don't know how much slop in the imabalance makes a difference, or how much or how little is problematic. Some of the macro ions we know more about through experience, and some of them have a large range of 'so what'.

 

[taricha]

If I use ICP test results to judge how much element to dose or whether to fix an ion imbalance, would you think the uncertainty in the ICP result would cause me to really screw things up or would it be more likely a “so what” event for my fish and coral? Now that I typed this question it looks like a big ask

I think we all are running survivorship biased experiments - we end up with great tanks of the stuff that can handle how we reef and forget about all the stuff that dies with our style of reefkeeping.
When we say "show me your tank" it should be followed up with "beautiful! now show me your drawer of the casualties that didn't survive your tank."
Anyway, If you sent an ICP every 3 weeks and assumed whatever you got back was accurate and adjusted all those >1ppm elements in the charts above by however much ICP said those elements had changed, I don't think the swings would be fatal to anything, probably not even harmful. The worst case scenario here is you swing your Br, Sr, B 20-50% once every 3 weeks. I don't envision anything getting killed off by that.
But that's just a made up guess.