How trustworthy are your test kits (Alk/Ca/Mg)? | BRStv Investigates

[randyBRS]

Happy Friday Reefers!

So, what’s more important your test kit’s accuracy or your testing consistency?

Let’s see if we can help with over 80 testing results from 5 different reefers (including Ryan and me!) ;Happy

 

[hotashes]

@brandon429 ^^^^^^^^^

A.

 

[Pete_the_Puma]

I'm not sure I agree with defining accuracy as the average of testing parameters and then comparing the tests themselves to that average. Seems like a bit of a redundancy, or self fulfilling prophecy to me (I know there is a scientific specific term for this, not confirmation bias but something similar that eludes me at this time).

Perhaps a number of outside ICP tests averaged would have been more accurate.


Edit: 11:24AM: I think "Circular reasoning" comes closest to what I'm trying to describe here.

 

[jschultzbass]

I'm not sure I agree with defining accuracy as the average of testing parameters and then comparing the tests themselves to that average. Seems like a bit of a redundancy, or self fulfilling prophecy to me (I know there is a scientific specific term for this, not confirmation bias but something similar that eludes me at this time).

Perhaps a number of outside ICP tests averaged would have been more accurate.

I was confused on that part as well. Maybe make a reference solution or get GHL's 7.5 dkh reference solution as your baseline.

 

[TexasTodd]

Always like the vids. Would have been nice to do an ICP on the same water sample.

 

[shred5]

both are important.

I have had many bad test kits over the years.

 

[Danj]

I was confused on that part as well. Maybe make a reference solution or get GHL's 7.5 dkh reference solution as your baseline.

Or even the Fauna Marin multireference solution that @ZackBRS said BRS had tested and would be soon be offering for sale

Product request - Alkalinity standard

Salifert has been including a reference for a few years now in their ALK test kits. I haven’t seen an alk reference in Salifert alk test for several years

www.reef2reef.com

 

[User1]

Good job as always. Thanks for taking the time to put this together.

@randyBRS Wow - just caught this from the video. TRIDENT IS IN STOCK - QUICK! Add to the cart!!!! Everybody is probably thinking what???? Some of us are still playing Dora the Explorer and looking for the mystical piece of equipment Again - thanks Randy - enjoyed it.

 

[Hemmdog]

 

[csb123]

I always appreciate your videos. BRS has been of great service to the whole reefing community. But, without a dKH standard to test, any discussion about accuracy is nullified. The comparisons regarding the test precision are useful.

 

[Sleepydoc]

Totally agree with @Pete_the_Puma. The only way to know how accurate the tests were would have been to have the water tested by a reference lab. Without doing that, it’s entirely possible that the ‘outlier’ was the only accurate test of the lot.

 

[MnFish1]

I was confused on that part as well. Maybe make a reference solution or get GHL's 7.5 dkh reference solution as your baseline.

Accuracy is determined by comparing results to a 'known gold standard' and measuring the difference. Not by comparing to a mean of all the results.

 

[MnFish1]

Usually, love your videos - this one has a couple mistakes -

1. Accuracy is determined by comparison with a known standard (not at all how it was done in the video) - the first part of the video really doesnt tell anything.
2. The second part comparing the variability of each test - is also done incorrectly - You should have the same person doing the same test 5 times - and seeing which one has the lowest deviation from the mean of those 5. Adding different testers adds another variable to the test - which makes it all uninterpretable. (i.e. Precision of the test).
3. There is also no rationale to remove the highest and lowest results - in fact doing this invalidates the whole thing.

 

[vangvace]

@MnFish1 They have made the accuracy mistake in a few of the recent videos now. The Trident video also sticks out talking about how accuracy increases as more tests are completed, not to a reference. I would like to have seen each person cycle through all the tests 5 times and show the results per person per test. That would show test to test as well as hobbiest to hobbiest. Think about the LaMotte scores they didn't show.

I find it interesting they don't have the Kh Director listed on their website under test kits. Understandable but interesting.

 

[User1]

@MnFish1 They have made the accuracy mistake in a few of the recent videos now. The Trident video also sticks out talking about how accuracy increases as more tests are completed, not to a reference. I would like to have seen each person cycle through all the tests 5 times and show the results per person per test. That would show test to test as well as hobbiest to hobbiest. Think about the LaMotte scores they didn't show.

I find it interesting they don't have the Kh Director listed on their website under test kits. Understandable but interesting.

With regards to the trident wouldn't the included calibration be a reference? Few people have also used it to compare their standalone test kits.

With regards to the Kh Director, I do not believe they are using it whereas they are using the Trident on several tanks. I do not know how long it took them to put the video together but if it was a short period or window then that could explain why it wasn't there in the list.

Have a good day!

 

[MnFish1]

With regards to the trident wouldn't the included calibration be a reference? Few people have also used it to compare their standalone test kits.

With regards to the Kh Director, I do not believe they are using it whereas they are using the Trident on several tanks. I do not know how long it took them to put the video together but if it was a short period or window then that could explain why it wasn't there in the list.

Have a good day!

Yes. But - They didnt use the Trident as the 'standard' - unless I misunderstood the video?

 

[eschulist]

Averaging test results to claim that is the correct value is a really bad scientific method to come to conclusions.

Having lab certified water or reference solution with known values to run all the manual tests against is the only way. It’s very possible one or two of the outlining results was actually very accurate and the other test kits were over or under reporting the true value.

There is probably something better out there to test against, and it’s probably very expensive, but the Trident Calibration Solution would have at least been a more reasonable baseline. It has known values that you can compare to. I’ve ran my Hanna using the calibration solution as my test sample and my Hanna checker was overshooting by almost a full dkh. I trusted it for so long when it told me 7-8 dkh but in actuality I was really in the 6-7 range and was running levels on the edge of too low.

The Trident can hit the calibration solution numbers almost exactly after it’s been calibrated. The last necessary info is just proving that the solution is exactly correct when it says it has the printed values on the bottle.

Compare to the values on the bottle listed above.

 

[User1]

Yes. But - They didnt use the Trident as the 'standard' - unless I misunderstood the video?

I honestly do not remember. I thought they focused more on the manual tests and not the automated kind and just threw it in there.

 

[Quietman]

I'm actually fairly impressed with the consistency between test kits shown here. The biggest differences seem to be due to readability or process complexity. Focusing on accurate readings can overshadow the fact the most test kits seem to produce reasonable results (less than 10% deviation in almost all cases) for the price (understanding that accuracy here is based on deviation from average and not a tested reference). Also found it interesting that those who used the additional equipment (pipettes, stirrers, etc) weren't always significantly closer or farther away from the average. Always debated with myself whether that investment would be worth it....doesn't seem to be.

That said I think consistency is key assuming that I've picked an easy for me to read and follow test kit. I'll just keep doing the best I can to do every test the same way, every time with the same test kit and ensuring it's stored in proper environment and within expiration dates.

 

[vangvace]

With regards to the trident wouldn't the included calibration be a reference? Few people have also used it to compare their standalone test kits.

With regards to the Kh Director, I do not believe they are using it whereas they are using the Trident on several tanks. I do not know how long it took them to put the video together but if it was a short period or window then that could explain why it wasn't there in the list.

Have a good day!

They used the Trident as just another test to include in the averages. They even dismissed the low Trident Mg results "as not enough tests". If they would have compared it to a calibrated Trident it would have still been flawed, but not as much as the the average of averages. Using a test standard fluid or comparing to an ICP test would have been even better. Another way to think about it is that they took multiple torque wrenches and technicians to get close enough for that bolt on your aircraft versus tighten to T.O. standard with a wrench that just came back from PMEL.

I have seen a few threads of people comparing Trident to kit X using tank water, but I have missed Trident vs kit x using the remainder of the calibration fluid. (aside from 2 posts down from yours. I'm glad I read the whole thread before replying ) I'm really excited for Trident users once that type of confirmation starts making their rounds.

For the Director, it is more of it missing from the website under test kits than not being in the video. If it was linked to test kits it might get an uptick on sales for them since it is a Alk tester.