GETTING IT RIGHT! ---MAKE TESTING COUNT BY USING A QUALITY SYSTEM APPROACH

[Rick Mathew]

GETTING IT RIGHT! ---MAKE TESTING COUNT BY USING A QUALITY SYSTEM APPROACH

INTRODUCTION

All of us want healthy and thriving Reef Tanks. This necessarily means excellent water quality. So I ask myself. How do I know the quality of my water? The answer of course is I need to test and measure. That being said, I asked myself some additional questions:

• Do I want to manage my water quality with inaccurate data?
• Do I want to be making adds of vital elements without having accurate tests
• Do I want my measuring devices (pH, ORP, Temperature ETC.) to report accurate data?
• Do I want to make changes in my system with a high degree of confidence that it is the correct thing to do?

The obvious answers to these questions and some others pointed me to look at my water testing from a Quality System Approach. I wanted to work to identify sources of errors in my testing methods and correct them as well as focus my error prevention where it really counts. This would give me a much higher degree of confidence in changes I decide to make and better ensure the health and vitality of my pets.
My journey with quality systems started many years ago in my past life as CTO (Chief Technical Officer) of a multi-national company I inherited the testing responsibility of seven local Quality labs. They were scattered across the globe... all doing their “own" thing and for the most part these were not good things...defiantly NOT High Quality Standards! As you can imagine this caused a lot of issues both internally as well as with our customers...Bad product---lots of returns and excessive costs to name a few. What was I to do that wouldn’t require coming up with a unique solution for each lab?
Quality System to the rescue!!!****** After two looooong years all but one of the labs were accredited for all of their critical customer facing and internal certifications tests...WOW did my life get easier!...Happy boss---Happy Customers---Happy me!
What made the difference? -----Standardized testing with the focus on accuracy and precision

A specific incident in the Reefing community made it clear to me, that a quality systems approach would benefit aquarists.
A fellow Reef Junkie that had a very sad story of almost 3 years of "killing" lots of wonderful pets and spending many hundreds of dollars, I asked him about his tank parameters. He was having the water tested at a LFS (no longer in business) as well as his own "testing". From the results of the tests he was making changes to his system with really bad outcomes! I volunteered to test his water and compare the results to that of the LFS as well as his results. …It is a wonder anything could survive in that water! I spent some time helping him adopt a Quality System Mindset and understand the basic principles of testing (accuracy & precision) and why they are important for the well-being of his pets and encouraged him to do his own testing. After some time his system stabilized and is doing much better...his pets stay alive!

What made the difference? -----Standardized testing with the focus on accuracy and precision in the Reefing Hobby
In 2017 a poll was conducted on Reef2Reef about “Water Testing Frequency” (Link here https://www.reef2reef.com/threads/poll-testing-frequency.322792/.) Almost 700 people responded. 68% of respondents tested at least once a week or more frequently. This to me is a clear indicator that testing is an important part of our hobby. If you frequent the Reef2Reef community you have observed many posts that focus on questions about test and measurement results…Are they correct…Which test kit or results do I believe? Why do test results vary when I repeat a test on the same sample? All of these questions concern measurement accuracy and precision. With so much testing going on it seems important to GET IT RIGHT.
This article will help you answer these questions, identify sources of errors and eliminate them.

Aquarium water testing is an analytical method. Just like any other analytical method it shares the challenge of getting good Accuracy and Precision.

SUMMARY​

We make many types of measurements on our reef systems. These would include but not limited to: Ph, Temperature, Salinity, ORP, O2, Alkalinity ETC. In addition we measure individual elements such as Calcium, Magnesium, Potassium, Iodine, and others. We also measure nutrients, such as ammonia, NO2, NO3, PO4 ETC. There are two primary means by which we make these measurements. In some cases we use probes, pH and Temperature for example. The second way is by colorimetric testing. Colorimetric testing is accomplished by reacting the test water with specific reagents. The reaction produces a color where the intensity of the color is proportional to the level of the element or nutrient present in the water. We then quantify the level by visually comparing the result to a reference chart or as measured electronically with a photometer like a Hanna Checker. There are generally several steps in this process and each is subject to errors that generally accumulate to potentially make our test result unreliable.
The fact that we often make decisions on making changes to our reef system based on these results, having unreliable results is problematic. We may be making changes that are not necessary resulting in over dosing, under dosing, unnecessary water changes and overall less stable conditions for our fish and corals.
I have applied a Quality System Approach so I can have a higher degree of confidence in my measurement results and thus more assurance that the changes I make are the correct ones. The Goal of this series of articles is to share with you a Quality System approach to testing your water so as to ensure a higher level of confidence in your test results and thus the actions you take as a result.
The article will be broken into 4 parts;

Part 1---UNDERSTANDING ACCURACY/PRECISION/ERRORS
WHY A QUALITY SYSTEM APPROACH
Part 2--- COLORIMETRIC VISUAL TESTING –Doing the Chemistry
Part 3--- COLORIMETRIC VISUAL TESTING –Assessment…Looking at the sample
Part 4---COLORIMETRIC INSTRUMENTAL TESTING & ICP TESTING


ACKNOWLEDGMENTS
I would like to personally thank @Dan_P and @taricha for their contributions and support in putting together this series of articles. Their input and direction helped to shape the form and content. It is great to be in community like Reef 2 Reef where quality people like these two are willing to come along side to encourage and support….Thanks for your help!


PART 1

UNDERSTANDING ACCURACY AND PRECISION
Before we jump into the error question we need to define two terms that we will use throughout this article. They are Accuracy and Precision. Both of these apply to our measurement whether we are using a probe, test kit or colorimeter (e.g. Hanna Checker) The focus of this will be on colorimetric testing and not probes but many of the same principles apply. HACH and HANNA have lots of great information on probe calibration, use and maintenance.

Accuracy is how close to the “real value” is the result. Let’s say a sample has a known amount of Magnesium in it (The Real Value). How close does our test measure to the known amount? The closer it is to this “Real Value” the more accurate the test.

Precision is how repeatability is the results. If I repeat the test on the same sample multiple times how close are the results to each other? The closer the results are to each other the better the precision.

We need both Accuracy and Precision. Accuracy without precision requires a large number of tests to get a statically valid result and Precision without Accuracy gives consistent results but results that are incorrect.
Both Accuracy and Precision can be measured.

For Accuracy we will use Relative Accuracy (RA). This value is expressed as a %. Any value less than or greater than 100% indicates a level of error. The relative Error can then be found by subtracting the Relative Accuracy from 100. The math looks like this:
(Actual value) - (Actual value – Measured Value) divide this by the Actual value

EXAMPLE: Known* Calcium Level is 430. Tested Value is 410
[430-(430-410)] /430 =.953
Relative Accuracy = .953 X 100 += 95.3%
Relative Error is (100-Relative Accuracy) = 4.7%

• * Known means a valid reference standard.

Here is a link to some additional details about Relative Accuracy
https://sciencing.com/calculate-relative-accuracy-6069718.html


For Precision we will use Relative Standard Deviation (RSD). This value expressed in % tells us if the calculated standard deviation is small or large compared to the average value for our data set. A small number means there is small variability in our data clustered around the average. This would indicate good repeatability. A large number would indicate a lot of variability in our measurement. The math looks like this
(Standard Deviation of the Data Set x 100) Divide by (The Average Value of the Data Set)

EXAMPLE: Data from measuring Calcium Level of the same sample 5 times.

MEASUREMENT #	1​	2​	3​	4​	5​
RESULTS​	410​	425​	418​	433​	420​

Standard Deviation of the Data Set (Calculated in Excel) = 8.53
Average (Mean) of the Data Set =421.2
Relative Standard Deviation = (8.53 X 100)/421.2 = 2.02%
Precision Statement = 421.2 ±2.02%

Here is a link if you would like some additional details on Relative Standard Deviation
https://www.statisticshowto.datasciencecentral.com/relative-standard-deviation/

By using these two values, Relative Accuracy and Relative Standard Deviation, we can get a good idea of how Accurate and Precise our test results are.


FIT FOR PURPOSE----How Accurate & Precise Do I need To Be?
Definition: “Appropriate and of necessary standard for its intended use.”

For our discussion this would mean that Accuracy and Precision necessary to measure a selected parameter within the required limits that we need or desire for the health of our Reef Systems. The key phrase here is “within the required limits that we need or desire”. For example: Is the system a Fish Only or Reef…Is it SPS dominant or LPS…Etc. These different systems would require different levels of Accuracy & Precision, therefore what may be “Fit for Purpose” for one would not be satisfactory for the other. In addition the “safe” range of a parameter could dictate the required Accuracy and Precision. For example if a parameter required control between .03ppm and .1ppm and a second parameter between 1200-1400ppm, these two would most certainly require different levels of Accuracy and Precision. The main point is there are a number of parameters we test for all having different Accuracy & Precision requirements as well as the requirements of our individual systems and our individual desires for accuracy. This is exactly why it is important to know the Accuracy and Precision of each of the tests to determine if they are “Fit for Purpose”. In the above example of the Calcium Test, if our desired accuracy is within 10% of the actual vale and ± 20ppm we can conclude our test is “Fit for Purpose”

UNDERSTANDING ERRORS

Each test that we do consists of two basic steps: First there is the step where we perform the chemistry…Measuring our water and reagents…Mixing them…Timing the reaction etc. We will call this the EVALUATION STEP. The second step is where we interpret the results…Check the color against a color chart…observe the amount of titrant used…read the value on the meter, etc. This is the value on which we make our decision. We will call this the ASSESSMENT STEP. Both of these steps have errors associated with them.
Just as an example the Evaluation step may have multiple components.

First------ the test device (Kit, Meter etc.)
Second--- the test procedure itself (Weighing, measuring, time, etc.)
Third----- the sample quality (uniformity, not contaminated etc.)
Fourth------the appraiser (person doing the test)
Fifth------ the environment the tests are conducted

All of these steps have potential errors. In general errors do not compensate, they accumulate. This accumulation of errors is called Error Propagation or Propagation of Uncertainty. Simply put Propagation of Uncertainty is the accumulation of errors in each of the above five elements that combine to produce a total measurement error generating “noisy” results. These “noisy” results are the variability we see in our test results. If the variability is large enough the test results become unreliable (random).
The two types of errors to consider are Random Errors and Systematic Errors.

Random Errors: Random errors in experimental measurements are caused by unknown and unpredictable changes in the experiment. These changes may occur in the measuring instruments, in the environmental conditions, operator error and or procedural errors.

Systematic Errors: Systematic errors in experimental observations usually come from the measuring instruments. They may occur because:

• There is something wrong with the instrument or its data handling system, or
• Because the instrument is wrongly used by the experimenter.

Two types of systematic error can occur with instruments having a linear response:
Offset or zero setting error in which the instrument does not read zero when the quantity to be measured is zero.
Multiplier or scale factor error in which the instrument consistently reads changes in the quantity to be measured greater or less than the actual changes.
Fig 1 gives a basic picture of these two types of errors and some examples of each.

Fig 1

All of the tests performed in aquarium water testing are subject to these types of errors. We will cover several of them in this article. Obtaining Accurate (Real Value) and Precise (How repeatable are the results) measurement results requires work to minimize these errors. Not doing so will produce uncertain results. These uncertain results can lead us to incorrect actions…Is the Calcium level correct or not? Did the nutrient level change or not? The quality of our results directly effect the management of our aquariums….Bad Results---Bad Management.
Understanding a tests accuracy and precision is important to an effective quality system.

If you are interested in the details here are some good articles to read. https://www.inorganicventures.com/accuracy-precision-mean-and-standard-deviation

https://www.thoughtco.com/difference-between-accuracy-and-precision-609328 http://www.chem1.com/acad/webtext/pre/pre-5.html

WHY A QUALITY SYSTEM APPROACH

A quality management system can be defined as “coordinated activities to direct and control an organization with regard to quality.”... In a quality management system, all aspects of the laboratory operation, including the organizational structure, processes, and procedures, need to be addressed to assure quality.
Although this definition is generally meant for organization laboratories we can apply the same principles to our testing of tank parameters. …The definition for us might look something like this:

A quality management system can be defined as “organized activities to direct and control tank testing parameters with regard to quality of results” ... In such a system, all aspects of the laboratory (testing) operation, including data records, processes, and procedures, need to be evaluated and addressed to assure the quality of our test results.

Many quality systems are founded on the 6 SIGMA concepts. Six Sigma is a method that explores ways to improve the quality of process outputs (Testing Results) by analyzing and reducing the source of defects (errors) and reducing variability in test results. It is a precise approach to cut down the existing errors or mistakes in terms of defects per million (DPM). The system focuses on the “6 M’s”. Methods, Materials, Machines (Equipment), Measurement, Man, and Mother Nature. (See Fig 2)

Fig 2

Looking at the 6 Ms we can see that all of them in one way or another come into play with our aquarium testing. By focusing on reducing errors in these we can improve our test results and have greater confidence in our aquarium management. Why is this important? Simply put Data outcomes direct action to be taken. If the data is incorrect the action will be incorrect. This incorrect action can cause problems with water quality and important tank parameters. In extreme cases it can result it a “Tank Crash” with over dosing or under dosing of an element, but at the very least it can result in less than stable tank conditions and create the “Yoyo Effect”, where values go up and down only because the test results are not accurate.

It is not the intent of this article to create a Quality System, but to introduce a Quality System Mindset and to point out some key elements to be considered in the design of such a system as well as important factors associated with the “6 M’s”. I will cover elements of both colorimetric test kits requiring visual evaluation (VISUAL ASSESSMENT) as well as instrumental testing, such as the Hanna Checkers. (INSTRUMENTAL--DIGITAL TESTING). I will also touch on ICP testing with regard to the Quality System Mindset.

Fig 3

The remaining parts of this article will focus on areas in our test methods where errors commonly occur that cause poor accuracy and precision and how to look at methods to minimize or eliminate them by using a Quality System Approach. (SEE Fig 3)

 

[Flippers4pups]

Very in-depth report!

 

[Rick Mathew]

Very in-depth report!

Thank you...Parts 2-4 to come

 

[Roosterjack]

My guess would be that for a lot of us we have decent precision with less than stellar accuracy. If I carefully follow the instructions and on three different test kits I will almost invariably get differing results from different test kit brands but identical results for multiple tests using the same brand.

Determining which kit is the most accurate is challenging when there is no practical way (that I am aware of) to obtain reference samples with known values.

The same can be said with salinity testing via various methods, and even something as basic as temperature.

I am eager to read more and learn how I can improve my methods and gear. I am a firm believer in the “If it can’t be measured it can’t be managed” school of thought.

 

[Rick Mathew]

My guess would be that for a lot of us we have decent precision with less than stellar accuracy. If I carefully follow the instructions and on three different test kits I will almost invariably get differing results from different test kit brands but identical results for multiple tests using the same brand.

Determining which kit is the most accurate is challenging when there is no practical way (that I am aware of) to obtain reference samples with known values.

The same can be said with salinity testing via various methods, and even something as basic as temperature.

I am eager to read more and learn how I can improve my methods and gear. I am a firm believer in the “If it can’t be measured it can’t be managed” school of thought.

Interesting observation. I have also observed that some get frustrated because the get two different results when they test the same sample and get two "different" results. The conclusion they come to is the "test is bad". Like you I have used different test kits testing for the same ion and got different results...I think now with the introduction of ICP the attention is moving to the question of "accuracy"....As illustrated in this thread https://www.reef2reef.com/threads/accuracy-of-phosphate-icp-analysis.381738/

You are correct about the difficulty of getting good reference samples...I will be touching on this topic in part 4 of this set of articles....You can check this one out in the meantime and tell me what you think.. http://cm-distributors.com/product/multi-reference-solution/

Although this is not a NIST certified standard it has some validation via ICP testing...Which itself has some looming questions about its Precision and accuracy...

I have the capabilities of creating my own reference standards although I in no way claim them to be high accuracy but non the less they give me "ball park" reference points.

Thanks for the reply hope you find the rest of the article helpful.

 

[Bruce60]

As a pathologist and laboratorian, I enjoyed the first part of your series, and look forward to R2R fixing the text justification and restoring the images.

The need for accuracy and precision is directly tied to what we are trying to measure and why. One example;

Over the course of 24 hours, the pH of my tank may vary by up to 0.1. So, the time of day I measure pH will impact the value of that result. Should I then try to perform this test at the same time each day it is performed? Or, is the difference between 8.25 and 8.15 really not that important? Is what is important about pH is that it remains within a specific range...or that it does not vary by more than a certain amount in a specific time frame...or both? Answering these questions helps inform the specific needs for accuracy and precision for this specific parameter.

 

[siggy]

If the first part was to instill a greater awareness and attention to detail while testing you succeeded.
I see the major challenge of testing in this hobby are the tools, ie. powder, plastic spoons and then inexpensive electronics
that are powered by batteries.
All i can do is be precise in the testing process to achieve accuracy in my chosen method. I also believe the size of sample the hobby uses is insufficient to achieve true accuracy of the sample with the methods we employ

 

[Rick Mathew]

If the first part was to instill a greater awareness and attention to detail while testing you succeeded.
I see the major challenge of testing in this hobby are the tools, ie. powder, plastic spoons and then inexpensive electronics
that are powered by batteries.
All i can do is be precise in the testing process to achieve accuracy in my chosen method. I also believe the size of sample the hobby uses is insufficient to achieve true accuracy of the sample with the methods we employ

Thank you. It was one of my intentions to create an awareness of the importance of the details of testing as well as to share my experiences of working towards my own quest of getting more reliable results. The questions I listed in the beginning we real questions for me. The thought being I am responsible for the well being of my tank critters and should be making the best decisions for them.

As to the sample size...You are correct!. These small sample sizes really puts the pressure on issues like contamination...procedural accuracy etc.

Thanks for the comments...Hope the rest of the article is helpful

rick

 

[reeftice]

As a pathologist and laboratorian, I enjoyed the first part of your series, and look forward to R2R fixing the text justification and restoring the images.

The need for accuracy and precision is directly tied to what we are trying to measure and why. One example;

Over the course of 24 hours, the pH of my tank may vary by up to 0.1. So, the time of day I measure pH will impact the value of that result. Should I then try to perform this test at the same time each day it is performed? Or, is the difference between 8.25 and 8.15 really not that important? Is what is important about pH is that it remains within a specific range...or that it does not vary by more than a certain amount in a specific time frame...or both? Answering these questions helps inform the specific needs for accuracy and precision for this specific parameter.

Sure thing! But how do you know if your measured pH is measured at a low point, or at a high point? 8.1 at low point is totally fine, 8.1 at a high point might be on the low side...

It's never easy, is it?

 

[Rick Mathew]

Sure thing! But how do you know if your measured pH is measured at a low point, or at a high point? 8.1 at low point is totally fine, 8.1 at a high point might be on the low side...

It's never easy, is it?

It most certainty is not easy...Just like your question about pH measurement...I would add the question...Is it the probe?...is the variation a probe fluctuation...so many questions?

 

[ReefGeezer]

NIST certified standards, calibrated instruments, validated processes, and certified reagents can definitely make testing more accurate. I suspect those who are managing public aquarium sized enterprises are already doing some or all of the above to ensure the validity of their testing results. As for my smallish system, I couldn't justify the cost. I use decent hobby grade test kits, pay attention to expiration dates, and understand their uncertainty in my hands. I also understand the risks associated with relying on these kits and have developed processes to manage them i.e. performing multiple tests, throwing out anomalous results, and relying on trend analysis rather than single sets of tests.

I'm a quality management system auditor. I assume those labs operate in accordance with some internal or industry standards. It might be fun to find out how well they are meeting these requirements.

 

[Rick Mathew]

NIST certified standards, calibrated instruments, validated processes, and certified reagents can definitely make testing more accurate. I suspect those who are managing public aquarium sized enterprises are already doing some or all of the above to ensure the validity of their testing results. As for my smallish system, I couldn't justify the cost. I use decent hobby grade test kits, pay attention to expiration dates, and understand their uncertainty in my hands. I also understand the risks associated with relying on these kits and have developed processes to manage them i.e. performing multiple tests, throwing out anomalous results, and relying on trend analysis rather than single sets of tests.

I'm a quality management system auditor. I assume those labs operate in accordance with some internal or industry standards. It might be fun to find out how well they are meeting these requirements.

It would be interesting to explore how they manage their system...The key is exactly what you stated...Understanding the uncertainty and managing it to get it as small as we practically can give what we have to work with...thanks for the reply!

rick

 

[Daniel@R2R]

Great article!! I think I got the picture issue fixed and even got the thread attributed correctly back to the author.

 

[Rick Mathew]

Great article!! I think I got the picture issue fixed and even got the thread attributed correctly back to the author.

Thank you...appreciate your help with this...Also appreciate the complement...

If I want to post part 2 what is the best way to go about it...I have it as a word document and could send it to a staff member for review ....or I could do as I did before an post on the forum and have a staff person review from there...just let me know...Thanks again so much for your help

Rick

 

[Daniel@R2R]

....or I could do as I did before an post on the forum and have a staff person review from there...

Go ahead and do it this way. It's easiest to set up correctly if it's already in a thread.

 

[Rick Mathew]

Go ahead and do it this way. It's easiest to set up correctly if it's already in a thread.

Shall Do...As soon as it is posted I will let you know...Thanks again for your help

 

[Soryu]

Definitely written from the management perspective! Hopefully more people read this and look into a quality mindset when it comes to establishing their testing methods with the hobby grade test equipment available to us. Having been in the industrial quality business for years I struggle with the relatively poor precision and accuracy of the hobby grade test equipment and would love to see more information published on best practices when using both specific test equipment and general tests. Knowing how other people overcome the deficiencies of the hobby grade equipment would be valuable to people willing to listen I think.

 

[Rick Mathew]

Definitely written from the management perspective! Hopefully more people read this and look into a quality mindset when it comes to establishing their testing methods with the hobby grade test equipment available to us. Having been in the industrial quality business for years I struggle with the relatively poor precision and accuracy of the hobby grade test equipment and would love to see more information published on best practices when using both specific test equipment and general tests. Knowing how other people overcome the deficiencies of the hobby grade equipment would be valuable to people willing to listen I think.

Thanks for your reply...I also come from an industrial background and agree with the need of a Quality System mind Set...It actually (IMO) is even more critical when using Hobby Grade Kits...Parts 2,3 & 4 of this article points out some of the areas where some of the deficiencies can be mitigated for....They will most likely never reach the true analytical level but improvements can be made...Part 2 will be first posted on the Reef Chemistry Forum and then later moved to the Article Forum...Hope to do this in the next day or so...thanks again for your reply

 

[User1]

Nice write up.

Not a fan of using averages. In fact, I don't. I use 90th percentile. Probably a personal bias on my side though.

 

[Rick Mathew]

Nice write up.

Not a fan of using averages. In fact, I don't. I use 90th percentile. Probably a personal bias on my side though.

Thank you...appreciate you reply