QC Overview
CMAR applies automated Quality Control (QC) tests and “human-in-the-loop” QC to the Coastal Monitoring Program Water Quality data.
An automated QC test is an algorithm that evaluates each data observation and assigns a flag to the observation indicating the test result. These flags are typically reviewed by human experts, which is referred to as “human-in-the-loop” QC. End users can then filter the data set for records that meet their quality criteria (UNESCO 2013).
Numerous QC flagging schemes and tests exist and have been applied to oceanographic data sets (e.g., Appendix A in UNESCO 2013). CMAR has adopted the well-known QARTOD flag scheme and several QARTOD tests.
It is beyond the scope of the Program to produce analysis-ready data products for all potential users. Some users may wish to apply additional QC. All users should carefully evaluate data and flags before analysis.
QARTOD Guidance
QARTOD stands for “Quality Assurance / Quality Control of Real-Time Oceanographic Data”. It is a project led by the U.S. Integrated Ocean Observing System (IOOS) that aims to develop guidelines for producing high-quality oceanographic data. QARTOD has developed QC manuals for core ocean variables that outline best practices and describe QC tests with codable instructions. QC manuals are formally reviewed by subject-matter experts, and are updated as needed to reflect new technology, additional knowledge, or growth of the project (IOOS 2020a).
QARTOD manuals focus on QC of real-time data1, but acknowledge that other data types may also benefit from these flags and tests (IOOS 2020b). The CMAR Water Quality data is not processed in real time due to technical, logistical, and budget constraints. Instead, data is logged and offloaded from the sensors for processing every 6–12 months. Some QARTOD guidance was therefore not applicable to this data, and procedures were adapted to reflect the nature of CMAR data and processing.
Flag Scheme
CMAR has adopted the QARTOD flag scheme (Table 1) with some adaptations. The QARTOD flag scheme provides information to data users on the expected quality of the data. This means that even “bad” data observations may be published, and it is up to the user to determine which records to include in their application. Details on how the QARTOD flag scheme was developed are provided in UNESCO (2013).
| Flag Label | Flag Value | Description |
|---|---|---|
| ■ Pass | 1 | The observation passed all applicable QC tests. |
| ■ Not Evaluated | 2 | At least one QC test could not be applied (e.g., at the start or end of a deployment). |
| ■ Suspect/Of Interest | 3 | The observation may be of poor quality (Suspect) or may reflect a real but unusual event (Of Interest). |
| ■ Fail | 4 | The observation failed at least one QC test. |
| ■ Missing Data | 9 | The observation is missing. Not currently used by CMAR. |
The main CMAR adaptation to this flag scheme is that the Missing Data flag is not used. This flag is meant to alert real-time operators that an expected observation was not received, and may trigger efforts to fix recording and transmission issues. Since CMAR does not receive real-time data, this placeholder flag was deemed unnecessary. Note that data gaps may still occur due to sensor failure, delays between retrieval and re-deployment, and vandalism. It is the responsibility of the data users to identify and address these data gaps if required.
Note that QARTOD uses a flag of 3 to denote observations that are Suspect (e.g., of dubious quality) or Of Interest (e.g., an unusual event). This is meant to encourage human-in-the-loop decision-making (IOOS 2020b). Where possible, CMAR has defined when these test results are likely Suspect vs. Of Interest, although data users should inspect these records carefully before deciding how to use (or discard) them.
Each observation receives a flag from each test. The published data include a summary flag for each observation, which is the worst (highest) flag across all tests. Observations at the start and end of each deployment are flagged Not Evaluated by the Spike and Rolling Standard Deviation tests, so their summary flag will be Not Evaluated even if they passed every other test.
Tests
CMAR applied five automated QC tests to the Coastal Monitoring Program Water Quality data (Table 2). Three are QARTOD tests (Gross Range, Climatological, and Spike), and the remaining two were developed by CMAR to address specific data quality concerns. Finally, a manual Human in the Loop test was applied, where experts reviewed the results of the automated tests and flagged additional observations where necessary2.
See the QC Tests & Thresholds page for more detail on each QC test.
| Test | Description | Reference |
|---|---|---|
| Gross Range | Flags observations that fall outside of the sensor measurement range and observations that are statistical outliers. | IOOS (2018), IOOS (2020c) |
| Climatological | Flags observations that are statistical outliers for a given month. | IOOS (2018), IOOS (2020c), OOI (2022) |
| Spike | Flags single-value spikes. | IOOS (2018), IOOS (2020c) |
| Rolling Standard Deviation | Flags observations with statistically high rolling standard deviation (e.g., multiple-value spikes). | CMAR |
| Depth Crosscheck | Flags deployments where the sensor depth at low tide is substantially different from the lowest measured depth. All observations from the deployment receive the resulting flag. | CMAR |
| Human in the Loop | Flags observations that human experts recognize as poor quality. | CMAR |
Thresholds
Each automated QC test requires threshold(s) that determine the results of the test. Choosing appropriate thresholds for each test and variable is a considerable part of the QC effort (IOOS 2018). Following best practices, CMAR has developed thresholds based on historical data where possible (IOOS 2020b; Taylor and Loescher 2013; OOI 2022). The QC Tests & Thresholds page provides an overview of each QC test and how the associated thresholds were calculated. The Thresholds page under each variable in the menu provides additional details on how the threshold(s) were determined.