Talk:Quality control coding
During our shear probe group meeting on March 28, 2022, we discussed the possibility of using a different flagging scheme for the shear probe data. Suggestions were to use a binary system that allows to relate the flagging number to one or more specific data quality control tests that would be unique even if multible quality control tests were failed by the data set. However, this flagging scheme would only apply to our data and international data centres or data display display servers (e.g. https://www.ocean-ops.org/board) would not recognize this. Perhaps a solution would be to use a two-level scheme, as also suggeted by the Intergovernmental Oceanographic Commission (IOC). On the primary level, we use a common quality flagging as shown below. The secondary level could then be a unique number that points to the quality test the data has failed. The flagging scheme below is taken from a best practice IOC document (link). The secondary level could then complement the primary level flags by reporting the results of specific QC tests performed and data processing history. As suggested by IOC best practices "The secondary level content varies in number and description and is chosen by those who implement the scheme, representing information on the applied quality tests (e.g., excessive spike check, regional data range check) and data processing history (e.g., interpolated values, corrected values)."
| Value | Primary-level flag short name | Definition |
|---|---|---|
| 1 | Good | Passed documented required QC tests |
| 2 | Not evaluated, not available or unknown | Used for data when no QC test performed or the information on quality is not available |
| 3 | Questionable/suspect | Failed non-critical documented metric or subjective test(s) |
| 4 | Bad | Failed critical documented QC test(s) or as assigned by the data provider |
| 9 | Missing data | Used as place holder when data are missing |
Would that work?
[Below are some old notes from the main page. They might still come in handy at a later stage: Amelie]
Ocean Sites Providing quality-control flags according to Ocean Sites is encouraged. These are described at Ocean Sites for quailty control (QC) coding. This flagging scheme is mostly compatible with the primary level flagging recommended by Intergovernmental Oceanographic Commission of UNESCO (2013). However, only the flags of 0, 1, and 4 make sense for dissipation estimates derived from shear-probe data.
| Flag | Meaning | Comment |
|---|---|---|
| 0 | unknown | No QC was performed. |
| 1 | good data | All QC tests passed. |
| 2 | probably good data | Data have failed one or more QC tests but detailed examination after processing (e.g. by visual examination) suggests data is good. |
| 3 | potentially correctable bad data | These data are not to be used without scientific correction or re-calibration (e.g. uncertain shear sensor sensitivity). |
| 4 | bad data | Data have failed one or more tests. |
| 5 | - | Not used |
| 6 | - | Not used |
| 7 | nominal value | Data were not observed but reported (e.g. instrument target depth.). |
| 8 | interpolated value | Missing data may be interpolated from neighboring data in space or time. |
| 9 | missing value | This is a fill value |
[Rolf left the stuff below in place because I do not know what to do with it.]
Climate and Forecast Metadata Convention (CF) requires that QC flags carry attributes. In netCDF (Network Common Data Form) data files, the following information for quality control flagging should be provided for each data variable <PARAM>.
<PARAM>_QC
<PARAM>_QC:long_name = “quality flag of <PARAM>”;
<PARAM>_QC:conventions = “OceanSITES QC Flags”;
<PARAM>_QC:flag_values = 0, 1, 2, 3, 4, 7, 8, 9;
<PARAM>_QC:flag_meanings = “0:unknown 1:good_data 2:probably_good_data 3:potentially_correctable_bad_data 4:bad_data 7:nominal_value 8:interpolated_value 9:missing_value”
