Replacement strategies for missing velocities: Difference between revisions
mNo edit summary |
|||
| Line 30: | Line 30: | ||
== Recommendations == | == Recommendations == | ||
* Use linear interpolation, and record the percent of good samples in each segment in the NetCDF [[Level 3 data (velocity point-measurements)| Level 3 data]]. | |||
* Reject and flag <math>\varepsilon</math> associated with segments with more than 10% data loss. | |||
* Record the threshold used for flagging and report it in the NetCDF file at [[Level 4 data (velocity point-measurements)| Level 4]]. | |||
** The rejection threshold may be relaxed if the data loss forms long continuous chunks of several seconds, after testing the actual time series used to establish if the spectra can tolerate more missing samples. Recording the threshold used at [[Level 4 data (velocity point-measurements)| Level 4]] would permit others to exclude these data at a later time. | |||
---- | ---- | ||
Return to [[Preparing quality-controlled velocities]] | Return to [[Preparing quality-controlled velocities]] | ||
Revision as of 14:16, 5 July 2022
Quality-control of raw velocities results in data loss, which usually must be replaced before computing the spectra necessary for obtaining [math]\displaystyle{ \varepsilon }[/math]. The number of missing samples that can be tolerated for computing reliable spectra was also investigated.
Data analysis tests
Techniques considered for replacing the missing samples
- Linear interpolation
- Using the variance of the signal, which is commonly used by those intending to compute eddy-covariances
- Unevenly spaced least-square Fourier transform (i.e., no replacement at all)
Replacement strategy for tests
These replacement strategies were trialed with one of the cleanest benchmarks (Underice MAVS sampling at 8 Hz) for different
- Number of missing samples to identify a threshold where the segment should be completely discarded from further analysis
- 10, 25, and 50% of the 30 min timeseries were removed
- Data loss (gap) duration
- 1 sample
- 8 samples (1 s)
- 480 samples (60 s)
- 960 samples (120 s)
A total section of 30 min was chosen as this coincides with the required segment length for deriving an estimate of [math]\displaystyle{ \varepsilon }[/math].
Test Results
Insert graphs with example spectra for different tests For all tests, the linear interpolation did the best job in recovering the original spectra, followed by the unevenly spaced techniques. However, the unevenly spaced fourier transforms behaved similarly (if not worse) than the variance replacement when the data loss was intermittent. Unsurprisingly, unevenly spaced techniques fair better if the data loss form long continuous gaps.
- For a given data loss (e.g., 10%), the original spectra was easier to recover with longer continuous gaps.
Recommendations
- Use linear interpolation, and record the percent of good samples in each segment in the NetCDF Level 3 data.
- Reject and flag [math]\displaystyle{ \varepsilon }[/math] associated with segments with more than 10% data loss.
- Record the threshold used for flagging and report it in the NetCDF file at Level 4.
- The rejection threshold may be relaxed if the data loss forms long continuous chunks of several seconds, after testing the actual time series used to establish if the spectra can tolerate more missing samples. Recording the threshold used at Level 4 would permit others to exclude these data at a later time.
Return to Preparing quality-controlled velocities
