Replacement strategies for missing velocities: Difference between revisions
mNo edit summary |
|||
| Line 11: | Line 11: | ||
* Unevenly spaced least-square Fourier transform (i.e., no replacement at all) | * Unevenly spaced least-square Fourier transform (i.e., no replacement at all) | ||
== Replacement strategy tests == | === Replacement strategy for tests === | ||
These replacement strategies were trialed with one of the cleanest benchmarks (Underice MAVS sampling at 8 Hz) for different | These replacement strategies were trialed with one of the cleanest benchmarks (Underice MAVS sampling at 8 Hz) for different | ||
* Number of missing samples to identify a threshold where the segment should be completely discarded from further analysis | * Number of missing samples to identify a threshold where the segment should be completely discarded from further analysis | ||
| Line 23: | Line 23: | ||
A total section of 30 min was chosen as this coincides with the required segment length for deriving an estimate of <math>\varepsilon</math>. | A total section of 30 min was chosen as this coincides with the required segment length for deriving an estimate of <math>\varepsilon</math>. | ||
== | ==Test Results== | ||
{{FontColor|fg=white|bg=red|text=Insert graphs with example spectra for different tests}} | {{FontColor|fg=white|bg=red|text=Insert graphs with example spectra for different tests}} | ||
For all tests, the linear interpolation did the best job in recovering the original spectra, followed by the unevenly spaced techniques. However, the unevenly spaced fourier transforms behaved similarly (if not worse) than the variance replacement when the data loss was intermittent. Unsurprisingly, unevenly spaced techniques fair better if the data loss form long continuous gaps. | For all tests, the linear interpolation did the best job in recovering the original spectra, followed by the unevenly spaced techniques. However, the unevenly spaced fourier transforms behaved similarly (if not worse) than the variance replacement when the data loss was intermittent. Unsurprisingly, unevenly spaced techniques fair better if the data loss form long continuous gaps. | ||
Revision as of 14:08, 5 July 2022
Quality-control of raw velocities results in data loss, which usually must be replaced before computing the spectra necessary for obtaining [math]\displaystyle{ \varepsilon }[/math]. The number of missing samples that can be tolerated for computing reliable spectra was also investigated.
Data analysis tests
Techniques considered for replacing the missing samples
- Linear interpolation
- Using the variance of the signal, which is commonly used by those intending to compute eddy-covariances
- Unevenly spaced least-square Fourier transform (i.e., no replacement at all)
Replacement strategy for tests
These replacement strategies were trialed with one of the cleanest benchmarks (Underice MAVS sampling at 8 Hz) for different
- Number of missing samples to identify a threshold where the segment should be completely discarded from further analysis
- 10, 25, and 50% of the 30 min timeseries were removed
- Data loss (gap) duration
- 1 sample
- 8 samples (1 s)
- 480 samples (60 s)
- 960 samples (120 s)
A total section of 30 min was chosen as this coincides with the required segment length for deriving an estimate of [math]\displaystyle{ \varepsilon }[/math].
Test Results
Insert graphs with example spectra for different tests For all tests, the linear interpolation did the best job in recovering the original spectra, followed by the unevenly spaced techniques. However, the unevenly spaced fourier transforms behaved similarly (if not worse) than the variance replacement when the data loss was intermittent. Unsurprisingly, unevenly spaced techniques fair better if the data loss form long continuous gaps.
- For a given data loss (e.g., 10%), the original spectra was easier to recover with longer continuous gaps.
Recommendations
Always use linear interpolation, and record the percent good samples in each segment so that this variable can be used to reject epsilon with more than 10% data loss. This threshold may be relaxed if the data loss forms long continuous chunks of several seconds, and tests with the timeseries shows that the estimated spectrum can tolerate more missing samples. The person processing the data should record the threshold used and report it in the NetCDF flags.
Return to Preparing quality-controlled velocities
