Segmenting datasets: Difference between revisions

From Atomix
m Tweaks
Line 1: Line 1:
Once the raw observations have been [[Data processing of raw measurements|quality-controlled]], then you must split the time series into shorter segments by considering:
Once the raw observations have been [[Data processing of raw measurements|quality-controlled]], then you must split the time series into shorter segments by considering:
* [[Time and length scales of turbulence]]
* [[Time and length scales of turbulence]]
* [[Stationarity]] of the segment
* [[Stationarity]] of the segment and [[Taylor's Frozen Turbulence| Taylor's frozen turbulence hypothesis]]
* [[Taylor's Frozen Turbulence| Taylor's frozen turbulence hypothesis]], etc ...
* Statistical significance of the resulting spectra
* Statistical significance of the resulting spectra



Revision as of 14:36, 30 November 2021

Once the raw observations have been quality-controlled, then you must split the time series into shorter segments by considering:

Measured velocities at 4 Hz from an Acoustic-Doppler Velocimeters have been detrended using three different techniques. Empirical modal decomposition (EMD) [1], linear trend, and a 2nd order low-pass Butterworth filter. A cut-off period of 10 min was targeted by both the filter and EMD

Application to measured velocities

Measurements are typically collected in the following two ways:

  • continuously, or in such long bursts that they can be considered continuous
  • short bursts that are typically at most 2-3x the expected largest turbulence time scales (e.g., 10 min in ocean environments)

This segmenting step dictates the minimum burst duration when setting up your equipment. The act of chopping a time series into smaller subsets, i.e., segments, is effectively a form of low-pass (box-car) filtering. How to segment the time series is usually a more important consideration than detrending the time series since estimating [math]\displaystyle{ \varepsilon }[/math] relies on resolving the inertial subrange in the final spectra computed over each segment.

  • Zoom of the first 512 s segment of the measured velocities shown above including the same trends
  • Example velocity spectra of the short 512 s of records before and after different detrending techniques applied to the original 6h time series. The impact of the detrending method can be seen at the lowest frequencies only

Trade-offs when choosing segment length

The shorter the segment, the higher the temporal resolution of the final [math]\displaystyle{ \varepsilon }[/math] time series.

Notes

  1. Zhaohua Wu, Norden E. Huang, Steven R. Long and and Chung-Kang Peng. 2007. On the trend, detrending, and variability of nonlinear and nonstationary time series. PNAS. doi:10.1073/pnas.0701020104