Flow chart for shear probes: Difference between revisions

From Atomix
Rolf (talk | contribs)
No edit summary
Aleboyer (talk | contribs)
 
(51 intermediate revisions by 4 users not shown)
Line 1: Line 1:


The processing of shear-probe data can be divided into the following five major steps and these steps apply to data collected with any platform or vehicle. There are many sub-steps to these major steps. The major steps are:
The processing of shear-probe data can be divided into five major steps, which apply to data collected with any platform or vehicle. There are many sub-steps to these major steps. The major steps are:


__TOC__
__TOC__


== Conversion to physical units ==
== Conversion to physical units ==
<div class="mw-collapsible" id="physical units" data-collapsetext="Collapse" data-expandtext="Expand">
<div class="mw-collapsible mw-collapsed" id="physical units" data-collapsetext="Collapse" data-expandtext="Expand">
<br>
As a first step, the raw binary data needs to be transformed into physical units.
As a first step, the raw binary data needs to be transformed into physical units.
:*     [[Determine the speed of profiling]] of the shear-probe through the water.
:#     [[Determine the speed of profiling]] of the shear-probe through the water.
:* [[Determine the temperature of the water]].  
:# [[Determine the temperature of the water]].
:* [[Convert the shear probe data]] samples into physical units  
:# [[Convert the shear probe data]] samples into physical units
:* Convert all other signals per the recommendations of the manufacturer of the sensor or instruments that produce these signals.  
:# Convert all other signals per the recommendations of the manufacturer of the sensor or instruments that produce these signals.
 
Please note that most choices made must be included in a data set, as described
in the [[Netcdf meta data (shear probes)|list of meta data]].
</div>
</div>


== "Section" selection ==
<div class="mw-collapsible" id="Section" data-collapsetext="Collapse" data-expandtext="Expand">


== "Section" selection ==
<div class="mw-collapsible  mw-collapsed" id="Section" data-collapsetext="Collapse" data-expandtext="Expand">
<br>
Before you can process your shear-probe data to derive the rate of dissipation you must select the [[section]] of data that you wish to process. For a vertically profiling instrument, this is traditionnally referred to as a "profile". We adopt the term "section" as this is platform independent and will include time series for dissipation estimates along horizontal or slanted trajectories as well as from moored shear probes. You must make sure that the selection is meaningful and sensible. For example, the shear probe must be profiling through the water with a speed, direction, and orientation that is fairly stationary. The selection of data can be partially automated by requiring that the kinematics of your instrument achieve certain minimum criteria. The steps to section selection are as follows:  
Before you can process your shear-probe data to derive the rate of dissipation you must select the [[section]] of data that you wish to process. For a vertically profiling instrument, this is traditionnally referred to as a "profile". We adopt the term "section" as this is platform independent and will include time series for dissipation estimates along horizontal or slanted trajectories as well as from moored shear probes. You must make sure that the selection is meaningful and sensible. For example, the shear probe must be profiling through the water with a speed, direction, and orientation that is fairly stationary. The selection of data can be partially automated by requiring that the kinematics of your instrument achieve certain minimum criteria. The steps to section selection are as follows:  
:*   Choose the [[minimum speed]] of profiling.
:#   Choose the [[minimum speed]] of profiling.
:*   Choose the [[direction of the vertical velocity]] of the profiler.
:#   Choose the [[direction of the vertical velocity]] of the profiler.
:*   Choose the [[minimum depth]].
:#   Choose the [[minimum depth]].
:*   Choose the [[maximum pitch and roll]] of the profiler.  
:#   Choose the [[maximum pitch and roll]] of the profiler.
:*   Choose the [[minimum duration]] over which the [[minimum speed]] through [[maximum pitch and roll]] must be satisfied.  
:#   Choose the [[minimum duration]] over which the [[minimum speed]] through [[maximum pitch and roll]] must be satisfied.
 
 
Please note that most choices made must be included in a data set, as described
in the [[Netcdf meta data (shear probes)|list of meta data]].
 
</div>
</div>


==    Choosing the [[processing parameters]] ==
<div class="mw-collapsible" id="processing parameters" data-collapsetext="Collapse" data-expandtext="Expand">


:* Choose the length of data (in meters) used for each dissipation estimate – [[diss-length]].
== Choosing the [[processing parameters]] ==
:* Choose the lowest wavenumber of spectral estimation – [[fft-length]].  
<div class="mw-collapsible mw-collapsed" id="processing parameters" data-collapsetext="Collapse" data-expandtext="Expand">
:* Translate [[diss-length]] and [[fft-length]] into [[duration]] (time).
<br>
:* Round these up to [[nearest power-of-two number]] of samples.
:# Choose the lowest wavenumber of spectral estimation (as a function of the vehicle's length) – [[fft-length]].
:* Choose a [[high-pass filter cut-off frequency]] to be consistent with duration of the [[fft-length]].  
:# Choose the length of data (in meters) used for each dissipation estimate – [[diss-length]].
:* Choose [[de-spiking parameters]].  
:# Choose a [[high-pass filter cut-off frequency]] to be consistent with duration of the [[fft-length]].
:* Choose [[vibration-coherent noise removal]].
:# Choose [[de-spiking parameters]].
:# Choose [[vibration-coherent noise removal]].
:# Choose [[quality-control parameters]].
 
 
Please note that most choices made must be included in a data set, as described
in the [[Netcdf meta data (shear probes)|list of meta data]].
 
</div>
</div>


==      Compute the [[dissipation rate estimates]] ==
<div class="mw-collapsible" id="estimates" data-collapsetext="Collapse" data-expandtext="Expand">


== Compute the [[dissipation rate estimates]] ==
<div class="mw-collapsible mw-collapsed" id="estimates" data-collapsetext="Collapse" data-expandtext="Expand">
<br>
The following steps are recommended to obtain estimates of the turbulent dissipation rate of kinetic energy (<math>\varepsilon</math>).   
The following steps are recommended to obtain estimates of the turbulent dissipation rate of kinetic energy (<math>\varepsilon</math>).   
   
   
#      Extract the section to estimate dissipation time series ("Section" selection).  
#      Extract the section to estimate dissipation time series ("Section" selection).  
#      High-pass filter the shear-probe and (optionally) the vibration data.   
#      High-pass filter the shear-probe and (optionally) the vibration data.   
#      Identify each diss-length segment in the section.  
#      Identify each diss-length segment in the section.  
#      [[De-spike the shear-probe data]], and track the fraction of data affected by de-spiking within each diss-length segment. This will become a [[Shear_probes_quality_control_metrics|quality-control metric]].  
#      [[De-spike the shear-probe data]], and track the fraction of data affected by de-spiking within each diss-length segment. This will become a [[Shear_probes_quality_control_metrics|quality-control metric]].  
#      Calculate the [[frequency spectra and cross-spectra of shear and vibrations]] for each diss-length segment.  
#      Calculate the [[frequency spectra and cross-spectra of shear and vibrations]] for each [[detrending time series|detrended]] diss-length segment.  
#      Extract the original and the vibration-coherent clean shear-probe frequency spectra with [[the Goodman algorithm]].  
#      Extract the original and the vibration-coherent clean shear-probe frequency spectra with [[the Goodman algorithm]].  
#      Correct shear and vibration frequency spectra for [[the high-pass filter]].  
#      Correct shear and vibration frequency spectra for [[the high-pass filter]].  
Line 54: Line 68:
#      Correct the spectra of shear for the [[wavenumber response of the shear probe]].  
#      Correct the spectra of shear for the [[wavenumber response of the shear probe]].  
#      Apply an [[iterative spectral integration algorithm]] to estimate the variance of shear.
#      Apply an [[iterative spectral integration algorithm]] to estimate the variance of shear.
#      If the dissipation estimate is larger than [[shear inertial subrange fit]] use the method fit to the inertial subrange
#      Calculate the turbulent dissipation rate by multiplying the shear variance by <math>\frac{15}{2}\nu</math> where <math>\nu </math> is the temperature-dependent kinematic viscosity.
#      Calculate the turbulent dissipation rate by multiplying the shear variance by <math>\frac{15}{2}\nu</math> where <math>\nu </math> is the temperature-dependent kinematic viscosity.
#      Determine the [[figure of merit (FM)]] for each shear-probe spectrum.  
#      Determine the [[figure of merit (FOM)]] for each shear-probe spectrum.  
#      Calculate the expected variance of each dissipation estimate.
#      Calculate the expected variance of each dissipation estimate.
Please note that most choices made must be included in a data set, as described
in the [[Netcdf meta data (shear probes)|list of meta data]].
</div>
</div>


==      Apply [[Shear_probes_quality_control_metrics|quality-control metrics]] ==
==      Apply [[Shear_probes_quality_control_metrics|quality-control metrics]] ==


<div class="mw-collapsible" id="quality_control" data-collapsetext="Collapse" data-expandtext="Expand">
<div class="mw-collapsible mw-collapsed" id="quality_control" data-collapsetext="Collapse" data-expandtext="Expand">
 
<br>
Shear-probe data can be corrupted or compromised in several different ways.
Shear-probe data can be corrupted or compromised in several different ways.
These include but are not limited to (''i'') collision with plankton and other materials, (''ii'') unremovable vibrational contamination. (''iii'') electronic noise, and (''iv'') interference from other instrumentation on a platform that carries the shear probes.
These include but are not limited to (''i'') collision with plankton and other materials, (''ii'') unremovable vibrational contamination. (''iii'') electronic noise, and (''iv'') interference from other instrumentation on a platform that carries the shear probes.
This section describes quality control metrics and the coding used to identify the quality-control metrics.
This section describes the quality control metrics and the coding used to [[Quality_control_coding|identify]] them.
 
Quality-control metrics that are currently identified include;
 
The purpose of quality-control metrics is to identify poor data , the reasons why it is poor, and to facilitate the Ellison of compromised data from an estimation of the rate of dissipation.
 
Spikes in epsilon estimates arise from a number of causes such as collisions of sensor tips with suspended particles (e.g. detritus, plankton, jelly fish, seaweed), electronic noise due to other sensors, or mechanical platform vibrations. This section describes quality control measures and its coding.  
 
In a ''first step'', epsilon estimates are flagged based on quality control metric and disagreement between dissipation estimates from redundant sensors.   
 
# Quality-control metrics that accompanied dissipation estimates are used to flag individual estimates. In particular, quality control thresholds for
#:* figure of merit (FM)
#:* fraction of shear-probe data altered by the de-spiking routine
#:* number of iterations of the de-spiking routine required to clean the data
#:* agreement between dissipation estimates from redundant sensors (i.e. two or more shear probes)


:* [[figure of merit (FOM)]]
:* [[fraction of shear-probe data altered by the de-spiking routine]]
:* number of [[iterations]] of the de-spiking routine required to clean the data
:* [[agreement between dissipation estimates]] from redundant sensors (i.e. two or more shear probes)


BELOW MUST BE UPDATED
The numerical threshold for these metrics should depend, as much as possible, on the known statistical properties of a turbulence shear measurement.
The numerical values of the QC codes (or flags) is found in [[Quality_control_coding|QC-flags]].


Flagged data will receive quality control coding 4.


In a ''second step'' of quality control, a review of ensembles that have been flagged is performed. Individual shear spectra, associated tilt and acceleration data and microstructure thermistor data/spectra are visually examined for consistency. Previously flagged data that appears to be good data will receive quality control coding 2.
Please note that most choices made must be included in a data set, as described
in the [[Netcdf meta data (shear probes)|list of meta data]].


Details for QC coding can be found [[Quality_control_coding|here]].


</div>
</div>

Latest revision as of 20:15, 6 June 2024

The processing of shear-probe data can be divided into five major steps, which apply to data collected with any platform or vehicle. There are many sub-steps to these major steps. The major steps are:

Conversion to physical units


As a first step, the raw binary data needs to be transformed into physical units.

  1. Determine the speed of profiling of the shear-probe through the water.
  2. Determine the temperature of the water.
  3. Convert the shear probe data samples into physical units
  4. Convert all other signals per the recommendations of the manufacturer of the sensor or instruments that produce these signals.

Please note that most choices made must be included in a data set, as described in the list of meta data.


"Section" selection


Before you can process your shear-probe data to derive the rate of dissipation you must select the section of data that you wish to process. For a vertically profiling instrument, this is traditionnally referred to as a "profile". We adopt the term "section" as this is platform independent and will include time series for dissipation estimates along horizontal or slanted trajectories as well as from moored shear probes. You must make sure that the selection is meaningful and sensible. For example, the shear probe must be profiling through the water with a speed, direction, and orientation that is fairly stationary. The selection of data can be partially automated by requiring that the kinematics of your instrument achieve certain minimum criteria. The steps to section selection are as follows:

  1. Choose the minimum speed of profiling.
  2. Choose the direction of the vertical velocity of the profiler.
  3. Choose the minimum depth.
  4. Choose the maximum pitch and roll of the profiler.
  5. Choose the minimum duration over which the minimum speed through maximum pitch and roll must be satisfied.


Please note that most choices made must be included in a data set, as described in the list of meta data.


Choosing the processing parameters


  1. Choose the lowest wavenumber of spectral estimation (as a function of the vehicle's length) – fft-length.
  2. Choose the length of data (in meters) used for each dissipation estimate – diss-length.
  3. Choose a high-pass filter cut-off frequency to be consistent with duration of the fft-length.
  4. Choose de-spiking parameters.
  5. Choose vibration-coherent noise removal.
  6. Choose quality-control parameters.


Please note that most choices made must be included in a data set, as described in the list of meta data.


Compute the dissipation rate estimates


The following steps are recommended to obtain estimates of the turbulent dissipation rate of kinetic energy ([math]\displaystyle{ \varepsilon }[/math]).

  1. Extract the section to estimate dissipation time series ("Section" selection).
  2. High-pass filter the shear-probe and (optionally) the vibration data.
  3. Identify each diss-length segment in the section.
  4. De-spike the shear-probe data, and track the fraction of data affected by de-spiking within each diss-length segment. This will become a quality-control metric.
  5. Calculate the frequency spectra and cross-spectra of shear and vibrations for each detrended diss-length segment.
  6. Extract the original and the vibration-coherent clean shear-probe frequency spectra with the Goodman algorithm.
  7. Correct shear and vibration frequency spectra for the high-pass filter.
  8. Correct the cleaned frequency spectra for the bias induced by the Goodman algorithm.
  9. Convert the frequency spectra into wavenumber spectra using the mean speed, [math]\displaystyle{ U }[/math], for each diss-length segment. That is, make the wavenumber [math]\displaystyle{ k=f/U }[/math] and the spectrum [math]\displaystyle{ E(k)=UE(f) }[/math] .
  10. Correct the spectra of shear for the wavenumber response of the shear probe.
  11. Apply an iterative spectral integration algorithm to estimate the variance of shear.
  12. If the dissipation estimate is larger than shear inertial subrange fit use the method fit to the inertial subrange
  13. Calculate the turbulent dissipation rate by multiplying the shear variance by [math]\displaystyle{ \frac{15}{2}\nu }[/math] where [math]\displaystyle{ \nu }[/math] is the temperature-dependent kinematic viscosity.
  14. Determine the figure of merit (FOM) for each shear-probe spectrum.
  15. Calculate the expected variance of each dissipation estimate.


Please note that most choices made must be included in a data set, as described in the list of meta data.


Apply quality-control metrics


Shear-probe data can be corrupted or compromised in several different ways. These include but are not limited to (i) collision with plankton and other materials, (ii) unremovable vibrational contamination. (iii) electronic noise, and (iv) interference from other instrumentation on a platform that carries the shear probes. This section describes the quality control metrics and the coding used to identify them. Quality-control metrics that are currently identified include;

The numerical threshold for these metrics should depend, as much as possible, on the known statistical properties of a turbulence shear measurement. The numerical values of the QC codes (or flags) is found in QC-flags.


Please note that most choices made must be included in a data set, as described in the list of meta data.



Return to Shear Probe Welcome Page