Atomix - User contributions [en]

Flow chart for shear probes

2023-09-15T11:18:24Z

Sarah: /* Compute the dissipation rate estimates */

The processing of shear-probe data can be divided into the following five major steps and these steps apply to data collected with any platform or vehicle. There are many sub-steps to these major steps. The major steps are:

__TOC__

== Conversion to physical units ==
<div class="mw-collapsible" id="physical units" data-collapsetext="Collapse" data-expandtext="Expand">
<br>
As a first step, the raw binary data needs to be transformed into physical units.
:* [[Determine the speed of profiling]] of the shear-probe through the water.
:* [[Determine the temperature of the water]].
:* [[Convert the shear probe data]] samples into physical units
:* Convert all other signals per the recommendations of the manufacturer of the sensor or instruments that produce these signals.
</div>

Please note that most choices made must be included in a data set, as described
in the [[Netcdf meta data (shear probes)|list of meta data]].

== "Section" selection ==
<div class="mw-collapsible" id="Section" data-collapsetext="Collapse" data-expandtext="Expand">
<br>
Before you can process your shear-probe data to derive the rate of dissipation you must select the [[section]] of data that you wish to process. For a vertically profiling instrument, this is traditionnally referred to as a "profile". We adopt the term "section" as this is platform independent and will include time series for dissipation estimates along horizontal or slanted trajectories as well as from moored shear probes. You must make sure that the selection is meaningful and sensible. For example, the shear probe must be profiling through the water with a speed, direction, and orientation that is fairly stationary. The selection of data can be partially automated by requiring that the kinematics of your instrument achieve certain minimum criteria. The steps to section selection are as follows:
:* Choose the [[minimum speed]] of profiling.
:* Choose the [[direction of the vertical velocity]] of the profiler.
:* Choose the [[minimum depth]].
:* Choose the [[maximum pitch and roll]] of the profiler.
:* Choose the [[minimum duration]] over which the [[minimum speed]] through [[maximum pitch and roll]] must be satisfied.

Please note that most choices made must be included in a data set, as described
in the [[Netcdf meta data (shear probes)|list of meta data]].

</div>

== Choosing the [[processing parameters]] ==
<div class="mw-collapsible" id="processing parameters" data-collapsetext="Collapse" data-expandtext="Expand">
<br>
:* Choose the length of data (in meters) used for each dissipation estimate – [[diss-length]].
:* Choose the lowest wavenumber of spectral estimation – [[fft-length]].
:* Translate [[diss-length]] and [[fft-length]] into [[duration]] (time).
:* Round these up to [[nearest power-of-two number]] of samples.
:* Choose a [[high-pass filter cut-off frequency]] to be consistent with duration of the [[fft-length]].
:* Choose [[de-spiking parameters]].
:* Choose [[vibration-coherent noise removal]].

Please note that most choices made must be included in a data set, as described
in the [[Netcdf meta data (shear probes)|list of meta data]].

</div>

== Compute the [[dissipation rate estimates]] ==

<div class="mw-collapsible" id="estimates" data-collapsetext="Collapse" data-expandtext="Expand">
<br>
The following steps are recommended to obtain estimates of the turbulent dissipation rate of kinetic energy (<math>\varepsilon</math>).

# Extract the section to estimate dissipation time series ("Section" selection).
# High-pass filter the shear-probe and (optionally) the vibration data.
# Identify each diss-length segment in the section.
# [[De-spike the shear-probe data]], and track the fraction of data affected by de-spiking within each diss-length segment. This will become a [[Shear_probes_quality_control_metrics|quality-control metric]].
# Calculate the [[frequency spectra and cross-spectra of shear and vibrations]] for each diss-length segment.
# Extract the original and the vibration-coherent clean shear-probe frequency spectra with [[the Goodman algorithm]].
# Correct shear and vibration frequency spectra for [[the high-pass filter]].
# Correct the cleaned frequency spectra for [[the bias induced by the Goodman algorithm]].
# Convert the frequency spectra into wavenumber spectra using the mean speed, <math>U</math>, for each diss-length segment. That is, make the wavenumber <math>k=f/U</math> and the spectrum <math>E(k)=UE(f)</math> .
# Correct the spectra of shear for the [[wavenumber response of the shear probe]].
# Apply an [[iterative spectral integration algorithm]] to estimate the variance of shear.
# If the dissipation estimate is larger than [[shear inertial subrange fit]] use the method fit to the inertial subrange
# Calculate the turbulent dissipation rate by multiplying the shear variance by <math>\frac{15}{2}\nu</math> where <math>\nu </math> is the temperature-dependent kinematic viscosity.
# Determine the [[figure of merit (FOM)]] for each shear-probe spectrum.
# Calculate the expected variance of each dissipation estimate.

Please note that most choices made must be included in a data set, as described
in the [[Netcdf meta data (shear probes)|list of meta data]].

</div>

== Apply [[Shear_probes_quality_control_metrics|quality-control metrics]] ==

<div class="mw-collapsible" id="quality_control" data-collapsetext="Collapse" data-expandtext="Expand">
<br>
Shear-probe data can be corrupted or compromised in several different ways.
These include but are not limited to (''i'') collision with plankton and other materials, (''ii'') unremovable vibrational contamination. (''iii'') electronic noise, and (''iv'') interference from other instrumentation on a platform that carries the shear probes.
This section describes the quality control metrics and the coding used to [[Quality_control_coding|identify]] them.
Quality-control metrics that are currently identified include;

:* [[figure of merit (FOM)]]
:* [[fraction of shear-probe data altered by the de-spiking routine]]
:* number of [[iterations]] of the de-spiking routine required to clean the data
:* [[agreement between dissipation estimates]] from redundant sensors (i.e. two or more shear probes)

The numerical threshold for these metrics should depend, as much as possible, on the known statistical properties of a turbulence shear measurement.
The numerical values of the QC codes (or flags) is found in [[Quality_control_coding|QC-flags]].

Please note that most choices made must be included in a data set, as described
in the [[Netcdf meta data (shear probes)|list of meta data]].

</div>
--------------------
Return to [[ Shear probes | Shear Probe Welcome Page]]
[[Category: Shear probes]]

Shear inertial subrange fit

2023-09-15T11:12:30Z

Sarah: Created page with "{{DefineConcept |parameter_name=shear_inertial_subrange_fit |description=For very high rates of dissipation, such as <math>\varepsilon=1\times 10^{-5}\ \mathrm{W\, kg^{-1}}</m..."

{{DefineConcept
|parameter_name=shear_inertial_subrange_fit
|description=For very high rates of dissipation, such as <math>\varepsilon=1\times 10^{-5}\ \mathrm{W\, kg^{-1}}</math>, the shear-probe cannot fully resolve the spectrum of shear. We recommend to estimate the rate by fitting to the spectrum in the inertial subrange.

In this range, the spectrum rises in proportion to <math>\varepsilon^{2/3}k^{1/3}</math> and, thus, its level provides an estimate of the rate of dissipation. The inertial subrange is confined to wavenumbers smaller than <math>k(\nu^3/\varepsilon)^{1/4}=0.02</math>, and thus will usually use fewer spectral points than the method of spectral integration. This reduces the statistical reliability of the dissipation estimate but it does avoid the bias introduced by not fully resolving the spectrum of shear.

<math></math>
|instrument_type=Shear probes
}}

Netcdf meta data (shear probes)

2023-09-15T10:15:36Z

Sarah: /* Optional Metadata */

<br>
Below we list the metadata '''required''' in a NetCDF data file, as well as the optional meta data we '''highly recommend''' that are included with the data set.

The parameter names should be used as listed (attention to underscores). The descriptions are example text or several options that are available for that parameter.

=Required Metadata=
<div class="mw-collapsible mw-expand" data-collapsetext="Collapse" data-expandtext="Expand">
<br>
{| class="wikitable sortable"
|-
! Parameter Name !! Description
|-
|title || e.g., Vertical micostructure profiler data from cruise XXX
|-
|summary || required by ACDD
|-
|comment|| required by CF
|-
|platform_type ||research vessel, sub-surface glider etc
|-
|creation_time || yyyy-mm-ddTHH:MM:SSZ
|-
|date_created || yyyy-mm-ddTHH:MM:SSZ
|-
|date_update || yyyy-mm-ddTHH:MM:SSZ
|-
|time_reference_year || year for time reference
|-
| fs_fast || sampling frequency for fast channels
|-
| fs_slow <math>^a</math> || sampling frequency for slow channels
|-
| profiling_direction || horizontal, vertical, glide
|-
| fft_length || as data points (note, fft_lengths_sec in seconds is optional)
|-
| diss_length || as data points
|-
| overlap || as data points
|-
| goodman || 0=not applied; 1=applied
|-
| HP_cut || the high-pass filter cutoff frequency in Hz. Can be zero for no filtering.
|-
| conventions || CF-1.6, ACDD-1.3, ATOMIX
|-
| history || Version 1
|-
| data_mode || D #(D)elayed
|}
<math>^a</math> or fast_ctd or similar if such records are included
</div>

=Optional Metadata=
<div class="mw-collapsible mw-expand" data-collapsetext="Collapse" data-expandtext="Expand">
<br>
{| class="wikitable sortable"
|-
! Parameter Name !! Description
|-
| fft_length_sec || seconds
|-
| diss_length_sec || seconds
|-
| overlap_sec || seconds
|-
| f_AA || Hz, the anti-aliasing frequency.
|-
| FOM_limit || non-dimensional, if absent the FOM QC-flag is not set. Typically, 1.1.
|-
| diss_ratio_limit || non-dimensional, if absent the dissipation ration QC-flag is not set. Typically, 2.77.
|-
| despike_shear_fraction_limit || non-dimensional, if absent the de-spike fraction QC-flag is not set. Typically, 0.05.
|-
| despike_shear_iterations_limit || non-dimensional, if absent the de-spike iteration passes QC-flag is not set. Typically 8.
|-
| variance_resolved_limit || threshold for the minimum percent of variance resolved in a shear spectrum. Typically 50%.
|-
| f_limit || upper limit to exclude frequencies that have contaminations. Typically infinity.
|-
| fit_2_isr || dissipation threshold to use the method of fitting to the inertial subrange. Typically 10-5 W/kg.
|-
|area || e.g., Arctic Ocean, Barents Sea
|-
|geospatial_lat_min || latitude minimum of data
|-
|geospatial_lat_max || latitude maximum of data
|-
|geospatial_lon_min || longitude minimum of data
|-
|geospatial_lon_max || longitude maximum of data
|-
|geospatial_vertical_min|| 0
|-
|geospatial_vertical_max|| in m
|-
|geospatial_vertical_positive|| down, up
|-
|time_coverage_start || yyyy-mm-ddTHH:MM:SSZ
|-
|time_coverage_end || yyyy-mm-ddTHH:MM:SSZ
|-
|institution || e.g. University of Bergen
|-
|principal_investigator || Name of Principal Investigator
|-
|authors || Names of authors
|-
|contact || email address of corresponding author (usually principal investigator)
|-
|project_name || name of project for which the data was collected
|-
|cruise || name of cruise from which the data was collected
|-
|vessel || name of vessel from which the data was collected
|-
|source || From the SeaVoX Platform Categories vocabulary (L06) list, e.g. “subsurface mooring”, ”ship”, ""sub-surface gliders"", ""autonomous underwater vehicle"" (CF)
|-
|references || key references
|-
|keywords || relevant keywords describing data e.g. shear probes
|-
|creator_name || name of person who generated the file
|-
|creator_email || email address of person who generated the file
|-
|creator_url || website address of the creator
|-
|acknowledgement|| acknowledgements for this data. e.g. this could include crew of ship or funders
|-
|station_name || name of station
|}

</div>

return to [[Dataset requirements for shear probes]]

[[Category:Shear probes]]

Flow chart for shear probes

2023-09-15T10:00:47Z

Sarah: /* Apply quality-control metrics */

The processing of shear-probe data can be divided into the following five major steps and these steps apply to data collected with any platform or vehicle. There are many sub-steps to these major steps. The major steps are:

__TOC__

== Conversion to physical units ==
<div class="mw-collapsible" id="physical units" data-collapsetext="Collapse" data-expandtext="Expand">
<br>
As a first step, the raw binary data needs to be transformed into physical units.
:* [[Determine the speed of profiling]] of the shear-probe through the water.
:* [[Determine the temperature of the water]].
:* [[Convert the shear probe data]] samples into physical units
:* Convert all other signals per the recommendations of the manufacturer of the sensor or instruments that produce these signals.
</div>

Please note that most choices made must be included in a data set, as described
in the [[Netcdf meta data (shear probes)|list of meta data]].

== "Section" selection ==
<div class="mw-collapsible" id="Section" data-collapsetext="Collapse" data-expandtext="Expand">
<br>
Before you can process your shear-probe data to derive the rate of dissipation you must select the [[section]] of data that you wish to process. For a vertically profiling instrument, this is traditionnally referred to as a "profile". We adopt the term "section" as this is platform independent and will include time series for dissipation estimates along horizontal or slanted trajectories as well as from moored shear probes. You must make sure that the selection is meaningful and sensible. For example, the shear probe must be profiling through the water with a speed, direction, and orientation that is fairly stationary. The selection of data can be partially automated by requiring that the kinematics of your instrument achieve certain minimum criteria. The steps to section selection are as follows:
:* Choose the [[minimum speed]] of profiling.
:* Choose the [[direction of the vertical velocity]] of the profiler.
:* Choose the [[minimum depth]].
:* Choose the [[maximum pitch and roll]] of the profiler.
:* Choose the [[minimum duration]] over which the [[minimum speed]] through [[maximum pitch and roll]] must be satisfied.

Please note that most choices made must be included in a data set, as described
in the [[Netcdf meta data (shear probes)|list of meta data]].

</div>

== Choosing the [[processing parameters]] ==
<div class="mw-collapsible" id="processing parameters" data-collapsetext="Collapse" data-expandtext="Expand">
<br>
:* Choose the length of data (in meters) used for each dissipation estimate – [[diss-length]].
:* Choose the lowest wavenumber of spectral estimation – [[fft-length]].
:* Translate [[diss-length]] and [[fft-length]] into [[duration]] (time).
:* Round these up to [[nearest power-of-two number]] of samples.
:* Choose a [[high-pass filter cut-off frequency]] to be consistent with duration of the [[fft-length]].
:* Choose [[de-spiking parameters]].
:* Choose [[vibration-coherent noise removal]].

Please note that most choices made must be included in a data set, as described
in the [[Netcdf meta data (shear probes)|list of meta data]].

</div>

== Compute the [[dissipation rate estimates]] ==

<div class="mw-collapsible" id="estimates" data-collapsetext="Collapse" data-expandtext="Expand">
<br>
The following steps are recommended to obtain estimates of the turbulent dissipation rate of kinetic energy (<math>\varepsilon</math>).

# Extract the section to estimate dissipation time series ("Section" selection).
# High-pass filter the shear-probe and (optionally) the vibration data.
# Identify each diss-length segment in the section.
# [[De-spike the shear-probe data]], and track the fraction of data affected by de-spiking within each diss-length segment. This will become a [[Shear_probes_quality_control_metrics|quality-control metric]].
# Calculate the [[frequency spectra and cross-spectra of shear and vibrations]] for each diss-length segment.
# Extract the original and the vibration-coherent clean shear-probe frequency spectra with [[the Goodman algorithm]].
# Correct shear and vibration frequency spectra for [[the high-pass filter]].
# Correct the cleaned frequency spectra for [[the bias induced by the Goodman algorithm]].
# Convert the frequency spectra into wavenumber spectra using the mean speed, <math>U</math>, for each diss-length segment. That is, make the wavenumber <math>k=f/U</math> and the spectrum <math>E(k)=UE(f)</math> .
# Correct the spectra of shear for the [[wavenumber response of the shear probe]].
# Apply an [[iterative spectral integration algorithm]] to estimate the variance of shear.
# Calculate the turbulent dissipation rate by multiplying the shear variance by <math>\frac{15}{2}\nu</math> where <math>\nu </math> is the temperature-dependent kinematic viscosity.
# Determine the [[figure of merit (FOM)]] for each shear-probe spectrum.
# Calculate the expected variance of each dissipation estimate.

Please note that most choices made must be included in a data set, as described
in the [[Netcdf meta data (shear probes)|list of meta data]].

</div>

== Apply [[Shear_probes_quality_control_metrics|quality-control metrics]] ==

<div class="mw-collapsible" id="quality_control" data-collapsetext="Collapse" data-expandtext="Expand">
<br>
Shear-probe data can be corrupted or compromised in several different ways.
These include but are not limited to (''i'') collision with plankton and other materials, (''ii'') unremovable vibrational contamination. (''iii'') electronic noise, and (''iv'') interference from other instrumentation on a platform that carries the shear probes.
This section describes the quality control metrics and the coding used to [[Quality_control_coding|identify]] them.
Quality-control metrics that are currently identified include;

:* [[figure of merit (FOM)]]
:* [[fraction of shear-probe data altered by the de-spiking routine]]
:* number of [[iterations]] of the de-spiking routine required to clean the data
:* [[agreement between dissipation estimates]] from redundant sensors (i.e. two or more shear probes)

The numerical threshold for these metrics should depend, as much as possible, on the known statistical properties of a turbulence shear measurement.
The numerical values of the QC codes (or flags) is found in [[Quality_control_coding|QC-flags]].

Please note that most choices made must be included in a data set, as described
in the [[Netcdf meta data (shear probes)|list of meta data]].

</div>
--------------------
Return to [[ Shear probes | Shear Probe Welcome Page]]
[[Category: Shear probes]]

Flow chart for shear probes

2023-09-15T10:00:24Z

Sarah: /* Compute the dissipation rate estimates */

The processing of shear-probe data can be divided into the following five major steps and these steps apply to data collected with any platform or vehicle. There are many sub-steps to these major steps. The major steps are:

__TOC__

== Conversion to physical units ==
<div class="mw-collapsible" id="physical units" data-collapsetext="Collapse" data-expandtext="Expand">
<br>
As a first step, the raw binary data needs to be transformed into physical units.
:* [[Determine the speed of profiling]] of the shear-probe through the water.
:* [[Determine the temperature of the water]].
:* [[Convert the shear probe data]] samples into physical units
:* Convert all other signals per the recommendations of the manufacturer of the sensor or instruments that produce these signals.
</div>

Please note that most choices made must be included in a data set, as described
in the [[Netcdf meta data (shear probes)|list of meta data]].

== "Section" selection ==
<div class="mw-collapsible" id="Section" data-collapsetext="Collapse" data-expandtext="Expand">
<br>
Before you can process your shear-probe data to derive the rate of dissipation you must select the [[section]] of data that you wish to process. For a vertically profiling instrument, this is traditionnally referred to as a "profile". We adopt the term "section" as this is platform independent and will include time series for dissipation estimates along horizontal or slanted trajectories as well as from moored shear probes. You must make sure that the selection is meaningful and sensible. For example, the shear probe must be profiling through the water with a speed, direction, and orientation that is fairly stationary. The selection of data can be partially automated by requiring that the kinematics of your instrument achieve certain minimum criteria. The steps to section selection are as follows:
:* Choose the [[minimum speed]] of profiling.
:* Choose the [[direction of the vertical velocity]] of the profiler.
:* Choose the [[minimum depth]].
:* Choose the [[maximum pitch and roll]] of the profiler.
:* Choose the [[minimum duration]] over which the [[minimum speed]] through [[maximum pitch and roll]] must be satisfied.

Please note that most choices made must be included in a data set, as described
in the [[Netcdf meta data (shear probes)|list of meta data]].

</div>

== Choosing the [[processing parameters]] ==
<div class="mw-collapsible" id="processing parameters" data-collapsetext="Collapse" data-expandtext="Expand">
<br>
:* Choose the length of data (in meters) used for each dissipation estimate – [[diss-length]].
:* Choose the lowest wavenumber of spectral estimation – [[fft-length]].
:* Translate [[diss-length]] and [[fft-length]] into [[duration]] (time).
:* Round these up to [[nearest power-of-two number]] of samples.
:* Choose a [[high-pass filter cut-off frequency]] to be consistent with duration of the [[fft-length]].
:* Choose [[de-spiking parameters]].
:* Choose [[vibration-coherent noise removal]].

Please note that most choices made must be included in a data set, as described
in the [[Netcdf meta data (shear probes)|list of meta data]].

</div>

== Compute the [[dissipation rate estimates]] ==

<div class="mw-collapsible" id="estimates" data-collapsetext="Collapse" data-expandtext="Expand">
<br>
The following steps are recommended to obtain estimates of the turbulent dissipation rate of kinetic energy (<math>\varepsilon</math>).

# Extract the section to estimate dissipation time series ("Section" selection).
# High-pass filter the shear-probe and (optionally) the vibration data.
# Identify each diss-length segment in the section.
# [[De-spike the shear-probe data]], and track the fraction of data affected by de-spiking within each diss-length segment. This will become a [[Shear_probes_quality_control_metrics|quality-control metric]].
# Calculate the [[frequency spectra and cross-spectra of shear and vibrations]] for each diss-length segment.
# Extract the original and the vibration-coherent clean shear-probe frequency spectra with [[the Goodman algorithm]].
# Correct shear and vibration frequency spectra for [[the high-pass filter]].
# Correct the cleaned frequency spectra for [[the bias induced by the Goodman algorithm]].
# Convert the frequency spectra into wavenumber spectra using the mean speed, <math>U</math>, for each diss-length segment. That is, make the wavenumber <math>k=f/U</math> and the spectrum <math>E(k)=UE(f)</math> .
# Correct the spectra of shear for the [[wavenumber response of the shear probe]].
# Apply an [[iterative spectral integration algorithm]] to estimate the variance of shear.
# Calculate the turbulent dissipation rate by multiplying the shear variance by <math>\frac{15}{2}\nu</math> where <math>\nu </math> is the temperature-dependent kinematic viscosity.
# Determine the [[figure of merit (FOM)]] for each shear-probe spectrum.
# Calculate the expected variance of each dissipation estimate.

Please note that most choices made must be included in a data set, as described
in the [[Netcdf meta data (shear probes)|list of meta data]].

</div>

== Apply [[Shear_probes_quality_control_metrics|quality-control metrics]] ==

<div class="mw-collapsible" id="quality_control" data-collapsetext="Collapse" data-expandtext="Expand">
<br>
Shear-probe data can be corrupted or compromised in several different ways.
These include but are not limited to (''i'') collision with plankton and other materials, (''ii'') unremovable vibrational contamination. (''iii'') electronic noise, and (''iv'') interference from other instrumentation on a platform that carries the shear probes.
This section describes the quality control metrics and the coding used to [[Quality_control_coding|identify]] them.
Quality-control metrics that are currently identified include;

:* [[figure of merit (FM)]]
:* [[fraction of shear-probe data altered by the de-spiking routine]]
:* number of [[iterations]] of the de-spiking routine required to clean the data
:* [[agreement between dissipation estimates]] from redundant sensors (i.e. two or more shear probes)

The numerical threshold for these metrics should depend, as much as possible, on the known statistical properties of a turbulence shear measurement.
The numerical values of the QC codes (or flags) is found in [[Quality_control_coding|QC-flags]].

Please note that most choices made must be included in a data set, as described
in the [[Netcdf meta data (shear probes)|list of meta data]].

</div>
--------------------
Return to [[ Shear probes | Shear Probe Welcome Page]]
[[Category: Shear probes]]

Shear probes quality control metrics

2021-06-25T12:16:35Z

Sarah: spelling mistake: matric to metric

Spikes in epsilon estimates arise from a number of causes such as collisions of sensor tips with suspended particles (e.g. detritus, plankton, jelly fish, seaweed), electronic noise due to other sensors, or mechanical platform vibrations. This section describes quality control measures and its coding.

In a ''first step'', epsilon estimates are flagged based on quality control metric and disagreement between dissipation estimates from redundant sensors.

# Quality-control metrics (see also Processing Steps section V) that accompanied dissipation estimates are used to flag individual estimates. In particular, quality control thresholds for
#:* figure of merit (FM)
#:* fraction of shear-probe data altered by the de-spiking routine
#:* number of iterations of the de-spiking routine required to clean the data
#:* (more to be discussed)
# Agreement between dissipation estimates from redundant sensors (i.e. two or more shear probes) does not exist.

Flagged data will receive quality control coding 4.

In a ''second step'' of quality control, a review of ensembles that have been flagged is performed. Individual shear spectra, associated tilt and acceleration data and microstructure thermistor data/spectra are visually examined for consistency. Previously flagged data that appears to be good data will receive quality control coding 2.

Details for QC coding can be found [[Quality_control_coding|here]].