View source for De-spike the shear-probe data

There is currently no standard method of de-spiking shear-probe data.


An algorithm that seems to effectively remove spikes from shear-probe data uses the following steps.

# Data are high-pass filtered with a first-order Butterworth filter with a cutoff frequency of <math>0.1\, \mathrm{Hz}</math> to remove any offset and very-low frequency signals.
# The shear data are rectified by taking their absolute value.
# A copy of the rectified shear-probe data is smoothed with a first-order low-pass filter with a cutoff frequency, that is usually in the range of <math>0.25</math> to <math>2\ \mathrm{Hz}</math>.
# Those samples for which the ratio of the absolute to the smoothed absolute shear exceeds a threshold (8 is a typical choice), are identified as spikes.
# A number <math>N</math> of samples after a spike and <math>N/2</math> samples before a spike are replace by a constant value equal to the mean shear of an interval of one-half second before and after this area of replacement.

The purpose of the low-pass filter is to establish the level of shear in a neighbourhood of duration that is roughly equal to the inverse of the low-pass filter cutoff frequency. 
A shear sample is anomalous if its magnitude exceeds the typical magnitude of its neighbourhood by more than a factor of the threshold. 
Thus, if the variance of shear is small, a small anomaly is detected, while the same anomaly remains undetected if the variance of shear is large.
That is, only anomalies that have the potential to bias the variance are removed.

What is a suitable neighbourhood and low-pass cutoff frequency?
Turbulent patches in the ocean are seldom thinner than about <math>0.5\ \mathrm{m}</math> in the vertical direction.
This can serve as a lower limit to the neighbourhood and an upper limit to the cutoff frequency.
Thus, if a vertical profiler is moving at a speed of <math>0.5\ \mathrm{m\, s^{-1}}</math>, then the cutoff frequency should be no higher than <math>1\ \mathrm{Hz}</math>.
Gliders that move more slowly and at an angle of <math>30^{\circ}</math> with respect to the horizontal should use a lower cutoff frequency to establish a neighbourhood for comparison of the ratio of signals, because they will take longer to pass through a patch of turbulence.
A cutoff frequency of <math>0.5\ \mathrm{Hz}</math> is often used for gliders.

A spike usually consists of a number of contiguous samples. 
A region surrounding the spike is then replaced by a local mean calculated using data from both sides of the spike (but excluding the <math>N</math> samples after and the <math>N/2</math> samples before the spike itself). 
Because the response of the shear probe to a collision with plankton  -- the ringing of anomalously large amplitude – is a temporal response, the amount of data replaced by a local mean is usually of fixed duration and not fixed in length. Typically, the amount of data replaced is 20ms before a spike and 40ms after a spike, because 40ms seems to be the typical relaxation time due to a collision with zooplankton. Some data before the collision is removed because of the run up to the peak of a spike. 

The iterative application permits longer anomalies to be removed, such as those that might occur because of collisions with jelly fish and seaweed. The fraction of the data that is altered by a de-spiking routine must be noted for each diss-length segment because this is a quality-control metric. Dissipation estimates should be treated with caution if the fraction of altered data exceeds a few percent. But, there is currently no standard for what is an acceptable fraction.



----------------------------
return to [[De-spiking parameters]]

return to [[Flow chart for shear probes]]