Velocity despiking - Revision history

CynthiaBluteau: /* Identifying the threshold */

2021-12-08T21:24:22Z

Identifying the threshold

← Older revision		Revision as of 21:24, 8 December 2021
Line 32:		Line 32:
	Some filter-based methods use thresholds based on a factor of the original time series' standard deviation (e.g., Tukey method discussed by Goring & Nikora<ref name="GoringNikora"/>). The results of the analysis then become sensitive to this choice of factor. A more automated means to identify the threshold was proposed by Brock (1986)<ref name="Brock"/> using a [[:Media:Threshold logic Brock1986.png\|histogram of velocity differences]] (the original minus the smoothed signal). The local minimums on either side from the center define the positive and negative thresholds. Velocity differences exceeding these thresholds are deemed to be spikes.		Some filter-based methods use thresholds based on a factor of the original time series' standard deviation (e.g., Tukey method discussed by Goring & Nikora<ref name="GoringNikora"/>). The results of the analysis then become sensitive to this choice of factor. A more automated means to identify the threshold was proposed by Brock (1986)<ref name="Brock"/> using a [[:Media:Threshold logic Brock1986.png\|histogram of velocity differences]] (the original minus the smoothed signal). The local minimums on either side from the center define the positive and negative thresholds. Velocity differences exceeding these thresholds are deemed to be spikes.

	[[File:Threshold logic Brock1986.png\|400px\|thumb\|Histogram of velocity differences. The local minimum denotes the lower and upper thresholds ~~<math>T</math>~~ for identifying spikes. The standard deviation of the original time series is 2 cm/s, while the thresholds are -4.8 and 5.3 cm/s.]]		[[File:Threshold logic Brock1986.png\|400px\|thumb\|Histogram of velocity differences between the original (spiky) and smoothed signals. The local minimum denotes the lower and upper thresholds for identifying spikes. The standard deviation of the original time series is 2 cm/s, while the thresholds are -4.8 and 5.3 cm/s.]]

	=Phase-space thresholding methods=		=Phase-space thresholding methods=

CynthiaBluteau: /* Identifying the threshold */

2021-12-08T21:23:07Z

Identifying the threshold

← Older revision		Revision as of 21:23, 8 December 2021
Line 30:		Line 30:

	===Identifying the threshold===		===Identifying the threshold===
	Some filter-based methods use thresholds based on a factor of the original time series' standard deviation (e.g., Tukey method discussed by Goring & Nikora<ref name="GoringNikora"/>). The results of the analysis then become sensitive to this choice of factor. A more automated means to identify the threshold was proposed by Brock (1986)<ref name="Brock"/> using a [[:Media:Threshold logic Brock1986.png\|histogram of velocity differences]]. The local minimums on either side from the center define the positive and negative thresholds. Velocity differences exceeding these thresholds are deemed to be spikes.		Some filter-based methods use thresholds based on a factor of the original time series' standard deviation (e.g., Tukey method discussed by Goring & Nikora<ref name="GoringNikora"/>). The results of the analysis then become sensitive to this choice of factor. A more automated means to identify the threshold was proposed by Brock (1986)<ref name="Brock"/> using a [[:Media:Threshold logic Brock1986.png\|histogram of velocity differences]] (the original minus the smoothed signal). The local minimums on either side from the center define the positive and negative thresholds. Velocity differences exceeding these thresholds are deemed to be spikes.

	[[File:Threshold logic Brock1986.png\|400px\|thumb\|Histogram of velocity differences. The local minimum denotes the lower and upper thresholds <math>T</math> for identifying spikes. The standard deviation of the original time series is 2 cm/s, while the thresholds are -4.8 and 5.3 cm/s.]]		[[File:Threshold logic Brock1986.png\|400px\|thumb\|Histogram of velocity differences. The local minimum denotes the lower and upper thresholds <math>T</math> for identifying spikes. The standard deviation of the original time series is 2 cm/s, while the thresholds are -4.8 and 5.3 cm/s.]]

CynthiaBluteau: /* Filter based methods */

2021-12-08T21:20:31Z

Filter based methods

← Older revision		Revision as of 21:20, 8 December 2021
Line 20:		Line 20:

	=Filter based methods=		=Filter based methods=
	This category of methods is quite common in the literature and has been reviewed extensively by others <ref name="GoringNikora"/><ref name="Starkenburg"/>. These methods compare the original signal against a smoothed version, and values exceeding a threshold ~~<math>T</math>~~ are deemed spikes. Most commonly, the threshold is applied to the time series of [[:Media:Smoothed velocities.png\|velocity differences]] obtained by subtracting the smoothed velocities from the original spiky signal.		This category of methods is quite common in the literature and has been reviewed extensively by others <ref name="GoringNikora"/><ref name="Starkenburg"/>. These methods compare the original signal against a smoothed version, and values exceeding a threshold are deemed spikes. Most commonly, the threshold is applied to the time series of [[:Media:Smoothed velocities.png\|velocity differences]] obtained by subtracting the smoothed velocities from the original spiky signal.

	[[File:Smoothed velocities.png\|400px\|thumb\|Example velocities before and after smoothing (top). Filter-based despiking methods use the difference between these two signals to identify spikes (bottom).]]		[[File:Smoothed velocities.png\|400px\|thumb\|Example velocities before and after smoothing (top). Filter-based despiking methods use the difference between these two signals to identify spikes (bottom).]]

CynthiaBluteau at 23:27, 30 November 2021

2021-11-30T23:27:45Z

← Older revision		Revision as of 23:27, 30 November 2021
Line 46:		Line 46:
	-----		-----
	Return to [[Velocity_point-measurements\|Velocity point-measurements' welcome page]]		Return to [[Velocity_point-measurements\|Velocity point-measurements' welcome page]]

	~~[[Category:Fundamentals]]~~

CynthiaBluteau: /* Overview */

2021-11-30T15:52:26Z

Overview

← Older revision		Revision as of 15:52, 30 November 2021
Line 11:		Line 11:
	* [[Phase-space despiking methods\|Phase-space thresholding techniques]] applied to [[Acoustic-Doppler Velocimeters\| acoustic-Doppler velocimeters]] <ref name="GoringNikora"/>		* [[Phase-space despiking methods\|Phase-space thresholding techniques]] applied to [[Acoustic-Doppler Velocimeters\| acoustic-Doppler velocimeters]] <ref name="GoringNikora"/>
	* [[Filter-based despiking methods]] such as those used in atmospheric turbulence studies <ref name="Starkenburg">{{Cite journal\|authors= D. Starkenburg, S. Metzger, G.J. Fochesatto, J.G Alfieri, R. Gens, A. Prakash and J. Cristobal\|journal_or_publisher= J. Atmos. Oceanic Technoly\|paper_or_booktitle= Assessment of Despiking Methods for Turbulence Data in Micrometeorology		* [[Filter-based despiking methods]] such as those used in atmospheric turbulence studies <ref name="Starkenburg">{{Cite journal\|authors= D. Starkenburg, S. Metzger, G.J. Fochesatto, J.G Alfieri, R. Gens, A. Prakash and J. Cristobal\|journal_or_publisher= J. Atmos. Oceanic Technoly\|paper_or_booktitle= Assessment of Despiking Methods for Turbulence Data in Micrometeorology
	\|year= 2016\|doi= 10.1175/JTECH-D-15-0154.1􏰐}}</ref>.		\|year= 2016\|doi= 10.1175/JTECH-D-15-0154.1􏰐}}</ref>. The subgroup is investigating in particular the median filter despiking technique <ref name="Brock">{{Cite journal\|authors=F.V. Brock\|journal_or_publisher= J. Atmos. Oceanic Technoly\|paper_or_booktitle= A nonlinear filter to remove impulse noise from meteorological data

	The subgroup ~~are~~ investigating in particular the median filter despiking technique <ref name="Brock">{{Cite journal\|authors=F.V. Brock\|journal_or_publisher= J. Atmos. Oceanic Technoly\|paper_or_booktitle= A nonlinear filter to remove impulse noise from meteorological data
	\|year= 1986\|doi= 10.1175/1520-0426(1986)003,0051:ANFTRI.2.0.CO;2􏰐}}</ref>.		\|year= 1986\|doi= 10.1175/1520-0426(1986)003,0051:ANFTRI.2.0.CO;2􏰐}}</ref>.

CynthiaBluteau: Grammar check

2021-11-30T15:51:57Z

Grammar check

CynthiaBluteau: /* Filter based methods */

2021-11-30T15:50:19Z

Filter based methods

← Older revision		Revision as of 15:50, 30 November 2021
Line 29:		Line 29:
	In the 1970s, scientists developed methods based on nonlinear median filters		In the 1970s, scientists developed methods based on nonlinear median filters
	<ref>{{Cite journal\|authors=A.E. Beaton, J.W. Tukey\|journal_or_publisher= Technometrics\|paper_or_booktitle= The fitting of power series, meaning polynomials, illustrated on band-spectroscopic data		<ref>{{Cite journal\|authors=A.E. Beaton, J.W. Tukey\|journal_or_publisher= Technometrics\|paper_or_booktitle= The fitting of power series, meaning polynomials, illustrated on band-spectroscopic data
	\|year= 1974\|doi=10.1080/00401706.1974.10489171}}</ref>. These filters are less biased by spikes than other linear operations such as low-pass ~~butterworth~~ filters<ref name="Starkenburg"/>. The moving median filter, for example, only requires a window size, i.e, the number of points over which to calculate the median. This size must be longer than the duration of spikes, which can span consecutive samples. This number must also be sufficiently short to compute a reasonable local median (see Table 1 of Starkenburg et al 2016<ref name="Starkenburg"/>). The challenge then becomes determining the threshold for identifying spikes.		\|year= 1974\|doi=10.1080/00401706.1974.10489171}}</ref>. These filters are less biased by spikes than other linear operations such as low-pass Butterworth filters<ref name="Starkenburg"/>. The moving median filter, for example, only requires a window size, i.e, the number of points over which to calculate the median. This size must be longer than the duration of spikes, which can span consecutive samples. This number must also be sufficiently short to compute a reasonable local median (see Table 1 of Starkenburg et al 2016<ref name="Starkenburg"/>). The challenge then becomes determining the threshold for identifying spikes.

	===Identifying the threshold===		===Identifying the threshold===

CynthiaBluteau: /* Overview */

2021-11-30T15:49:32Z

Overview

← Older revision		Revision as of 15:49, 30 November 2021
Line 9:		Line 9:
	Spikes in velocities measured by [[Acoustic-Doppler Velocimeters\| acoustic-Doppler velocimeters]] can result from aliasing of the Doppler signal, in particular when pulses become contaminated by reflecting from complex objects and boundaries <ref name="GoringNikora">{{Cite journal\|authors= D. G. Goring and V.I. Nikora\|journal_or_publisher= Journal of Hydraulic Engineering\|paper_or_booktitle= Despiking Acoustic Doppler Velocimeter Data		Spikes in velocities measured by [[Acoustic-Doppler Velocimeters\| acoustic-Doppler velocimeters]] can result from aliasing of the Doppler signal, in particular when pulses become contaminated by reflecting from complex objects and boundaries <ref name="GoringNikora">{{Cite journal\|authors= D. G. Goring and V.I. Nikora\|journal_or_publisher= Journal of Hydraulic Engineering\|paper_or_booktitle= Despiking Acoustic Doppler Velocimeter Data
	\|year= 2002\|doi= 10.1061/􏰠ASCE􏰐0733-9429􏰠2002􏰐128:1􏰠117􏰐}}</ref>. Several techniques exist for despiking:		\|year= 2002\|doi= 10.1061/􏰠ASCE􏰐0733-9429􏰠2002􏰐128:1􏰠117􏰐}}</ref>. Several techniques exist for despiking:
	* [[Phase-space despiking methods\|Phase-space thresholding techniques]] ~~<ref name="GoringNikora"/> for~~ [[Acoustic-Doppler Velocimeters\| acoustic-Doppler velocimeters]]~~; and~~		* [[Phase-space despiking methods\|Phase-space thresholding techniques]] applied to [[Acoustic-Doppler Velocimeters\| acoustic-Doppler velocimeters]] <ref name="GoringNikora"/>
	* [[Filter-based despiking methods]] such as those used in atmospheric turbulence studies <ref name="Starkenburg">{{Cite journal\|authors= D. Starkenburg, S. Metzger, G.J. Fochesatto, J.G Alfieri, R. Gens, A. Prakash and J. Cristobal\|journal_or_publisher= J. Atmos. Oceanic Technoly\|paper_or_booktitle= Assessment of Despiking Methods for Turbulence Data in Micrometeorology		* [[Filter-based despiking methods]] such as those used in atmospheric turbulence studies <ref name="Starkenburg">{{Cite journal\|authors= D. Starkenburg, S. Metzger, G.J. Fochesatto, J.G Alfieri, R. Gens, A. Prakash and J. Cristobal\|journal_or_publisher= J. Atmos. Oceanic Technoly\|paper_or_booktitle= Assessment of Despiking Methods for Turbulence Data in Micrometeorology
	\|year= 2016\|doi= 10.1175/JTECH-D-15-0154.1􏰐}}</ref>.		\|year= 2016\|doi= 10.1175/JTECH-D-15-0154.1􏰐}}</ref>.

CynthiaBluteau: /* Overview */

2021-11-30T15:48:55Z

Overview

← Older revision		Revision as of 15:48, 30 November 2021
Line 8:		Line 8:
	=Overview =		=Overview =
	Spikes in velocities measured by [[Acoustic-Doppler Velocimeters\| acoustic-Doppler velocimeters]] can result from aliasing of the Doppler signal, in particular when pulses become contaminated by reflecting from complex objects and boundaries <ref name="GoringNikora">{{Cite journal\|authors= D. G. Goring and V.I. Nikora\|journal_or_publisher= Journal of Hydraulic Engineering\|paper_or_booktitle= Despiking Acoustic Doppler Velocimeter Data		Spikes in velocities measured by [[Acoustic-Doppler Velocimeters\| acoustic-Doppler velocimeters]] can result from aliasing of the Doppler signal, in particular when pulses become contaminated by reflecting from complex objects and boundaries <ref name="GoringNikora">{{Cite journal\|authors= D. G. Goring and V.I. Nikora\|journal_or_publisher= Journal of Hydraulic Engineering\|paper_or_booktitle= Despiking Acoustic Doppler Velocimeter Data
	\|year= 2002\|doi= 10.1061/􏰠ASCE􏰐0733-9429􏰠2002􏰐128:1􏰠117􏰐}}</ref>. Several techniques exist for despiking~~. The most used with~~ [[~~Acoustic~~-~~Doppler Velocimeters~~\| ~~acoustic-Doppler velocimeters]] are the phase~~-space thresholding techniques <ref name="GoringNikora"/>~~. Other techniques currently being investigated by the subgroup are~~ those used in atmospheric turbulence studies <ref name="Starkenburg">{{Cite journal\|authors= D. Starkenburg, S. Metzger, G.J. Fochesatto, J.G Alfieri, R. Gens, A. Prakash and J. Cristobal\|journal_or_publisher= J. Atmos. Oceanic Technoly\|paper_or_booktitle= Assessment of Despiking Methods for Turbulence Data in Micrometeorology		\|year= 2002\|doi= 10.1061/􏰠ASCE􏰐0733-9429􏰠2002􏰐128:1􏰠117􏰐}}</ref>. Several techniques exist for despiking:
	\|year= 2016\|doi= 10.1175/JTECH-D-15-0154.1􏰐}}</ref>, in particular the median filter despiking technique <ref name="Brock">{{Cite journal\|authors=F.V. Brock\|journal_or_publisher= J. Atmos. Oceanic Technoly\|paper_or_booktitle= A nonlinear filter to remove impulse noise from meteorological data		* [[Phase-space despiking methods\|Phase-space thresholding techniques]] <ref name="GoringNikora"/> for [[Acoustic-Doppler Velocimeters\| acoustic-Doppler velocimeters]]; and
			* [[Filter-based despiking methods]] such as those used in atmospheric turbulence studies <ref name="Starkenburg">{{Cite journal\|authors= D. Starkenburg, S. Metzger, G.J. Fochesatto, J.G Alfieri, R. Gens, A. Prakash and J. Cristobal\|journal_or_publisher= J. Atmos. Oceanic Technoly\|paper_or_booktitle= Assessment of Despiking Methods for Turbulence Data in Micrometeorology
			\|year= 2016\|doi= 10.1175/JTECH-D-15-0154.1􏰐}}</ref>.

			The subgroup are investigating in particular the median filter despiking technique <ref name="Brock">{{Cite journal\|authors=F.V. Brock\|journal_or_publisher= J. Atmos. Oceanic Technoly\|paper_or_booktitle= A nonlinear filter to remove impulse noise from meteorological data
	\|year= 1986\|doi= 10.1175/1520-0426(1986)003,0051:ANFTRI.2.0.CO;2􏰐}}</ref>.		\|year= 1986\|doi= 10.1175/1520-0426(1986)003,0051:ANFTRI.2.0.CO;2􏰐}}</ref>.

CynthiaBluteau: /* Overview */

2021-11-29T21:03:04Z

Overview

← Older revision		Revision as of 21:03, 29 November 2021
Line 13:		Line 13:

	<div><ul>		<div><ul>
	<li style="display: inline-block; vertical-align: top;"> [[File:Velocities spike.png\|thumb\|none\|~~400px~~\|Example velocities measured by an [[Acoustic-Doppler Velocimeters\| acoustic-Doppler velocimeters]]]] </li>		<li style="display: inline-block; vertical-align: top;"> [[File:Velocities spike.png\|thumb\|none\|350px\|Example velocities measured by an [[Acoustic-Doppler Velocimeters\| acoustic-Doppler velocimeters]]]] </li>
	<li style="display: inline-block; vertical-align: top;"> [[File:Despike spectra.png\|thumb\|none\|~~400px~~\|Example velocity spectra before and after removing spikes. The variance of the despiked signal has reduced by 12% even though there were 30 spikes]] </li>		<li style="display: inline-block; vertical-align: top;"> [[File:Despike spectra.png\|thumb\|none\|350px\|Example velocity spectra before and after removing spikes. The variance of the despiked signal has reduced by 12% even though there were 30 spikes]] </li>
	</ul></div>		</ul></div>



	=Filter based methods=		=Filter based methods=

← Older revision		Revision as of 15:52, 30 November 2021
Line 11:		Line 11:
	* [[Phase-space despiking methods\|Phase-space thresholding techniques]] applied to [[Acoustic-Doppler Velocimeters\| acoustic-Doppler velocimeters]] <ref name="GoringNikora"/>		* [[Phase-space despiking methods\|Phase-space thresholding techniques]] applied to [[Acoustic-Doppler Velocimeters\| acoustic-Doppler velocimeters]] <ref name="GoringNikora"/>
	* [[Filter-based despiking methods]] such as those used in atmospheric turbulence studies <ref name="Starkenburg">{{Cite journal\|authors= D. Starkenburg, S. Metzger, G.J. Fochesatto, J.G Alfieri, R. Gens, A. Prakash and J. Cristobal\|journal_or_publisher= J. Atmos. Oceanic Technoly\|paper_or_booktitle= Assessment of Despiking Methods for Turbulence Data in Micrometeorology		* [[Filter-based despiking methods]] such as those used in atmospheric turbulence studies <ref name="Starkenburg">{{Cite journal\|authors= D. Starkenburg, S. Metzger, G.J. Fochesatto, J.G Alfieri, R. Gens, A. Prakash and J. Cristobal\|journal_or_publisher= J. Atmos. Oceanic Technoly\|paper_or_booktitle= Assessment of Despiking Methods for Turbulence Data in Micrometeorology
	\|year= 2016\|doi= 10.1175/JTECH-D-15-0154.1􏰐}}</ref>.		\|year= 2016\|doi= 10.1175/JTECH-D-15-0154.1􏰐}}</ref>. The subgroup is investigating in particular the median filter despiking technique <ref name="Brock">{{Cite journal\|authors=F.V. Brock\|journal_or_publisher= J. Atmos. Oceanic Technoly\|paper_or_booktitle= A nonlinear filter to remove impulse noise from meteorological data

	The subgroup ~~are~~ investigating in particular the median filter despiking technique <ref name="Brock">{{Cite journal\|authors=F.V. Brock\|journal_or_publisher= J. Atmos. Oceanic Technoly\|paper_or_booktitle= A nonlinear filter to remove impulse noise from meteorological data
	\|year= 1986\|doi= 10.1175/1520-0426(1986)003,0051:ANFTRI.2.0.CO;2􏰐}}</ref>.		\|year= 1986\|doi= 10.1175/1520-0426(1986)003,0051:ANFTRI.2.0.CO;2􏰐}}</ref>.

← Older revision		Revision as of 15:49, 30 November 2021
Line 9:		Line 9:
	Spikes in velocities measured by [[Acoustic-Doppler Velocimeters\| acoustic-Doppler velocimeters]] can result from aliasing of the Doppler signal, in particular when pulses become contaminated by reflecting from complex objects and boundaries <ref name="GoringNikora">{{Cite journal\|authors= D. G. Goring and V.I. Nikora\|journal_or_publisher= Journal of Hydraulic Engineering\|paper_or_booktitle= Despiking Acoustic Doppler Velocimeter Data		Spikes in velocities measured by [[Acoustic-Doppler Velocimeters\| acoustic-Doppler velocimeters]] can result from aliasing of the Doppler signal, in particular when pulses become contaminated by reflecting from complex objects and boundaries <ref name="GoringNikora">{{Cite journal\|authors= D. G. Goring and V.I. Nikora\|journal_or_publisher= Journal of Hydraulic Engineering\|paper_or_booktitle= Despiking Acoustic Doppler Velocimeter Data
	\|year= 2002\|doi= 10.1061/􏰠ASCE􏰐0733-9429􏰠2002􏰐128:1􏰠117􏰐}}</ref>. Several techniques exist for despiking:		\|year= 2002\|doi= 10.1061/􏰠ASCE􏰐0733-9429􏰠2002􏰐128:1􏰠117􏰐}}</ref>. Several techniques exist for despiking:
	* [[Phase-space despiking methods\|Phase-space thresholding techniques]] ~~<ref name="GoringNikora"/> for~~ [[Acoustic-Doppler Velocimeters\| acoustic-Doppler velocimeters]]~~; and~~		* [[Phase-space despiking methods\|Phase-space thresholding techniques]] applied to [[Acoustic-Doppler Velocimeters\| acoustic-Doppler velocimeters]] <ref name="GoringNikora"/>
	* [[Filter-based despiking methods]] such as those used in atmospheric turbulence studies <ref name="Starkenburg">{{Cite journal\|authors= D. Starkenburg, S. Metzger, G.J. Fochesatto, J.G Alfieri, R. Gens, A. Prakash and J. Cristobal\|journal_or_publisher= J. Atmos. Oceanic Technoly\|paper_or_booktitle= Assessment of Despiking Methods for Turbulence Data in Micrometeorology		* [[Filter-based despiking methods]] such as those used in atmospheric turbulence studies <ref name="Starkenburg">{{Cite journal\|authors= D. Starkenburg, S. Metzger, G.J. Fochesatto, J.G Alfieri, R. Gens, A. Prakash and J. Cristobal\|journal_or_publisher= J. Atmos. Oceanic Technoly\|paper_or_booktitle= Assessment of Despiking Methods for Turbulence Data in Micrometeorology
	\|year= 2016\|doi= 10.1175/JTECH-D-15-0154.1􏰐}}</ref>.		\|year= 2016\|doi= 10.1175/JTECH-D-15-0154.1􏰐}}</ref>.

← Older revision		Revision as of 15:48, 30 November 2021
Line 8:		Line 8:
	=Overview =		=Overview =
	Spikes in velocities measured by [[Acoustic-Doppler Velocimeters\| acoustic-Doppler velocimeters]] can result from aliasing of the Doppler signal, in particular when pulses become contaminated by reflecting from complex objects and boundaries <ref name="GoringNikora">{{Cite journal\|authors= D. G. Goring and V.I. Nikora\|journal_or_publisher= Journal of Hydraulic Engineering\|paper_or_booktitle= Despiking Acoustic Doppler Velocimeter Data		Spikes in velocities measured by [[Acoustic-Doppler Velocimeters\| acoustic-Doppler velocimeters]] can result from aliasing of the Doppler signal, in particular when pulses become contaminated by reflecting from complex objects and boundaries <ref name="GoringNikora">{{Cite journal\|authors= D. G. Goring and V.I. Nikora\|journal_or_publisher= Journal of Hydraulic Engineering\|paper_or_booktitle= Despiking Acoustic Doppler Velocimeter Data
	\|year= 2002\|doi= 10.1061/􏰠ASCE􏰐0733-9429􏰠2002􏰐128:1􏰠117􏰐}}</ref>. Several techniques exist for despiking~~. The most used with~~ [[~~Acoustic~~-~~Doppler Velocimeters~~\| ~~acoustic-Doppler velocimeters]] are the phase~~-space thresholding techniques <ref name="GoringNikora"/>~~. Other techniques currently being investigated by the subgroup are~~ those used in atmospheric turbulence studies <ref name="Starkenburg">{{Cite journal\|authors= D. Starkenburg, S. Metzger, G.J. Fochesatto, J.G Alfieri, R. Gens, A. Prakash and J. Cristobal\|journal_or_publisher= J. Atmos. Oceanic Technoly\|paper_or_booktitle= Assessment of Despiking Methods for Turbulence Data in Micrometeorology		\|year= 2002\|doi= 10.1061/􏰠ASCE􏰐0733-9429􏰠2002􏰐128:1􏰠117􏰐}}</ref>. Several techniques exist for despiking:
	\|year= 2016\|doi= 10.1175/JTECH-D-15-0154.1􏰐}}</ref>, in particular the median filter despiking technique <ref name="Brock">{{Cite journal\|authors=F.V. Brock\|journal_or_publisher= J. Atmos. Oceanic Technoly\|paper_or_booktitle= A nonlinear filter to remove impulse noise from meteorological data		* [[Phase-space despiking methods\|Phase-space thresholding techniques]] <ref name="GoringNikora"/> for [[Acoustic-Doppler Velocimeters\| acoustic-Doppler velocimeters]]; and
			* [[Filter-based despiking methods]] such as those used in atmospheric turbulence studies <ref name="Starkenburg">{{Cite journal\|authors= D. Starkenburg, S. Metzger, G.J. Fochesatto, J.G Alfieri, R. Gens, A. Prakash and J. Cristobal\|journal_or_publisher= J. Atmos. Oceanic Technoly\|paper_or_booktitle= Assessment of Despiking Methods for Turbulence Data in Micrometeorology
			\|year= 2016\|doi= 10.1175/JTECH-D-15-0154.1􏰐}}</ref>.

			The subgroup are investigating in particular the median filter despiking technique <ref name="Brock">{{Cite journal\|authors=F.V. Brock\|journal_or_publisher= J. Atmos. Oceanic Technoly\|paper_or_booktitle= A nonlinear filter to remove impulse noise from meteorological data
	\|year= 1986\|doi= 10.1175/1520-0426(1986)003,0051:ANFTRI.2.0.CO;2􏰐}}</ref>.		\|year= 1986\|doi= 10.1175/1520-0426(1986)003,0051:ANFTRI.2.0.CO;2􏰐}}</ref>.

← Older revision		Revision as of 15:51, 30 November 2021
Line 4:		Line 4:
	}}		}}

	Any measured signal may be contaminated by spikes. These spikes are typically short-lived and transient, which result in sudden change in the [[:Media:Velocities spike.png\| measured velocities]]. These spikes must be removed because they can alter quite dramatically the [[:Media:Despike spectra.png\|velocity spectra]], which is then used for fitting [[Velocity inertial subrange model\|inertial subrange model]].		Any measured signal may be contaminated by spikes. These spikes are typically short-lived and transient, which result in sudden change in the [[:Media:Velocities spike.png\| measured velocities]]. These spikes must be removed because they can alter quite dramatically the [[:Media:Despike spectra.png\|velocity spectra]], which is then used for fitting [[Velocity inertial subrange model\|inertial subrange model]].

	=Overview =		=Overview =
	Spikes in velocities measured by [[Acoustic-Doppler Velocimeters\| acoustic-Doppler velocimeters]] can result from aliasing of the Doppler signal, in particular when pulses become contaminated by reflecting from complex objects and boundaries <ref name="GoringNikora">{{Cite journal\|authors= D. G. Goring and V.I. Nikora\|journal_or_publisher= Journal of Hydraulic Engineering\|paper_or_booktitle= Despiking Acoustic Doppler Velocimeter Data		Spikes in velocities measured by [[Acoustic-Doppler Velocimeters\| acoustic-Doppler velocimeters]] can result from aliasing of the Doppler signal, in particular when pulses become contaminated by reflecting from complex objects and boundaries <ref name="GoringNikora">{{Cite journal\|authors= D. G. Goring and V.I. Nikora\|journal_or_publisher= Journal of Hydraulic Engineering\|paper_or_booktitle= Despiking Acoustic Doppler Velocimeter Data
	\|year= 2002\|doi= 10.1061/􏰠ASCE􏰐0733-9429􏰠2002􏰐128:1􏰠117􏰐}}</ref>. Several techniques exist for despiking:		\|year= 2002\|doi= 10.1061/􏰠ASCE􏰐0733-9429􏰠2002􏰐128:1􏰠117􏰐}}</ref>. Several techniques exist for despiking:
	* [[Phase-space despiking methods\|Phase-space thresholding techniques]] applied to [[Acoustic-Doppler Velocimeters\| acoustic-Doppler velocimeters]] <ref name="GoringNikora"/>		* [[Phase-space despiking methods\|Phase-space thresholding techniques]] applied to [[Acoustic-Doppler Velocimeters\| acoustic-Doppler velocimeters]] <ref name="GoringNikora"/>
Line 22:		Line 22:

	=Filter based methods=		=Filter based methods=
	This category of methods is quite common in the literature and ~~have~~ been reviewed extensively by others <ref name="GoringNikora"/><ref name="Starkenburg"/>. These methods compare the original signal against a smoothed version, and values exceeding a threshold <math>T</math> are deemed ~~to be~~ spikes. Most commonly, the threshold is applied to the ~~timeseries~~ of [[:Media:Smoothed velocities.png\|velocity differences]] obtained by subtracting the smoothed velocities from the original spiky signal.		This category of methods is quite common in the literature and has been reviewed extensively by others <ref name="GoringNikora"/><ref name="Starkenburg"/>. These methods compare the original signal against a smoothed version, and values exceeding a threshold <math>T</math> are deemed spikes. Most commonly, the threshold is applied to the time series of [[:Media:Smoothed velocities.png\|velocity differences]] obtained by subtracting the smoothed velocities from the original spiky signal.

	[[File:Smoothed velocities.png\|400px\|thumb\|Example velocities before and after smoothing (top). Filter-based despiking methods use the difference between these two signals to identify spikes (bottom).]]		[[File:Smoothed velocities.png\|400px\|thumb\|Example velocities before and after smoothing (top). Filter-based despiking methods use the difference between these two signals to identify spikes (bottom).]]
Line 29:		Line 29:
	In the 1970s, scientists developed methods based on nonlinear median filters		In the 1970s, scientists developed methods based on nonlinear median filters
	<ref>{{Cite journal\|authors=A.E. Beaton, J.W. Tukey\|journal_or_publisher= Technometrics\|paper_or_booktitle= The fitting of power series, meaning polynomials, illustrated on band-spectroscopic data		<ref>{{Cite journal\|authors=A.E. Beaton, J.W. Tukey\|journal_or_publisher= Technometrics\|paper_or_booktitle= The fitting of power series, meaning polynomials, illustrated on band-spectroscopic data
	\|year= 1974\|doi=10.1080/00401706.1974.10489171}}</ref>. These filters are less biased by spikes than other linear operations such as low-pass Butterworth filters<ref name="Starkenburg"/>. The moving median filter, for example, only requires a window ~~size~~, i.e, the number of points over which to calculate the median. This ~~size~~ must be longer than the duration of spikes, which can span consecutive samples. This number must also be sufficiently short to compute a reasonable local median (see Table 1 of Starkenburg et al 2016<ref name="Starkenburg"/>). The challenge then becomes determining the threshold for identifying spikes.		\|year= 1974\|doi=10.1080/00401706.1974.10489171}}</ref>. These filters are less biased by spikes than other linear operations such as low-pass Butterworth filters<ref name="Starkenburg"/>. The moving median filter, for example, only requires a window length, i.e, the number of points over which to calculate the median. This window length must be longer than the duration of spikes, which can span consecutive samples. This number must also be sufficiently short to compute a reasonable local median (see Table 1 of Starkenburg et al 2016<ref name="Starkenburg"/>). The challenge then becomes determining the threshold for identifying spikes.

	===Identifying the threshold===		===Identifying the threshold===
	Some filter-based methods use thresholds based on a factor of the original ~~timeseries~~' standard deviation (e.g., Tukey method discussed by Goring & Nikora<ref name="GoringNikora"/>). The results of the analysis then ~~becomes~~ sensitive to this choice of factor. A more automated means to identify the threshold was proposed by Brock (1986)<ref name="Brock"/> using a [[:Media:Threshold logic Brock1986.png\|histogram of velocity differences]]. The local minimums on either side from the center ~~defines~~ the positive and negative thresholds. Velocity differences exceeding these thresholds are deemed to be spikes.		Some filter-based methods use thresholds based on a factor of the original time series' standard deviation (e.g., Tukey method discussed by Goring & Nikora<ref name="GoringNikora"/>). The results of the analysis then become sensitive to this choice of factor. A more automated means to identify the threshold was proposed by Brock (1986)<ref name="Brock"/> using a [[:Media:Threshold logic Brock1986.png\|histogram of velocity differences]]. The local minimums on either side from the center define the positive and negative thresholds. Velocity differences exceeding these thresholds are deemed to be spikes.

	[[File:Threshold logic Brock1986.png\|400px\|thumb\|Histogram of velocity differences. The local minimum denotes the lower and upper thresholds <math>T</math> for identifying spikes. The standard deviation of the original ~~timeseries~~ is 2 cm/s, while the thresholds are -4.8 and 5.3 cm/s.]]		[[File:Threshold logic Brock1986.png\|400px\|thumb\|Histogram of velocity differences. The local minimum denotes the lower and upper thresholds <math>T</math> for identifying spikes. The standard deviation of the original time series is 2 cm/s, while the thresholds are -4.8 and 5.3 cm/s.]]

	=Phase-space thresholding methods=		=Phase-space thresholding methods=
	'''Elaborate here''' on the methods of Goring and Nikora<ref name="GoringNikora"/>, which was developed specifically for acoustic-doppler velocity measurements. The other papers referenced above include methods for turbulence measurements for other systems such as eddy-correlation. The JTECH paper by Starkenburg et al. (2016)<ref name="Starkenburg"/> review a number of methods, including those of Goring and Nikora. Starkenburg et al. (2016) deemed median based filtering a more robust means of filtering data since it can handle missing points more easily, and handles better ~~timeseries~~ with evidence of low-frequency coherent turbulent structures.		'''Elaborate here''' on the methods of Goring and Nikora<ref name="GoringNikora"/>, which was developed specifically for acoustic-doppler velocity measurements. The other papers referenced above include methods for turbulence measurements for other systems such as eddy-correlation. The JTECH paper by Starkenburg et al. (2016)<ref name="Starkenburg"/> review a number of methods, including those of Goring and Nikora. Starkenburg et al. (2016) deemed median-based filtering a more robust means of filtering data since it can handle missing points more easily, and handles better time series with evidence of low-frequency coherent turbulent structures.