Replacement strategies for missing velocities - Revision history

CynthiaBluteau: /* Test Results */

2022-07-05T19:10:36Z

Test Results

← Older revision		Revision as of 19:10, 5 July 2022
Line 30:		Line 30:
	For all tests, the linear interpolation did the best job in recovering the original spectra, followed by the unevenly spaced techniques. However, when the data loss was intermittent, the unevenly spaced Fourier transforms behaved similarly (if not worse) than the variance replacement. Unsurprisingly, unevenly spaced techniques fair better if the data loss creates long continuous gaps.		For all tests, the linear interpolation did the best job in recovering the original spectra, followed by the unevenly spaced techniques. However, when the data loss was intermittent, the unevenly spaced Fourier transforms behaved similarly (if not worse) than the variance replacement. Unsurprisingly, unevenly spaced techniques fair better if the data loss creates long continuous gaps.

	[[File:Spectra_replacement_strategies.png\|center\|thumbnail\|~~700px~~\|Spectral estimated from the data presented above after using the different replacement strategies: unevenly spaced least-square Fourier transforms (uneven), the variance of the signal (var), and linear interpolation (interp). The original time series refers to the data before removing (randomly) samples. For a given data loss (e.g., 10%), the original spectra were easier to recover with longer continuous gaps (d vs a)]]		[[File:Spectra_replacement_strategies.png\|center\|thumbnail\|800px\|Spectral estimated from the data presented above after using the different replacement strategies: unevenly spaced least-square Fourier transforms (uneven), the variance of the signal (var), and linear interpolation (interp). The original time series refers to the data before removing (randomly) samples. For a given data loss (e.g., 10%), the original spectra were easier to recover with longer continuous gaps (d vs a)]]

	== Recommendations ==		== Recommendations ==

CynthiaBluteau: /* Test Results */

2022-07-05T15:25:33Z

Test Results

← Older revision		Revision as of 15:25, 5 July 2022
Line 30:		Line 30:
	For all tests, the linear interpolation did the best job in recovering the original spectra, followed by the unevenly spaced techniques. However, when the data loss was intermittent, the unevenly spaced Fourier transforms behaved similarly (if not worse) than the variance replacement. Unsurprisingly, unevenly spaced techniques fair better if the data loss creates long continuous gaps.		For all tests, the linear interpolation did the best job in recovering the original spectra, followed by the unevenly spaced techniques. However, when the data loss was intermittent, the unevenly spaced Fourier transforms behaved similarly (if not worse) than the variance replacement. Unsurprisingly, unevenly spaced techniques fair better if the data loss creates long continuous gaps.

	[[File:Spectra_replacement_strategies.png\|~~left~~\|thumbnail\|~~600px~~\|Spectral estimated from the data presented above after using the different replacement strategies: unevenly spaced least-square Fourier transforms (uneven), the variance of the signal (var), and linear interpolation (interp). The original time series refers to the data before removing (randomly) samples. For a given data loss (e.g., 10%), the original spectra were easier to recover with longer continuous gaps (d vs a)]]		[[File:Spectra_replacement_strategies.png\|center\|thumbnail\|700px\|Spectral estimated from the data presented above after using the different replacement strategies: unevenly spaced least-square Fourier transforms (uneven), the variance of the signal (var), and linear interpolation (interp). The original time series refers to the data before removing (randomly) samples. For a given data loss (e.g., 10%), the original spectra were easier to recover with longer continuous gaps (d vs a)]]

	== Recommendations ==		== Recommendations ==

CynthiaBluteau: /* Data analysis tests */

2022-07-05T15:23:08Z

Data analysis tests

← Older revision		Revision as of 15:23, 5 July 2022
Line 10:		Line 10:
	* Using the variance of the signal, which is commonly used by those intending to compute eddy-covariances		* Using the variance of the signal, which is commonly used by those intending to compute eddy-covariances
	* Unevenly spaced least-square Fourier transform (i.e., no replacement at all)		* Unevenly spaced least-square Fourier transform (i.e., no replacement at all)

			[[File:Timeseries_replacement_strategies.png\|thumbnail\|600px\|Example velocity time series where we randomly removed data in varying length gaps. Only the example of 1min (480 samples at 8Hz sampling) are illustrated. Removing chunks of 8 continuous samples (1s) looks identical to the original time series and is not illustrated.]]

	=== Replacement strategy for tests ===		=== Replacement strategy for tests ===
Line 21:		Line 23:
	** 960 samples (120 s)		** 960 samples (120 s)

	[[File:Timeseries_replacement_strategies.png\|thumbnail\|600px\|Example velocity time series where we randomly removed data in varying length gaps. Only the example of 1min (480 samples at 8Hz sampling) are illustrated. Removing chunks of 8 continuous samples (1s) looks identical to the original time series and is not illustrated.]]

	A total section of 30 min was chosen as this coincides with the required segment length for deriving an estimate of <math>\varepsilon</math>.		A total section of 30 min was chosen as this coincides with the required segment length for deriving an estimate of <math>\varepsilon</math>.

CynthiaBluteau: /* Test Results */

2022-07-05T15:21:51Z

Test Results

← Older revision		Revision as of 15:21, 5 July 2022
Line 26:		Line 26:

	==Test Results==		==Test Results==
	~~{{FontColor\|fg=white\|bg=red\|text=Insert graphs with example spectra for different tests}}~~
	For all tests, the linear interpolation did the best job in recovering the original spectra, followed by the unevenly spaced techniques. However, the unevenly spaced fourier transforms behaved similarly (if not worse) than the variance replacement when the data loss was intermittent. Unsurprisingly, unevenly spaced techniques fair better if the data loss form long continuous gaps.

	* For a given data loss (e.g., 10%), the original spectra ~~was~~ easier to recover with longer continuous gaps.		For all tests, the linear interpolation did the best job in recovering the original spectra, followed by the unevenly spaced techniques. However, when the data loss was intermittent, the unevenly spaced Fourier transforms behaved similarly (if not worse) than the variance replacement. Unsurprisingly, unevenly spaced techniques fair better if the data loss creates long continuous gaps.

			[[File:Spectra_replacement_strategies.png\|left\|thumbnail\|600px\|Spectral estimated from the data presented above after using the different replacement strategies: unevenly spaced least-square Fourier transforms (uneven), the variance of the signal (var), and linear interpolation (interp). The original time series refers to the data before removing (randomly) samples. For a given data loss (e.g., 10%), the original spectra were easier to recover with longer continuous gaps (d vs a)]]

	== Recommendations ==		== Recommendations ==

CynthiaBluteau: /* Replacement strategy for tests */

2022-07-05T15:14:28Z

Replacement strategy for tests

← Older revision		Revision as of 15:14, 5 July 2022
Line 20:		Line 20:
	** 480 samples (60 s)		** 480 samples (60 s)
	** 960 samples (120 s)		** 960 samples (120 s)

			[[File:Timeseries_replacement_strategies.png\|thumbnail\|600px\|Example velocity time series where we randomly removed data in varying length gaps. Only the example of 1min (480 samples at 8Hz sampling) are illustrated. Removing chunks of 8 continuous samples (1s) looks identical to the original time series and is not illustrated.]]

	A total section of 30 min was chosen as this coincides with the required segment length for deriving an estimate of <math>\varepsilon</math>.		A total section of 30 min was chosen as this coincides with the required segment length for deriving an estimate of <math>\varepsilon</math>.

CynthiaBluteau: /* Recommendations */

2022-07-05T14:16:10Z

Recommendations

← Older revision		Revision as of 14:16, 5 July 2022
Line 30:		Line 30:

	== Recommendations ==		== Recommendations ==
	~~Always use~~ linear interpolation, and record the percent good samples in each segment ~~so that this variable can be used to reject epsilon~~ with more than 10% data loss. ~~This~~ threshold may be relaxed if the data loss forms long continuous chunks of several seconds, ~~and tests with~~ the ~~timeseries shows that~~ the ~~estimated spectrum~~ can tolerate more missing samples. ~~The person processing the data should record~~ the threshold used ~~and report it in the NetCDF flags~~.		* Use linear interpolation, and record the percent of good samples in each segment in the NetCDF [[Level 3 data (velocity point-measurements)\| Level 3 data]].
			* Reject and flag <math>\varepsilon</math> associated with segments with more than 10% data loss.
			* Record the threshold used for flagging and report it in the NetCDF file at [[Level 4 data (velocity point-measurements)\| Level 4]].
			** The rejection threshold may be relaxed if the data loss forms long continuous chunks of several seconds, after testing the actual time series used to establish if the spectra can tolerate more missing samples. Recording the threshold used at [[Level 4 data (velocity point-measurements)\| Level 4]] would permit others to exclude these data at a later time.
	----		----
	Return to [[Preparing quality-controlled velocities]]		Return to [[Preparing quality-controlled velocities]]

CynthiaBluteau at 14:08, 5 July 2022

2022-07-05T14:08:24Z

← Older revision		Revision as of 14:08, 5 July 2022
Line 11:		Line 11:
	* Unevenly spaced least-square Fourier transform (i.e., no replacement at all)		* Unevenly spaced least-square Fourier transform (i.e., no replacement at all)

	== Replacement strategy tests ==		=== Replacement strategy for tests ===
	These replacement strategies were trialed with one of the cleanest benchmarks (Underice MAVS sampling at 8 Hz) for different		These replacement strategies were trialed with one of the cleanest benchmarks (Underice MAVS sampling at 8 Hz) for different
	* Number of missing samples to identify a threshold where the segment should be completely discarded from further analysis		* Number of missing samples to identify a threshold where the segment should be completely discarded from further analysis
Line 23:		Line 23:
	A total section of 30 min was chosen as this coincides with the required segment length for deriving an estimate of <math>\varepsilon</math>.		A total section of 30 min was chosen as this coincides with the required segment length for deriving an estimate of <math>\varepsilon</math>.

	===Results===		==Test Results==
	{{FontColor\|fg=white\|bg=red\|text=Insert graphs with example spectra for different tests}}		{{FontColor\|fg=white\|bg=red\|text=Insert graphs with example spectra for different tests}}
	For all tests, the linear interpolation did the best job in recovering the original spectra, followed by the unevenly spaced techniques. However, the unevenly spaced fourier transforms behaved similarly (if not worse) than the variance replacement when the data loss was intermittent. Unsurprisingly, unevenly spaced techniques fair better if the data loss form long continuous gaps.		For all tests, the linear interpolation did the best job in recovering the original spectra, followed by the unevenly spaced techniques. However, the unevenly spaced fourier transforms behaved similarly (if not worse) than the variance replacement when the data loss was intermittent. Unsurprisingly, unevenly spaced techniques fair better if the data loss form long continuous gaps.

CynthiaBluteau: /* Considerations */

2022-07-05T14:07:22Z

Considerations

← Older revision		Revision as of 14:07, 5 July 2022
Line 6:		Line 6:

	== Data analysis tests ==		== Data analysis tests ==
	===~~Considerations===~~		===Techniques considered for replacing the missing samples===
	~~Several techniques were~~ considered for replacing the missing samples:
	* Linear interpolation		* Linear interpolation
	* Using the variance of the signal, which is commonly used by those intending to compute eddy-covariances		* Using the variance of the signal, which is commonly used by those intending to compute eddy-covariances
	* Unevenly spaced least-square Fourier transform (i.e., no replacement at all)		* Unevenly spaced least-square Fourier transform (i.e., no replacement at all)

			== Replacement strategy tests ==
	These replacement strategies were trialed with one of the cleanest benchmarks (Underice MAVS sampling at 8 Hz) for different		These replacement strategies were trialed with one of the cleanest benchmarks (Underice MAVS sampling at 8 Hz) for different
	* Number of missing samples to identify a threshold where the segment should be completely discarded from further analysis		* Number of missing samples to identify a threshold where the segment should be completely discarded from further analysis

CynthiaBluteau: /* Recommendations */

2022-07-05T12:54:12Z

Recommendations

← Older revision		Revision as of 12:54, 5 July 2022
Line 32:		Line 32:
	Always use linear interpolation, and record the percent good samples in each segment so that this variable can be used to reject epsilon with more than 10% data loss. This threshold may be relaxed if the data loss forms long continuous chunks of several seconds, and tests with the timeseries shows that the estimated spectrum can tolerate more missing samples. The person processing the data should record the threshold used and report it in the NetCDF flags.		Always use linear interpolation, and record the percent good samples in each segment so that this variable can be used to reject epsilon with more than 10% data loss. This threshold may be relaxed if the data loss forms long continuous chunks of several seconds, and tests with the timeseries shows that the estimated spectrum can tolerate more missing samples. The person processing the data should record the threshold used and report it in the NetCDF flags.

	---		----
	Return to [[Preparing quality-controlled velocities]]		Return to [[Preparing quality-controlled velocities]]

CynthiaBluteau: /* Recommendations */

2022-07-05T12:53:46Z

Recommendations

← Older revision		Revision as of 12:53, 5 July 2022
Line 31:		Line 31:
	== Recommendations ==		== Recommendations ==
	Always use linear interpolation, and record the percent good samples in each segment so that this variable can be used to reject epsilon with more than 10% data loss. This threshold may be relaxed if the data loss forms long continuous chunks of several seconds, and tests with the timeseries shows that the estimated spectrum can tolerate more missing samples. The person processing the data should record the threshold used and report it in the NetCDF flags.		Always use linear interpolation, and record the percent good samples in each segment so that this variable can be used to reject epsilon with more than 10% data loss. This threshold may be relaxed if the data loss forms long continuous chunks of several seconds, and tests with the timeseries shows that the estimated spectrum can tolerate more missing samples. The person processing the data should record the threshold used and report it in the NetCDF flags.

			---
			Return to [[Preparing quality-controlled velocities]]