I have a set of experimental data s(t) which consists of a vector (with 81 points as a function of time t).
From the physics, this is the result of the convolution of the system response e(t) with a probe p(t), which is a Gaussian (actually a laser pulse). In terms of vector, its FWHM covers approximately 15 points in time.
I want to deconvolve this data in Matlab using the convolution theorem: FT{e(t)*p(t)}=FT{e(t)}xFT{p(t)} (where * is the convolution, x the product and FT the Fourier transform).
The procedure itself is no problem, if I suppose a Dirac function as my probe, I recover exactly the initial signal (which makes sense, measuring a system with a Dirac gives its impulse response)
However, the Gaussian case as a probe, as far as I understood turns out to be a critical one. When I divide the signal in the Fourier space by the FT of the probe, the wings of the Gaussian highly amplifies those frequencies and I completely loose my initial signal instead of having a deconvolved one.
From your experience, which method could be used here (like Hamming windows or any windowing technique, or...) ? This looks rather pretty simple but I did not find any easy way to follow in signal processing and this is not my field.
You have noise in your experimental data, do you? The problem is ill-posed then (non-uniquely solvable) and you need regularization.
If the noise is Gaussian the keywords are Tikhonov regularization or Wiener filtering.
Basically, add a positive regularization factor that acts as a lowpass filter. In your notation the estimation of the true curve o(t) then becomes:
o(t) = FT^-1(FT(e)*conj(FT(p))/(abs(FT(p))^2+l))
with a suitable l>0.
You're trying to do Deconvolution process by assuming the Filter Model is Gaussian Blur.
Few notes for doing Deconvolution:
Since your data is real (Not synthetic) data it includes some kind of Noise.
Hence it is better to use the Wiener Filter (Even with the assumption of low variance noise). Otherwise, the "Deconvolution Filter" will increase the noise significantly (As it is an High Pass basically).
When doing the division in the Fourier Domain zero pad the signals to the correct size or better yet create the Gaussian Filter in the time domain with the same number of samples as the signal.
Boundaries will create artifact, Windowing might be useful.
There are many more sophisticated methods for Deconvolution by defining a more sophisticated model on the signal and the noise. If you have more prior data about them, you should look for this kind of framework.
You can always set a threshold on the amplification level for certain frequencies, do that if needed.
Use as much samples as you can.
I hope this will assist you.
Related
I'm dealing with CWT, and I have a big problem converting scales to frequencies. In the MAtlab Wavelet Tutorial they use this expression to convert scales to frequencies
But if i use the default function scal2freq I obtain different result.
I don't understand the role of the Morlet Fourier Factor
Thanks in advance
It is a pretty complicated concept, which I somewhat understand it. I'll write some points here so that you might figure it out yourself, rather easier.
A simple fact is that:
Scale is inversely proportional to frequency.
For example, imagine we have a 1-100 Hz range of frequencies in some time series data such as stock markets data or earthquake data. Scale is "supposed to be" the inverse of that. For instance, if scale would be in range of 1 to 100, we'd have had:
Scale(1/Hz) Frequency (Hz)
1 100
50 50
100 1
Therefore,
The frequency is not the real frequency of those time series data (e.g., stock market, earthquake) that we know of. They are only related, inversely.
And we can safely say that here we are calculating some "pseudo-frequencies", which MATLAB does that (by approximating that). You can read about the approximation process in the documentation in the section pseudo-frequencies:
MATlAB does calculate those pseudo-frequencies based on:
In wavelet analysis, the way to relate scales to frequencies is to determine the center frequency of the wavelet function:
which you can visually see in this image and of-course it would differ, when we would change the types of our function in the calculation. Thus, that center frequency will change everytime in our approximation process:
That "MorletFourierFactor" is a variable to approximate a constant so that when you would do the 1/scale, it would closely approximate those "pseudo-frequencies".
I thought this image about shifting (time axis) and scaling (frequency axis) might be a little helpful to look into as well:
The bottom line is that don't worry about pseudo-frequencies, you wouldn't probably need those. If you would want any frequency spectrum, you can likely go towards applying some of those frequency methods (such as Fast Fourier Transform) on whatever time series data that you have.
If you really really want to map that, you can also try to design some methods to approximate it yourself.
Source
Harvard Seismology
I was using PCA for dimensionality reduction of MD (molecular dynamics) trajectory data of some protein simulations. Basically my data is xyz coordinates of protein atoms which change with time (that means I have lot of frames of this xyz coordinates). The dimension of this data is something like 20000 frames of 200x3 (atoms by coordinates). I implemented PCA using princomp command in Matlab.
I was wondering if I can do FFT on my data. I have experience of doing FFT on audio signals (1D signal). Here my data has both time and space in picture. It must be theoretically possible to implement FFT on my data and then filter it using a LPF (low pass filter). But I am unsure.
Can someone give me some direction/code snippets/references towards implementing FFT on my data?
Why people are preferring PCA more often compared to FFT and filtering. Is it because of computational efficiency of algorithm or is it because of the statistical nature of underlying data?
For the first question "Can someone give me some direction/code snippets/references towards implementing FFT on my data?":
I should say fft is implemented in matlab and you do not need to implement it by your own. Also, for your case you should use fftn (fft documentation)to transform and after applying lowpass filtering by dessignfilt (design filter in matalab), the apply ifftn (inverse fft in matlab)to inverse the transform.
For the second question "Why people are preferring PCA more often compared to FFT and filtering ...":
I should say as the filtering in fft is done in signal space, after filtering you can't generalize it in time space. You can more details about this drawback in this article.
But, Fourier analysis has also some other
serious drawbacks. One of them may be that time
information is lost in transforming to the frequency
domain. When looking at a Fourier transform of a
signal, it is impossible to tell when a particular event
has taken place. If it is a stationary signal - this
drawback isn't very important. However, most
interesting signals contain numerous non-stationary or
transitory characteristics: drift, trends, abrupt changes,
and beginnings and ends of events. These
characteristics are often the most important part of the
signal, and Fourier analysis is not suitable in detecting
them.
I have audio record.
I want to detect sinusoidal pattern.
If i do regular fft i have result with bad SNR.
for example
my signal contents 4 high frequencies:
fft result:
To reduce noise i want to do Coherent integration as described in this article: http://flylib.com/books/en/2.729.1.109/1/
but i cant find any MATLAB examples how to do it. Sorry for bad english. Please help )
I look at spectra almost every day, but I never heard of 'coherent integration' as a method to calculate one. As also mentioned by Jason, coherent integration would only work when your signal has a fixed phase during every FFT you average over.
It is more likely that you want to do what the article calls 'incoherent integration'. This is more commonly known as calculating a periodogram (or Welch's method, a slightly better variant), in which you average the squared absolute value of the individual FFTs to obtain a power-spectral-density. To calculate a PSD in the correct way, you need to pay attention to some details, like applying a suitable Fourier window before doing each FFT, doing the proper normalization (so that the result is properly calibrated in i.e. Volt^2/Hz) and using half-overlapping windows to make use of all your data. All of this is implemented in Matlab's pwelch function, which is part of the signal-processing toolbox. See my answer to a similar question about how to use pwelch.
Integration or averaging of FFT frames just amounts to adding the frames up element-wise and dividing by the number of frames. Since MATLAB provides vector operations, you can just add the frames with the + operator.
coh_avg = (frame1 + frame2 + ...) / Nframes
Where frameX are the complex FFT output frames.
If you want to do non-coherent averaging, you just need to take the magnitude of the complex elements before adding the frames together.
noncoh_avg = (abs(frame1) + abs(frame2) + ...) / Nframes
Also note that in order for coherent averaging to work the best, the starting phase of the signal of interest needs to be the same for each FFT frame. Otherwise, the FFT bin with the signal may add in such a way that the amplitudes cancel out. This is usually a tough requirement to ensure without some knowledge of the signal or some external triggering so it is more common to use non-coherent averaging.
Non-coherent integration will not reduce the noise power, but it will increase signal to noise ratio (how the signal power compares to the noise power), which is probably what you really want anyway.
I think what you are looking for is the "spectrogram" function in Matlab, which computes the short time Fourier transform(STFT) of an input signal.
STFT
Spectrogram
FIR filter has to be used for removing the noise.
I don't know the frequencies of the noise that might be adding up into the analog feedback signal I am taking.
My apparatus consists analog feedback signal then i am using ADC to digitize the value now I have to apply FIR filter to remove the noise, Now I am not sure which noise the noise which added up in the analog signal from the environment or some sort of noise comes there due to ADC ?
I have to code this in vhdl.(this part is easy I can do that).
My main problem is in deciding the frequencies.
Thanks in Advance !
I am tagging vhdl as some people who are working in vhdl might know about the filter.
Let me start by stating the obvious: An ADC samples at a fixed rate and can not represent any frequency higher than the Nyquist frequency
Step one: understand aliasing, and that any frequency higher than the Nyquist will alias into your signal as noise. Once you get this you understand that you need an anti aliasing filter in your hardware, in your analog signal path before you digitize it. Depending on the noise requirements of the application you may implement a very complicated 4 pole filter using op-amps; the simplest is to use an RC filter.
Step two: setting the filter cut off. Don't set the cutoff right at the Nyquist frequency, make sure the filter is cutting well before the nyquist (1/2x... 1/10x, depends really how clean and how much noise is present)
So now you're actually kind of over sampling your signal: The filter is cutting above your signal, and the sample rate is high enough such that the Nyquist frequency is sufficiently higher. Over sampling is kind of extra data, that you captured with the intent of filtering further, and possibly even decimating (keeping on in N samples and throwing the rest out)
Step three: use a filter to further remove the noise between the initial cut off of the anti-aliasing filter and the nyquist frequency. This is a science on it's own really, but let me start by suggesting a good decimation filter: Averaging 2 values. It's a box-car filter of order 2, also known as a SINC filter, and can be re applied N times. After N times it is the equivalent of an FIR using the values of the Nth row in pascal's triangle (and divided by their sum).
Again, the filter choice is a science on it's own really. To the extreme is the decimation filters of a sigma-delta ADC. The CS5376A datasheet clearly explains what they're doing; I learn quite a bit just from reading that datasheet!
For a homework assignment I have to design a simple bandpass filter in Matlab that filters out everything between 250Hz and 1000 Hz. What I did so far:
- using the 'enframe' function to create half overlapping windows with 512 samples each. On the windows I apply the hann window function.
- On each window I apply an fft. After this I reconstruct the original signal with the function ifft, that all goes well.
But the problem is how I have to interpret the result of the fft function and how to filter out a frequency band.
Unless I'm mistaken, it sounds like you're taking the wrong approach to this.
If your assignment is to manipulate a signal specifically by manipulating its FFT then ignore me. Otherwise.. read on.
The FFT is normally used to analyse a signal in the frequency domain. If you start fiddling with the complex coefficients that an FFT returns then you're getting into a complicated mathematical situation. This is particularly the case since your cut-off frequencies aren't going to lie nicely on FFT bin frequencies. Also, remember that the FFT is not a perfect transform of the signal you're analysing. It will always introduce artefacts of its own due to scalloping error, and convolution with your hann window.
So.. let's leave the FFT for analysis, and build a filter.
If you're doing band-pass design in your class I'm going to assume you understand what they do. There's a number of functions in Matlab to generate the coefficients for different types of filter i.e. butter, kaiser cheby1. Look up their help pages in Matlab for loads more info. The values you plug in to these functions will be dependent on your filter specification, i.e. you want "X"dB rolloff and "Y"dB passband ripple. You'll need some idea of the how these filters work, and knowledge of their transfer functions to understand how their filter order relates to your specification.
Once you have your coefficients, it's just a case of running them through the filter function (again.. check the help page if you're not sure how this works).
The mighty JOS has a great walkthrough of bandpass filter design here.
One other little niggle.. in your question you mentioned that you want your filter to "filter out" everything between 250Hz and 1000Hz. This is a bit ambiguous. If you're designing a bandpass filter you would want to "pass" everything between 250Hz and 1000Hz. If you do in fact want to "filter out" everything in this range you want a band-stop filter instead.
It all depends on the sampling rate you use.
If you sample right according to the Nyquist-Shannon sampling theorem then you can try and interpret the samples of your fft using the definition of the DFT.
For understanding which frequencies correspond with which samples in the dft results, I think it's best to look at the inverse transformation. You multiply coefficient k with
exp(i*2*pi*k/N*n)
which can be interpreted to be a cosine with Euler's Formula. So each coefficient gets multiplied by a sine of a certain frequency.
Good luck ;)