I have the following signal which contains some distorted data
I'm trying to remove those spikes without damaging my signal, I've tried the medfilt1 function but it smoothed out the correct signal as well which is not wanted. Filtering cannot be used because of the frequency overlap between the wanted and unwanted signal. I also tried a moving window which will compare the value with the median of this window and if the point is much higher than it it will set it to the median as shown bellow:
%moving cleaning window
y1_1= y1(1:100);%first window
x=1;
%cleaning loop
while x<= length(y1)
if(y1(x)> 1.01*(median(y1_1))||y1(x) < 0.95*(median(y1_1)))
y1(x)= median(y1_1);
end
if(x>= length(y1)-100)
y1_1= y1(length(y1)-100:length(y1));
else
y1_1 = y1(x:x+100);
end
x=x+1;
end
I've gotten rid of the spikes but also some of the signal's distinct peaks were gone as shown in the figure bellow
How do I achieve the best denoising in a simple way?
Thanks
You can use median filter or moving average filter. Whatever filter it is, you need to use some kind of threshold. Threshold the spikes and replace them with your filter result.
s=rand(500,1)*5;
s(ceil(rand(1,20)*500))=rand(1,20)*100;
maxs=max(s);
figure
subplot(211); plot(s);
thr=10;
med_s=medfilt2(s,[10,1]);
s(s>med_s+thr)=med_s(s>med_s+thr);
subplot(212); plot(s); ylim([0 maxs])
If you can upgrade to R2017a, you may want to check out the filloutliers function. There are various methods you can use to detect the spikes, and similarly a variety of choices as to how to fill them.
If you want to basically get a median filter but only on the spikes, then you can specify 'movmedian' as the find method and 'center' as the fill method.
Related
I have a signal like this
And now i need to mark the points that are considered noisy in the signal. How to do that?
Noisy parts are just as shown in the zoomed in version of the plot and how to mark the noisy points in this signal by using matlab?
Following code might work:
array=abs([0 diff(signal)])
threshold_for_noisy=15
for i=1:size(array,2)
if array(i)>threshold_for_noisy
noisy_ind(i)=1;
else
noisy_ind(i)=0;
end
end
abnormal=zeros(1,size(array,2));
ch_pt=find(noisy_ind==1)
for i=1:2:size(ch_pt,2)-1
abnormal(ch_pt(i):ch_pt(i+1))=1
end
The elements in abnormal will be 1 whenever your signal exceeds the threshold and continues until it reaches back to normal (threshold_for_noisy).
Since the dataset you presented here doesn't have too much variance, simple comparison of past-future values as I suggested in my code will work. But if you have higher variance in your data, you can consider using a built-in function from Signal Processing Toolbox.
See Matlab ischange function documentation
Is the Matlab code available anywhere?
I'm trying to understand what exactly does it do. As I understood, it divides the data into segments with length tau (when tau increases each time) and then averages the data within each segment. After that, it compares the value attained from successive segments.
Am I correct in my understanding?
Thanks in advance!
You can view the code by typing edit allanvar into the command window.
Of course, this assumes you've already downloaded the associated toolbox.
I tried the same approach, but was also UNable to open the allanvar.p file. Despite the documentation I've found, if I use simple white noise and run allanvar on it, when I plot sqrt(this output), I get a line with a slope of -1/2 on a log-log scale (one order of magnitude drop in ADEV for two orders of magnitude increase in averaging interval value). A simple program to compute the OVERLAPPING Allan variance yields a -1/1 slope. Thus I am suspicious that the allanvar function does not compute the overlapping version!
I am trying to find power spectrum of the signal. The length of the signal is 100000, sample frequency is 1000Hz,and the number of points is 100000. I found the power spectrum using two approaches. The first one is by taking all the length as one part and found power spectrum for it while the second approach is by dividing the signal into 100*1000and find spectrum for each row then get the mean for all rows. My problem is that I must get the same answer in both approaches but I got different answers. I do not know what is the error in my code.
N=100000;
SF=1000;
a=0.1;
b=0.3;
amplitude1=1;
amplitude2=0.5;
t=0:1/SF:100;
f1=SF*a;
f2=SF*b;
A=amplitude1*sin(2*pi*f1*t)+amplitude2*sin(2*pi*f2*t);
Y=2*randn(1,length(A))+A;
bin=[0 :N/2];
fax_Hz=(bin*SF)/N;
FFT=fft(Y);
spectra=2/(SF*length(Y))*(FFT.*conj(FFT));
plot(fax_Hz,spectra(1,1:50001));
D=reshape(Y(1,1:100000),[100,1000]);
M=length(D(1,:));
for i=1:100
FFT_1(i,:)=fft(D(i,:));
S(i,:)=(2/(SF*M))*(FFT_1(i,:).*conj(FFT_1(i,:)));
end
S_f=mean(S);
figure
plot (S_f);
I just update the code. I do not know but when I added noise to signal the two plots looks shifted.
The main problem is with reshape you are working with each row being a separate sequence. Reshape however fills the first column before moving to the second one.
You can use the following instead.
D=reshape(A(1,1:100000),[1000,100]).';
Normalization is another problem. You can either use ifft instead of fft as it is normalized by default (not sure why). Or alternatively keep your normalization and instead of using mean you should can use sum, maybe that is due to a mistake you might have made. There still seems to be a small discrepancy in the amplitudes, not sure where that is coming from.
At the end to plot use the following:
bin=[0 :N];
fax_Hz=(bin*SF)/N;
FFT=ifft(A);
spectra=FFT.*conj(FFT);
plot(fax_Hz,spectra); hold on
D=reshape(A(1,1:100000),[1000,100]).';
M=length(D(1,:));
for i=1:100
FFT_1(i,:)=ifft(D(i,:));
S(i,:)=FFT_1(i,:).*conj(FFT_1(i,:));
end
S_f=mean(S);
plot(fax_Hz(1:100:end-1), S_f);
Note: the fax_Hz(1:100:end-1) is a hacky way of getting the length of the vectors to be the same.
I want to simulate an interpolator in MATLAB using upsampling followed by a low pass filter. First I have up-sampled my signal by introducing 0's.
Now I want to apply a low pass filter in order to interpolate. I have designed the following filter:
The filter is exactly 1/8 of the normalized frequency because I need to downsample afterward. (it's a specific excersise to upsample interpolate and downsample in this particular order.)
However when I apply this filter to my data using the function filter(myfilter, data) the following signal is generated:
I really don't know what is happening to my signal because in theory I know an interpolated signal should appear. This is the first time I'm working in MATLAB with filters because till now we only had the theory and had to assume ideal filters and analytical solutions.
Can someone give me an indication what might be wrong? I use the following code:
clear all; close all;
% Settings parameters
fs=10e6;
N=10;
c=3/fs;
k=3;
M=8;
% Settings time and signal
t=0:fs^-1:N*fs^-1;
x=exp(-(t.^2)./(2.*c.^2)); % Gaussian signal
% Upsampling
tu=0:(fs*M)^-1:N*fs^-1;
xu=zeros(1,size(tu,2));
sample_range=1:M:size(xu,2);
for i=1:size(x,2);
xu(sample_range(i))=x(i);
end;
%% Direct Method
xf=filter(lpf5mhz,xu);
As suggested by hotpaw2's answer, the low-pass filter needs some time to ramp up to the input signal values. This is particularly obvious with signal with sharp steps such as yours (the signal implicitly includes a large step at the first sample since past samples are assumed to be zeros by the filter call). Also, with your design parameters the delay of the filter is greater than the maximum time range shown on your output plot (1e-6), and correspondingly the output remains very small for the time range shown.
To illustrate the point, we can take a look at the filtered output with smaller filter lengths (and correspondingly smaller delays), using filters generated with fir1(length,0.125):
Given a signal with a smoother transition such as a Gaussian pulse which has been sufficiently time delayed:
delay = 10/fs;
x=exp(-((t-delay).^2)./(2.*c.^2)); % Gaussian signal
the filter can better ramp up to the signal value:
The next thing you may noticed, is that the filtered output has 1/Mth the amplitude as the unfiltered signal. To get an interpolated signal with similar amplitude as the unfiltered signal you would have to scale the filter output with:
xf=M*filter(lpf5mhz,1,xu);
Finally, the signal is delayed by the filtering operation. So for comparison purposes you may want plot a time shifted version with:
filter_delay = (1/(M*fs))*(length(lpf5mhz)-1)/2;
plot(tu-(1/(M*fs))*(length(b)-1)/2, xf);
A low-pass filter only helps interpolate signals that are much longer than the length of the impulse response of the low-pass filter. Otherwise the output can be dominated by the filter transient.
I am trying to compare two data sets in MATLAB. To do this I need to filter the data sets by Fourier transforming the data, filtering it and then inverse Fourier transforming it.
When I inverse Fourier transform the data however I get a spike at either end of the red data set (picture shows the first spike), it should be close to zero at the start, like the blue line. I am comparing many data sets and this only happens occasionally.
I have three questions about this phenomenon. First, what may be causing it, secondly, how can I remedy it, and third, will it affect the data further along the time series or just at the beginning and end of the time series as it appears to from the picture.
Any help would be great thanks.
When using DFT you must remember the DFT assumes a Periodic Signal (As a Superposition of Harmonic Functions).
As you can see, the start point is exact continuation of the last point in harmonic function manner.
Did you perform any Zero Padding in the Spectrum Domain?
Anyhow, Windowing might reduce the Overshooting.
Knowing more about the filter and the Original data might be helpful.
If you say spike near zero frequencies, I answer check the DC component.
You seem interested by the shape, so doing
x = x - mean(x)
or
x -= mean(x)
or
x -= x.mean()
(I love numpy!)
will just constrain the dataset to begin with null amplitude at zero-frequency and to go ahead with comapring the spectra's amplitude.
(as a side-note: did you check that you approprately use fftshift and ifftshift? this has always been the source of trouble for me)
Could be the numerical equivalent of Gibbs' phenomenon. If that's correct, there's no way to remedy it except for filtering.