I have a signal like this
And now i need to mark the points that are considered noisy in the signal. How to do that?
Noisy parts are just as shown in the zoomed in version of the plot and how to mark the noisy points in this signal by using matlab?
Following code might work:
array=abs([0 diff(signal)])
threshold_for_noisy=15
for i=1:size(array,2)
if array(i)>threshold_for_noisy
noisy_ind(i)=1;
else
noisy_ind(i)=0;
end
end
abnormal=zeros(1,size(array,2));
ch_pt=find(noisy_ind==1)
for i=1:2:size(ch_pt,2)-1
abnormal(ch_pt(i):ch_pt(i+1))=1
end
The elements in abnormal will be 1 whenever your signal exceeds the threshold and continues until it reaches back to normal (threshold_for_noisy).
Since the dataset you presented here doesn't have too much variance, simple comparison of past-future values as I suggested in my code will work. But if you have higher variance in your data, you can consider using a built-in function from Signal Processing Toolbox.
See Matlab ischange function documentation
Related
I have written a simple code that performs a 3-point moving average smoothing algorithm. It is meant to follow the same basic algorithm as Matlab's smooth(...) function as described here.
However, the result of my code is very different from that of Matlab's. Matlab's 3-point filter appears to perform a much more aggressive smoothing.
Here is a comparison of a noisy data smoothed using my code (red) and Matlab's function (blue):
Here is my code written in the form of a function:
function [NewSignal] = smoothing(signal)
NewSignal = signal;
for i = 2 : length(signal)-1
NewSignal(i,:) = (NewSignal(i,:)+NewSignal(i-1,:)+NewSignal(i+1,:))./3;
end
end
Matlab's function is used as follows:
signal = smooth(time, signal, 3, 'moving');
As far as I understand Matlab's function works the same way; it averages 3 adjacent bins to a single bin. So I expected both algorithms to produce the same results.
So, what is the reason for the discrepancy? And how could I tweak my code to produce the same results?
Edit:
My sample data can be found here. It can be accessed using:
M = csvread('DS0009.csv');
time = M(:,1);
signal = M(:,2);
Here is the new result (red plot) using rinkert's correction:
One reason for the difference could be that you are partially using your smoothed signal during smoothing. In your loop, you store the smoothed value in NewSignal(i,:), and for the next sample to smooth this value will be called by NewSignal(i-1,:).
Let NewSignal be determined by the original signal only:
function [NewSignal] = smoothing(signal)
NewSignal = signal;
for i = 2 : length(signal)-1
NewSignal(i,:) = (signal(i,:)+signal(i-1,:)+signal(i+1,:))./3;
end
end
Update: To show that the function above in fact does the same as Matlab's smooth function, let's consider this MVCE:
t = (0:0.01:10).'; % time vector
y = sin(t) + 0.5*randn(size(t));
y_smooth1 = smooth(t,y,3,'moving');
y_smooth2 = smoothing(y);
difference_methods = abs(y_smooth1-y_smooth2);
So creating a sine wave, add some noise, and determine the absolute difference between the two methods. If you take the sum of all the differences you will see that this adds up to something like 7.5137e-14, which cannot explain the differences you see.
Plotting the smooth signal (blue original, red smoothed):
figure(1); clf; hold on
plot(t,y)
plot(t,y_smooth2)
And then plotting the difference between the two methods:
figure(2); clf; hold on;
plot(t,y_smooth1-y_smooth2)
As you can see, the difference is of the order 1e-16, so influenced by the Floating-point relative accuracy (see eps).
To answer your question in the comments: the Function filter and smooth perform arithmetically the same (in the case that they are applied for moving average). however, there are the special cases at the beginning and endpoints which are handled differently.
This is also stated in the documentation of smooth "Because of the way smooth handles endpoints, the result differs from the result returned by the filter function."
Here you see it in an example:
%generate randonm data
signal=rand(1,50);
%plot data
plot(signal,'LineWidth',2)
hold on
%plot filtered data
plot(filter(ones(3,1)/3,1,signal),'r-','LineWidth',2)
%plot smoothed data
plot( smooth(signal,3,'moving'),'m--','LineWidth',2)
%plot smoothed and delayed
plot([zeros(1,1); smooth(signal,3,'moving')],'k--','LineWidth',2)
hold off
legend({'Data','Filter','Smooth','Smooth-Delay'})
As you can see the filtered data (in red) is just a delayed version of the smoothed data (in magenta). Additionally, they differ in the beginning. Delaying the smoothed data results in an identical waveform as the filtered data (besides the beginning). As rinkert pointed out, your approach overwrites the data points which you are accessing in the next step. This is a different issue.
In the next example you will see that rinkerts implementation (smooth-rinkert) is identical to matlabs smooth, and that your approach differs from both due to overwriting the values:
So it is your function which low passes the input stronger. (as pointed out by Cris)
I want to calculate the Fourier series of a signal using FFT on matlab. And I came across the following unexpected issue. Mere example
If I define a grid and then compute the fft as:
M=59;
x= deal(1*(0:M-1)/M);
y=3*cos(2*pi*x);
Yk=fftshift(fft2(y)/(M));
which gives me the exact analytic values expected: Yk(29)=1.5; Yk(31)=1.5; zeros anything else
but if I define the grid as, and repeat the fft calculation:
x=0:1/(M-1):1;
y=3*cos(2*pi*x);
Yk=fftshift(fft2(y)/(M));
got the Yk's values completely screwed up
This is an annoying issue since I have to analyse many signals data that was sampled as in the second method so the Yk's values will be wrong. Is there a way to workaround this? an option to tell something to the fft function about the way the signal was sampled. Have no way to resample the data in the correct way.
The main reason to avoid have spectral leaking, is that I do further operations with these Fourier terms individually Real and Imag parts. And the spectral leaking is messing the final results.
The second form of sampling includes one sample too many in the period of the cosine. This causes some spectral leaking, and adds a small shift to your signal (which leads to non-zero imaginary values). If you drop the last point, you'll cosine will again be sampled correctly, and you'll get rid of both of these effects. Your FFT will have one value less, I don't know if this will affect your analyses in any way.
x = 0:1/(M-1):1;
y = 3*cos(2*pi*x);
Yk = fftshift(fft2(y(1:end-1))/(M-1));
>> max(abs(imag(Yk)))
ans =
1.837610523517500e-16
I have the following signal which contains some distorted data
I'm trying to remove those spikes without damaging my signal, I've tried the medfilt1 function but it smoothed out the correct signal as well which is not wanted. Filtering cannot be used because of the frequency overlap between the wanted and unwanted signal. I also tried a moving window which will compare the value with the median of this window and if the point is much higher than it it will set it to the median as shown bellow:
%moving cleaning window
y1_1= y1(1:100);%first window
x=1;
%cleaning loop
while x<= length(y1)
if(y1(x)> 1.01*(median(y1_1))||y1(x) < 0.95*(median(y1_1)))
y1(x)= median(y1_1);
end
if(x>= length(y1)-100)
y1_1= y1(length(y1)-100:length(y1));
else
y1_1 = y1(x:x+100);
end
x=x+1;
end
I've gotten rid of the spikes but also some of the signal's distinct peaks were gone as shown in the figure bellow
How do I achieve the best denoising in a simple way?
Thanks
You can use median filter or moving average filter. Whatever filter it is, you need to use some kind of threshold. Threshold the spikes and replace them with your filter result.
s=rand(500,1)*5;
s(ceil(rand(1,20)*500))=rand(1,20)*100;
maxs=max(s);
figure
subplot(211); plot(s);
thr=10;
med_s=medfilt2(s,[10,1]);
s(s>med_s+thr)=med_s(s>med_s+thr);
subplot(212); plot(s); ylim([0 maxs])
If you can upgrade to R2017a, you may want to check out the filloutliers function. There are various methods you can use to detect the spikes, and similarly a variety of choices as to how to fill them.
If you want to basically get a median filter but only on the spikes, then you can specify 'movmedian' as the find method and 'center' as the fill method.
I want to simulate an interpolator in MATLAB using upsampling followed by a low pass filter. First I have up-sampled my signal by introducing 0's.
Now I want to apply a low pass filter in order to interpolate. I have designed the following filter:
The filter is exactly 1/8 of the normalized frequency because I need to downsample afterward. (it's a specific excersise to upsample interpolate and downsample in this particular order.)
However when I apply this filter to my data using the function filter(myfilter, data) the following signal is generated:
I really don't know what is happening to my signal because in theory I know an interpolated signal should appear. This is the first time I'm working in MATLAB with filters because till now we only had the theory and had to assume ideal filters and analytical solutions.
Can someone give me an indication what might be wrong? I use the following code:
clear all; close all;
% Settings parameters
fs=10e6;
N=10;
c=3/fs;
k=3;
M=8;
% Settings time and signal
t=0:fs^-1:N*fs^-1;
x=exp(-(t.^2)./(2.*c.^2)); % Gaussian signal
% Upsampling
tu=0:(fs*M)^-1:N*fs^-1;
xu=zeros(1,size(tu,2));
sample_range=1:M:size(xu,2);
for i=1:size(x,2);
xu(sample_range(i))=x(i);
end;
%% Direct Method
xf=filter(lpf5mhz,xu);
As suggested by hotpaw2's answer, the low-pass filter needs some time to ramp up to the input signal values. This is particularly obvious with signal with sharp steps such as yours (the signal implicitly includes a large step at the first sample since past samples are assumed to be zeros by the filter call). Also, with your design parameters the delay of the filter is greater than the maximum time range shown on your output plot (1e-6), and correspondingly the output remains very small for the time range shown.
To illustrate the point, we can take a look at the filtered output with smaller filter lengths (and correspondingly smaller delays), using filters generated with fir1(length,0.125):
Given a signal with a smoother transition such as a Gaussian pulse which has been sufficiently time delayed:
delay = 10/fs;
x=exp(-((t-delay).^2)./(2.*c.^2)); % Gaussian signal
the filter can better ramp up to the signal value:
The next thing you may noticed, is that the filtered output has 1/Mth the amplitude as the unfiltered signal. To get an interpolated signal with similar amplitude as the unfiltered signal you would have to scale the filter output with:
xf=M*filter(lpf5mhz,1,xu);
Finally, the signal is delayed by the filtering operation. So for comparison purposes you may want plot a time shifted version with:
filter_delay = (1/(M*fs))*(length(lpf5mhz)-1)/2;
plot(tu-(1/(M*fs))*(length(b)-1)/2, xf);
A low-pass filter only helps interpolate signals that are much longer than the length of the impulse response of the low-pass filter. Otherwise the output can be dominated by the filter transient.
I'm trying to limit a signal by multiplying the signal by an array to limit the amplitude within a certain range. The signal in (green) the tot array isn't suppose to go past the limiting array in (red) the ptest array, but it does, how can I fix this so the green signal doesn't go past the red line (ptest array)? see code and plot below. Please note this is a simple test the signals will be more complicated.
I'm using octave 3.8.1 which is like matlab
fs=200
x=[0,.2,.5,1]
y=[1,.5,.1,0]
t=linspace(0,x(end),fs);
peq=polyfit(x,y,length(y)); %create polynomial from points
ptest=polyval(peq,t); %plug numbers into polynomial
plot(ptest,t,'-ro')
hold on
plot(x,y,'b*')
t2=linspace(0,2*pi,fs);
s1 = sin(10*t2) ; %signal to limit
tot=ptest.*s1;
plot(t,tot,'-g')
Typo in your code for plotting. You flipped t and ptest in your first plot call:
plot(ptest,t,'-ro');
This should be:
plot(t,ptest,'-ro');
The independent axis is time or t, but you have made that ptest instead.
When I flip this and run the code again, I get:
BTW, your question is a bit misleading. I wouldn't call this "limiting"... you are actually modulating the amplitude of the sinusoidal signal with the amplitude of the ptest signal.