I am trying to estimate the amplitude, frequency, and phase of an incoming signal of about 50Hz based on measurement of only a few cycles. The frequency needs to be precise to .01 Hz. Since the signal itself is going to be a pretty clear sine wave, I am trying parameter fitting with SciPy's curve_fit. I've never used it before, so I wrote a quick test function.
I start by generating samples of a single cycle of a dummy cosine wave
from math import *
import numpy as np
from scipy.optimize import curve_fit
import matplotlib.pyplot as plt
fs = 1000 # Sampling rate (Hz)
T = .1 # Length of collection (s)
windowlength = int(fs*T) # Number of samples
f0 = 10 # Fundamental frequency of our wave (Hz)
wave = [0]*windowlength
for x in range(windowlength):
wave[x] = cos(2*pi*f0*x/fs)
t = np.linspace(0,T,int(fs*T)) # This will be our x-axis for plotting
Then I try to fit those samples to a function, adapting the code from the official example provided by scipy: https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.curve_fit.html
# Define function to fit
def sinefit(x, A, ph, f):
return A * np.sin(2*pi*f * x + ph)
# Call curve_fit
popt,cov = curve_fit(sinefit, t, wave, p0=[1,np.pi/2,10])
# Plot the result
plt.plot(t, wave, 'b-', label='data')
plt.plot(t, sinefit(t, *popt), 'r-', label='fit')
print("[Amplitude,phase,frequency]")
print(popt)
This gives me popt = [1., 1.57079633, 9.9] and the plot
plot output
My question is: why is my frequency off? I initialized the curve_fit function with the exact parameters of the cosine wave, so shouldn't the first iteration of the LM algorithm realize that there is zero residual and that it has already arrived at the correct answer? That seems to be the case for amplitude and phase, but frequency is 0.1Hz too low.
I expect this is a dumb coding mistake, since the original wave and the fit are clearly lined up in the plot. I also confirmed that the difference between them was zero across the entire sample. If they really were .1 Hz out of phase, there would be a phase shift of 3.6 degrees over my 100ms window.
Any thoughts would be very much appreciated!
The problem is that your array t is not correct. The last value in your t is 0.1, but with a sampling period of 1/fs = 0.001, the last value in t should be 0.099. That is, the times of the 100 samples are [0, 0.001, 0.002, ..., 0.098, 0.099].
You can create t correctly with either
t = np.linspace(0, T, int(fs*T), endpoint=False)
or
t = np.arange(windowlength)/fs # Use float(fs) if you are using Python 2
Related
I am trying to perform an FFT from time series data of DC motor current from "F.A.I.R. open dataset of brushed DC motor faults for testing of AI algorithms". However, the result does not show any dominant frequency bands. It just resembles broadband noise. The first image is a zoomed in snap shot of the time series data (the entire series is over 100,000 data points), after the DC portion has been substracted.
Timeseries graph
The second image is the fft graph and my code is below. The time period is not yet set correctly but this does not effect the form of the data, only the frequency values assigned to it.
FFT graph
import matplotlib.pyplot as plt
import h5py
filename = "MOTOR-DC_2020_12_02_17_59_47_Analogico.hdf5"
#MOTOR-DC_2020_12_02_17_59_47_Analogico.hdf5
#MOTOR-DC_2020_12_02_17_30_42_Analogico.hdf5
with h5py.File(filename, "r") as f:
# List all groups
print("Keys: %s" % f.keys())
a_group_key = list(f.keys())[0]
# Get the data
data = list(f[a_group_key])
vibration =[(data[i][0]) for i in range(0,len(data))]
current =[(data[i][1]) for i in range(0,len(data))]
voltage=current =[(data[i][2]) for i in range(0,len(data))]
x=list(range(0,len(vibration)))
from scipy.fft import fft, fftfreq
import numpy as np
# Number of sample points
N = len(data)#600
# sample spacing
T = 0.0001
x = np.linspace(0.0, N*T, N, endpoint=False)
y = current
y_mean=np.mean(y)
y_med=np.median(y)
print('Mean',y_mean,'Median=',y_med)
for i in range(0,len(y)):
y[i]=y[i]-y_mean
#plt.plot(x,current)
yf = fft(y)
xf = fftfreq(n=N, d=T)[:N//2]
import matplotlib.pyplot as plt
plt.plot(xf, 2.0/N * np.abs(yf[0:N//2]))
plt.grid()
plt.show()
Try
yf = fft(y)
xf = fftfreq(N)
xf[np.argmax(np.abs(yf))]
This will give you the normalized frequency of the most prominent harmonic.
You can then multiply it by the sampling frequency to get the actual frequency.
I am processing Long term afib dataset - https://physionet.org/content/ltafdb/1.0.0/
When I test the 30s strips of this data, my model is not correcting predicting the signals. So I am trying to deal with noise in this dataset. Here how it looks
Here is the code to plot -
def plot_filter_graphs(data,xmin,xmax,order):
from numpy import sin, cos, pi, linspace
from numpy.random import randn
from scipy import signal
from scipy.signal import lfilter, lfilter_zi, filtfilt, butter
from matplotlib.pyplot import plot, legend, show, grid, figure, savefig,xlim
lowcut=1
highcut=35
nyq = 0.5 * 300
low = lowcut / nyq
high = highcut / nyq
b, a = signal.butter(order, [low, high], btype='band')
# Apply the filter to xn. Use lfilter_zi to choose the initial condition
# of the filter.
z = lfilter(b, a,data)
# Use filtfilt to apply the filter.
y = filtfilt(b, a, data)
y = np.flipud(y)
y = signal.lfilter(b, a, y)
y = np.flipud(y)
# Make the plot.
figure(figsize=(16,5))
plot(data,'b',linewidth=1.75)
plot(z, 'r--', linewidth=1.75)
plot( y, 'k', linewidth=1.75)
xlim(xmin,xmax)
legend(('actual',
'lfilter',
'filtfilt'),
loc='best')
grid(True)
show()
I am using butter band pass filter to filter the noise. I also checked with filtfilt and lfilt but that is also not giving good result.
Any suggestion, how noise can be removed so that signal accuracy is good and hense it can be used for model prediction
I have adapted the code in Comparing FFT of Function to Analytical FT Solution in Matlab for this question. I am trying to do FFTs and comparing the result with analytical expressions in the Wikipedia tables.
My code is:
a = 1.223;
fs = 1e5; %sampling frequency
dt = 1/fs;
t = 0:dt:30-dt; %time vector
L = length(t); % no. sample points
t = t - 0.5*max(t); %center around t=0
y = ; % original function in time
Y = dt*fftshift(abs(fft(y))); %numerical soln
freq = (-L/2:L/2-1)*fs/L; %freq vector
w = 2*pi*freq; % angular freq
F = ; %analytical solution
figure; subplot(1,2,1); hold on
plot(w,real(Y),'.')
plot(w,real(F),'-')
xlabel('Frequency, w')
title('real')
legend('numerical','analytic')
xlim([-5,5])
subplot(1,2,2); hold on;
plot(w,imag(Y),'.')
plot(w,imag(F),'-')
xlabel('Frequency, w')
title('imag')
legend('numerical','analytic')
xlim([-5,5])
If I study the Gaussian function and let
y = exp(-a*t.^2); % original function in time
F = exp(-w.^2/(4*a))*sqrt(pi/a); %analytical solution
in the above code, looks like there is good agreement when the real and imaginary parts of the function are plotted:
But if I study a decaying exponential multiplied with a Heaviside function:
H = #(x)1*(x>0); % Heaviside function
y = exp(-a*t).*H(t);
F = 1./(a+1j*w); %analytical solution
then
Why is there a discrepancy? I suspect it's related to the line Y = but I'm not sure why or how.
Edit: I changed the ifftshift to fftshift in Y = dt*fftshift(abs(fft(y)));. Then I also removed the abs. The second graph now looks like:
What is the mathematical reason behind the 'mirrored' graph and how can I remove it?
The plots at the bottom of the question are not mirrored. If you plot those using lines instead of dots you'll see the numeric results have very high frequencies. The absolute component matches, but the phase doesn't. When this happens, it's almost certainly a case of a shift in the time domain.
And indeed, you define the time domain function with the origin in the middle. The FFT expects the origin to be at the first (leftmost) sample. This is what ifftshift is for:
Y = dt*fftshift(fft(ifftshift(y)));
ifftshift moves the origin to the first sample, in preparation for the fft call, and fftshift moves the origin of the result to the middle, for display.
Edit
Your t does not have a 0:
>> t(L/2+(-1:2))
ans =
-1.5000e-05 -5.0000e-06 5.0000e-06 1.5000e-05
The sample at t(floor(L/2)+1) needs to be 0. That is the sample that ifftshift moves to the leftmost sample. (I use floor there in case L is odd in size, not the case here.)
To generate a correct t do as follows:
fs = 1e5; % sampling frequency
L = 30 * fs;
t = -floor(L/2):floor((L-1)/2);
t = t / fs;
I first generate an integer t axis of the right length, with 0 at the correct location (t(floor(L/2)+1)==0). Then I convert that to seconds by dividing by the sampling frequency.
With this t, the Y as I suggest above, and the rest of your code as-is, I see this for the Gaussian example:
>> max(abs(F-Y))
ans = 4.5254e-16
For the other function I see larger differences, in the order of 6e-6. This is due to the inability to sample the Heaviside function. You need t=0 in your sampled function, but H doesn't have a value at 0. I noticed that the real component has an offset of similar magnitude, which is caused by the sample at t=0.
Typically, the sampled Heaviside function is set to 0.5 for t=0. If I do that, the offset is removed completely, and max difference for the real component is reduced by 3 orders of magnitude (largest errors happen for values very close to 0, where I see a zig-zag pattern). For the imaginary component, the max error is reduced to 3e-6, still quite large, and is maximal at high frequencies. I attribute these errors to the difference between the ideal and sampled Heaviside functions.
You should probably limit yourself to band-limited functions (or nearly-band-limited ones such as the Gaussian). You might want to try to replace the Heaviside function with an error function (integral of Gaussian) with a small sigma (sigma = 0.8 * fs is the smallest sigma I would consider for proper sampling). Its Fourier transform is known.
I'm working on sound signals of a walking pattern, which has obvious regular patterns:
Then I thought I can get the frequency of walking (approximately 1.7Hz from the image) using FFT function:
x = walk_5; % Walking sound with a size of 711680x2 double
Fs = 48000; % sound frquency
L=length(x);
t=(1:L)/Fs; %time base
plot(t,x);
figure;
NFFT=2^nextpow2(L);
X=fft(x,NFFT);
Px=X.*conj(X)/(NFFT*L); %Power of each freq components
fVals=Fs*(0:NFFT/2-1)/NFFT;
plot(fVals,Px(1:NFFT/2),'b','LineSmoothing','on','LineWidth',1);
title('One Sided Power Spectral Density');
xlabel('Frequency (Hz)')
ylabel('PSD');
But then it doesn't give me what I expected:
FFT result:
zoom image has lots of noises:
and there is no information near 1.7Hz
Here is the graph from log domain using
semilogy(fVals,Px(1:NFFT));
It's pretty symmetric though:
I couldn't find anything wrong with my code. Do you have any solutions to easily extract the 1.7Hz from the walking pattern?
here is the link for the audio file in mat
https://www.dropbox.com/s/craof8qkz9n5dr1/walk_sound.mat?dl=0
Thank you very much!
Kai
I suggest you to forget about DFT approach since your signal is not appropriate for this type of analysis due to many reasons. Even by looking on the spectrum in range of frequencies that you are interested in, there is no easy way to estimate the peak:
Of course you could try with PSD/STFT and other funky methods, but this is an overkill. I can think of two, rather simple methods, for this task.
First one is based simply on the Auto Correlation Function.
Calculate the ACF
Define the minimum distance between them. Since you know that expected frequency is around 1.7Hz, then it corresponds to 0.58s. Let's make it 0.5s as the minimum distance.
Calculate the average distance between peaks found.
This gave me an approximate frequency of 1.72 Hz .
Second approach is based on the observation to your signal already has some peaks which are periodic. Therefore we can simply search for them using findpeaks function.
Define the minimum peak distance in a same way as before.
Define the minimum peak height. For example 10% of maximum peak.
Get the average difference.
This gave me an average frequency of 1.7 Hz.
Easy and fast method. There are obviously some things that can be improved, such as:
Refining thresholds
Finding both positive and negative peaks
Taking care of some missing peaks, i.e. due to low amplitude
Anyway that should get you started, instead of being stuck with crappy FFT and lazy semilogx.
Code snippet:
load walk_sound
fs = 48000;
dt = 1/fs;
x = walk_5(:,1);
x = x - mean(x);
N = length(x);
t = 0:dt:(N-1)*dt;
% FFT based
win = hamming(N);
X = abs(fft(x.*win));
X = 2*X(1:N/2+1)/sum(win);
X = 20*log10(X/max(abs(X)));
f = 0:fs/N:fs/2;
subplot(2,1,1)
plot(t, x)
grid on
xlabel('t [s]')
ylabel('A')
title('Time domain signal')
subplot(2,1,2)
plot(f, X)
grid on
xlabel('f [Hz]')
ylabel('A [dB]')
title('Signal Spectrum')
% Autocorrelation
[ac, lag] = xcorr(x);
min_dist = ceil(0.5*fs);
[pks, loc] = findpeaks(ac, 'MinPeakDistance', min_dist);
% Average distance/frequency
avg_dt = mean(gradient(loc))*dt;
avg_f = 1/avg_dt;
figure
plot(lag*dt, ac);
hold on
grid on
plot(lag(loc)*dt, pks, 'xr')
title(sprintf('ACF - Average frequency: %.2f Hz', avg_f))
% Simple peak finding in time domain
[pkst, loct] = findpeaks(x, 'MinPeakDistance', min_dist, ...
'MinPeakHeight', 0.1*max(x));
avg_dt2 = mean(gradient(loct))*dt;
avg_f2 = 1/avg_dt2;
figure
plot(t, x)
grid on
hold on
plot(loct*dt, pkst, 'xr')
xlabel('t [s]')
ylabel('A')
title(sprintf('Peak search in time domain - Average frequency: %.2f Hz', avg_f2))
Here's a nifty solution:
Take the absolute value of your raw data before taking the FFT. The data has a ton of high frequency noise that is drowning out whatever low frequency periodicity is present in the signal. The amplitude of the high frequency noise gets bigger every 1.7 seconds, and the increase in amplitude is visible to the eye, and periodic, but when you multiply the signal by a low frequency sine wave and sum everything you still end up with something close to zero. Taking the absolute value changes this, making those amplitude modulations periodic at low frequencies.
Try the following code comparing the FFT of the regular data with the FFT of abs(data). Note that I took a few liberties with your code, such as combining what I assume were the two stereo channels into a single mono channel.
x = (walk_5(:,1)+walk_5(:,2))/2; % Convert from sterio to mono
Fs = 48000; % sampling frquency
L=length(x); % length of sample
fVals=(0:L-1)*(Fs/L); % frequency range for FFT
walk5abs=abs(x); % Take the absolute value of the raw data
Xold=abs(fft(x)); % FFT of the data (abs in Matlab takes complex magnitude)
Xnew=abs(fft(walk5abs-mean(walk5abs))); % FFT of the absolute value of the data, with average value subtracted
figure;
plot(fVals,Xold/max(Xold),'r',fVals,Xnew/max(Xnew),'b')
axis([0 10 0 1])
legend('old method','new method')
[~,maxInd]=max(Xnew); % Index of maximum value of FFT
walkingFrequency=fVals(maxInd) % print max value
And plotting the FFT for both the old method and the new, from 0 to 10 Hz gives:
As you can see it detects a peak at about 1.686 Hz, and for this data, that's the highest peak in the FFT spectrum.
This is the first time I'm using the fft function and I'm trying to plot the frequency spectrum of a simple cosine function:
f = cos(2*pi*300*t)
The sampling rate is 220500. I'm plotting one second of the function f.
Here is my attempt:
time = 1;
freq = 220500;
t = 0 : 1/freq : 1 - 1/freq;
N = length(t);
df = freq/(N*time);
F = fftshift(fft(cos(2*pi*300*t))/N);
faxis = -N/2 / time : df : (N/2-1) / time;
plot(faxis, real(F));
grid on;
xlim([-500, 500]);
Why do I get odd results when I increase the frequency to 900Hz? These odd results can be fixed by increasing the x-axis limits from, say, 500Hz to 1000Hz. Also, is this the correct approach? I noticed many other people didn't use fftshift(X) (but I think they only did a single sided spectrum analysis).
Thank you.
Here is my response as promised.
The first or your questions related to why you "get odd results when you increase the frequency to 900 Hz" is related to the Matlab's plot rescaling functionality as described by #Castilho. When you change the range of the x-axis, Matlab will try to be helpful and rescale the y-axis. If the peaks lie outside of your specified range, matlab will zoom in on the small numerical errors generated in the process. You can remedy this with the 'ylim' command if it bothers you.
However, your second, more open question "is this the correct approach?" requires a deeper discussion. Allow me to tell you how I would go about making a more flexible solution to achieve your goal of plotting a cosine wave.
You begin with the following:
time = 1;
freq = 220500;
This raises an alarm in my head immediately. Looking at the rest of the post, you appear to be interested in frequencies in the sub-kHz range. If that is the case, then this sampling rate is excessive as the Nyquist limit (sr/2) for this rate is above 100 kHz. I'm guessing you meant to use the common audio sampling rate of 22050 Hz (but I could be wrong here)?
Either way, your analysis works out numerically OK in the end. However, you are not helping yourself to understand how the FFT can be used most effectively for analysis in real-world situations.
Allow me to post how I would do this. The following script does almost exactly what your script does, but opens some potential on which we can build . .
%// These are the user parameters
durT = 1;
fs = 22050;
NFFT = durT*fs;
sigFreq = 300;
%//Calculate time axis
dt = 1/fs;
tAxis = 0:dt:(durT-dt);
%//Calculate frequency axis
df = fs/NFFT;
fAxis = 0:df:(fs-df);
%//Calculate time domain signal and convert to frequency domain
x = cos( 2*pi*sigFreq*tAxis );
F = abs( fft(x, NFFT) / NFFT );
subplot(2,1,1);
plot( fAxis, 2*F )
xlim([0 2*sigFreq])
title('single sided spectrum')
subplot(2,1,2);
plot( fAxis-fs/2, fftshift(F) )
xlim([-2*sigFreq 2*sigFreq])
title('whole fft-shifted spectrum')
You calculate a time axis and calculate your number of FFT points from the length of the time axis. This is very odd. The problem with this approach, is that the frequency resolution of the fft changes as you change the duration of your input signal, because N is dependent on your "time" variable. The matlab fft command will use an FFT size that matches the size of the input signal.
In my example, I calculate the frequency axis directly from the NFFT. This is somewhat irrelevant in the context of the above example, as I set the NFFT to equal the number of samples in the signal. However, using this format helps to demystify your thinking and it becomes very important in my next example.
** SIDE NOTE: You use real(F) in your example. Unless you have a very good reason to only be extracting the real part of the FFT result, then it is much more common to extract the magnitude of the FFT using abs(F). This is the equivalent of sqrt(real(F).^2 + imag(F).^2).**
Most of the time you will want to use a shorter NFFT. This might be because you are perhaps running the analysis in a real time system, or because you want to average the result of many FFTs together to get an idea of the average spectrum for a time varying signal, or because you want to compare spectra of signals that have different duration without wasting information. Just using the fft command with a value of NFFT < the number of elements in your signal will result in an fft calculated from the last NFFT points of the signal. This is a bit wasteful.
The following example is much more relevant to useful application. It shows how you would split a signal into blocks and then process each block and average the result:
%//These are the user parameters
durT = 1;
fs = 22050;
NFFT = 2048;
sigFreq = 300;
%//Calculate time axis
dt = 1/fs;
tAxis = dt:dt:(durT-dt);
%//Calculate frequency axis
df = fs/NFFT;
fAxis = 0:df:(fs-df);
%//Calculate time domain signal
x = cos( 2*pi*sigFreq*tAxis );
%//Buffer it and window
win = hamming(NFFT);%//chose window type based on your application
x = buffer(x, NFFT, NFFT/2); %// 50% overlap between frames in this instance
x = x(:, 2:end-1); %//optional step to remove zero padded frames
x = ( x' * diag(win) )'; %//efficiently window each frame using matrix algebra
%// Calculate mean FFT
F = abs( fft(x, NFFT) / sum(win) );
F = mean(F,2);
subplot(2,1,1);
plot( fAxis, 2*F )
xlim([0 2*sigFreq])
title('single sided spectrum')
subplot(2,1,2);
plot( fAxis-fs/2, fftshift(F) )
xlim([-2*sigFreq 2*sigFreq])
title('whole fft-shifted spectrum')
I use a hamming window in the above example. The window that you choose should suit the application http://en.wikipedia.org/wiki/Window_function
The overlap amount that you choose will depend somewhat on the type of window you use. In the above example, the Hamming window weights the samples in each buffer towards zero away from the centre of each frame. In order to use all of the information in the input signal, it is important to use some overlap. However, if you just use a plain rectangular window, the overlap becomes pointless as all samples are weighted equally. The more overlap you use, the more processing is required to calculate the mean spectrum.
Hope this helps your understanding.
Your result is perfectly right. Your frequency axis calculation is also right. The problem lies on the y axis scale. When you use the function xlims, matlab automatically recalculates the y scale so that you can see "meaningful" data. When the cosine peaks lie outside the limit you chose (when f>500Hz), there are no peaks to show, so the scale is calculated based on some veeeery small noise (here at my computer, with matlab 2011a, the y scale was 10-16).
Changing the limit is indeed the correct approach, because if you don't change it you can't see the peaks on the frequency spectrum.
One thing I noticed, however. Is there a reason for you to plot the real part of the transform? Usually, it is abs(F) that gets plotted, and not the real part.
edit: Actually, you're frequency axis is only right because df, in this case, is 1. The faxis line is right, but the df calculation isn't.
The FFT calculates N points from -Fs/2 to Fs/2. So N points over a range of Fs yields a df of Fs/N. As N/time = Fs => time = N/Fs. Substituting that on the expression of df you used: your_df = Fs/N*(N/Fs) = (Fs/N)^2. As Fs/N = 1, the final result was right :P