I'm trying to remove the high frequency noise from the following file.
It's a file of a woman reading the news, with a high pitched noise playing loudly over it. Towards the end of the file, someone else begins to speak, but in a different language.
I want to filter out this high pitched noise, and be able to clearly hear the woman reading the news. Looking at the frequency domain:
I have tried using a low pass filter, and band stop filter. The bandstop filter produces a signal that no longer has the high pitch ringing, but the audio isn't very clear and it's hard to make out what is being said - the same goes for the low pass filter. I surmise that this is due to me filtering out not only the noise, but the harmonics of the speech as well. It was also necessary that I amplify the audio signal after I filtered it, because it was quieter than before.
Is there some clever way for me to reconstruct the harmonics of the speech in order to hear what is being said more clearly? Or is there a clever way for me to filter the signal without losing too much audio clarity?
I can include any code I used in matlab if needed.
Note:
I shifted the signal to 0 in the image I linked
I did use filtfilt() instead of filter()
I used butter() for the filters
Given the fairly dynamic nature of the interference in your sample, stationary filters are not going to yield very satisfying results. To improve performance, you would need to dynamically adjust the filtering parameters based on estimates of the interference.
Fortunately in this case the interference is pretty strong and exhibits a fairly regular pattern which makes it easier to estimate. This can be seen from the signal's spectrogram.
For the following derivations we will be assuming the samples of the wavfile has been stored in the array x and that the sampling rate is fs (which is 8000Hz in the provided sample).
[Sx,f,t] = spectrogram(x, triang(1024), 1023, [], fs, 'onesided');
Given that the interference is strong, obtaining the frequency of the interference can be done by locating the peak frequency in each time slice:
frequency = zeros(size(Sx,2),1);
for k = 1:size(Sx,2)
[pks,loc] = findpeaks(Sx(:,k));
frequency(k) = fs * (loc(1)-1);
end
Seeing that the interference is periodic we can use the Discrete Fourier Transform to decompose this signal:
M = 32*fs;
Ff = fft(frequency, M);
plot(fs*[0:M-1]/M, 20*log10(abs(Ff));
axis(0, 2);
xlabel('frequency (Hz)');
ylabel('amplitude (dB)');
Using the first two harmonics as an approximation, we can model the frequency of the interference signal as:
T = 1.0/fs
t = [0:length(x)-1]*T
freq = 750.0127340203496
+ 249.99913423501602*cos(2*pi*0.25*t - 1.5702946346796276)
+ 250.23974282864816*cos(2*pi*0.5 *t - 1.5701043282285363);
At this point we would have enough to create a narrowband filter with a center frequency (which would change dynamically as we keep updating the filter coefficients) given by that frequency model. Note however that constantly recomputing and updating the filter coefficient is a fairly expensive process and given that the interference is strong, it is possible to do better by locking on to the interference phase. This can be done by correlating small blocks of the original signal with sine and cosine at desired frequency. We can then slightly tweak the phase to align the sine/cosine with the original signal.
% Compute the phase of the sine/cosine to correlate the signal with
delta_phi = 2*pi*freq/fs;
phi = cumsum(delta_phi);
% We scale the phase adjustments with a triangular window to try to reduce
% phase discontinuities. I've chosen a window of ~200 samples somewhat arbitrarily,
% but it is large enough to cover 8 cycles of the interference around its lowest
% frequency sections (so we can get a better estimate by averaging out other signal
% contributions over multiple interference cycles), and is small enough to capture
% local phase variations.
step = 50;
L = 200;
win = triang(L);
win = win/sum(win);
for i = 0:floor((length(x)-(L-step))/step)
% The phase tweak to align the sine/cosine isn't linear, so we run a few
% iterations to let it converge to a phase locked to the original signal
for iter = 0:1
xseg = x[(i*step+1):(i*step+L+1)];
phiseg = phi[(i*step+1):(i*step+L+1)];
r1 = sum(xseg .* cos(phiseg));
r2 = sum(xseg .* sin(phiseg));
theta = atan2(r2, r1);
delta_phi[(i*step+1):(i*step+L+1)] = delta_phi[(i*step+1):(i*step+L+1)] - theta*win;
phi = cumsum(delta_phi);
end
end
Finally, we need to estimate the amplitude of the interference. Here we choose to perform the estimation over the initial 0.15 seconds where there is a little pause before the speech starts so that the estimation is not biased by the speech's amplitude:
tmax = 0.15;
nmax = floor(tmax * fs);
amp = sqrt(2*mean(x[1:nmax].^2));
% this should give us amp ~ 0.250996990794946
These parameters then allow us to fairly precisely reconstruct the interference, and correspondingly remove the interference from the original signal by direct subtraction:
y = amp * cos(phi)
x = x-y
Listening to the resulting output, you may notice a remaining faint whooshing noise, but nothing compared to the original interference. Obviously this is a fairly ideal case where the parameters of the interference are so easy to estimate that the results almost look too good to be true. You may not get the same performance with more random interference patterns.
Note: the python script used for this processing (and the corresponding .wav file output) can be found here.
Assume I have a smooth function (represented as a vector):
x=0:0.1:1000;
y=sin(2*x);
and I want to find its periodicity - pi (or even its frequency -2 ) .
I have tried the following:
nfft=1024;
Y=fft(y,nfft);
Y=abs(Y(1:nfft/2));
plot(Y);
but obviously it doesn't work (the plot does not give me a peak at "2" ).
Will you please help me find a way to find the value "2"?
Thanks in advance
You have several issues here:
You are computing the fft of x when your actual signal is y
x should be in radians
You need to define a sampling rate and use that to determine the frequency values along the x axis
So once we correct all of these things, we get:
samplingRate = 1000; % Samples per period
nPeriods = 10;
nSamples = samplingRate * nPeriods;
x = linspace(0, 2*pi*nPeriods, nSamples);
y = sin(2*x);
F = fft(y);
amplitude = abs(F / nSamples);
f = samplingRate / nSamples*[0:(nSamples/2-1),-nSamples/2:-1];
plot(f, amplitude)
In general, you can't use an FFT alone to find the period of a periodic signal. That's because an FFT does sinusoidal basis decomposition (or basis transform), and lots of non-sinusoidal waveforms (signals that look absolutely nothing like a sinewave or single sinusoidal basis vector) can be repeated to form a periodic function, waveform, or signal. Thus, it's quite possible for the frequency of a periodic function or waveform to not show up at all in an FFT result (it's called the missing fundamental problem).
Only in the case of a close or near sinusoidal signal will an FFT reliably report the reciprocal of the period of that periodic function.
There are lots of pitch detection/estimation algorithms. You can use an FFT as a sub-component of some composite methods, including cepstrums or cepstral analysis, and Harmonic Product Spectrum pitch detection methods.
While trying to understand Fast Fourier Transform I encountered a problem with the phase. I have broken it down to the simple code below. Calculating one period of a 50Hz sinewave, and applying an fft algorithm:
fs = 1600;
dt = 1/fs;
L = 32;
t=(0:L-1)*dt;
signal = sin(t/0.02*2*pi);
Y = fft(signal);
myAmplitude = abs(Y)/L *2 ;
myAngle = angle(Y);
Amplitude_at_50Hz = myAmplitude(2);
Phase_at_50Hz = myAngle(2);
While the amplitude is ok, I don't understand the phase result. Why do I get -pi/2 ? As there is only one pure sinewave, I expected the phase to be 0. Either my math is wrong, or my use of Matlab, or both of them... (A homemade fft gives me the same result. So I guess I am stumbling over my math.)
There is a similar post here: MATLAB FFT Phase plot. However, the suggested 'unwrap' command doesn't solve my problem.
Thanks and best regards,
DanK
The default waveform for an FFT phase angle of zero is a cosine wave which starts and ends in the FFT window at 1.0 (not a sinewave which starts and ends in the FFT window at 0.0, or at its zero crossings.) This is because the common nomenclature is to call the cosine function components of the FFT basis vectors (the complex exponentials) the "real" components. The sine function basis components are called "imaginary", and thus infer a non-zero complex phase.
That is what it should be. If you used cosine, you would have found a phase of zero.
Ignoring numerical Fourier transforms for a moment and taking a good old Fourier transform of sin(x), which I am too lazy to walk through, we get a pair of purely imaginary deltas.
As for an intuitive reason, recall that a discrete Fourier transform is averaging a bunch of points along a curve in the complex plane while turning at the angular frequency of the bin you're computing and using the amplitude corresponding to the sample. If you sample a sine curve while turning at its own frequency, the shape you get is a circle centered on the imaginary axis (see below). The average of that is of course going to be right on the imaginary axis.
Plot made with wolfram alpha.
Fourier transform of a sine function such as A*sin((2*pi*f)*t) where f is the frequency will yield 2 impulses of magnitude A/2 in the frequency domain at +f and -f where the associated phases are -pi/2 and pi/2 respectively.
You can take a look at its proof here:
http://mathworld.wolfram.com/FourierTransformSine.html
So the code is working fine.
I'm trying to find a factor using matlab that requires me to compute the Fourier transform of an input signal. The problem was stated to me this way:
fbin = 50HZ
0 <= n <= 1999
alpha = F {Blackman[2000] . cos[-2pi . fbin . n/2000]} (f)
where F is the Continous Time Fourier Transform operator.
My matlab code looks like this:
blackman_v = blackman(2000);
signal_x = cos(-2 * pi() .* fbin * (0:(1999)) ./ 2000) .* blackman_v';
fft_real = abs(fft(signal_x, 2000));
alpha = fft_real(51); %51 is the bin for 50hz => or {(f * N/Fs)+1}==51
My problem is that I'm supposed to get a value of around 412 for 49hz but I get about 250 (I'm actually verifying some previous results). Did I wrongly translate the problem? I've been battling for quite a while and I really don't see anything wrong here. Thought the value a 50Hz (430) is ok.
Would really appreciate any hint!
EDIT
blackman_v = blackman(2000);
signal_x = cos(-2 * pi() .* fbin * (0:(1999)) ./ 2000) .* blackman_v';
alpha = abs(freqz(signal_x , 1, 2*pi*50/10000))
Do you know what the freqz is? I read matlab doc and it is still not to clear in my head.
Maybe I misinterpreted your question but Matlab is not for continuous time analysis. It's for numerical analysis only, with discrete values. You can however calculate the discrete time fourier transform (DFT) of your signal, the resolution of which will depend on the length of your signal. Are you using a Blackmann window because your signal is non-periodic?
How to calculate the FFT (DFT) in Matlab: http://www.mathworks.se/help/matlab/ref/fft.html
Any discrete Fourier transform will assume that your signal is periodic. If it isn't you will obtain spectral leakage where certain frequency peaks "leak" their energy to the sides resulting in less defined peak with smeared out frequency values. Thus, the time-domain signal is preferably made periodic before calculting the DFT - periodic to the extent that a general pattern is repeated, values does not have to exact between periods since noise can/will be inherent in the signal. Applying a window function to the time-domain signal before calculating the DFT will make the signal periodic but you will have change amplitude values and introduced a low frequency component.
I've been playing around a little with the Exocortex implementation of the FFT, but I'm having some problems.
Whenever I modify the amplitudes of the frequency bins before calling the iFFT the resulting signal contains some clicks and pops, especially when low frequencies are present in the signal (like drums or basses). However, this does not happen if I attenuate all the bins by the same factor.
Let me put an example of the output buffer of a 4-sample FFT:
// Bin 0 (DC)
FFTOut[0] = 0.0000610351563
FFTOut[1] = 0.0
// Bin 1
FFTOut[2] = 0.000331878662
FFTOut[3] = 0.000629425049
// Bin 2
FFTOut[4] = -0.0000381469727
FFTOut[5] = 0.0
// Bin 3, this is the first and only negative frequency bin.
FFTOut[6] = 0.000331878662
FFTOut[7] = -0.000629425049
The output is composed of pairs of floats, each representing the real and imaginay parts of a single bin. So, bin 0 (array indexes 0, 1) would represent the real and imaginary parts of the DC frequency. As you can see, bins 1 and 3 both have the same values, (except for the sign of the Im part), so I guess bin 3 is the first negative frequency, and finally indexes (4, 5) would be the last positive frequency bin.
Then to attenuate the frequency bin 1 this is what I do:
// Attenuate the 'positive' bin
FFTOut[2] *= 0.5;
FFTOut[3] *= 0.5;
// Attenuate its corresponding negative bin.
FFTOut[6] *= 0.5;
FFTOut[7] *= 0.5;
For the actual tests I'm using a 1024-length FFT and I always provide all the samples so no 0-padding is needed.
// Attenuate
var halfSize = fftWindowLength / 2;
float leftFreq = 0f;
float rightFreq = 22050f;
for( var c = 1; c < halfSize; c++ )
{
var freq = c * (44100d / halfSize);
// Calc. positive and negative frequency indexes.
var k = c * 2;
var nk = (fftWindowLength - c) * 2;
// This kind of attenuation corresponds to a high-pass filter.
// The attenuation at the transition band is linearly applied, could
// this be the cause of the distortion of low frequencies?
var attn = (freq < leftFreq) ?
0 :
(freq < rightFreq) ?
((freq - leftFreq) / (rightFreq - leftFreq)) :
1;
// Attenuate positive and negative bins.
mFFTOut[ k ] *= (float)attn;
mFFTOut[ k + 1 ] *= (float)attn;
mFFTOut[ nk ] *= (float)attn;
mFFTOut[ nk + 1 ] *= (float)attn;
}
Obviously I'm doing something wrong but can't figure out what.
I don't want to use the FFT output as a means to generate a set of FIR coefficients since I'm trying to implement a very basic dynamic equalizer.
What's the correct way to filter in the frequency domain? what I'm missing?
Also, is it really needed to attenuate negative frequencies as well? I've seen an FFT implementation where neg. frequency values are zeroed before synthesis.
Thanks in advance.
There are two issues: the way you use the FFT, and the particular filter.
Filtering is traditionally implemented as convolution in the time domain. You're right that multiplying the spectra of the input and filter signals is equivalent. However, when you use the Discrete Fourier Transform (DFT) (implemented with a Fast Fourier Transform algorithm for speed), you actually calculate a sampled version of the true spectrum. This has lots of implications, but the one most relevant to filtering is the implication that the time domain signal is periodic.
Here's an example. Consider a sinusoidal input signal x with 1.5 cycles in the period, and a simple low pass filter h. In Matlab/Octave syntax:
N = 1024;
n = (1:N)'-1; %'# define the time index
x = sin(2*pi*1.5*n/N); %# input with 1.5 cycles per 1024 points
h = hanning(129) .* sinc(0.25*(-64:1:64)'); %'# windowed sinc LPF, Fc = pi/4
h = [h./sum(h)]; %# normalize DC gain
y = ifft(fft(x) .* fft(h,N)); %# inverse FT of product of sampled spectra
y = real(y); %# due to numerical error, y has a tiny imaginary part
%# Depending on your FT/IFT implementation, might have to scale by N or 1/N here
plot(y);
And here's the graph:
The glitch at the beginning of the block is not what we expect at all. But if you consider fft(x), it makes sense. The Discrete Fourier Transform assumes the signal is periodic within the transform block. As far as the DFT knows, we asked for the transform of one period of this:
This leads to the first important consideration when filtering with DFTs: you are actually implementing circular convolution, not linear convolution. So the "glitch" in the first graph is not really a glitch when you consider the math. So then the question becomes: is there a way to work around the periodicity? The answer is yes: use overlap-save processing. Essentially, you calculate N-long products as above, but only keep N/2 points.
Nproc = 512;
xproc = zeros(2*Nproc,1); %# initialize temp buffer
idx = 1:Nproc; %# initialize half-buffer index
ycorrect = zeros(2*Nproc,1); %# initialize destination
for ctr = 1:(length(x)/Nproc) %# iterate over x 512 points at a time
xproc(1:Nproc) = xproc((Nproc+1):end); %# shift 2nd half of last iteration to 1st half of this iteration
xproc((Nproc+1):end) = x(idx); %# fill 2nd half of this iteration with new data
yproc = ifft(fft(xproc) .* fft(h,2*Nproc)); %# calculate new buffer
ycorrect(idx) = real(yproc((Nproc+1):end)); %# keep 2nd half of new buffer
idx = idx + Nproc; %# step half-buffer index
end
And here's the graph of ycorrect:
This picture makes sense - we expect a startup transient from the filter, then the result settles into the steady state sinusoidal response. Note that now x can be arbitrarily long. The limitation is Nproc > 2*min(length(x),length(h)).
Now onto the second issue: the particular filter. In your loop, you create a filter who's spectrum is essentially H = [0 (1:511)/512 1 (511:-1:1)/512]'; If you do hraw = real(ifft(H)); plot(hraw), you get:
It's hard to see, but there are a bunch of non-zero points at the far left edge of the graph, and then a bunch more at the far right edge. Using Octave's built-in freqz function to look at the frequency response we see (by doing freqz(hraw)):
The magnitude response has a lot of ripples from the high-pass envelope down to zero. Again, the periodicity inherent in the DFT is at work. As far as the DFT is concerned, hraw repeats over and over again. But if you take one period of hraw, as freqz does, its spectrum is quite different from the periodic version's.
So let's define a new signal: hrot = [hraw(513:end) ; hraw(1:512)]; We simply rotate the raw DFT output to make it continuous within the block. Now let's look at the frequency response using freqz(hrot):
Much better. The desired envelope is there, without all the ripples. Of course, the implementation isn't so simple now, you have to do a full complex multiply by fft(hrot) rather than just scaling each complex bin, but at least you'll get the right answer.
Note that for speed, you'd usually pre-calculate the DFT of the padded h, I left it alone in the loop to more easily compare with the original.
Your primary issue is that frequencies aren't well defined over short time intervals. This is particularly true for low frequencies, which is why you notice the problem most there.
Therefore, when you take really short segments out of the sound train, and then you filter these, the filtered segments wont filter in a way that produces a continuous waveform, and you hear the jumps between segments and this is what generates the clicks you here.
For example, taking some reasonable numbers: I start with a waveform at 27.5 Hz (A0 on a piano), digitized at 44100 Hz, it will look like this (where the red part is 1024 samples long):
So first we'll start with a low pass of 40Hz. So since the original frequency is less than 40Hz, a low-pass filter with a 40Hz cut-off shouldn't really have any effect, and we will get an output that almost exactly matches the input. Right? Wrong, wrong, wrong – and this is basically the core of your problem. The problem is that for the short sections the idea of 27.5 Hz isn't clearly defined, and can't be represented well in the DFT.
That 27.5 Hz isn't particularly meaningful in the short segment can be seen by looking at the DFT in the figure below. Note that although the longer segment's DFT (black dots) shows a peak at 27.5 Hz, the short one (red dots) doesn't.
Clearly, then filtering below 40Hz, will just capture the DC offset, and the result of the 40Hz low-pass filter is shown in green below.
The blue curve (taken with a 200 Hz cut-off) is starting to match up much better. But note that it's not the low frequencies that are making it match up well, but the inclusion of high frequencies. It's not until we include every frequency possible in the short segment, up to 22KHz that we finally get a good representation of the original sine wave.
The reason for all of this is that a small segment of a 27.5 Hz sine wave is not a 27.5 Hz sine wave, and it's DFT doesn't have much to do with 27.5 Hz.
Are you attenuating the value of the DC frequency sample to zero? It appears that you are not attenuating it at all in your example. Since you are implementing a high pass filter, you need to set the DC value to zero as well.
This would explain low frequency distortion. You would have a lot of ripple in the frequency response at low frequencies if that DC value is non-zero because of the large transition.
Here is an example in MATLAB/Octave to demonstrate what might be happening:
N = 32;
os = 4;
Fs = 1000;
X = [ones(1,4) linspace(1,0,8) zeros(1,3) 1 zeros(1,4) linspace(0,1,8) ones(1,4)];
x = ifftshift(ifft(X));
Xos = fft(x, N*os);
f1 = linspace(-Fs/2, Fs/2-Fs/N, N);
f2 = linspace(-Fs/2, Fs/2-Fs/(N*os), N*os);
hold off;
plot(f2, abs(Xos), '-o');
hold on;
grid on;
plot(f1, abs(X), '-ro');
hold off;
xlabel('Frequency (Hz)');
ylabel('Magnitude');
Notice that in my code, I am creating an example of the DC value being non-zero, followed by an abrupt change to zero, and then a ramp up. I then take the IFFT to transform into the time domain. Then I perform a zero-padded fft (which is done automatically by MATLAB when you pass in an fft size bigger than the input signal) on that time-domain signal. The zero-padding in the time-domain results in interpolation in the frequency domain. Using this, we can see how the filter will respond between filter samples.
One of the most important things to remember is that even though you are setting filter response values at given frequencies by attenuating the outputs of the DFT, this guarantees nothing for frequencies occurring between sample points. This means the more abrupt your changes, the more overshoot and oscillation between samples will occur.
Now to answer your question on how this filtering should be done. There are a number of ways, but one of the easiest to implement and understand is the window design method. The problem with your current design is that the transition width is huge. Most of the time, you will want as quick of transitions as possible, with as little ripple as possible.
In the next code, I will create an ideal filter and display the response:
N = 32;
os = 4;
Fs = 1000;
X = [ones(1,8) zeros(1,16) ones(1,8)];
x = ifftshift(ifft(X));
Xos = fft(x, N*os);
f1 = linspace(-Fs/2, Fs/2-Fs/N, N);
f2 = linspace(-Fs/2, Fs/2-Fs/(N*os), N*os);
hold off;
plot(f2, abs(Xos), '-o');
hold on;
grid on;
plot(f1, abs(X), '-ro');
hold off;
xlabel('Frequency (Hz)');
ylabel('Magnitude');
Notice that there is a lot of oscillation caused by the abrupt changes.
The FFT or Discrete Fourier Transform is a sampled version of the Fourier Transform. The Fourier Transform is applied to a signal over the continuous range -infinity to infinity while the DFT is applied over a finite number of samples. This in effect results in a square windowing (truncation) in the time domain when using the DFT since we are only dealing with a finite number of samples. Unfortunately, the DFT of a square wave is a sinc type function (sin(x)/x).
The problem with having sharp transitions in your filter (quick jump from 0 to 1 in one sample) is that this has a very long response in the time domain, which is being truncated by a square window. So to help minimize this problem, we can multiply the time-domain signal by a more gradual window. If we multiply a hanning window by adding the line:
x = x .* hanning(1,N).';
after taking the IFFT, we get this response:
So I would recommend trying to implement the window design method since it is fairly simple (there are better ways, but they get more complicated). Since you are implementing an equalizer, I assume you want to be able to change the attenuations on the fly, so I would suggest calculating and storing the filter in the frequency domain whenever there is a change in parameters, and then you can just apply it to each input audio buffer by taking the fft of the input buffer, multiplying by your frequency domain filter samples, and then performing the ifft to get back to the time domain. This will be a lot more efficient than all of the branching you are doing for each sample.
First, about the normalization: that is a known (non) issue. The DFT/IDFT would require a factor 1/sqrt(N) (apart from the standard cosine/sine factors) in each one (direct an inverse) to make them simmetric and truly invertible. Another possibility is to divide one of them (the direct or the inverse) by N, this is a matter of convenience and taste. Often the FFT routines do not perform this normalization, the user is expected to be aware of it and normalize as he prefers. See
Second: in a (say) 16 point DFT, what you call the bin 0 would correspond to the zero frequency (DC), bin 1 low freq... bin 4 medium freq, bin 8 to the highest frequency and bins 9...15 to the "negative frequencies". In you example, then, bin 1 is actually both the low frequency and medium frequency. Apart from this consideration, there is nothing conceptually wrong in your "equalization". I don't understand what you mean by "the signal gets distorted at low frequencies". How do you observe that ?