I'm working on lossless data compression in MATLAB. I wish to encode a signal of about 60000 length. Here's my code:
function sig = huffman (Y, fs)
%getting array of unique values
Z = unique (Y);
%counting occurences of each element and listing it to a new array
countElY=histc(Y,Z); %# get the count of elements
p = countElY/numel(Y); %getting the probability distribution
[dict,avglen] = huffmandict(Z,p); % Create dictionary.
comp = huffmanenco(Y,dict) % Encode the data.
dsig = huffmandeco(comp, dict) %Decode the data
sound(dsig, fs)
Problem is, for a signal of such length, I exceed the 500 recursion limit at MATLAB, and that error occurs while creating the dictionary. I have already tried to break the signal into parts, but that took hell lot of time, and for only a small part of it. Any ideas how to make it work? Apart from extending the recursion limit, which is rather pointless and time consuming?
First you need to determine why you think it's possible to compress the data. Is the signal smooth? Is the range limited? Is the quantization limited? What makes it compressible will determine how to compress it.
Simply applying Huffman coding to a series of real values will not compress the data, since each of the values appears once, or maybe a few appear twice. Huffman depends on taking advantage of many occurrences of the same symbol, and a bias in the frequency, where some symbols are much more common than others.
Compressing a waveform would use different approaches. First would be to convert each sample to as few bits as are significant and that cover the range of inputs. Second would be to take differences between samples or use more advanced predictors to take advantage of smoothness in the waveform (if it is smooth). Third would be to find a way to group differences to encode more likely ones in fewer bits. That last step might use Huffman coding, but only after you've constructed something that Huffman coding can take advantage of.
Related
I'm looking for an efficient encoding for storing simulated coefficients.
The data has thousands of curves with each 512 contiguous numbers with single precision. The data may be stored as fixed point while it should preserve about 23-bit precision (compared to unity level).
The curves could look like those:
My best approach was to convert the numbers to 24-bit fixed point. Repeatedly I took the adjacent difference as long as the sum-of-squares decreases. When compressing the resulting data using LZMA (xz,lzip) I get about 7.5x compression (compared to float32).
Adjacent differences are good at the beginning, but they emphasize the quantization noise at each turn.
I've also tried the cosine transform after subtracting the slope/curve at the boundaries. The resulting compression was much weaker.
I tried AEC but LZMA compressed much stronger. The highest compression was using bzip3 (after adjacent differences).
I found no function to fit the data with high precision and a limited parameter count.
Is there a way to reduce the penalty of quantization noise when using adjacent differences?
Are there encodings which are better suited for this type of data?
You could try a higher-order predictor. Your "adjacent difference" is a zeroth-order predictor, where the next sample is predicted to be equal to the last sample. You take the differences between the actuals and the predictions, and then compress those differences.
You can try first, second, etc. order predictors. A first-order predictor would look at the last two samples, draw a line between those, and predict that the next sample will fall on the line. A second-order predictor would look at the last three samples, fit those to a parabola, and predict that the next sample will fall on the parabola. And so on.
Assuming that your samples are equally spaced on your x-axis, then the predictors for x[0] up through cubics are:
x[-1] (what you're using now)
2*x[-1] - x[-2]
3*x[-1] - 3*x[-2] + x[-3]
4*x[-1] - 6*x[-2] + 4*x[-3] - x[-4]
(Note that the coefficients are alternating-sign binomial coefficients.)
I doubt that the cubic polynomial predictor will be useful for you, but experiment with all of them to see if any help.
Assuming that the differences are small, you should use a variable-length integer to represent them. The idea would be to use one byte for each difference most of the time. For example, you could code seven bits of difference, say -64 to 63, in one byte with the high bit clear. If the difference doesn't fit in that, then make the high bit set, and have a second byte with another seven bits for a total of 14 with that second high bit clear. And so on for larger differences.
I am trying to use FFT to decode morse code, but I'm finding that when I examine the resulting frequency bin/bucket I'm interested in, the absolute value is varying quite significantly even when a constant tone is presented. This makes it impossible for me to use the rise and fall around a threshold and therefore decode audio morse.
I've even tried the simple example that seems to be copied everywhere, but it also varies...
I can't work out what I'm doing wrong, and my maths is not clever enough to understand all the formulas associated with FFT.
I now it must be possible, but I can't find out how... can anyone help please?
Make sure you are using the magnitude of the FFT result, not just the real or imaginary component of a complex result.
In general, when a longer constant amplitude sinusoid is fed to a series of shorter FFTs (windowed STFT), the magnitude result will only be constant if the period of the sinusoid is exactly integer periodic in the FFT length. e.g.
f_tone modulo (f_sampling_rate / FFT_length) == 0
If you are only interested in the magnitude of one selected tone frequency, the Goertzel algorithm would serve as a more efficient filter than a full FFT. And, depending on the setup and length restrictions required by your chosen FFT library, it may be easier to vary the length of a Goertzel to match the requirements for your target tone frequency, as well as the time/frequency resolution trade-off needed.
In compressed sensing, how to verify if a vector is recovered or how could one plot the figures on recovery rate? since in numerical experiments, there is always a difference between the original vector and the vector produced by compressed algorithms.
The only way to confirm that your vector is actually recovered would be to
keep one copy of the vector
compress another copy
decompress the compressed vector
compare with the original
This could be useful to do when you are trying to design or select a lossless compression method.
However, if you are in the setting where you have a vector, need to compress it. Later decompress it and want to know whether there was any loss. Then there is no practical way to be sure. Fortunately, you can get some indications whether your new vector resembles the old one by storing and comparing some statistics.
Some statistics that may be interesting, depending on your vector:
Length
Moments (Mean, variance etc.)
Number of positive values
Highest, and lowest value
Smallest nonzero difference between two values
The exact value of every 10000th element
...
Basically whatever really matters to you
I would like be able to use a vector as an envelope to apply fft equalization to rather large chunks of sound, with varying sizes.
To be able to multiply the frequency domain bins by the envelope, the envelope needs to have the same resolution as the fft data, which will vary with the size of the sound chunks.
So, I need a function to resample my envelope vector. Do you know whether vDSP features a function for that purpose? I browsed the reference back and forth, but found nothing. Which doesn't mean there is nothing there - it's easy to miss something while searching the vDSP reference...
It's not that I couldn't implement something myself, but if there was a vDSP function, it would propably be much faster than anything I could possibly come up with. Which is relevant as this project is targeted at iOS devices as well.
And there's no need to reinvent the wheel :)
Thanks!!
If I understand correctly, you have a 1D array of envelope values which you want to vector multiply with a 1D array of frequency bins. The problem you are trying to solve is to scale the envelope array to the same length as the FFT array. It would be helpful to know how you are generating the envelope array in the first place, can you not simply generate it at the correct length? If so, problem solved :)
If not, then how about using vDSP_vtabi to generate the envelope vector from the lookup table of values that you currently have? You can generate the lookup table input vector A using vDSP_vramp.
This seems rather complicated and expensive to me though, with a fair amount of buffer mallocing / reallocing. It might be simpler to calculate how many FFT samples should be multiplied by each envelope value, then loop for each envelope sample using vDSP_vsmul to multiply chunks of the FFT vector by the envelope value.
Which solution will perform better really depends a lot on the relative sizes of each vector. It would also be helpful to know why the FFT vectors are different sizes, and how you are generating the envelope array in the first place to give a more accurate answer.
I'd suggest to go through a different way.
Because your input signal comes from hardware at a fixed sample rate and you exactly
know the number of envelope values , just make a sample rate conversion using
AudioConverterFillComplexBuffer
It's fast and it take care about filtering and interpolation when resampling.
I'm trying to write a simple measurement plug-in for Audacity and it's about as much fun as pounding rocks against my skull. All I want to do is take a chunk of audio and find the average of all the samples (the chunk's DC offset) so that I can present it as a number to the user, and so that I can subtract the DC offset from the samples for further processing. I know and understand the math I want to do, but I don't understand how to do it in Lisp/XLisp/Nyquist/whatever.
Background information in this thread
As far as I know, there is no function to do this. For some reason, the snd-avg function does not actually compute the average of the sound, as you might expect. It does the absolute value first, and then computes the average computes the average and then does the absolute value. Even though there's a separate snd-abs function that could do it. >:(
So I guess I have to write my own? This means converting a sound into an array and then computing the average of that?
(snd-fetch-array sound len step)
Reads
sequential arrays of samples from
sound, returning either an array of
FLONUMs or NIL when the sound
terminates.
(snd-samples sound limit)
Converts the
samples into a lisp array.
Nyquist Functions
And there's not even an average function, so I'll have to do a sum myself, too? But the math functions only work on lists? So I need to convert the array into a list?
And this will also use up a huge amount of memory for longer waveforms (18 bytes per sample), so it would be best to process it in chunks and do a cumulative average. But I don't even know how to do the unoptimized version.
No, (hp s 0.1) won't work, since:
I want to remove DC only, and keep arbitrarily low frequencies. 0.01 Hz should pass through unchanged, DC should be removed.
The high-pass filter is causal and the first samples of the waveform remain unchanged, no matter what knee frequency you use, making it useless for measuring peak samples, etc.
NEVERMIND
snd-maxsamp is computing the absolute value, not snd-avg. snd-avg works just fine. Here's how to squeeze a number ("FLONUM") out of it:
(snd-fetch (snd-avg s (round len) (round len) OP-AVERAGE))
This produces a negative number for negative samples and a positive number for positive samples, as it should.
Should this question be deleted or left as an example to others?