Floating point precision matlab filter2 - matlab

Question

I'm having round-off errors caused by filter2.
Here is a minimal code example:
format long
x=[ 0 0 0 0 0
64 65 72 74 72
104 111 109 106 112];
h=[ 0 0 0 0 0
0 0.500000000000000 0 0.500000000000000 0
0 0 0 0 0]
y=filter2(h,x, 'valid')
y_= x(2,2)/2 + x(2,4)/2
y__= sum(sum(x .* h))
round(y)
round(y_)
round(y__)
results in
y = 69.499999999999986
y_ = 69.500000000000000
y_ = 69.500000000000000
ans = 69
ans = 70
ans = 70
I'm guessing this is a result of doing the filtering in the fft domain (or something similar). Unfortunately this is giving me issues when verifying the test vectors I'm generating agains an FPGA implementation.
Any ideas how to fix/avoid this error?
PS I'm using matlab 2007b.
Edit: 2007a to 2007b
Edit2: Added y__ example

Answer 1

It is expected behavior that operations performed with floating-point arithmetic produce approximate results. Floating-point arithmetic uses a fixed number of bits to represent numbers, and the result of each floating-point operation is rounded to the nearest representable number (unless another rounding mode is set).
In particular, performing an FFT requires sines and cosines of many values, and the sines and cosines are not exactly representable, and the arithmetic upon those values and the values in the FFT data produces many intermediate results that are not exactly representable. Consequently, the results of FFTs are expected to be approximate.
Studies of errors in floating-point FFTs have shown the error behavior is generally good. However, you cannot expect that a result will land on the “correct” side of 69.5 to cause the rounding you desire. Essentially, it is an error to expect that rounding the results of an FFT will produce exact results.
Generally, using floating-point formats with greater precision can reduce the magnitudes of errors. Thus, using greater precision could produce FFT results closer to ideal results. However, consider what is necessary to make rounding work. Any number 69.5 or slightly greater will round to 70. Any number slightly less than 69.5 will round to 69, which you do not want. Therefore, to round as you desire, no error that produces a number less than 69.5 is acceptable, regardless of how small that error is. However, every floating-point format has some error. Therefore, there is no precision that is guaranteed to produce results that can be rounded in the way you desire. (Errors can be controlled somewhat by setting rounding modes, to cause rounding upward or downward as desired. However, the FFT is a complex operation, and getting the desired rounding in the final product would require controlling the rounding in every intermediate operation and is impractical.)
So, a floating-point FFT will not produce the results you desire. Some options available to you are:
Change your expectations; do not expect the filter to produce results identical to exact arithmetic.
Perform the filter using direct arithmetic, not an FFT. (I do not use Matlab and cannot advise on good ways to do this in Matlab.) Note that doing this with floating-point arithmetic will produce exact results only so long as all intermediate values are representable in the floating-point format. This is generally true only for small filters with “simple” values in the filter and in the data. (For this purpose, and supposing a binary floating-point format, “simple” values are those representable with just a few bits in the fraction portion of the floating-point format, i.e., those that are sums of a few integer powers of two that are close to each other, such as .625, which is 2-1+2-3.)
Use exact arithmetic. Some mathematics software, such as Maple and Mathematica, support exact arithmetic. As far as I know, Matlab does not. This generally requires performing the filter with direct arithmetic, not an FFT, since performing an FFT exactly requires greater mathematical capabilities (manipulating sines and cosines exactly).
Since you state this is for testing, I suggest that you either:
Allow small errors in the results. Small errors are typical of floating-point arithmetic and generally do not indicate errors in the filter you are testing. The bugs you are looking for in your tests will generally produce large and numerous errors.
Use direct arithmetic, if it is sufficient, or exact arithmetic if necessary. This will consume more processor time. However, testing generally does not need to be high-performance, so it is okay to use more processor time.

Answer 2

The standard way to deal with this is to avoid floating point comparison.
Rather than checking whether two things are equal, check whether the absolute difference is smaller than some epsilon.
So if you want to see whether your two numbers match you can do:
abs(y-y_)