Problems while rectifying stereo images in matlab - matlab

Question

I am trying to use stereo imaging for 3D reconstruction, however when I use the tutorials and tools in matlab for stereo vision, I get erroneous results. I use a Loreo 3D macro lens to take images of small instruments at a distance of around 23mm. Then after cropping the images to create left and right images I use the stereo calibration app (I have also used code from the matlab tutorial which does pretty much the same thing). These are the kinds of results I get.
Stereo calibration using matlab's app
I am aware that the reprojection errors are quite high, but have tried a lot of things like changes in image quantity, illumination, checkerboard size and the skew, tangential distortion, and coefficients in the app to reduce this value without any luck. At a first glance, the Extrinsic reconstruction in the bottom right looks accurate, as the dimensions are quite correct. Therefore when I use the exported stereoParameters with a new image and the next code:
Isv = imread('IMG_0036.JPG');
I1 = imcrop(Isv, [0 0 2592 3456]);
I2 = imcrop(Isv, [2593 0 2592 3456]);
% Rectify the images.
[J1, J2] = rectifyStereoImages(I1, I2, stereoParams, 'OutputView', 'valid');
% Display the images before rectification.
figure;
imshow(stereoAnaglyph(I1, I2), 'InitialMagnification', 30);
title('Before Rectification');
% Display the images after rectification.
figure;
imshow(stereoAnaglyph(J1, J2), 'InitialMagnification', 30);
title('After Rectification');
disparityRange = [0, 64];
disparityMap = disparity(rgb2gray(J1), rgb2gray(J2), 'DisparityRange', ...
disparityRange);
figure;
imshow(disparityMap, disparityRange, 'InitialMagnification', 30);
colormap('jet');
colorbar;
title('Disparity Map');
point3D = reconstructScene(disparityMap, stereoParams);
% Convert from millimeters to meters.
point3D = point3D / 1000;
% Plot points between 3 and 7 meters away from the camera.
z = point3D(:, :, 3);
maxZ = 7;
minZ = 3;
zdisp = z;
zdisp(z < minZ | z > maxZ) = NaN;
point3Ddisp = point3D;
point3Ddisp(:,:,3) = zdisp;
figure
pcshow(point3Ddisp, J1, 'VerticalAxis', 'Y', 'VerticalAxisDir', 'Down' );
xlabel('X');
ylabel('Y');
zlabel('Z');
I get these erroneous rectification, disparity and 3D reconstruction.
Rectification, disparity and erroneous 3D reconstruction
As it can be seen, the rectification looks bad, as it the objects are too separated in my opinion; also the disparity results look very random, and finally the 3D reconstruction simply has no discernible outcome.
Please, I ask for any possible help, comments or recommendations regarding this issue.

Answer 1

Your reprojection errors are indeed high... Leaving that aside for the moment, your most immediate problem is that the disparityRange is too small.
The rectified images look fine. Corresponding points appear to be on the same pixel rows in both images, which is what you want. Display the anaglyph of the rectified images using imtool, and use the ruler widget to measure the distances between some of the corresponding points. That should give you an idea of what your disparity range should be. [0 64] is definitely too small.
To improve the reprojection errors, normally, I would say get more images. But you already have 30 pairs, which is a good number. You can specify the initial intrinsics and distortion, if you can get them off the camera manufacturer's spec, but I doubt they would help here. Try turning on tangential distortion estimation, and try using 3 radial distortion coefficients instead of two.
Also, see if you can more your cameras farther away from the scene. It may be that at such a short distance the pinhole camera model starts to break down.
There are also things you can do to improve your disparity and 3D reconstruction:
Try varying the BlockSize parameter of the disparity function
Try applying histogram equalization and/or low-pass filtering to your rectified images before computing disparity
Try median filtering the resulting disparity map to reduce the noise
Another tip: since you know approximately where the objects of interest are relative to the cameras, you can simply exclude the 3D points whose z-coordinate is too big or two small (or negative). This should give you a much cleaner 3D plot. You already have this in your code, but you should modify it to have appropriate units and thresholds for Z.