I am trying to use the Computer Vision System Toolbox to calibrate the pair of cameras below in order to be able to generate a 3-D point cloud of a vehicle at a range between 1 to 5m. The output image size was approximately 1 MB per image for the checkerboard calibration images and the checkerboard square size was 25 mm. The cameras used were a pair of SJ4000 HD1080P cameras. The cameras were placed as parallel to each other as possible with no angle in the vertical axis. The checkboard calibration was done with the aid of a bright light and whiteboard. The mean error per pixel using the stereo camera calibrator code was 3.31 with 31/32 successful pairings. The approximate distance to the checkerboard was 30 cm and the distance between the cameras was 20 cm.
The problem I am encountering upon rectification is during the 3D Reconstruction of the scene. The figure below is what was outputted. I’m not sure if there is a parameter that is missing in the camera setup or if there is something that is missing / needs to be added in the script. Below is the code that was used for the Stereo anaglyph and scene reconstruction that was adapted from the Matlab Stereo Camera Calibration tutorial.
% Read in the stereo pair of images.
I1 = imread('left.jpg');
I2 = imread('right.jpg');
% Rectify the images.
[J1, J2] = rectifyStereoImages(I1, I2, stereoParams);
% Display the images before rectification.
figure;
imshow(stereoAnaglyph(I1, I2), 'InitialMagnification', 50);
title('Before Rectification');
% Display the images after rectification.
figure;
imshow(stereoAnaglyph(J1, J2), 'InitialMagnification', 50);
title('After Rectification');
%
% Compute Disparity for 3-D Reconstruction
% The distance in pixels between corresponding points in the rectified images is called disparity.
% The disparity is used for 3-D reconstruction, because it is proportional to the distance between the cameras and the 3-D world point.
disparityMap = disparity(rgb2gray(J1), rgb2gray(J2));
figure;
imshow(disparityMap, [0, 64], 'InitialMagnification', 50);
colormap('jet');
colorbar;
title('Disparity Map');
%Reconstruct the 3-D Scene
%Reconstruct the 3-D world coordinates of points corresponding to each pixel from the disparity map.
point3D = reconstructScene(disparityMap, stereoParams);
% Convert from millimeters to meters.
point3D = point3D / 1000;
% Visualize the 3-D Scene
% Plot points between 3 and 7 meters away from the camera.
z = point3D(:, :, 3);
zdisp = z;
point3Ddisp = point3D;
point3Ddisp(:,:,3) = zdisp;
showPointCloud(point3Ddisp, J1, 'VerticalAxis', 'Y',...
'VerticalAxisDir', 'Down' );
xlabel('X');
ylabel('Y');
zlabel('Z');
I have included the images of the Scene Reconstruction, Disparity Map, Mean Error Per Pixel and After Rectification. The version of Matlab I am using is R2014b Student Edition purchased from the Matlab Website.
I think that the most clear problem here is that the re-projection
error (more than 3 pixels) you got in the stereo calibration, points
to a calibration issue. I would advise you to re-calibrate to get a
smaller re-projection error (should be significantly below 1 pixel
for good reconstruction results).
Another question regarding your calibration: what lens distortion
model do you use? I believe you've got fish-eye lenses there - I
am not sure that the Matlab toolbox knows how to handle these.
You have two problems here. One, as #ezfn pointed out, is that the lens distortion may simply be too severe. The best thing to try here is to take even more calibration images so that you have the checkerboard close to the edges and corners of the field of view. Also, try placing the checkerboard at varying distances from the camera. See if you can get those reprojection errors down.
The second issue here is that you need to change the 'DisparityRange' parameter of the disparity function. Display the anaglyph image using imtool, and use the ruler widget to measure distances between some pairs of corresponding points. That should give you the idea of what the disparity range should be. Just looking at the image I can see that [0 64] is way too small.
Related
I'm trying to print out QR codes onto the surface of water bottles using matlab. However, the QR bottles cannot be recognized because of the curved surface of the bottle, and the therefore deformed shape of the QR codes.
I was wondering what mathematical equation I might have to apply to deform the picture (for both the general geometry of the picture, and the spacing deformation inside the picture), so that when printed onto a cylindrical surface, it seems like its on a flat surface to a camera.
Thanks!
I had some success with this and though it doesn't use Matlab, it may give you, or someone else an idea on a possible way to get started.
I generated a QR code with qrencode in Terminal like this:
qrencode -o qr.png -d 300 'http://www.thesetchells.com'
I resized that to 500x500 and then generated a "displacement map" which is just a linear gradient with a contrast stretch created to match the size of the QR code. I used ImageMagick in Terminal, but obviously you can use Matlab or other tools:
I then applied the displacement map to my QR code like this:
convert qr.png map.png -fx 'p{v*w,j}' result.png
And got a distorted QR code, which I printed and wrapped round a large bottle and my iPhone was able to read it.
Anthony Thyssen has some very useful information on "displacement maps" here.
So I've tried an approach to my problem using the basic "car.jpg" image included in Matlab.
So far, I've taken a square part of the car picture, and wrapped it around the surface of part of a cylinder.
Here is my code:
close all, clf, clear all
%% Creating cylinder
r = 6; %centimeter
h = 25; %centimeter
[X,Y,Z] = cylinder(r,100);
x = X(:,1:ceil(length(X)/3));
y = Y(:,1:ceil(length(Y)/3));
z = Z(:,1:ceil(length(Z)/3));
%% Plotting cylinder surface
figure(1), clf
h = surf(x,y,z); hold on
axis([-r r -r r]*2.2)
plot3([-r -r], get(gca,'ylim'), [0 0]);
plot3([r r], get(gca,'ylim'), [0 0]);
plot3(get(gca,'xlim'), [-r -r], [0 0]);
plot3(get(gca,'xlim'), [r r], [0 0]);
xlabel('x');
ylabel('y');
zlabel('z');
rotate3d on
axis vis3d
%% Car image
img = imread('car.jpg');
img = imrotate(img,180);
figure(2), clf
%imshow(img(1:340, 1:340, :));
imagesc(img(1:340, 1:340, :));
figure(), clf
warped_plot = warp(x,y,z,img(1:340, 1:340, :))
The next step could be to now project that warped image onto a flat surface (but I'm not sure how to do that mathematically, nor have I found a in-built Matlab function to do this).
Another possibility I was hoping to get feedback on, is to find a mathematical relation between corresponding horizontal points on the flat and cylindrical image, and then to apply the inverse of that equation to the flat image, so that once it is printed out and stuck onto a cylindrical surface it appears flat.
I am using the extrinsics function for a project that aims to determine distances of objects from the camera that have checkerboards attached to them. This is done for every frame in a video. My workflow is fairly simple:
Attach an asymmetric checkerboard to each object
Calibrate the camera using the Image Processing Toolbox on ~ 20 images
For each frame
while checkerboardsFound < numCheckerboards
% Undistort the image
data = undistortImage(data, cameraParams);
% Get the checkerboard points in the frame
[imagePoints, boardSize] = detectCheckerboardPoints(data);
worldPoints = generateCheckerboardPoints(boardSize, squareSize);
% Get the rotation and translation
[rot, trans] = extrinsics(imagePoints,worldPoints, cameraParams);
% <Draw a black rectangle over the area where the checkerboard points are
% using the imagePoints matrix. This "hides" the checkerboard so we can
% run the detectCheckerboardPoints(data) function again and find a different
% one.>
data = hideCheckerboard(data,...
[imagePoints(1,:);...
imagePoints(end,:)]);
checkerboardsFound = checkerboardsFound + 1;
end;
My problem is that when I find, say, two particular checkerboards that are about 5cm in depth apart and look at the Z value of their respective translation vectors as returned by the extrinsics function, I expect to see a difference of about 50 units (5cm). However, I see a considerably greater difference - e.g. about 400 units (40cm). I am certain the square sizes for the checkerboards are correctly defined (5mm each). The frames come from a Samsung smart phone capturing in video mode (which applies magnification).
Could anyone please help determine what could be causing this discrepancy in the Z values? Is it too ambitious to use the extrinsics function in such a manner?
I am trying to use stereo imaging for 3D reconstruction, however when I use the tutorials and tools in matlab for stereo vision, I get erroneous results. I use a Loreo 3D macro lens to take images of small instruments at a distance of around 23mm. Then after cropping the images to create left and right images I use the stereo calibration app (I have also used code from the matlab tutorial which does pretty much the same thing). These are the kinds of results I get.
Stereo calibration using matlab's app
I am aware that the reprojection errors are quite high, but have tried a lot of things like changes in image quantity, illumination, checkerboard size and the skew, tangential distortion, and coefficients in the app to reduce this value without any luck. At a first glance, the Extrinsic reconstruction in the bottom right looks accurate, as the dimensions are quite correct. Therefore when I use the exported stereoParameters with a new image and the next code:
Isv = imread('IMG_0036.JPG');
I1 = imcrop(Isv, [0 0 2592 3456]);
I2 = imcrop(Isv, [2593 0 2592 3456]);
% Rectify the images.
[J1, J2] = rectifyStereoImages(I1, I2, stereoParams, 'OutputView', 'valid');
% Display the images before rectification.
figure;
imshow(stereoAnaglyph(I1, I2), 'InitialMagnification', 30);
title('Before Rectification');
% Display the images after rectification.
figure;
imshow(stereoAnaglyph(J1, J2), 'InitialMagnification', 30);
title('After Rectification');
disparityRange = [0, 64];
disparityMap = disparity(rgb2gray(J1), rgb2gray(J2), 'DisparityRange', ...
disparityRange);
figure;
imshow(disparityMap, disparityRange, 'InitialMagnification', 30);
colormap('jet');
colorbar;
title('Disparity Map');
point3D = reconstructScene(disparityMap, stereoParams);
% Convert from millimeters to meters.
point3D = point3D / 1000;
% Plot points between 3 and 7 meters away from the camera.
z = point3D(:, :, 3);
maxZ = 7;
minZ = 3;
zdisp = z;
zdisp(z < minZ | z > maxZ) = NaN;
point3Ddisp = point3D;
point3Ddisp(:,:,3) = zdisp;
figure
pcshow(point3Ddisp, J1, 'VerticalAxis', 'Y', 'VerticalAxisDir', 'Down' );
xlabel('X');
ylabel('Y');
zlabel('Z');
I get these erroneous rectification, disparity and 3D reconstruction.
Rectification, disparity and erroneous 3D reconstruction
As it can be seen, the rectification looks bad, as it the objects are too separated in my opinion; also the disparity results look very random, and finally the 3D reconstruction simply has no discernible outcome.
Please, I ask for any possible help, comments or recommendations regarding this issue.
Your reprojection errors are indeed high... Leaving that aside for the moment, your most immediate problem is that the disparityRange is too small.
The rectified images look fine. Corresponding points appear to be on the same pixel rows in both images, which is what you want. Display the anaglyph of the rectified images using imtool, and use the ruler widget to measure the distances between some of the corresponding points. That should give you an idea of what your disparity range should be. [0 64] is definitely too small.
To improve the reprojection errors, normally, I would say get more images. But you already have 30 pairs, which is a good number. You can specify the initial intrinsics and distortion, if you can get them off the camera manufacturer's spec, but I doubt they would help here. Try turning on tangential distortion estimation, and try using 3 radial distortion coefficients instead of two.
Also, see if you can more your cameras farther away from the scene. It may be that at such a short distance the pinhole camera model starts to break down.
There are also things you can do to improve your disparity and 3D reconstruction:
Try varying the BlockSize parameter of the disparity function
Try applying histogram equalization and/or low-pass filtering to your rectified images before computing disparity
Try median filtering the resulting disparity map to reduce the noise
Another tip: since you know approximately where the objects of interest are relative to the cameras, you can simply exclude the 3D points whose z-coordinate is too big or two small (or negative). This should give you a much cleaner 3D plot. You already have this in your code, but you should modify it to have appropriate units and thresholds for Z.
I have built a 3D model from a 2D image. I want to know how much accurate my model is using some statistical test. I think there are many available methods to do this like correlation and mean squares as mentioned in this question, Is it possible to compare 3D images?.
I couldn't find a clear description of the available tests in other sites. I've found an implementation which compares 2D images using square means here, http://www.mathworks.com/matlabcentral/answers/81048-mse-mean-square-error. I'm not sure if this can be used to calculate the model accuracy. Also, I didn't find an explanation of how the tests work, i.e. what parameters are compared (color, intensity, etc.) ?
EDIT: For more clarity, the 3D model represents every pixel in the 2D image as a voxel which has a color associated with it. The purpose of this model is to reconstruct the different color regions found in the 2D image into the 3D representation. So, the number of pixels that has some color (they represent a region) is calculated from the 2D image. A similar number of voxels will be constructed in the 3D model and given the same color. What matters in this modeling problem is the following,
1- size of the regions (must be almost similar in the 2D image and the model).
2-Connectivity level of a region in the 2D image and its corresponding region constructed in the 3D image must be similar. By connectivity I mean to check if the region components are scattered through the image or they are connected forming one large connected region instead of many small scattered regions of the same color.
EDIT2: I think color correlogram is suitable. I have found a code that implements it, but it is not clear to me. Here is the code,
% Soumyabrata Dev
% E-mail: soumyabr001#e.ntu.edu.sg
% http://www3.ntu.edu.sg/home2012/soumyabr001/
I= imread ('img.jpg');
correlogram_vector=[];
[Y,X]=size(rgb2gray(I));
% quantize image into 64 colors = 4x4x4, in RGB space
[img_no_dither, ~] = rgb2ind(I, 64, 'nodither');
% figure, imshow(img_no_dither, map);
%rgb = ind2rgb(img_no_dither, map); % rgb = double(rgb)
distance_vector= [1 3];
[~,d]=size(distance_vector);
count_matrix=zeros(64,d); total_matrix=zeros(64,d);
prob_dist=cell(1,d);
for serial_no=1:1:d
for x=1:X
for y=1:Y
color=img_no_dither(y,x);
% At the given distance
[positive_count,total_count]=get_n(distance_vector(serial_no),x,y,color,img_no_dither,X,Y);
count_matrix(color+1,serial_no)=count_matrix(color+1,serial_no)+positive_count;
total_matrix(color+1,serial_no)=total_matrix(color+1,serial_no)+total_count;
end
end
prob_dist{serial_no}=count_matrix(:,serial_no)./(1+total_matrix(:,serial_no));
end
for serial_no=1:d
correlogram_vector=cat(1,correlogram_vector,prob_dist{serial_no});
end
end
This is the method get_n,
function [positive_count,total_count]=get_n(n,x,y,color,img_no_dither,X,Y)
% This function is useful to get the validity map of the neighborhood case.
% It can handle any number of neighborhood distances.
% Input
% n=The order of the neighborhood
% x & y= x y co-ordinates of the given pixel
% color= particular quantized color
% img_no_dither= The color quantized image matrix
% X & Y= The original dimensions of the input image
% Output
% positive_count= The number of occurences which have the same color
% total_count= The total number of valid cases for this particular instant
valid_vector8n=zeros(1,8*n); % This is because of the propoerty of inf-norm. Each distance has 8 times the order
positive_count=0; total_count=0;
nbrs_x=zeros(1,8*n); nbrs_y=zeros(1,8*n);
% The counting of the pixels is done in the following manner: From the
% given pixel, go left-->up-->right-->down-->left-->up
% Y co-ordinates of nbrs
nbrs_y(1)=y;
d=1;
for k=2:1+n
nbrs_y(k)=y-d;
d=d+1;
end
nbrs_y(1+n:1:3*n+1)=y-n;
d=0;
for k=3*n+1:5*n+1
nbrs_y(k)=y-n+d;
d=d+1;
end
nbrs_y(5*n+1:1:7*n+1)=y+n;
d=0;
for k=7*n+1:1:7*n+1+(n-1)
nbrs_y(k)=y+n-d;
d=d+1;
end
% X co-ordinates of nbrs
nbrs_x(1)=x-n;
nbrs_x(2:1:1+n)=x-n;
d=0;
for k=1+n:1:3*n+1
nbrs_x(k)=x-n+d;
d=d+1;
end
nbrs_x(3*n+1:5*n+1)=x+n;
d=0;
for k=5*n+1:7*n+1
nbrs_x(k)=x+n-d;
d=d+1;
end
nbrs_x(7*n+1:7*n+1+(n-1))=x-n;
% Assigning the validity of the neighborhood
for i=1:8*n
if nbrs_x(i)>0 && nbrs_x(i)<=X && nbrs_y(i)>0 && nbrs_y(i)<=Y
valid_vector8n(i)=1;
else
valid_vector8n(i)=0;
end
end
% Couting the number of common colors in the valid areas of the
% neighborhood.
for j=1:8*n
if valid_vector8n(j)==1
data= img_no_dither(nbrs_y(j),nbrs_x(j));
if (data==color)
positive_count=positive_count+1;
end
total_count=total_count+1;
end
end
end
Can anyone please clarify how this code works?
The code above is for autocorrelogram not correlogram. I've read that the first is better, but it can only calculate the spatial probability using pixels of the same colors (can't be applied on pairs of pixels which have different colors). Is this right?
Thank You.
TLDR: Classical workflow:
find matching features in both models,
calculate the distance,
??????,
PROFIT!!
I want to extract (x,y) pixel coordinates out of the SURF points returned, as an example in the example provided here using Matlab. It is clear that using 'ptsIn(1).Location' I can return the (x,y) coordinates of the point. But the points retuned included some decimal points as well, as an example (102.9268, 51.7285). Is there any way to convert this to pixel positions in the image plane, or just averaging these values will give the pixel positions? Thank you.
To understand it further I tried the following code in this link.
% Extract SURF features
I = imread('cameraman.tif');
points = detectSURFFeatures(I);
[features, valid_points] = extractFeatures(I, points);
% Visualize 10 strongest SURF features, including their
% scales and orientation which were determined during the
% descriptor extraction process.
imshow(I); hold on;
strongestPoints = valid_points.selectStrongest(10);
strongestPoints.plot('showOrientation',true);
Then, tried the command strongestPoints.Location in the Matlab console, which returned the following (x,y) coordinates.
139.7482 95.9542
107.4502 232.0347
116.6112 138.2446
105.5152 172.1816
113.6975 48.7220
104.4210 75.7348
111.3914 154.4597
106.2879 175.2709
131.1298 98.3900
124.2933 64.4942
Since there is a coordinate (107.4502 232.0347), I tried to mark the row 232 as black (I(232,:)=0;) to see whether it matches 232.0347 y coordinate in the SURF point, and received the following figure. So it seems rounded values of the SURF points give the (x,y) pixel coordinates of the image.