Having this coordinate system:
And this dominant vertical vanishing point:
I would like to rotate the image around x axis so the vanishing point is at infinity. That means that all vertical lines are parallel.
I am using matlab. I find the line segmentes using LSD and the vanishing point using homogeneous coordinates. I would like to use angle-axis representation, then convert it to a rotation matrix and pass this to imwarp and get the rotated image. Also would be good to know how to rotate the segments. The segments are as (x1,y1,x2,y2).
Image above example:
Vanishin point in homogenous coordinates:
(x,y,z) = 1.0e+05 * [0.4992 -2.2012 0.0026]
Vanishin point in cartesian coordinates (what you see in the image):
(x,y) = [190.1335 -838.3577]
Question: With this vanishing point how do I compute the rotation matrix in the world x axis as explained above?
If all you're doing is rotating the image so that the vector from the origin to the vanishing point, is instead pointing directly vertical, here's an example.
I = imread('cameraman.tif');
figure;imagesc(I);set(gcf,'colormap',gray);
vp=-[190.1335 -838.3577,0]; %3d version,just for cross-product use,-ve ?
y=[0,1,0]; %The vertical axis on the plot
u = cross(vp,y); %you know it's going to be the z-axis
theta = -acos(dot(vp/norm(vp),y)); %-ve ?
rotMat = vrrotvec2mat([u, theta]);
J=imwarp(I,affine2d (rotMat));
figure;imagesc(J);set(gcf,'colormap',gray); %tilted image
You can play with the negatives, and plotting, since I'm not sure about those parts applying to your situation. The negatives may come from plotting upside down, or from rotation of the world vs. camera coordinate system, but I don't have time to think about it right now.
EDIT
If you want to rotation about the X-axis, this might work (adapted from https://www.mathworks.com/matlabcentral/answers/113074-how-to-rotate-an-image-along-y-axis), or check out: Rotate image over X, Y and Z axis in Matlab
[rows, columns, numberOfColorChannels] = size(I);
newRows = rows * cos(theta);
rotatedImage = imresize(I, [newRows, columns]);
Related
This question already has answers here:
How to straighten a tilted square shape in an image?
(2 answers)
Closed 5 years ago.
I have a cropped image of a card:
The card is a rectangle with rounded corners, is brightly colored, and sits on a relatively dark background.
It is, therefore, easy to differentiate between pixels belonging to the card and pixels belonging to the background.
I want to use MATLAB to rotate the card so its sides are vertical and horizontal (and not diagonal) and create an image of nothing but the straightened card.
I need this to work for any reasonable card angle (say +45 to -45 degrees of initial card rotation).
What would be the best way of doing this?
Thanks!
You can do this by finding the lines made by the edges of the card. The angle of rotation is then the angle between one of the lines and the horizontal (or vertical).
In MATLAB, you can use the Hough line detector to find lines in a binary image.
0. Read the input image
I downloaded your image and renamed it card.png.
A = imread('card.png');
We don't need color information, so convert to grayscale.
I = rgb2gray(A);
1. Detect edges in the image
A simple way is to use the Canny edge detector. Adjust the threshold to reject noise and weak edges.
BW = edge(I, 'canny', 0.5);
Display the detected edges.
figure
imshow(BW)
title('Canny edges')
2. Use the Hough line detector
First, you need to use the Hough transform on the black and white image, with the hough function. Adjust the resolution so that you detect all lines you need later.
[H,T,R] = hough(BW, 'RhoResolution', 2);
Second, find the strongest lines in the image by finding peaks in the Hough transform with houghpeaks.
P = houghpeaks(H, 100); % detect a maximum of 100 lines
Third, detect lines with houghlines.
lines = houghlines(BW, T, R, P);
Display the detected lines to make sure you find at least one along the edge of the card. The white border around the black background in your image makes detecting the right edges a bit more difficult.
figure
imshow(A)
hold on
for k = 1:length(lines)
xy = [lines(k).point1; lines(k).point2];
plot(xy(:,1), xy(:,2), 'LineWidth', 2, 'Color', 'red');
end
title('Detected lines')
3. Calculate the angle of rotation
lines(3) is the left vertical edge of the card. lines(3).point2 is the end of the line that is at the bottom. We want this point to stay where it is, but we want to vector along the line to be aligned with the vector v = [0 -1]'. (The origin is the top-left corner of the image, x is horizontal to the right and y is vertical down.)
lines(3)
ans =
struct with fields:
point1: [179 50]
point2: [86 455]
theta: 13
rho: 184
Simply calculate the angle between the vector u = lines(3).point1 - lines(3).point2 and the vertical vector v.
u = lines(3).point1 - lines(3).point2; % vector along the vertical left edge.
v = [0 -1]; % vector along the vertical, oriented up.
theta = acos( u*v' / (norm(u) * norm(v)) );
The angle is in radians.
4. Rotate
The imrotate function lets you rotate an image by specifying an angle in degrees. You could also use imwarp with a rotation transform.
B = imrotate(A, theta * 180 / pi);
Display the rotated image.
figure
imshow(B)
title('Rotated image')
Then you would have to crop it.
I have this 3D image generated from the simple code below.
% Input Image size
imageSizeY = 200;
imageSizeX = 120;
imageSizeZ = 100;
%# create coordinates
[rowsInImage, columnsInImage, pagesInImage] = meshgrid(1:imageSizeY, 1:imageSizeX, 1:imageSizeZ);
%# get coordinate array of vertices
vertexCoords = [rowsInImage(:), columnsInImage(:), pagesInImage(:)];
centerY = imageSizeY/2;
centerX = imageSizeX/2;
centerZ = imageSizeZ/2;
radius = 28;
%# calculate distance from center of the cube
sphereVoxels = (rowsInImage - centerY).^2 + (columnsInImage - centerX).^2 + (pagesInImage - centerZ).^2 <= radius.^2;
%# Now, display it using an isosurface and a patch
fv = isosurface(sphereVoxels,0);
patch(fv,'FaceColor',[0 0 .7],'EdgeColor',[0 0 1]); title('Binary volume of a sphere');
view(45,45);
axis equal;
grid on;
xlabel('x-axis [pixels]'); ylabel('y-axis [pixels]'); zlabel('z-axis [pixels]')
I have tried plotting the image with isosurface and some other volume visualization tools, but there remains quite a few surprises for me from the plots.
The code has been written to conform to the image coordinate system (eg. see: vertexCoords) which is a left-handed coordinate system I presume. Nonetheless, the image is displayed in the Cartesian (right-handed) coordinate system. I have tried to see this displayed as the figure below, but that’s simply not happening.
I am wondering if the visualization functions have been written to display the image the way they do.
Image coordinate system:
Going forward, there are other aspects of the code I am to write for example if I have an input image sphereVoxels as in above, in addition to visualizing it, I would want to find north, south east, west, top and bottom locations in the image, as well as number and count the coordinates of the vertices, plus more.
I foresee this would likely become confusing for me if I don’t stick to one coordinate system, and considering that the visualization tools predominantly use the right-hand coordinate system, I would want to stick with that from the onset. However, I really do not know how to go about this.
Right-hand coordinate system:
Any suggestions to get through this?
When you call meshgrid, the dimensions x and y axes are switched (contrary to ndgrid). For example, in your case, it means that rowsInImage is a [120x100x200] = [x,y,z] array and not a [100x120x200] = [y,x,z] array even if meshgrid was called with arguments in the y,x,z order. I would change those two lines to be in the classical x,y,z order :
[columnsInImage, rowsInImage, pagesInImage] = meshgrid(1:imageSizeX, 1:imageSizeY, 1:imageSizeZ);
vertexCoords = [columnsInImage(:), rowsInImage(:), pagesInImage(:)];
I use camera calibration in matlab to detect some checkerboard patterns, after
figure; showExtrinsics(cameraParams, 'CameraCentric');
Now, I want to rotate the checkerboard patterns around the x-axis such that all of them have nearly the same y coordinates in the camera frame.
Method:
I get the positions of all patterns in the camera's frame. Then I do optimization,where the objective function is to minimize variance in y and the variable is rotation about x ranging from o to 360.
Problem:
But when I plot the transformed y-coordinates, they are even nearly in a line.
Code:
Get the checkerboad points:
%% Get rotation and translation matrices for each image;
T_cw=cell(num_imgs,1); % stores camera to world rotation and translation for each image
pixel_coordinates=zeros(num_imgs,2); % stores the pixel coordinates of each checkerboard origin
for ii=1:num_imgs,
% Calibrate the camera
im=imread(list_imgs_path{ii});
[imagePoints, boardSize] = detectCheckerboardPoints(im);
[r_wc, t_wc] = extrinsics(imagePoints, worldPoints, cameraParams);
T_wc=[r_wc,t_wc';0 0 0 1];
% World to camera matrix
T_cw{ii} = inv(T_wc);
t_cw{ii}=T_cw{ii}(1:3,4); % x,y,z coordinates in camera's frame
end
Data(num_imgs=10):
t_cw
[-1072.01388542262;1312.20387622761;-1853.34408157349]
[-1052.07856598756;1269.03455126794;-1826.73576892251]
[-1091.85978641218;1351.08261414473;-1668.88197803184]
[-1337.56358084648;1373.78548638383;-1396.87603554914]
[-1555.19509876309;1261.60428874489;-1174.63047408086]
[-1592.39596647158;1066.82210015055;-1165.34417772659]
[-1523.84307918660;963.781819272748;-1207.27444716506]
[-1614.00792252030;893.962075837621;-1114.73528985018]
[-1781.83112607964;708.973204727939;-797.185326205240]
[-1781.83112607964;708.973204727939;-797.185326205240]
Main code (Optimization and transformation):
%% Get theta for rotation
f_obj = #(x)var_ycors(x,t_cw);
opt_theta = fminbnd(f_obj,0,360);
%% Plotting (rotate ycor and check to fix theta)
y_rotated=zeros(1,num_imgs);
for ii=1:num_imgs,
y_rotated(ii)=rotate_cor(opt_theta,t_cw{ii});
end
plot(1:numel(y_rotated),y_rotated);
function var_computed=var_ycors(theta,t_cw)
ycor=zeros(1,numel(t_cw));
for ii =1:numel(t_cw),
ycor(ii)=rotate_cor(theta,t_cw{ii});
end
var_computed=var(ycor);
end
function ycor=rotate_cor(theta,mat)
r_x=[1 0 0; 0 cosd(theta) -sind(theta); 0 sind(theta) cosd(theta)];
rotate_mat=mat'*r_x;
ycor=rotate_mat(2);
end
This is a clear eigenvector problem!
Take your centroids:
t_cw=[-1072.01388542262;1312.20387622761;-1853.34408157349
-1052.07856598756;1269.03455126794;-1826.73576892251
-1091.85978641218;1351.08261414473;-1668.88197803184
-1337.56358084648;1373.78548638383;-1396.87603554914
-1555.19509876309;1261.60428874489;-1174.63047408086
-1592.39596647158;1066.82210015055;-1165.34417772659
-1523.84307918660;963.781819272748;-1207.27444716506
-1614.00792252030;893.962075837621;-1114.73528985018
-1781.83112607964;708.973204727939;-797.185326205240
-1781.83112607964;708.973204727939;-797.185326205240];
t_cw=reshape(t_cw,[3,10])';
compute PCA on them, so we know the principal conponents:
[R]=pca(t_cw);
And.... thats it! R is now the transformation matrix between your original points and the rotated coordinate system. As an example, I will draw in red the old points and in blue the new ones:
hold on
plot3(t_cw(:,1),t_cw(:,2),t_cw(:,3),'ro')
trans=t_cw*R;
plot3(trans(:,1),trans(:,2),trans(:,3),'bo')
You can see that now the blue ones are in a plane, with the best possible fit to the X direction. If you want them in Y direction, just rotate 90 degrees in Z (I am sure you can figure out how to do this with 2 minutes of Google ;) ).
Note: This is mathematically the best possible fit. I know they are not as "in a row" as one would like, but this is because of the data, this is honestly the best possible fit, as that is what the eigenvectors are!
Calibration:
I have calibrated the camera using this vision toolbox in Matlab. I used checkerboard images to do so. After calibration I get the cameraParams
which contains:
Camera Extrinsics
RotationMatrices: [3x3x18 double]
TranslationVectors: [18x3 double]
and
Camera Intrinsics
IntrinsicMatrix: [3x3 double]
FocalLength: [1.0446e+03 1.0428e+03]
PrincipalPoint: [604.1474 359.7477]
Skew: 3.5436
Aim:
I have recorded trajectories of some objects in motion using this camera. Each object corresponds to a single point in a frame. Now, I want to project the points such that I get a top-view.
Note all these points I wish to transform are are the on the same plane.
ex: [xcor_i,ycor_i ]
-101.7000 -77.4040
-102.4200 -77.4040
KEYPOINT: This plane is perpendicular to one of images of checkerboard used for calibration. For that image(below), I know the height of origin of the checkerboard of from ground(193.040 cm). And the plane to project the points on is parallel to the ground and perpendicular to this image.
Code
(Ref:https://stackoverflow.com/a/27260492/3646408 and answer by #Dima below):
function generate_homographic_matrix()
%% Calibrate camera
% Define images to process
path=['.' filesep 'Images' filesep];
list_imgs=dir([path '*.jpg']);
list_imgs_path=strcat(path,{list_imgs.name});
% Detect checkerboards in images
[imagePoints, boardSize, imagesUsed] = detectCheckerboardPoints(list_imgs_path);
imageFileNames = list_imgs_path(imagesUsed);
% Generate world coordinates of the corners of the squares
squareSize = 27; % in units of 'mm'
worldPoints = generateCheckerboardPoints(boardSize, squareSize);
% Calibrate the camera
[cameraParams, imagesUsed, estimationErrors] = estimateCameraParameters(imagePoints, worldPoints, ...
'EstimateSkew', true, 'EstimateTangentialDistortion', true, ...
'NumRadialDistortionCoefficients', 3, 'WorldUnits', 'mm');
%% Compute homography for peripendicular plane to checkerboard
% Detect the checkerboard
im=imread(['.' filesep 'Images' filesep 'exp_19.jpg']); %exp_19.jpg is the checkerboard orthogonal to the floor
[imagePoints, boardSize] = detectCheckerboardPoints(im);
% Compute rotation and translation of the camera.
[Rc, Tc] = extrinsics(imagePoints, worldPoints, cameraParams);
% Rc(rotation of the calibration view w.r.t the camera) = [x y z])
%then the floor has rotation Rf = [z x -y].(Normal vector of the floor goes up.)
Rf=[Rc(:,3),Rc(:,1),Rc(:,2)*-1];
% Translate it to the floor
H=452;%distance btw origin and floor
Fc = Rc * [0; H; 0];
Tc = Tc + Fc';
% Combine rotation and translation into one matrix:
Rf(3, :) = Tc;
% Compute the homography between the checkerboard and the image plane:
H = Rf * cameraParams.IntrinsicMatrix;
save('homographic_matrix.mat','H')
end
%% Transform points
function [x_transf,y_transf] =transform_points(xcor_i,ycor_i)
% creates a projective2D object and then transforms the points forward to
% get a top-view
% xcor_i and ycor_i are 1d vectors comprising of the x-coordinates and
% y-coordinates of trajectories.
data=load('homographic_matrix.mat');
homo_matrix=data.H;
tform=projective2d(inv(homo_matrix));
[x_transf,y_transf] = transformPointsForward(tform,xcor_i,ycor_i);
end
Quoting text from OReilly Learning OpenCV Pg 412:
"Once we have the homography matrix and the height parameter set as we wish, we could
then remove the chessboard and drive the cart around, making a bird’s-eye view video
of the path..."
This what I essentially wish to achieve.
Abhishek,
I don't entirely understand what you are trying to do. Are your points on a plane, and are you trying to create a bird's eye view of that plane?
If so, then you need to know the extrinsics, R and t, describing the relationship between that plane and the camera. One way to get R and t is to place a checkerboard on the plane, and then use the extrinsics function.
After that, you can follow the directions in the question you cited to get the homography. Once you have the homography, you can create a projective2D object, and use its transformPointsForward method to transform your points.
Since you have the size of squares on the grid, then given 2 points that you know are connected by an edge of size E (in real world units), you can calculate their 3D position.
Taking the camera intrinsic matrix K and the 3D position C and the camera orientation matrix R, you can calculate a ray to each of the points p by doing:
D = R^T * K^-1 * p
Each 3D point is defined as:
P = C + t*D
and you have the constraint that ||P1-P2|| = E
then it's a matter of solving for t1,t2 and finding the 3D position of the two points.
In order to create a top view, you can take the 3D points and project them using a camera model for that top view to generate a new image.
If all your points are on a single plane, it's enough to calculate the position of 3 points, and you can extrapolate the rest.
If your points are located on a plane that you know one coordinate of, you can do it simply for each point. For example, if you know that your camera is located at height h=C.z, and you want to find the 3D location of points in the frame, given that they are on the floor (z=0), then all you have to do is calculate the direction D as above, and then:
t=abs( (h-0)/D.z )
The 0 represent the height of the plane. Substitute for any other value for other planes.
Now that you have the value of t, you can calculate the 3D position of each point: P=C+t*D.
Then, to create a top view, create a new camera position and rotation to match your required projection, and you can project each point onto this camera's image plane.
If you want a full image, you can interpolate positions and fill in the blanks where no feature point was present.
For more details, you can always read: http://www.robots.ox.ac.uk/~vgg/hzbook/index.html
Imagine a dome with its centre in the +z direction. What I want to do is to move that dome's centre to a different axis (e.g. 20 degrees x axis, 20 degrees y axis, 20 degrees z axis). How can I do that ? Any hint/tip helps.
Add more info:
I've been dabbling with rotation matrices in wiki for a while. The problem is, it is not a commutative operation. RxRyRz is not same as RzRyRx. So based on the way I multiple it I get a different final results. For example, I want my final projection to have 20 degrees from the original X axis, 20 degrees from original Y axis and 20 degrees from original Z axis. Based on the matrix, giving alpha, beta, gamma values 20 (or its corresponding radian) does NOT result the intended rotation. Am I missing something? Is there a matrix that I can just put the intended angles and get it at the end ?
Using a rotation matrix is an easy way to rotate a collection of (x,y,z) points. You can calculate a rotation matrix for your case using the equations in the general rotation section. Note that figuring out the angle values to plug into those equations can be tricky. Think of it as rotating about one axis at a time and remember that the order of your rotations (order of multiplications) does matter.
An alternative to the general rotation equations is to calculate a rotation matrix from axis and angle. It may be easier for you to define correct parameters with this method.
Update: After perusing Wikipedia, I found a simple way to calculate rotation axis and angle between two vectors. Just fill in your starting and ending vectors for a and b here:
a = [0.0 0.0 1.0];
b = [0.5 0.5 0.0];
vectorMag = #(x) sqrt(sum(x.^2));
rotAngle = acos(dot(a,b) / (vectorMag(a) * vectorMag(b)))
rotAxis = cross(a,b)
rotAxis =
-0.5 0.5 0
rotAngle =
1.5708