fisheye extrinsics vs estimateWorldCameraPose - matlab

Question

When calibrating a fisheye camera using Matlab, I get the fisheye intrinsics parameters:
% Calibrate the camera using fisheye parameters
[cameraParams, imagesUsed, estimationErrors] = estimateFisheyeParameters(imagePoints, worldPoints, ...
[mrows, ncols], ...
'EstimateAlignment', true, ...
'WorldUnits', 'millimeters');
The output of this function are the Scaramuzza parameters, in a shape very different from the classical 3x3 intrinsics matrix.
From this intrinsic parameters, I would like to estimate the pose of one my calibration pattern. So far I found two solutions but I do not know which one is more accurate.
First, I found I can directly feed the extrinsics function with the current intrinsics :
% Extract intrinsics parameters
intrinsics = cameraParams.Intrinsics;
% Compute Rt matrix
[R,t] = extrinsics(imagePoints,worldPoints,intrinsics);
Looking inside the function, I can see this method uses the homography but kind of process by magic the Scaramuzza's intrinsics parameters. It is the same function used for both fisheye and non-fisheye models. Any idea the Scaramuzza parameters are processed here ?
Second solution is to use the function estimateWorldCameraPose which uses P3P and a RANSAC underneath. This function does not accept the raw fisheye parameters. One solution I found (https://fr.mathworks.com/matlabcentral/answers/548787-function-estimateworldcamerapose-or-extrinsics-for-fisheyeparameters-is-missing-is-it-possible?s_tid=answers_rc1-2_p2_MLT) uses the function undistortFisheyeImage as a workaround to extract the 3x3 intrinsic parameters :
[J,camIntrinsics] = undistortFisheyeImage(I,intrinsics)
Then I can feed the new intrinsics in the estimateWorldCameraPose.
Is this solution better ? How realiable this new intrinsics matrix is ?

Answer 1

In all calibration methods (either a normal lens or fisheye lens), the extrinsic estimation uses the undistorted image points. So, the matlab function "extrinsics" should have the undistortion part(undistortion using Scaramuzza's lens distortion model) as well.
Looking at the code "extrinsics", I see it is seperately handling the intrinsic when it is fisheye model.
if ~isa(cameraParams, 'fisheyeIntrinsics')
if isa(cameraParams, 'cameraParameters')
intrinsicParams = cameraParams;
else
intrinsicParams = cameraParams.CameraParameters;
end
intrinsics = intrinsicParams.IntrinsicMatrix;
[rotationMatrix, translationVector] = vision.internal.calibration.extrinsicsPlanar(...
imagePoints, worldPoints, intrinsics);
else
That is the only solution and only way to do that.

Answer 2

extrinsics function both accept fisheye and non-fisheye models,if accept fisheye model,the input argument imagePoints is distort points，while pin-hole(non-fisheye) model, it must accept undistort image points. it's internal implementation is both vision.internal.calibration.extrinsicsPlanar.m function.
function [R,T] = extrinsicsPlanar(imagePoints, worldPoints, intrinsics)
A = intrinsics';
% Compute homography.
H = fitgeotrans(worldPoints, imagePoints, 'projective');
H = H.T';
h1 = H(:, 1);
h2 = H(:, 2);
h3 = H(:, 3);
lambda = 1 / norm(A \ h1);
% Compute rotation
r1 = A \ (lambda * h1);
r2 = A \ (lambda * h2);
r3 = cross(r1, r2);
R = [r1'; r2'; r3'];
% R may not be a true rotation matrix because of noise in the data.
% Find the best rotation matrix to approximate R using SVD.
[U, ~, V] = svd(R);
R = U * V';
% Compute translation vector.
T = (A \ (lambda * h3))';
EXTRINSICS Compute camera extrinsics from a planar calibration pattern,so this function is suit for you.
Pixel point de-aliasing was performed using the hidden method in the fisheyeIntrinsics class before processing the fisheye pixel points(before extrinsicsPlanar function).
%------------------------------------------------------------------
% Determine the normalized 3D vector on the unit sphere
%------------------------------------------------------------------
function worldPoints = imageToNormalizedVector(this, imagePoints)
%imageToNormalizedVector Determine the normalized 3D vector on
%the unit sphere.
% worldPoints = imageToNormalizedWorld(intrinsics,imagePoints)
% maps image points to the normalized 3D vector emanating from
% the single effective viewpoint on the unit sphere.
%
% Inputs:
% -------
% intrinsics - fisheyeIntrinsics object.
%
% imagePoints - M-by-2 matrix containing [x, y]
% coordinates of image points. M is the
% number of points.
%
% Output:
% -------
% worldPoints - M-by-3 matrix containing corresponding
% [X,Y,Z] coordinates on the unit sphere.
points = vision.internal.inputValidation.checkAndConvertPoints(...
imagePoints, 'fisheyeIntrinsics', 'imagePoints');
if isa(points, 'single')
points = double(points);
end
center = double(this.DistortionCenter);
stretch = double(this.StretchMatrix);
coeffs = double(this.MappingCoeffsInternal);
% Convert image points to sensor coordinates
points(:, 1) = points(:, 1) - center(1);
points(:, 2) = points(:, 2) - center(2);
points = stretch \ points';
rho = sqrt(points(1, :).^2 + points(2, :).^2);
f = polyval(coeffs(end:-1:1), rho);
% Note, points could be invalid if f < 0
worldPoints = [points; f]';
nw = sqrt(sum(worldPoints.^2, 2));
nw(nw == 0) = eps;
worldPoints = worldPoints ./ horzcat(nw,nw, nw);
worldPoints = cast(worldPoints, class(imagePoints));
end
estimateWorldCameraPose function input argument worldPoints accept 3 columns positions. but you use coplaner coordinates, z=0,may be this function return status is 1 or 2. So your later solution may not better.