I am using the extrinsics function for a project that aims to determine distances of objects from the camera that have checkerboards attached to them. This is done for every frame in a video. My workflow is fairly simple:
Attach an asymmetric checkerboard to each object
Calibrate the camera using the Image Processing Toolbox on ~ 20 images
For each frame
while checkerboardsFound < numCheckerboards
% Undistort the image
data = undistortImage(data, cameraParams);
% Get the checkerboard points in the frame
[imagePoints, boardSize] = detectCheckerboardPoints(data);
worldPoints = generateCheckerboardPoints(boardSize, squareSize);
% Get the rotation and translation
[rot, trans] = extrinsics(imagePoints,worldPoints, cameraParams);
% <Draw a black rectangle over the area where the checkerboard points are
% using the imagePoints matrix. This "hides" the checkerboard so we can
% run the detectCheckerboardPoints(data) function again and find a different
% one.>
data = hideCheckerboard(data,...
[imagePoints(1,:);...
imagePoints(end,:)]);
checkerboardsFound = checkerboardsFound + 1;
end;
My problem is that when I find, say, two particular checkerboards that are about 5cm in depth apart and look at the Z value of their respective translation vectors as returned by the extrinsics function, I expect to see a difference of about 50 units (5cm). However, I see a considerably greater difference - e.g. about 400 units (40cm). I am certain the square sizes for the checkerboards are correctly defined (5mm each). The frames come from a Samsung smart phone capturing in video mode (which applies magnification).
Could anyone please help determine what could be causing this discrepancy in the Z values? Is it too ambitious to use the extrinsics function in such a manner?
Related
I am trying to use stereo imaging for 3D reconstruction, however when I use the tutorials and tools in matlab for stereo vision, I get erroneous results. I use a Loreo 3D macro lens to take images of small instruments at a distance of around 23mm. Then after cropping the images to create left and right images I use the stereo calibration app (I have also used code from the matlab tutorial which does pretty much the same thing). These are the kinds of results I get.
Stereo calibration using matlab's app
I am aware that the reprojection errors are quite high, but have tried a lot of things like changes in image quantity, illumination, checkerboard size and the skew, tangential distortion, and coefficients in the app to reduce this value without any luck. At a first glance, the Extrinsic reconstruction in the bottom right looks accurate, as the dimensions are quite correct. Therefore when I use the exported stereoParameters with a new image and the next code:
Isv = imread('IMG_0036.JPG');
I1 = imcrop(Isv, [0 0 2592 3456]);
I2 = imcrop(Isv, [2593 0 2592 3456]);
% Rectify the images.
[J1, J2] = rectifyStereoImages(I1, I2, stereoParams, 'OutputView', 'valid');
% Display the images before rectification.
figure;
imshow(stereoAnaglyph(I1, I2), 'InitialMagnification', 30);
title('Before Rectification');
% Display the images after rectification.
figure;
imshow(stereoAnaglyph(J1, J2), 'InitialMagnification', 30);
title('After Rectification');
disparityRange = [0, 64];
disparityMap = disparity(rgb2gray(J1), rgb2gray(J2), 'DisparityRange', ...
disparityRange);
figure;
imshow(disparityMap, disparityRange, 'InitialMagnification', 30);
colormap('jet');
colorbar;
title('Disparity Map');
point3D = reconstructScene(disparityMap, stereoParams);
% Convert from millimeters to meters.
point3D = point3D / 1000;
% Plot points between 3 and 7 meters away from the camera.
z = point3D(:, :, 3);
maxZ = 7;
minZ = 3;
zdisp = z;
zdisp(z < minZ | z > maxZ) = NaN;
point3Ddisp = point3D;
point3Ddisp(:,:,3) = zdisp;
figure
pcshow(point3Ddisp, J1, 'VerticalAxis', 'Y', 'VerticalAxisDir', 'Down' );
xlabel('X');
ylabel('Y');
zlabel('Z');
I get these erroneous rectification, disparity and 3D reconstruction.
Rectification, disparity and erroneous 3D reconstruction
As it can be seen, the rectification looks bad, as it the objects are too separated in my opinion; also the disparity results look very random, and finally the 3D reconstruction simply has no discernible outcome.
Please, I ask for any possible help, comments or recommendations regarding this issue.
Your reprojection errors are indeed high... Leaving that aside for the moment, your most immediate problem is that the disparityRange is too small.
The rectified images look fine. Corresponding points appear to be on the same pixel rows in both images, which is what you want. Display the anaglyph of the rectified images using imtool, and use the ruler widget to measure the distances between some of the corresponding points. That should give you an idea of what your disparity range should be. [0 64] is definitely too small.
To improve the reprojection errors, normally, I would say get more images. But you already have 30 pairs, which is a good number. You can specify the initial intrinsics and distortion, if you can get them off the camera manufacturer's spec, but I doubt they would help here. Try turning on tangential distortion estimation, and try using 3 radial distortion coefficients instead of two.
Also, see if you can more your cameras farther away from the scene. It may be that at such a short distance the pinhole camera model starts to break down.
There are also things you can do to improve your disparity and 3D reconstruction:
Try varying the BlockSize parameter of the disparity function
Try applying histogram equalization and/or low-pass filtering to your rectified images before computing disparity
Try median filtering the resulting disparity map to reduce the noise
Another tip: since you know approximately where the objects of interest are relative to the cameras, you can simply exclude the 3D points whose z-coordinate is too big or two small (or negative). This should give you a much cleaner 3D plot. You already have this in your code, but you should modify it to have appropriate units and thresholds for Z.
I am working on a project in image processing which is based on importance of phase only reconstruction. For more information you can read the answer given by geometrikal in https://dsp.stackexchange.com/questions/16462/how-moving-part-pixel-intensity-values-of-video-frames-becomes-dominant-compared
Project has 2 parts.
Detect moving objects from the video of Traffic on road ( Please download the 1.47 MB video by ( step1) click on the play button then (step2) right clicking on video then ( step3 ) click on save as option )
Perform tracking of any single moving object from the video.
Now,I am quite successful in the 1st part of the project.Algorithm for it is
Algorithm No. 1
The proposed approach
Requirement: An input image sequence I(x, y, n) (where x and y are image dimensions and n represent frame number in a video) which is extracted from video.
Outcome: The segmentation mask of moving object for each frame
For each frame in a input video perform step 2, append step 2 result in resultant array ‘I(x, y, n)’
Smoothen the current frame using 2D Gaussian filter
Perform 3D FFT for the whole sequence I(x, y, n) using (Eq.4.1)
Calculate the phase spectrum using the real and imaginary parts of 3D DFT
Calculate the reconstructed sequence Î(x, y, n) using (Eq.4.2)
For each frame in a input video perform step 7 to step 10 to get segmentation mask for each frame and append step 10 result in resultant segmentation mask array BW(x,y,n)’
Smooth the reconstructed frame of Î(x, y, n) using the averaging filter.
Compute the mean value of the current frame
Convert the current frame into binary image using mean value as the threshold
Perform morphological processing, i.e., filling and closing, to obtain segmented mask of moving object for the current frame
End algorithm.
With the above algorithm I could find all moving object from the video.
Now, I want to move to 2nd part i.e.Perform tracking of any single moving object from the video .
As you can see in the figure,I have achieved the results as shown in the 1st column .I have reached till the 1st column results only. But my goal is to track a single vehicle like as shown in the 2nd column of the figure. (I have made results shown in 2nd column using Photoshop)
So can anybody help me ?
tic
clc;
clear all;
close all;
%read video file
video = VideoReader('D:\dvd\Matlab code\test videos\5.mp4');
T= video.NumberOfFrames ; %number of frames%
frameHeight = video.Height; %frame height
frameWidth = video.Width ; %frameWidth
get(video); %return graphics properties of video
i=1;
for t=300:15:550 %select frames between 300 to 550 with interval of 15 from the video
frame_x(:,:,:,i)= read(video, t);
frame_y=frame_x(:,:,:,i);
%figure,
%imshow(f1),title(['test frames :' num2str(i)]);
frame_z=rgb2gray(frame_y); %convert each colour frame into gray
frame_m(:,:,:,i)=frame_y; %Store colour frames in the frame_m array
%Perform Gaussian Filtering
h1=(1/8)*(1/8)*[1 3 3 1]'*[1 3 3 1] ; % 4*4 Gaussian Kernel
convn=conv2(frame_z,h1,'same');
g1=uint8(convn);
Filtered_Image_Array(:,:,i)=g1; %Store filtered images into an array
i=i+1;
end
%Apply 3-D Fourier Transform on video sequences
f_transform=fftn(Filtered_Image_Array);
%Compute phase spectrum array from f_transform
phase_spectrum_array =exp(1j*angle(f_transform));
%Apply 3-D Inverse Fourier Transform on phase spectrum array and
%reconstruct the frames
reconstructed_frame_array=(ifftn(phase_spectrum_array));
k=i;
i=1;
for t=1:k-1
%Smooth the reconstructed frame of Î(x, y, n) using the averaging filter.
Reconstructed_frame_magnitude=abs(reconstructed_frame_array(:,:,t));
H = fspecial('disk',4);
circular_avg(:,:,t) = imfilter(Reconstructed_frame_magnitude,H);
%Convert the current frame into binary image using mean value as the threshold
mean_value=mean2(circular_avg(:,:,t));
binary_frame = im2bw(circular_avg(:,:,t),1.6*mean_value);
%Perform Morphological operations
se = strel('square',3);
morphological_closing = imclose(binary_frame,se);
morphological_closing=imclearborder(morphological_closing); %clear noise present at the borders of the frames
%Superimpose segmented masks on it's respective frames to obtain moving
%objects
moving_object_frame = frame_m(:,:,:,i);
moving_object_frame(morphological_closing) = 255;
figure,
imshow(moving_object_frame,[]), title(['Moving objects in Frame :' num2str(i)]);
i=i+1;
end
toc
Once you've identified your object to track implement a Kalman filter and you'll get the desired tracking you're looking for.
There are quite a bit of resources out there, but if you want to do it by hand I suggest you use StudentDave's explanation on YouTube its pretty good. https://www.youtube.com/watch?v=FkCT_LV9Syk
I want to detect edges (with sub-pixel accuracy) in images like the one displayed:
The resolution would be around 600 X 1000.
I came across a comment by Mark Ransom here, which mentions about edge detection algorithms for vertical edges. I haven't come across any yet. Will it be useful in my case (since the edge isn't strictly a straight line)? It will always be a vertical edge though. I want it to be accurate till 1/100th of a pixel at least. I also want to have access to these sub-pixel co-ordinate values.
I have tried "Accurate subpixel edge location" by Agustin Trujillo-Pino. But this does not give me a continuous edge.
Are there any other algorithms available? I will be using MATLAB for this.
I have attached another similar image which the algorithm has to work on:
Any inputs will be appreciated.
Thank you.
Edit:
I was wondering if I could do this:
Apply Canny / Sobel in MATLAB and get the edges of this image (note that it won't be a continuous line). Then, somehow interpolate this Sobel edges and get the co-ordinates in subpixel. Is it possible?
A simple approach would be to project your image vertically and fit the projected profile with an appropriate function.
Here is a try, with an atan shape:
% Load image
Img = double(imread('bQsu5.png'));
% Project
x = 1:size(Img,2);
y = mean(Img,1);
% Fit
f = fit(x', y', 'a+b*atan((x0-x)/w)', 'Startpoint', [150 50 10 150])
% Display
figure
hold on
plot(x, y);
plot(f);
legend('Projected profile', 'atan fit');
And the result:
I get x_0 = 149.6 pix for your first image.
However, I doubt you will be able to achieve a subpixel accuracy of 1/100th of pixel with those images, for several reasons:
As you can see on the profile, your whites are saturated (grey levels at 255). As you cut the real atan profile, the fit is biased. If you have control over the experiments, I suggest you do it again again with a smaller exposure time for instance.
There are not so many points on the transition, so there is not so many information on where the transition is. Typically, your resolution will be the square root of the width of the atan (or whatever shape you prefer). In you case this limits the subpixel resolution at 1/5th of a pixel, at best.
Finally, your edges are not stricly vertical, they are slightly titled. If you choose to use this projection method, to increase the accuracy you should look for a way to correct this tilt before projecting. This won't increase your accuracy by several orders of magnitude, though.
Best,
There is a problem with your image. At pixel level, it seems like there are four interlaced subimages (odd and even rows and columns). Look at this zoomed area close to the edge.
In order to avoid this artifact, I just have taken the even rows and columns of your image, and compute subpixel edges. And finally, I look for the best fitting straight line, using the function clsq whose code is in this page:
%load image
url='http://i.stack.imgur.com/bQsu5.png';
image = imread(url);
imageEvenEven = image(1:2:end,1:2:end);
imshow(imageEvenEven, 'InitialMagnification', 'fit');
% subpixel detection
threshold = 25;
edges = subpixelEdges(imageEvenEven, threshold);
visEdges(edges);
% compute fit line
A = [ones(size(edges.x)) edges.x edges.y];
[c n] = clsq(A,2);
y = [1,200];
x = -(n(2)*y+c) / n(1);
hold on;
plot(x,y,'g');
When executing this code, you can see the green line that best aproximate all the edge points. The line is given by the equation c + n(1)*x + n(2)*y = 0
Take into account that this image has been scaled by 1/2 when taking only even rows and columns, so the right coordinates must be scaled.
Besides, you can try with the other tree subimages (imageEvenOdd, imageOddEven and imageOddOdd) and combine the four straigh lines to obtain the best solution.
How can i calculate the average of a certain area in an image using mat-lab?
For example, if i have an intensity image with an area that is more alight and i want to know what is the average of the intensity there- how do i calculate it?
I think i can find the coordinates of the alight area by using the 'impixelinfo' command.
If there is another more efficient way to find the coordinates i will also be glad to know.
After i know the coordinates how do i calculate the average of part of the image?
You could use one of the imroi type functions in Matlab such as imfreehand
I = imread('cameraman.tif');
h = imshow(I);
e = imfreehand;
% now select area on image - do not close image
% this makes a mask from the area you just drew
BW = createMask(e);
% this takes the mean of pixel values in that area
I_mean = mean(I(BW));
Alternatively, look into using regionprops, especially if there's likely to be more than one of these features in the image. Here, I'm finding points in the image above some threshold intensity and then using imdilate to pick out a small area around each of those points (presuming the points above the threshold are well separated, which may not be the case - if they are too close then imdilate will merge them into one area).
se = strel('disk',5);
BW = imdilate(I>thresh,se);
s = regionprops(BW, I, 'MeanIntensity');
As a preface: this is my first question - I've tried my best to make it as clear as possible, but I apologise if it doesn't meet the required standards.
As part of a summer project, I am taking time-lapse images of an internal melt figure growing inside a crystal of ice. For each of these images I would like to measure the perimeter of, and area enclosed by the figure formed. Linked below is an example of one of my images:
The method that I'm trying to use is the following:
Load image, crop, and convert to grayscale
Process to reduce noise
Find edge/perimeter
Attempt to join edges
Fill perimeter with white
Measure Area and Perimeter using regionprops
This is the code that I am using:
clear; close all;
% load image and convert to grayscale
tyrgb = imread('TyndallTest.jpg');
ty = rgb2gray(tyrgb);
figure; imshow(ty)
% apply a weiner filter to remove noise.
% N is a measure of the window size for detecting coherent features
N=20;
tywf = wiener2(ty,[N,N]);
tywf = tywf(N:end-N,N:end-N);
% rescale the image adaptively to enhance contrast without enhancing noise
tywfb = adapthisteq(tywf);
% apply a canny edge detection
tyedb = edge(tywfb,'canny');
%join edges
diskEnt1 = strel('disk',8); % radius of 4
tyjoin1 = imclose(tyedb,diskEnt1);
figure; imshow(tyjoin1)
It is at this stage that I am struggling. The edges do not quite join, no matter how much I play around with the morphological structuring element. Perhaps there is a better way to complete the edges? Linked is an example of the figure this code outputs:
The reason that I am trying to join the edges is so that I can fill the perimeter with white pixels and then use regionprops to output the area. I have tried using the imfill command, but cannot seem to fill the outline as there are a large number of dark regions to be filled within the perimeter.
Is there a better way to get the area of one of these melt figures that is more appropriate in this case?
As background research: I can make this method work for a simple image consisting of a black circle on a white background using the below code. However I don't know how edit it to handle more complex images with edges that are less well defined.
clear all
close all
clc
%% Read in RGB image from directory
RGB1 = imread('1.jpg') ;
%% Convert RPG image to grayscale image
I1 = rgb2gray(RGB1) ;
%% Transform Image
%CROP
IC1 = imcrop(I1,[74 43 278 285]);
%BINARY IMAGE
BW1 = im2bw(IC1); %Convert to binary image so the boundary can be traced
%FIND PERIMETER
BWP1 = bwperim(BW1);
%Traces perimeters of objects & colours them white (1).
%Sets all other pixels to black (0)
%Doing the same job as an edge detection algorithm?
%FILL PERIMETER WITH WHITE IN ORDER TO MEASURE AREA AND PERIMETER
BWF1 = imfill(BWP1); %This opens figure and allows you to select the areas to fill with white.
%MEASURE PERIMETER
D1 = regionprops(BWF1, 'area', 'perimeter');
%Returns an array containing the properties area and perimeter.
%D1(1) returns the perimeter of the box and an area value identical to that
%perimeter? The box must be bounded by a perimeter.
%D1(2) returns the perimeter and area of the section filled in BWF1
%% Display Area and Perimeter data
D1(2)
I think you might have room to improve the effect of edge detection in addition to the morphological transformations, for instance the following resulted in what appeared to me a relatively satisfactory perimeter.
tyedb = edge(tywfb,'sobel',0.012);
%join edges
diskEnt1 = strel('disk',7); % radius of 4
tyjoin1 = imclose(tyedb,diskEnt1);
In addition I used bwfill interactively to fill in most of the interior. It should be possible to fill the interior programatically but I did not pursue this.
% interactively fill internal regions
[ny nx] = size(tyjoin1);
figure; imshow(tyjoin1)
tyjoin2=tyjoin1;
titl = sprintf('click on a region to fill\nclick outside window to stop...')
while 1
pts=ginput(1)
tyjoin2 = bwfill(tyjoin2,pts(1,1),pts(1,2),8);
imshow(tyjoin2)
title(titl)
if (pts(1,1)<1 | pts(1,1)>nx | pts(1,2)<1 | pts(1,2)>ny), break, end
end
This was the result I obtained
The "fractal" properties of the perimeter may be of importance to you however. Perhaps you want to retain the folds in your shape.
You might want to consider Active Contours. This will give you a continous boundary of the object rather than patchy edges.
Below are links to
A book:
http://www.amazon.co.uk/Active-Contours-Application-Techniques-Statistics/dp/1447115570/ref=sr_1_fkmr2_1?ie=UTF8&qid=1377248739&sr=8-1-fkmr2&keywords=Active+shape+models+Andrew+Blake%2C+Michael+Isard
A demo:
http://users.ecs.soton.ac.uk/msn/book/new_demo/Snakes/
and some Matlab code on the File Exchange:
http://www.mathworks.co.uk/matlabcentral/fileexchange/28149-snake-active-contour
and a link to a description on how to implement it: http://www.cb.uu.se/~cris/blog/index.php/archives/217
Using the implementation on the File Exchange, you can get something like this:
%% Load the image
% You could use the segmented image obtained previously
% and then apply the snake on that (although I use the original image).
% This will probably make the snake work better and the edges
% in your image is not that well defined.
% Make sure the original and the segmented image
% have the same size. They don't at the moment
I = imread('33kew0g.jpg');
% Convert the image to double data type
I = im2double(I);
% Show the image and select some points with the mouse (at least 4)
% figure, imshow(I); [y,x] = getpts;
% I have pre-selected the coordinates already
x = [ 525.8445 473.3837 413.4284 318.9989 212.5783 140.6320 62.6902 32.7125 55.1957 98.6633 164.6141 217.0749 317.5000 428.4172 494.3680 527.3434 561.8177 545.3300];
y = [ 435.9251 510.8691 570.8244 561.8311 570.8244 554.3367 476.3949 390.9586 311.5179 190.1085 113.6655 91.1823 98.6767 106.1711 142.1443 218.5872 296.5291 375.9698];
% Make an array with the selected coordinates
P=[x(:) y(:)];
%% Start Snake Process
% You probably have to fiddle with the parameters
% a bit more that I have
Options=struct;
Options.Verbose=true;
Options.Iterations=1000;
Options.Delta = 0.02;
Options.Alpha = 0.5;
Options.Beta = 0.2;
figure(1);
[O,J]=Snake2D(I,P,Options);
If the end result is an area/diameter estimate, then why not try to find maximal and minimal shapes that fit in the outline and then use the shapes' area to estimate the total area. For instance, compute a minimal circle around the edge set then a maximal circle inside the edges. Then you could use these to estimate diameter and area of the actual shape.
The advantage is that your bounding shapes can be fit in a way that minimizes error (unbounded edges) while optimizing size either up or down for the inner and outer shape, respectively.