Related
I need to extract image patches of size s x s x 3 around specified 2D locations from an image (3 channels).
How can I do this efficiently without a for loop? I know I can extract one patch around (x,y) location as:
apatch = I(y-s/2:y+s/2, x-s/2:x+s/2, :)
How can I do this for many patches? I know I can use MATLAB's function blockproc but I can't specify the locations.
You can use im2col from the image processing toolbox to transform each pixel neighbourhood into a single column. The pixel neighbourhoods are selected such that each block is chose on a column-basis, which means that the blocks are constructed by traversing down the rows first, then proceeding to the next column and getting the neighbourhoods there.
You call im2col this way:
B = im2col(A, [M N]);
I'm assuming you'll want sliding / overlapping neighbourhoods and not distinct neighbourhoods, which are what is normally used when performing any kind of image filtering. A is your image and you want to find M x N pixel neighbourhoods transformed as columns. B would be the output where each neighbourhood is a single column and horizontally-tiled together. However, you'll probably want to handle the case where you want to grab pixel neighbourhoods along the borders of the image. In this case, you'll want to pad the image first. We're going to assume that M and N are odd to allow the padding to be easier. Specifically, you want to be sure that there are floor(M/2) rows padded on top of the image as well as the bottom as well as floor(N/2) columns padded to the left of the image as well as the right. As such, we should pad A first by using padarray. Let's assume that the border pixels will be replicated, which means that the padded rows and columns will simply be those grabbed from the top or bottom row, or the left and right column, depending on where we need to pad. Therefore:
Apad = padarray(A, floor([M N]/2), 'replicate');
For the next part, if you want to choose specify neighbourhoods, you can use sub2ind to convert your 2D co-ordinates into linear indices so you can select the right columns to get the right pixel blocks. However, because you have a colour image, you'll want to perform im2col on each colour channel. Unfortunately, im2col only works on grayscale images, and so you'd have to repeat this for each channel in your image.
As such, to get ready for patch sampling, do something like this:
B = arrayfun(#(x) im2col(Apad(:,:,x), [M N]), 1:size(A,3), 'uni', 0);
B = cat(3, B{:});
The above code will create a 3D version of im2col, where each 3D slice would be what im2col produces for each colour channel. Now, we can use sub2ind to convert your (x,y) co-ordinates into linear indices so that we can choose which pixel neighbourhoods we want. Therefore, assuming your positions are stored in vectors x and y, you would do something like this:
%// Generate linear indices
ind = sub2ind([size(A,1) size(A,2)], y, x);
%// Select neighbourhoods
%// Should be shaped as a MN x len(ind) x 3 matrix
neigh = B(:,ind,:);
%// Create cell arrays for each patch
patches = arrayfun(#(x) reshape(B(:,x,:), [M N 3]), 1:numel(ind), 'uni', 0);
patches will be a cell array where each element contains your desired patch at each location of (x,y) that you specify. Therefore, patches{1} would be the patch located at (x(1), y(1)), patches{2} would be the patch located at (x(2), y(2)), etc. For your copying and pasting pleasure, this is what we have:
%// Define image, M and N here
%//...
%//...
Apad = padarray(A, floor([M N]/2), 'replicate');
B = arrayfun(#(x) im2col(Apad(:,:,x), [M N]), 1:size(A,3), 'uni', 0);
B = cat(3, B{:});
ind = sub2ind([size(A,1) size(A,2)], y, x);
neigh = B(:,ind,:);
patches = arrayfun(#(x) reshape(neigh(:,x,:), [M N 3]), 1:numel(ind), 'uni', 0);
As unexpected as this may seem, but for me the naive for-loop is actually the fastest. This might depend on your version of MATLAB though, as with newer versions they keep on improving the JIT compiler.
Common data:
A = rand(30, 30, 3); % Image
I = [5,2,3,21,24]; % I = y
J = [3,7,5,20,22]; % J = x
s = 3; % Block size
Naive approach: (faster than im2col and arrayfun!)
Patches = cell(size(I));
steps = -(s-1)/2:(s-1)/2;
for k = 1:numel(Patches);
Patches{k} = A(I(k)+steps, ...
J(k)+steps, ...
:);
end
Approach using arrayfun: (slower than the loop)
steps = -(s-1)/2:(s-1)/2;
Patches = arrayfun(#(ii,jj) A(ii+steps,jj+steps,:), I, J, 'UniformOutput', false);
I want to get uniformity value of an image intensity. In the description sheet it described as the following:
General Description
If Z is a random variable indicating image intensity, its nth moment around the mean is
where m is the mean of Z, p(.) its histogram and L is the number of intensity levels.
For Uniformity
Uniformity, defined as
uniformity's maximum value is reached when all intensity levels are equal.
What I do not understand and I want to know is p and L actually. I do not know how to calculate them. Additionally, what is (.) for, in p(.)?
EDIT
% V component of HSV image, contains 0~1 values in double type
% size(Iv) = 960 720
Z = Iv(:);
IL = unique(Z); % Get all intensity levels
noinle = numel(IL); % Number of intensity levels
p = hist(Ivc, noinle); % Histogram
U = 0;
for i = 1:noinle
U = U + sum( p(i,:).^2 );
end
This operation results very big numbers which are not compatible in the sample dataset given. They all are double numbers between 0 and 1.
Assuming your image is stored in an array called Img, you can obtain L and p with the following code:
IL = unique(Img(:)); % Get all intensity levels
L = numel(IL); % Number of intensity levels
p = histogram(Img, IL); % Histogram
Best,
Unless its defined in the paper otherwise generally these are the definitions:
p is the histogram of the image. What this is a break down of the colors in the image put into buckets. In matlab the function for this is h = histogram(x,nbins).
L is the number of buckets. Generally this is something you decide based on your application.
I want to classify pixels of one tiff image according to pixel's RGB colour. The input is an image and three predefined colours for water(r0,g0,b0), forest(r1,g1,b1) and building(r2,g2,c2). The classification is based on the distance between image pixel and these three colors. If a pixel is closet to the water, the pixel is water and changed it the water RGB. The distance is calculated as (one sample) sqrt((x-r0)^2+(y-g0)^2+(z0-b0)^2)
The sample implementation is:
a=imread(..);
[row col dim] = size(a); % dim =3
for i=1:row
for j=1:col
dis1=sqrtCal(a(i,j,:)-water)
dis2=sqrtCal(a(i,j,:)-forest)
dis3=sqrtCal(a(i,j,:)-build)
a(i,j,:) = water;
if dis2< dis1
dis1 = dis2
a(i,j,:) = forest
end
dis3=sqrt(a(i,j,:)-build)
if dis3<dis1
a(i,j,:) = build
end
end
end
This implementation should work. The problem is that the two for loops is not a good choice.
So is there any good alternatives in the Matlab? The
D = pdist(X,distance)
Seems not appliable for my case.
I think this does what you want. The number of reference colors is arbitrary (3 in your example). Each reference color is defined as a row of the matrix ref_colors. Let MxN denote the number of pixels in the original image. Thus the original image is an MxNx3 array.
result_index is an MxN array which for each original pixel contains the index of the closest reference color.
result is a MxNx3 array in which each pixel has been assigned the RBG values of the closest reference color.
im = rand(10,12,3); %// example image
ref_colors = [ .1 .1 .8; ... %// water
.3 .9 .2; ... %// forest
.6 .6 .6 ]; %// build: example reference colors
dist = sum(bsxfun(#minus, im, permute(ref_colors, [3 4 2 1])).^2, 3);
%// 4D array containing the distance from each pixel to each reference color.
%// Dimensions are: 1st: pixel row, 2nd: pixel col, 3rd: not used,
%// 4th: index of reference color
[~, result_index] = min(dist, [], 4); %// compute arg min along 4th dimension
result = reshape(ref_colors(result_index,:), size(im,1), size(im,2), 3);
This one is another bsxfun based solution, but stays in 3D and could be more efficient -
%// Concatenate water, forest, build as a 2D array for easy processing
wfb = cat(2,water(:),forest(:),build(:))
%// Calculate square root of squared distances & indices corresponding to min vals
[~,idx] = min(sum(bsxfun(#minus,reshape(a,[],3),permute(wfb,[3 1 2])).^2,2),[],3)
%// Get the output with the minimum from the set of water, forest & build
a_out = reshape(wfb(:,idx).',size(a))
If you have the statistics toolbox installed, you can use this version based on knnsearch, that scales well for a large number of colors.
result_index = knnsearch(ref_colors, reshape(im,[],3));
result = reshape(ref_colors(result_index,:), size(im));
I wonder whether it would be possible to extract only hands from a video with matlab. In the video hands perform some gesture. Because first frames are only background I tried in this way:
readerObj = VideoReader('VideoWithHands.mp4');
nFrames = readerObj.NumberOfFrames;
fr = get(readerObj, 'FrameRate');
writerObj = VideoWriter('Hands.mp4', 'MPEG-4');
set(writerObj, 'FrameRate', fr);
open(writerObj);
bg = read(readerObj, 1); %background
for k = 1 : nFrames
frame = read(readerObj, k);
hands = imabsdiff(frame,bg);
writeVideo(writerObj,hands);
end
close(writerObj);
But I realized that colors of the hands are not "real" and they are transparent. Is there a better way to extract them from video keeping colors and opacity level exploiting the first frames (background)?
EDIT: Well, I have found a good setting for vision.ForegroundDetector object, now hands are white logical regions but when I try to visualize them with:
videoSource = vision.VideoFileReader('VideoWithHands.mp4', 'VideoOutputDataType', 'uint8');
detector = vision.ForegroundDetector('NumTrainingFrames', 46, 'InitialVariance', 4000, 'MinimumBackgroundRatio', 0.2);
videoplayer = vision.VideoPlayer();
hands = uint8(zeros(720,1280,3));
while ~isDone(videoSource)
frame = step(videoSource);
fgMask = step(detector, frame);
[m,n] = find(fgMask);
a = [m n];
if isempty(a)==true
hands(:,:,:) = uint8(zeros(720,1280,3));
else
hands(m,n,1) = frame(m,n,1);
hands(m,n,2) = frame(m,n,2);
hands(m,n,3) = frame(m,n,3);
end
step(videoplayer, hands)
end
release(videoplayer)
release(videoSource)
or put them into a videofile with:
eaderObj = VideoReader('Video 9.mp4');
nFrames = readerObj.NumberOfFrames;
fr = get(readerObj, 'FrameRate');
writerObj = VideoWriter('hands.mp4', 'MPEG-4');
set(writerObj, 'FrameRate', fr);
detector = vision.ForegroundDetector('NumTrainingFrames', 46, 'InitialVariance', 4000, 'MinimumBackgroundRatio', 0.2);
open(writerObj);
bg = read(readerObj, 1);
frame = uint8(zeros(size(bg)));
for k = 1 : nFrames
frame = read(readerObj, k);
fgMask = step(detector, frame);
[m,n] = find(fgMask);
hands = uint8(zeros(720,1280));
if isempty([m n]) == true
hands(:,:) = uint8(zeros(720,1280));
else
hands(m,n) = frame(m,n);
end
writeVideo(writerObj,mani);
end
close(writerObj);
...my PC crashes. Some suggestion?
So you're trying to cancel out the background, making it black, right?
The easiest way to do this should be to filter it, you can do that by comparing your difference data to a threshold value and then using the result as indices to set a custom background.
filtered = imabsdiff(frame,bg);
bgindex = find( filtered < 10 );
frame(bgindex) = custombackground(bgindex);
where custombackground is whatever image file you want to put into the background. If you want it to be just black or white, use 0 or 255 instead of custombackground(bgindex). Note that the numbers depend on your video data's format and could be inaccurate (except 0, this one should always be right). If too much gets filtered out, lower the 10 above, if too much remains unfiltered, increase the 10.
At the end, you write your altered frame back into the video, so it just replaces the hands variable in your code.
Also, depending on your format, you might have to do the comparison across RGB values. This is slightly more complicated as it involves checking 3 values at the same time and doing some magic with the indices. This is the RGB version (works with anything containing 3 color bands):
filtered = imabsdiff(frame,bg); % differences at each pixel in each color band
totalfiltered = sum(filtered,3); % sums up the differences
% in each color band (RGB)
bgindex = find( totalfiltered < 10 ); % extracts indices of pixels
% with color close to bg
allind = sub2ind( [numel(totalfiltered),3] , repmat(bgindex,1,3) , ...
repmat(1:3,numel(bgindex),1) ); % index magic
frame(allind) = custombackground(allind); % copy custom background into frame
EDIT :
Here's a detailed explanation of the index magic.
Let's assume a 50x50 image. Say the pixel at row 2, column 5 is found to be background, then bgindex will contain the number 202 (linear index corresponding to [2,5] = (5-1)*50+2 ). What we need is a set of 3 indices corresponding to the matrix coordinates [2,5,1], [2,5,2] and [2,5,3]. That way, we can change all 3 color bands corresponding to that pixel. To make calculations easier, this approach actually assumes linear indexing for the image and thus converts it to a 2500x1 image. Then it expands the 3 color bands, creating a 2500x3 matrix. We now construct the indices [202,1], [202,2] and [202,3] instead.
To do that, we first construct a matrix of indices by repeating our values. repmat does this for us, it creates the matrices [202 202 202] and [1 2 3]. If there were more pixels in bgindex, the first matrix would contain more rows, each repeating the linear pixel coordinates 3 times. The second matrix would contain additional [1 2 3] rows. The first argument to sub2ind is the size of the matrix, in this case, 2500x3, so we calculate the number of pixels with numel applied to the sum vector (which collapses the image's 3 bands into 1 value and thus has 1 value per pixel) and add a static 3 in the second dimension.
sub2ind now takes each element from the first matrix as a row index, each corresponding element from the second matrix as a column index and converts them to linear indices into a matrix of the size we determined earlier. In our example, this results in the indices [202 2702 5202]. sub2ind preserves the shape of the inputs, so if we had 10 background pixels, this result would have the size 10x3. But since linear indexing doesn't care about the shape of the index matrix, it just takes all of those values.
To confirm this is correct, let's revert the values in the example. The original image data would have the size 50x50x3. For an NxMxP matrix, a linear index to the subscript [n m p] can be calculated as ind = (p-1)*M*N + (m-1)*N + n. Using our values, we get the following:
[2 5 1] => 202
[2 5 2] => 2702
[2 5 3] => 5202
ind2sub confirms this.
Yes, there is a better way. The computer vision system toolbox includes a vision.ForegroundDetector object that does what you need. It implements the Gaussian Mixture Model algorithm for background subtraction.
I have the following error in my code
Assignment has more non-singleton rhs
dimensions than non-singleton
This is the code:
string =['d:\face\face\ffw' int2str(r) '_' int2str(Sel(nSelRow,t)) '.bmp'];
A = imread(string);
B = im2double(A);
Train_Dat(:,:,s)=B;
Update 1:
When I updated the code we got new error in the next row
my code
for r=1:Class_Num
for t=1:Class_Train_Num
%string=['e:\face_lib\feret_80\ff' int2str(r) '_' int2str(t) '.tif'];
string =['d:\face\face\ffw' int2str(r) '_' int2str(Sel(nSelRow,t)) '.bmp'];
A=imread(string);
B=im2double(A);
Train_Dat(:,:,:,s)=B;
Train_Dat_Vector(:,s)=B(:); %here new error Subscripted assignment dimension mismatch.
s=s+1;
end
Update 2:
my define for
nImgW = 40;
nImgH = 40;
nImgSize = nImgW*nImgH;
Train_Dat_Vector = zeros( nImgSize, Train_Num );
A=imread(string);
B=im2double(A);
Train_Dat(:,:,:,s)=B;
Train_Dat_Vector(:,s)=B(:);%here i want convert matrix to 40x40,Train_Num
s=s+1;
I think the problem is that B is most likely a 3-D RGB image, and you're trying to assign it to a single 2-D plane of your 3-D matrix Train_Dat. If you're trying to collect a set of 3-D images to use as training data, you will have to make Train_Dat either a 4-D matrix (if all of your images have the exact same dimensions) or a cell array (with one image per cell):
Example #1: a 4-D matrix...
nRows = 100; %# The number of rows in the images
nCols = 100; %# The number of columns in the images
nDepth = 3; %# The depth of the images (3 color planes for RGB images)
nImages = 5; %# The number of images you will use
Train_Dat = zeros(nRows,nCols,nDepth,nImages); %# Initialize to zeros
Train_Dat(:,:,:,1) = B; %# Assign B as the first image
If you want to use this option, but all of your images are not the same size, you will have to resize them all to a given size. One way to do this, if you have the Image Processing Toolbox, is to use the function IMRESIZE:
newSize = [40 40]; %# The new size the image will be
C = imresize(B,newSize); %# Resize image B
If you don't have access to the Image Processing Toolbox, one alternative is to use the function INTERP2 to resize your image. Here's one example of resizing a 3-D RGB image of type UINT8:
B = double(B); %# Convert B to double (needed to use INTERP2)
[nRows,nCols,nDepth] = size(B); %# Get the old image size
C = zeros(40,40,3,'uint8'); %# Initialize the new 3-D 40-by-40 uint8 image
xi = linspace(1,nCols,40); %# Create a down-sampled set of x points
yi = linspace(1,nRows,40); %# Create a down-sampled set of y points
[X,Y] = meshgrid(xi,yi); %# Create 40-by-40 grids of x and y points
C(:,:,1) = interp2(B(:,:,1),X,Y,'spline'); %# Interpolate the red color plane
C(:,:,2) = interp2(B(:,:,2),X,Y,'spline'); %# Interpolate the green color plane
C(:,:,3) = interp2(B(:,:,3),X,Y,'spline'); %# Interpolate the blue color plane
The image C will now be a down-sampled 40-by-40 version of B.
Example #2: a cell array...
nImages = 5; %# The number of images you will use
Train_Dat = cell(1,nImages); %# Initialize the cell array
Train_Dat{1} = B; %# Assign B as the first image
In this case, the images you add to each cell can be different sizes and types, so no resizing is needed.