I wonder whether it would be possible to extract only hands from a video with matlab. In the video hands perform some gesture. Because first frames are only background I tried in this way:
readerObj = VideoReader('VideoWithHands.mp4');
nFrames = readerObj.NumberOfFrames;
fr = get(readerObj, 'FrameRate');
writerObj = VideoWriter('Hands.mp4', 'MPEG-4');
set(writerObj, 'FrameRate', fr);
open(writerObj);
bg = read(readerObj, 1); %background
for k = 1 : nFrames
frame = read(readerObj, k);
hands = imabsdiff(frame,bg);
writeVideo(writerObj,hands);
end
close(writerObj);
But I realized that colors of the hands are not "real" and they are transparent. Is there a better way to extract them from video keeping colors and opacity level exploiting the first frames (background)?
EDIT: Well, I have found a good setting for vision.ForegroundDetector object, now hands are white logical regions but when I try to visualize them with:
videoSource = vision.VideoFileReader('VideoWithHands.mp4', 'VideoOutputDataType', 'uint8');
detector = vision.ForegroundDetector('NumTrainingFrames', 46, 'InitialVariance', 4000, 'MinimumBackgroundRatio', 0.2);
videoplayer = vision.VideoPlayer();
hands = uint8(zeros(720,1280,3));
while ~isDone(videoSource)
frame = step(videoSource);
fgMask = step(detector, frame);
[m,n] = find(fgMask);
a = [m n];
if isempty(a)==true
hands(:,:,:) = uint8(zeros(720,1280,3));
else
hands(m,n,1) = frame(m,n,1);
hands(m,n,2) = frame(m,n,2);
hands(m,n,3) = frame(m,n,3);
end
step(videoplayer, hands)
end
release(videoplayer)
release(videoSource)
or put them into a videofile with:
eaderObj = VideoReader('Video 9.mp4');
nFrames = readerObj.NumberOfFrames;
fr = get(readerObj, 'FrameRate');
writerObj = VideoWriter('hands.mp4', 'MPEG-4');
set(writerObj, 'FrameRate', fr);
detector = vision.ForegroundDetector('NumTrainingFrames', 46, 'InitialVariance', 4000, 'MinimumBackgroundRatio', 0.2);
open(writerObj);
bg = read(readerObj, 1);
frame = uint8(zeros(size(bg)));
for k = 1 : nFrames
frame = read(readerObj, k);
fgMask = step(detector, frame);
[m,n] = find(fgMask);
hands = uint8(zeros(720,1280));
if isempty([m n]) == true
hands(:,:) = uint8(zeros(720,1280));
else
hands(m,n) = frame(m,n);
end
writeVideo(writerObj,mani);
end
close(writerObj);
...my PC crashes. Some suggestion?
So you're trying to cancel out the background, making it black, right?
The easiest way to do this should be to filter it, you can do that by comparing your difference data to a threshold value and then using the result as indices to set a custom background.
filtered = imabsdiff(frame,bg);
bgindex = find( filtered < 10 );
frame(bgindex) = custombackground(bgindex);
where custombackground is whatever image file you want to put into the background. If you want it to be just black or white, use 0 or 255 instead of custombackground(bgindex). Note that the numbers depend on your video data's format and could be inaccurate (except 0, this one should always be right). If too much gets filtered out, lower the 10 above, if too much remains unfiltered, increase the 10.
At the end, you write your altered frame back into the video, so it just replaces the hands variable in your code.
Also, depending on your format, you might have to do the comparison across RGB values. This is slightly more complicated as it involves checking 3 values at the same time and doing some magic with the indices. This is the RGB version (works with anything containing 3 color bands):
filtered = imabsdiff(frame,bg); % differences at each pixel in each color band
totalfiltered = sum(filtered,3); % sums up the differences
% in each color band (RGB)
bgindex = find( totalfiltered < 10 ); % extracts indices of pixels
% with color close to bg
allind = sub2ind( [numel(totalfiltered),3] , repmat(bgindex,1,3) , ...
repmat(1:3,numel(bgindex),1) ); % index magic
frame(allind) = custombackground(allind); % copy custom background into frame
EDIT :
Here's a detailed explanation of the index magic.
Let's assume a 50x50 image. Say the pixel at row 2, column 5 is found to be background, then bgindex will contain the number 202 (linear index corresponding to [2,5] = (5-1)*50+2 ). What we need is a set of 3 indices corresponding to the matrix coordinates [2,5,1], [2,5,2] and [2,5,3]. That way, we can change all 3 color bands corresponding to that pixel. To make calculations easier, this approach actually assumes linear indexing for the image and thus converts it to a 2500x1 image. Then it expands the 3 color bands, creating a 2500x3 matrix. We now construct the indices [202,1], [202,2] and [202,3] instead.
To do that, we first construct a matrix of indices by repeating our values. repmat does this for us, it creates the matrices [202 202 202] and [1 2 3]. If there were more pixels in bgindex, the first matrix would contain more rows, each repeating the linear pixel coordinates 3 times. The second matrix would contain additional [1 2 3] rows. The first argument to sub2ind is the size of the matrix, in this case, 2500x3, so we calculate the number of pixels with numel applied to the sum vector (which collapses the image's 3 bands into 1 value and thus has 1 value per pixel) and add a static 3 in the second dimension.
sub2ind now takes each element from the first matrix as a row index, each corresponding element from the second matrix as a column index and converts them to linear indices into a matrix of the size we determined earlier. In our example, this results in the indices [202 2702 5202]. sub2ind preserves the shape of the inputs, so if we had 10 background pixels, this result would have the size 10x3. But since linear indexing doesn't care about the shape of the index matrix, it just takes all of those values.
To confirm this is correct, let's revert the values in the example. The original image data would have the size 50x50x3. For an NxMxP matrix, a linear index to the subscript [n m p] can be calculated as ind = (p-1)*M*N + (m-1)*N + n. Using our values, we get the following:
[2 5 1] => 202
[2 5 2] => 2702
[2 5 3] => 5202
ind2sub confirms this.
Yes, there is a better way. The computer vision system toolbox includes a vision.ForegroundDetector object that does what you need. It implements the Gaussian Mixture Model algorithm for background subtraction.
Related
I am implementing a simple algorithm to do in-painting on a "damaged" image. I have a predefined mask that specifies the area which needs to be fixed. My strategy is to start at the border of the masked area and in-paint each pixel with the central mean of its neighboring non-zero pixels, repeating until there's no unknown pixels left.
function R = inPainting(I, mask)
H = [1 2 1; 2 0 2; 1 2 1];
R = I;
n = 1;
[row,col,~] = find(~mask); %Find zeros in mask (area to be inpainted)
unknown = horzcat(row, col)';
while size(unknown,2) > 0
new_unknown = [];
new_R = R;
for u = unknown
r = u(1);
c = u(2);
nb = R(max((r-n), 1):min((r+n), end), max((c-n),1):min((c+n),end));
nz = nb~=0;
nzs = sum(nz(:));
if nzs ~= 0 %We have non-zero neighbouring pixels. In-paint with average.
new_R(r,c) = sum(nb(:)) / nzs;
else
new_unknown = horzcat(new_unknown, u);
end
end
unknown = new_unknown;
R = new_R;
end
This works well, but it's not very efficient. Is it possible to vectorize such an approach, using mostly matrix operations? Does someone know of a more efficient way to implement this algorithm?
If I understand your problem statement, you are given a mask and you wish to fill in these pixels in this mask with the mean of the neighbourhood pixels that surround each pixel in the mask. Another constraint is that the image is defined such that any pixels that belong to the mask in the same spatial locations are zero in this mask. You are starting from the border of the mask and are propagating information towards the innards of the mask. Given this algorithm, there is unfortunately no way you can do this with standard filtering techniques as the current time step is dependent on the previous time step.
Image filtering mechanisms, like imfilter or conv2 can't work here because of this dependency.
As such, what I can do is help you speed up what is going on inside your loop and hopefully this will give you some speed up overall. I'm going to introduce you to a function called im2col. This is from the image processing toolbox, and given that you can use imfilter, we can use this function.
im2col creates a 2D matrix such that each column is a pixel neighbourhood unrolled into a single vector. How it works is that each pixel neighbourhood in column major order is grabbed, so we get a pixel neighbourhood at the top left corner of the image, then move down one row, and another row and we keep going until we reach the last row. We then move one column over and repeat the same process. For each pixel neighbourhood that we have, it gets unrolled into a single vector, and the output would be a MN x K matrix where you have a neighbourhood size of M x N for each pixel neighbourhood and there are K neighbourhoods.
Therefore, at each iteration of your loop, we can unroll the current inpainted image's pixel neighbourhoods into single vectors, determine which pixel neighborhoods are non-zero and from there, determine how many zero values there are for each of these selected pixel neighbourhood. After, we compute the mean for these non-zero columns disregarding the zero elements. Once we're done, we update the image and move to the next iteration.
What we're going to need to do first is pad the image with a 1 pixel border so that we're able to grab neighbourhoods that extend beyond the borders of the image. You can use padarray, also from the image processing toolbox.
Therefore, we can simply do this:
function R = inPainting(I, mask)
R = double(I); %// For precision
n = 1;
%// Change - column major indices
unknown = find(~mask); %Find zeros in mask (area to be inpainted)
%// Until we have searched all unknown pixels
while numel(unknown) ~= 0
new_R = R;
%// Change - take image at current iteration and
%// create columns of pixel neighbourhoods
padR = padarray(new_R, [n n], 'replicate');
cols = im2col(padR, [2*n+1 2*n+1], 'sliding');
%// Change - Access the right pixel neighbourhoods
%// denoted by unknown
nb = cols(:,unknown);
%// Get total sum of each neighbourhood
nbSum = sum(nb, 1);
%// Get total number of non-zero elements per pixel neighbourhood
nzs = sum(nb ~= 0, 1);
%// Replace the right pixels in the image with the mean
new_R(unknown(nzs ~= 0)) = nbSum(nzs ~= 0) ./ nzs(nzs ~= 0);
%// Find new unknown pixels to look at
unknown = unknown(nzs == 0);
%// Update image for next iteration
R = new_R;
end
%// Cast back to the right type
R = cast(R, class(I));
I want to classify pixels of one tiff image according to pixel's RGB colour. The input is an image and three predefined colours for water(r0,g0,b0), forest(r1,g1,b1) and building(r2,g2,c2). The classification is based on the distance between image pixel and these three colors. If a pixel is closet to the water, the pixel is water and changed it the water RGB. The distance is calculated as (one sample) sqrt((x-r0)^2+(y-g0)^2+(z0-b0)^2)
The sample implementation is:
a=imread(..);
[row col dim] = size(a); % dim =3
for i=1:row
for j=1:col
dis1=sqrtCal(a(i,j,:)-water)
dis2=sqrtCal(a(i,j,:)-forest)
dis3=sqrtCal(a(i,j,:)-build)
a(i,j,:) = water;
if dis2< dis1
dis1 = dis2
a(i,j,:) = forest
end
dis3=sqrt(a(i,j,:)-build)
if dis3<dis1
a(i,j,:) = build
end
end
end
This implementation should work. The problem is that the two for loops is not a good choice.
So is there any good alternatives in the Matlab? The
D = pdist(X,distance)
Seems not appliable for my case.
I think this does what you want. The number of reference colors is arbitrary (3 in your example). Each reference color is defined as a row of the matrix ref_colors. Let MxN denote the number of pixels in the original image. Thus the original image is an MxNx3 array.
result_index is an MxN array which for each original pixel contains the index of the closest reference color.
result is a MxNx3 array in which each pixel has been assigned the RBG values of the closest reference color.
im = rand(10,12,3); %// example image
ref_colors = [ .1 .1 .8; ... %// water
.3 .9 .2; ... %// forest
.6 .6 .6 ]; %// build: example reference colors
dist = sum(bsxfun(#minus, im, permute(ref_colors, [3 4 2 1])).^2, 3);
%// 4D array containing the distance from each pixel to each reference color.
%// Dimensions are: 1st: pixel row, 2nd: pixel col, 3rd: not used,
%// 4th: index of reference color
[~, result_index] = min(dist, [], 4); %// compute arg min along 4th dimension
result = reshape(ref_colors(result_index,:), size(im,1), size(im,2), 3);
This one is another bsxfun based solution, but stays in 3D and could be more efficient -
%// Concatenate water, forest, build as a 2D array for easy processing
wfb = cat(2,water(:),forest(:),build(:))
%// Calculate square root of squared distances & indices corresponding to min vals
[~,idx] = min(sum(bsxfun(#minus,reshape(a,[],3),permute(wfb,[3 1 2])).^2,2),[],3)
%// Get the output with the minimum from the set of water, forest & build
a_out = reshape(wfb(:,idx).',size(a))
If you have the statistics toolbox installed, you can use this version based on knnsearch, that scales well for a large number of colors.
result_index = knnsearch(ref_colors, reshape(im,[],3));
result = reshape(ref_colors(result_index,:), size(im));
I have been using short (20s) videos as stimulus material in a recent study. I would now like to compare the videos regarding the amount of movements that each video contains.
This is my code until now:
VidObj = VideoReader(video_file);
VidFrames = read(VidObj);
returns 4D Matrix = 2d array of 2d images = a x b x c x VidNoFrames
VidNoFrames = VidObj.NumberofFrames;
VidHeight = VidObj.Height;
VidWidth = VidObj.Width;
for k = 1 : VidNoFrames
k mov(k).cdata = VidFrames(:,:,:,k); %720 x 1280 x 3
end
Now I guess the next step would constructing the loop for correlation/SSD. However, cdata is still 3D (height x width x 3). I do not understand what this third dimension is and how to go on comparing the images... Thanks so much for your help!
Thanks a lot!
Christine
You might want to use correlation to compare 2 images, or in your case I would opt for computing the Sum Square Intensity Difference, which tells you in a more quantitative way maybe how much pixel intensity differs from one image to another. Here is a simple example for both cases;
clear
clc
%// Correlation coefficient. Close to 1 == more similarity
A = imread('coins.png');
B = medfilt2(A);
CorrCoeff = corr2(A,B)
%// Sum Square Intensity Difference
SquareIntDiff = (B-A).^2; %//Compute the square of the pixel intensity difference.
SSID = sum(SquareIntDiff(:)) %// Sum it to get the SSID. A value of 0 means that both images are similar in terms of pixel intensity.
%// Using sum(sum(SquareIntDiff)) would yield the same result in a less efficient manner. MATLAB takes the sum along the columns and then another time to get a single value.
CorrCoeff =
0.9964
SSID =
550746
Of course you can easily implement this in a loop to compare consecutive frames in your video. Hope that helps you a bit :)
I need to create an nth-order Hadamard matrix, row double it, within each row randomly permute the elements of the matrix, and then display it. So far, I have accomplished all of these things. What I end up with when I imshow(matrix) is a nice picture of black and white boxes. But I haven't figured out how to insert a fine line to divide each row. I can create something like the first image on the left, but not the image on the right (these are Figures 1 and 2 from this paper)
Any help or comments would be thoroughly appreciated.
I've found using vector approaches (e.g., patch and rectangle) for this sort of problem unnecessarily challenging. I think that it's more straightforward to build a new image. This avoids floating-point rounding issues and other things that crop up with vector graphics. My solution below relies on some functions in the Image Processing Toolbox, but is simple and fast:
% Create data similarly to #TryHard
H = hadamard(48);
C = (1+[H;-H])/2;
rng(0); % Set seed
C(:) = C(randperm(numel(C))); % For demo, just permute all values, not rows
% Scale image and lines
scl = 10; % Amount to vertically scale each row
pad = 2; % Number of pixels to add between each row
C = imresize(C,scl,'nearest');
C = blockproc(C,[scl size(C,2)],#(x)[x.data;zeros(pad,size(C,2))]);
C = C(1:end-pad,:); % Remove last line added
% Dispay image
imshow(C)
This results in an image like this
The scl and pad parameters can be easily adjusted to obtain different sizes and relative sizes. You can call imresize(...,'nearest') again after adding the lines to further scale the image if desired. The blocproc line could potentially be made more efficient with various options (see the help). It could also be replaced by calls to im2col and col2im, which possibly could be faster, if messier.
I did not try the code, but I think that something like that should work:
sizeOfACube = 6;
numberOfRows = 47;
RGB = imread('image.png');
RGB = imresize(A, [(numRows+numberOfRows) numCols]);
for i=1:1:NumberOfRows
RGB(i*6,:,:) = 0;
end
imagesc(RGB);
imwrite(RGB,'newImage.png');
with:
sizeOfAcube the size of one cube on the QRcode.
numRows and numCols the number of Rows and Column of the original image.
One solution is to use patches, for instance as follows:
% set up example array
xl = 24; yl = xl;
[X Y] = find(hadamard(xl)==1);
% generate figure
figure, hold on
for ii=1:length(X)
patch(X(ii) + [0 0 1 1],Y(ii) + [0.1 0.9 0.9 0.1],[1 1 1],'Edgecolor',[1 1 1])
end
axis([0 xl+1 0 yl+1])
axis('square')
The patch command patch(x,y, color) accepts the vertices of the polygon element as x and y. In this example you can modify the term [0.1 0.9 0.9 0.1] to set the thickness of the bounding black line.
This generates
Edited
For the particular instance provided by the OP:
H=Hadamard(48); %# now to row-double the matrix
A=(1+H)/2;
B=(1-H)/2;
C=[A; B]; %# the code below randomly permutes elements within the rows of the matrix
[nRows,nCols] = size(C);
[junk,idx] = sort(rand(nRows,nCols),2); %# convert column indices into linear indices
idx = (idx-1)*nRows + ndgrid(1:nRows,1:nCols); %# rearrange whatever matrix
E = C;
E(:) = E(idx);
[X Y] = find(logical(E));
xl = length(X);
yl = length(Y);
figure, hold on
for ii=1:xl
rectangle('Position',[X(ii) Y(ii)+.2 1 0.8],'facecolor',[1 1 1],'edgecolor',[1 1 1])
end
axis([0 max(X)+1 0 max(Y)+1])
axis('square')
set(gca,'color',[0 0 0])
set(gca,'XTickLabel',[],'YTickLabel',[],'XTick',[],'YTick',[])
This example uses rectangle instead of patch to generate sharp corners.
The image:
I have a 5000x5000 grid, and I'm trying to implement a simple model of cancer division in MATLAB. Initially, it picks a random point (x,y) and makes that cell a cancer cell. On the first iteration, it divides - the parent cell stays in it's place, the daughter cell is randomly assigned to any neighbouring cell.
Easy so far.
My problem is this: on successive iterations, a daughter cell will often be assigned to a cell that already has a cancer cell. In this case, I want the daughter cell to take its place and "bump" the cell already there to an adjacent cell. If that adjacent cell is empty, it is filled and the process stops. If not, the cell already in that place is bumped and so on until the last cell finds an empty space and the process stops.
This should be simple, but I have no idea how to code it up and what kind of loops to use.
I'm a physical scientists rather than a programmer, so please treat me like a simpleton!
Here is a function I hacked together that roughly meets the specs you provided.
I does slow down as the number of cancerous cells gets large.
Basically I have a few variables, the NxN matrix that represents the grid of cell locations (i call this a plate as grid is the name of an existing matlab function)
A vector of points that I can iterate through quickly. I pick a seed location and then run a while loop until the grid is full.
On each loop iteration I perform the following for each cell:
Generate a random number to determine if that cell should divide
Generate a random direction to divide
Find the first open plate position in that direction
Populate that position
I haven't tested it extensively but it appears to work.
function simulateCancer(plateSize, pDivide)
plate = zeros(plateSize, plateSize);
nCells = 1;
cellLocations = zeros(plateSize*plateSize,2);
initX = randi(plateSize);
initY = randi(plateSize);
cellLocations(nCells,:) = [initX, initY];
plate(initX, initY) = 1;
f = figure;
a = axes('Parent', f);
im = imagesc(plate, 'Parent', a);
while(nCells < (plateSize * plateSize))
currentGeneration = currentGeneration+1;
for i = 1:nCells
divide = rand();
if divide <= pDivide
divideLocation = cellLocations(i,:);
divideDir = randi(4);
[x, y, v] = findNewLocation(divideLocation(1), divideLocation(2), plate, divideDir);
if (v==1)
nCells = nCells+1;
plate(x,y) = 1;
cellLocations(nCells,:) = [x,y];
end
end
end
set(im,'CData', plate);
pause(.1);
end
end
function [x,y, valid] = findNewLocation(xin, yin, plate, direction)
x = xin;
y = yin;
valid = 1;
% keep looking for new spot if current spot is occupied
while( plate(x, y) == 1)
switch direction
case 1 % divide up
y = y-1;
case 2 % divide down
y = y+1;
case 3 % divide left
x = x-1;
case 4 % divide down
x = x+1;
otherwise
warning('Invalid direction')
x = xin;
y = yin;
return;
end
%if there has been a collision with a wall then just quit
if y==0 || y==size(plate,2)+1 || x==0 || x==size(plate,1)+1 % hit the top
x = xin; %return original values to say no division happend
y = yin;
valid = 0;
return;
end
end
end
Note: Instead of thinking of pushing cells, I coded this in a way that leaves cells where they currently are and creates the new cell at the end of the row/column. Semantically its different but logically it has the same end result, as long as you don't care about the generations.
Inspired by an another question, I though of using image processing techniques to implement this simulation. Specifically we can use morphological dilation to spread the cancerous cells.
The idea is to dilate each pixel using a structuring element that looks like:
1 0 0
0 1 0
0 0 0
where the center is fixed, and the other 1 is placed at random at one of the other eight remaining positions. This would effectively extend the pixel in that direction.
The way the dilation is performed is by created a blank image, with only one pixel set, then accumulating all the results using a simple OR operation.
To speed things up, we don't need to consider every pixel, only those on the perimeter of the current blocks formed by the clusters of cancerous cells. The pixels on the inside are already surrounded by cancer cells, and would have no effect if dilated.
To speed even further, we perform the dilation on all pixels that are chosen to be extended in the same direction in one call. Thus every iteration, we perform at most 8 dilation operations.
This made the code relatively fast (I tested up to 1000x1000 grid). Also it maintains the same timing across all iterations (will not slow down as the grid starts to fill up).
Here is my implementation:
%# initial grid
img = false(500,500);
%# pick 10 random cells, and set them as cancerous
img(randi(numel(img),[10 1])) = true;
%# show initial image
hImg = imshow(img, 'Border','tight', 'InitialMag',100);
%# build all possible structing elements
%# each one dilates in one of the 8 possible directions
SE = repmat([0 0 0; 0 1 0; 0 0 0],[1 1 8]);
SE([1:4 6:9] + 9*(0:7)) = 1;
%# run simulation until all cells have cancer
BW = false(size(img));
while ~all(img(:)) && ishandle(hImg)
%# find pixels on the perimeter of all "blocks"
on = find(bwperim(img,8));
%# percentage chance of division
on = on( rand(size(on)) > 0.5 ); %# 50% probability of cell division
if isempty(on), continue; end
%# decide on a direction for each pixel
d = randi(size(SE,3),[numel(on) 1]);
%# group pixels according to direction chosen
dd = accumarray(d, on, [8 1], #(x){x});
%# dilate each group of pixels in the chosen directions
%# to speed up, we perform one dilation for all pixels with same direction
for i=1:8
%# start with an image with only those pixels set
BW(:) = false;
BW(dd{i}) = true;
%# dilate in the specified direction
BW = imdilate(BW, SE(:,:,i));
%# add results to final image
img = img | BW;
end
%# show new image
set(hImg, 'CData',img)
drawnow
end
I also created an animation of the simulation on a 500x500 grid, with 10 random initial cancer cells (warning: the .gif image is approximately 1MB in size, so may take some time to load depending on your connection)