Please, I'm trying to detect the fingers' movement in a video. Firstly, I'd like to apply a skin color detection to separate the hand from the background and then I'll find the hand counter followed by calculating the convex points to detect the fingers. What I want now from this video is a new video showing only the movement of the two tapping fingers (or their contour), as shown in this figure.
I used this code to detect the skin color:
function OutImg=Skin_Detect(I)
% clear
% I=imread('D:\New Project\Movie Frames from RLC_L_FTM_IP60\Frame
0002.png');
I=double(I);
[hue,s,v]=rgb2hsv(I);
cb = 0.148* I(:,:,1) - 0.291* I(:,:,2) + 0.439 * I(:,:,3) + 128;
cr = 0.439 * I(:,:,1) - 0.368 * I(:,:,2) -0.071 * I(:,:,3) + 128;
[w h]=size(I(:,:,1));
for i=1:w
for j=1:h
if 138<=cr(i,j) && cr(i,j)<=169 && 136<=cb(i,j) && cb(i,j)<=200 &&
0.01<=hue(i,j) && hue(i,j)<=0.2
segment(i,j)=1;
else
segment(i,j)=0;
end
end
end
% imshow(segment);
OutImg(:,:,1)=I(:,:,1).*segment;
OutImg(:,:,2)=I(:,:,2).*segment;
OutImg(:,:,3)=I(:,:,3).*segment;
% figure,imshow(uint8(im));
The same code is working perfectly when I apply it to an image, but I detect nothing when I apply it to a video, as follows:
videoFReader = vision.VideoFileReader('RLC_L_FT_IP60.m4v');
% Create a video player object for displaying video frames.
videoPlayer = vision.DeployableVideoPlayer;
% Display the original video
while ~isDone(videoFReader)
videoFrame = step(videoFReader);
% Track using the Hue channel data
Out=Skin_Detect(videoFrame);
step(videoPlayer,Out);
end
Please, any suggestion and idea to solve this problem?
I'll be so grateful if anyone can help with this even with different codes.
Thank you in advance.
I had a similar problem. I think a proper way is using a classifier, even it is a "simple" classifier... Here the steps I have followed in my solution:
1) I have used the RGB color space and a Mahalanobis distance for the skin color model. It is fast and work pretty well.
2) Connected components: A simple morphological close operation with a small structuring element can be used to join areas that might be disconnected during imperfect thresholding, like the fingers of the hand.
3) Feature extraction: area, perimeter, and ratio of area over perimeter, for example.
4) Classification: use an SVM classifier to perform the final classification. I hope you have labeled training data for the process.
I am not solving exactly yor concrete problem, but maybe it could give you some idea... :)
If you don't insist on writing on your own, you can use Google's MediaPipe for hand- and finger-tracking.
Info:
https://ai.googleblog.com/2019/08/on-device-real-time-hand-tracking-with.html
Examples for Desktop and Android:
https://github.com/google/mediapipe/blob/master/mediapipe/docs/hand_tracking_mobile_gpu.md
Related
I have been researching methods to detect people only in the video from a security camera. I want to use vision.peopledetector with vision.BlobAnalysis and vision.ForegroundDetector. But it doesn't work.
It should be like Motion-Based Multiple Object Tracking example, but only for detecting humans. Cant seem to get it work.
What I have done so far without using vision.BlobAnalysis and `vision.ForegroundDetector. It is not accurate at all and cant count
video = VideoReader('atrium.mp4');
peopleDetector = vision.PeopleDetector;
videoPlayer = vision.VideoPlayer;
while hasFrame(video)
img = readFrame(video);
[bboxes,scores] = step(peopleDetector,img);
D = 1;
frame = insertObjectAnnotation(img,'rectangle',bboxes,D);
step(videoPlayer,frame);
end
OK. So here's what I think is happening: the resolution of the atrium.mp4 video is not high enough to make reliable detections using vision.PeopleDetector. Here's what I did to modify your code:
video = VideoReader('atrium.mp4');
peopleDetector = vision.PeopleDetector;
videoPlayer = vision.VideoPlayer;
while hasFrame(video)
img = readFrame(video);
img = imresize(img, 3); % resize the frame to make people large enough
[bboxes,scores] = step(peopleDetector,img);
D = 1;
frame = insertObjectAnnotation(img,'rectangle',bboxes,D);
step(videoPlayer,frame);
end
I now see fairly consistent detections in the video, but they are still not tracked all the time, and there seems to be some erroneous detections (one, specifically at the right bottom corner of the video). To avoid these issues, I would do something like what this demo does:
https://www.mathworks.com/help/vision/examples/track-face-raspberry-pi2.html
In essence, this demo uses face detection only when there are no detections, and switches over to tracking once a detection has been made. That way, your processing loop is significantly faster (tracking is less computationally demanding than detection), and you can have (in general) higher fidelity tracking than independent detections in each frame. You could also include some other heuristics such as no movements at all in > 50 frames implies a false positive, and such.
I am currently working on a project where I need to find the
square shape tiles in pile which are not overlapped,
am currently working on a project where
I need to determine the orientation , position (center ) ,and color
of each square tile . These orientation and positions
will be used as input for a robot to be picked
and the robot will sort them in a specific locations .
I am using Matlab and i should transfer the data using TCP/IP.
I've experimenting with edge detection(canny,sobel) ,
found the boundaries,segmentation using threshold and FCM but
I haven't found a reliable way to determine the tiles which are
not overlapped ,i am trying to use template shape matching but
i don't know how to do that . This needs to be done in real time
as i will be using frame which is taken from a USB camera that
attached to PC . I was wondering if someone could offer a
reliable solution ? Here is a sample image.
I was wondering if someone could offer
a reliable solution to determine the square shape tiles
which are not overlapped? Here is a sample imageoverlapped Tiles
You've separated the image into tiles and background. So now simply label all the connected components. Take each one and test for single tile-ness. If you know the approximate size of the tiles, first exclude by area. Then calculate the centroid and the extreme left, right, top and bottom. If it is tile, the intersection of top bottom and left-right will be approximately in the centroid, and the half angles will be perpendicular to the tile edge. So rotate, take the bounding box, and count unset pixels, which should be almost zero for a rectangluar tile.
(You'll probably need to do a morphological operation or two to clean up the images if the tile / background separation is a bit dicey).
Check out the binary image processing library http://malcolmmclean.github.io/binaryimagelibrary 1
thanks for your quick replay.i already did some morphological operation and found connected component and below is my code in matlab ,and each tile has 2.5*2.5 cm area
a = imread('origenal image.jpg');
I = rgb2gray(a);
imshow(I)
thresold = graythresh(I);
se1=strel('diamond',2);
I1=imerode(I,se1);
figure(1)
imshow(I1);
bw = imclose(I1 , ones(25) );
imshow(bw)
CC = bwconncomp(bw);
L = labelmatrix(CC);
I was trying to animate different moves in Matlab. This post Matlab for loop animations helped my a lot , but I would like to change moves after some time. Thus, after defining the trajectories I animated them. Could you please have a look?
I wanted to keep the speed of the dot fixed and thus I solved it with differential equations, which are defined at other files. I have defined also the times tf , tf1... I used exactly the same way as it was suggested at the above link, with hPoint.
tf=4*pi/15; % time at which 4pi are completed. speed=15
tf1= 2+tf;
tf2= pi/15 +tf1;
[t,X]=ode45(#dif,[0 tf],[0 -15 -15])
p1 = [X(:,2) X(:,3)];
[t,X2]=ode45(#dif2,[tf tf1],[-15 -15])
p1a = [X2(:,1) X2(:,2)];
[t,X3]=ode45(#dif,[tf1 tf2],[0 -15 +15])
p1b = [X3(:,2) X3(:,3)];
D = [p1(:,1) p1(:,2)
p1a(:,1) p1a(:,2)
p1b(:,1) p1b(:,2)];
hPoint = line('XData',D(1,1), 'YData',D(1,2), 'EraseMode',ERASEMODE, ...
'Color','r', 'Marker','o', 'MarkerSize',50, 'LineWidth',1);
However, when I am trying to animate it, the dot stops a bit and then continues. Especially for the vector p1b, which is the third part( the upper circle). Any ideas about this behavior? Is there any way to make it stable and animate with the same speed? Thank you in advance !
I am currently trying to reconstruct a 3D trajectory of a falling object like a ball or a rock out of a sequence of images taken from an iPhone video.
Where should I start looking? I know I have to calibrate the camera (I think I'll use the matlab calibration toolbox by Jean-Yves Bouguet) and then find the vanishing point from the same sequence, but then I'm really stuck.
read this: http://www.cs.auckland.ac.nz/courses/compsci773s1c/lectures/773-GG/lectA-773.htm
it explains 3d reconstruction using two cameras. Now for a simple summary, look at the figure from that site:
You only know pr/pl, the image points. By tracing a line from their respective focal points Or/Ol you get two lines (Pr/Pl) that both contain the point P. Because you know the 2 cameras origin and orientation, you can construct 3d equations for these lines. Their intersection is thus the 3d point, voila, it's that simple.
But when you discard one camera (let's say the left one), you only know for sure the line Pr. What's missing is depth. Luckily you know the radius of your ball, this extra information can give you the missing depth information. see next figure (don't mind my paint skills):
Now you know the depth using the intercept theorem
I see one last issue: the shape of ball changes when projected under an angle (ie not perpendicular on your capture plane). However you do know the angle, so compensation is possible, but I leave that up to you :p
edit: #ripkars' comment (comment box was too small)
1) ok
2) aha, the correspondence problem :D Typically solved by correlation analysis or matching features (mostly matching followed by tracking in a video). (other methods exist too)
I haven't used the image/vision toolbox myself, but there should definitely be some things to help you on the way.
3) = calibration of your cameras. Normally you should only do this once, when installing the cameras (and every other time you change their relative pose)
4) yes, just put the Longuet-Higgins equation to work, ie: solve
P = C1 + mu1*R1*K1^(-1)*p1
P = C2 + mu2*R2*K2^(-1)*p2
with
P = 3D point to find
C = camera center (vector)
R = rotation matrix expressing the orientation of the first camera in the world frame.
K = calibration matrix of the camera (containing internal parameters of the camera, not to be confused with the external parameters contained by R and C)
p1 and p2 = the image points
mu = parameter expressing the position of P on the projection line from camera center C to P (if i'm correct R*K^-1*p expresses a line equation/vector pointing from C to P)
these are 6 equations containing 5 unknowns: mu1, mu2 and P
edit: #ripkars' comment (comment box too small once again)
The only computer vison library that pops up in my mind is OpenCV (http://opencv.willowgarage.com/wiki ). But that's a C library, not matlab... I guess google is your friend ;)
About the calibration: yes, if those two images contain enough information to match some features. If you change the relative pose of the cameras, you'll have to recalibrate of course.
The choice of the world frame is arbitrary; it only becomes important when you want to analyze the retrieved 3d data afterwards: for example you could align one of the world planes with the plane of motion -> simplified motion equation if you want to fit one.
This world frame is just a reference frame, changeable with a 'change of reference frame transformation' (translation and/or rotation transformation)
Unless you have a stereo camera, you will never be able to know the position for sure, even with calibrated camera. Because you don't know whether the ball is small and close or large and far away.
There are other methods with single camera, based on series of images with different focus. But I doubt that you can control the camera of your cell phone in that way.
Edit(1):
as #GuntherStruyf points out correctly, you can know the position if one of your inputs is the size of the ball.
I have written a code to create an animation (satellite movement around the Earth). When I run it, it works fine. However, when it is modified to be part of a code much more complex present in a Matlab GUI, the results produced changes (mainly because of the bigger number of points to plot). I also have noticed that if I use the OpenGL renderer the movement of the satellite is quicker than when the other renderers (Painters and Zbuffer) are used. I do not know if there are further possibilities to achieve an improvement in the rendering of the satellite movement. I think the key is, perhaps, changing the code that creates the actual position of the satellite (handles.psat) and its trajectory along the time (handles.tray)
handles.tray = zeros(1,Fin);
handles.psat = line('parent',ah4,'XData',Y(1,1), 'YData',Y(1,2),...
'ZData',Y(1,3),'Marker','o', 'MarkerSize',10,'MarkerFaceColor','b');
...
while (k<Fin)
az = az + 0.01745329252;
set(hgrot,'Matrix',makehgtform('zrotate',az));
handles.tray(k) = line([Y(k-1,1) Y(k,1)],[Y(k-1,2) Y(k,2)],...
[Y(k-1,3) Y(k,3)],...
'Color','red','LineWidth',3);
set(handles.psat,'XData',Y(k,1),'YData',Y(k,2),'ZData',Y(k,3));
pause(0.02);
k = k + 1;
if (state == 1)
state = 0;
break;
end
end
...
Did you consider to apply a rotation transform matrix on your data instead of the axis?
I think <Though I haven't checked it> that it can speedup your code.
You've used the typical tricks that I use to speed things up, like precomputing the frames, setting XData and YData rather than replotting, and selecting a renderer. Here are a couple more tips though:
1) One thing I noticed in your description is that different renderers and different complexities changed how fast your animation appeared to be running. This is often undesirable. Consider using the actual interval between frames (i.e. use tic; dt = toc) to calculate how far to advance the animation, rather than relying on pause(0.2) to generate a steady frame rate.
2) If the complexity is such that your frame rate is undesirably low, consider replacing pause(0.02) with drawnow, or at least calculate how long to pause on each frame.
3) Try to narrow down the source of your bottleneck a bit further by measuring how long the various steps take. That will let you optimize the right stage of the operation.