I'm trying to capture the "floor texture" based on a ARCore detected plane and the environment (camera) texture. Then reapply this floor texture in a plane mesh, creating a digital floor based on reality.
I've uploaded an image to illustrate this:
This is not a ARCore specific question, I think it can be resolved with math and graphics programming, maybe something like unprojecting the plane based on the camera matrix, but I do not know exactly how to do that.
Can someone help me?
Thanks!!
Essentially, we have three important coordinate systems in this problem:
We have the usual 3D world coordinate system that can be defined arbitrarily. We have a 2D coordinate system for the plane. The origin of that coordinate system is in the plane's center (note that I use the term plane synonymous for rectangle for this purpose) and the coordinates range from -1 to +1. And finally, we have a 2D coordinate for the image. Actually, we have two for the image: An unsigned coordinate system (as shown in the figure) with the origin in the bottom left and coordinates ranging from 0 to 1, and a signed one with coordinates ranging from -1 to 1.
We know the four corners of our plane in world space and the 3x4 view/projection matrix P that allows us to project any point in world space to image space using homogeneous coordinates:
p_image,signed = P * p_world
If your projection matrix is 4x4, simply drop the third row (last-but 1) as we are not interested in image-space depth.
We do not really care about world space as this is somewhat arbitrary. Given a 2D point in plane space, we can transform it to world space using:
p_world = 1/4 (p0 + p1 + p2 + p3) + u * 1/2 * (p1 - p0) + v * 1/2 * (p3 - p0)
The first part is the plane's origin and the point differences in the second and third terms are the coordinate axes. We can represent this in matrix form as
/ x \ / p1_x - p0_x p2_x - p0_x 1/4 (p0_x + p1_x + p2_x + p3_x) \ / u \
| y | = | p1_y - p0_y p2_x - p0_x 1/4 (p0_y + p1_y + p2_y + p3_y) | * | v |
| z | | p1_z - p0_z p2_x - p0_x 1/4 (p0_z + p1_z + p2_z + p3_z) | \ 1 /
\ 1 / \ 0 0 1 /
Let us call this matrix M.
Now, we can go directly from plane space to image space with:
p_image,signed = P * M * p_plane
The matrix P * M is now a 3x3 matrix. It is the homography between your ground plane and the image plane.
So, what can we do with it? We can use it to draw the image in plane space. So, here is what we are going to do:
We will generate a render target that we will fill with a single draw call. This render target will then contain the texture of our plane. To draw this, we:
Upload the camera image to the GPU as a texture
Bind the render target
Draw a full-screen quad with corners (-1, -1), (1, -1), (1, 1), (-1, 1)
In the vertex shader, calculate texture coordinates in image space from plane space
In the pixel shader, sample the camera image at the interpolated texture coordinates
The interesting part is number 4. We almost know what we need to do. We already know how to go to signed image space. Now, we just need to go to unsigned image space. And this is a simple shift and scale:
/ 1/2 0 1/2 \
p_image,unsigned = | 0 1/2 1/2 | * p_image,signed
\ 0 0 1 /
If we call this matrix S, we can then calculate S * P * M to get a single 3x3 matrix T. This matrix can be used in the vertex shader to calculate the texture coordinates from the plane-space points that you pass in:
texCoords = p_image,unsigned = T * p_plane
It is important that you pass the entire 3D vector to the fragment shader and do the perspective divide only in the pixel shader to produce a correct perspective.
Related
I'm working on a program on hololens2 research mode on unity. Hololens give us a depth image that is distance from depth sensor to object in front, for every pixel.
What I do is for every pixel I project pixel to image plane, then backproject it according to depth distance captured by depth sensor and it gives me the xyz in depth sensor coordinate frame. now it is needed to transform this coordinate to global coordinate system. to do so I get camera coordinate from unity by cam_pose = Camera.main.transform and in the other hand saved depth sensor extrinsic matrix.
From these two matrices I create a depth_to_world = cam_pose # inv(extrinsic). Now for every xyz on depth I perform global_xyz = depth_to_world # xyz to get point in real world. Problem is it return a point with 10-15 cm error. What am I doing wrong? (code is in python)
x = self.us[Depth_i, Depth_j] # projection from pixels to image plane
y = self.vs[Depth_i, Depth_j] # projection from pixels to image plane
D = distance_img[Depth_i, Depth_j] #distance_img is depth image
distance = 1000*float(D) / np.sqrt(x * x + y * y + 1) #distance according to spherical image plane D is in millimeter
depth_to_world = cam_pose # np.linalg.inv(Constants.camera_extrinsic)
X = (np.array([x * distance, y * distance, 1.0 * distance, 1])).reshape(4, 1)
point = (depth_to_world # X )[0:3, 0]
I got it! according to (https://github.com/petergu684/HoloLens2-ResearchMode-Unity) first I passed unity world origin to a winrt plugin, and depth_to_world was depth_to_world = inv(extrinsic) * cam_pose witch cam_pose is given by TryLocateAtTimeStamp. And other point is that unity coordinate is left handed (surprisingly!) so we should multiply a (-1) to z. (z <- -z)
my depth_to_world transformation was near but not correct.
I have two samples of 3D point cloud of human face. The blue point cloud denote target face and the red point cloud indicates the template. The image below shows that the target and template face are aligned in different directions (target face roughly along x-axis, template face roughly along y-axis).
Figure 1:
The region around the nose is displayed in Figure 1.
I want to rotate my target face (blue face) with nasal tip as the center of rotation (I translated the target to the template prior to Figure1 so that the tip of nose, i.e., the centerpt, for both faces are superimposed) to grossly align with the template face (red face). I rotated the target face with the following MATLAB code:
% PCA for the target face
targetFaceptfmt = pointCloud(targetFace); % Convert to point cloud format
point = [templateFace(3522, 1), templateFace(3522, 2), templateFace(3522, 3)]; % The 3522th point in the templateFace is the nasal tip point used as center of rotation later on
radius = 20; % 20mm
[NNTarIndex, NNTarDist] = findNeighborsInRadius(Locationptfmt, point, radius); % Find all vertices within 20 of the nasal tip point on the target face
NNTar = select(Locationptfmt, NNTarIndex); % Select the identified points for PCA
[TarVec,TarSCORE,TarVal] = pca(NNTar.Location); % Do PCA for target face using vertices close to the nasal tip
% PCA for the template face
templateFaceptfmt = pointCloud(templateFace); % Convert to point cloud format
[NNTemIndex, NNTemDist] = findNeighborsInRadius( templateFaceptfmt, point, radius); % Find all vertices within 20 of the nasal tip point on the template
NNTem = select(templateFaceptfmt, NNTemIndex); % Select the identified points for PCA
[TemVec,TemSCORE,TemVal] = pca(NNTem.Location); % Do PCA for template face using vertices close to the nasal tip
% Rotate target face with nasal tip point as the center of rotation
targetFace_r = R * (targetFace-cenertpt)' + centerpt';
targetFace_new = targetFace_r';
where targetFace and templateFace contains coordinates for the unrotated target face and the template face, respectively. The targetFace_r contains coordinates for the target face after rotation around nasal tip, R is the rotation matrix calculated through PCA (See here for source of formula for rotation), and centerpt is the nasal tip point which is used as the center of rotation. I then plotted the transposed targetFace_r, i.e., the targetFace_new, with normals added to each vertex:
Figure 2:
Before rotation, the normals for the target face and template face are generally pointing toward similar directions (Figure 1). After rotation, the target and template face are both aligned along the y-axis (which is what I want), however, normals for the target face and template face point toward opposite directions. Bearing in mind that no changes were made to the template face, I realized that normals of the target face calculated after rotation are flipped. But I do not know why. I used the checkFaceOrientation function of the Rvcg package in R to check if expansion along normals increases centroid size. I was returned TRUE for the template face but FALSE for the target face, which confirms that vertex normals for the target face are flipped.
Vertex normals were calculated in MATLAB as follows:
TR = triangulation(Faces, Vertices); % Triangulation based on face and vertex information
VN = vertexNormal(TR); % Calculate vertext normal
where Faces contains face information, i.e., the connectivity list, and Vertices contains coordiantes for vertices. For target face before rotation, target face after rotation, and template face, vertex normals were calcuated separately. I used the same Faces data for calculation of vertex normal before and after rotating the target face.
The flipped vertex normals resulted in errors for some further analyses. As a result, I have to manually flip the normals to make them pointing similarly to normals of the template face.
Figure 3:
Figure 3 shows that after manually flip the normals, normals of the target and template face are generally pointing similarly in direction.
My question is why does the normals of the target face calculated after rotation flipped? In what case does rotation of 3D point cloud result in flipping of vertex normals?
Some further informaiton that may be useful: the rotation matrix R I obtained is as follows for your reference:
0.0473096146726546 0.867593376108813 -0.495018720950670
0.987013081649028 0.0355601323276586 0.156654567895508
-0.153515396665006 0.496001220483328 0.854643675613313
Since trace(R) = 1 + 2cos(alpha), I calcualted alpha through acos((trace(R)-1)/2)*180/pi, which yielded an angle of rotation of 91.7904, relative to the nasal tip point.
If I'm understanding everything correctly, it looks like your rotation matrix is actually encoding a rotation plus a reflection. If your matrix is approximately:
0.04 0.86 -0.49
0.98 0.03 0.15
-0.15 0.49 0.85
Then the image of each unit vector pointing along the positive axes are:
x = [ 0.04 0.98 -0.15]
y = [ 0.86 0.03 0.49]
z = [-0.49 0.15 0.85]
However, if you take the cross-product of x and y (cross(x, y)), you get approximately [0.49 -0.15 -0.85], which is the negation of z, which implies that the matrix is encoding both a rotation and a reflection. Naturally, multiplying a mesh's vertices by a reflection matrix will reverse the winding order of its polygons, yielding inverted normals.
In the slides that you referenced, it states that the PCA method of generating a rotation matrix should only be considering four different combinations of axes in the 3D case, to ensure that the output matrix obeys the right-hand rule. If all combinations of axes were checked, that would allow PCA to consider both rotated and reflected spaces when searching for a best match. If that were the case, and if there some noise in the data such that the left half of the template is a slightly better match to the right half of the target and vice versa, then the PCA method might generate a reflection matrix like the one you observe. Perhaps you might want to reexamine the logic of how R is generated from the PCA results?
As alluded to in the comments, the direction of your vertex normals will depend on how you've ordered the triangular facets in your Faces matrix. This will follow a right-hand rule, where your fingers follow the vertex order around the triangle and your thumb indicates the surface normal direction. Here's a simple example to help illustrate:
Vertices = [0 0; 0 1; 1 1; 1 0]; % Points clockwise around a unit square in x-y plane
Faces = [1 2 3; 1 3 4]; % Two triangular facets, clockwise vertex ordering
TR = triangulation(Faces, Vertices);
VN = vertexNormal(TR)
VN =
0 0 -1
0 0 -1
0 0 -1
0 0 -1
In this example, Vertices contains the 4 vertices of a unit square in the x-y plane, ordered clockwise if you're looking down from positive z. Two triangular facets are defined in Faces, and the order of the indices in each row traces along the vertices in a clockwise fashion as well. This results in a surface normal for each face that points in the negative z direction. When the vertex normals are computed, they are pointing in the negative z direction as well.
What happens when we flip the order of one triangle so that its points are counter-clockwise?...
Faces = [1 2 3; 1 4 3]; % Second facet is 1 4 3 instead of 1 3 4
TR = triangulation(Faces, Vertices);
VN = vertexNormal(TR)
VN =
0 0 0
0 0 -1
0 0 0
0 0 1
The surface normal of the second triangle will now point in the positive z direction. The vertices that are only used by one triangle (rows 2 and 4) will have vertex normals that match the surface normals, while the vertices shared by each (rows 1 and 3) will have vertex normals of 0 (the two surface normals cancel).
How will this help you with your problem? Well, it's hard to say since I don't know exactly how you are defining Faces and Vertices. However, if you know for certain that every vertex normal in your mesh is pointing in the wrong direction, you can easily flip them all by swapping two columns in your Faces matrix before computing the normals:
Faces = [1 2 3; 1 3 4]; % Clockwise-ordered vertices
TR = triangulation(Faces(:, [1 3 2]), Vertices); % Change to counter-clockwise
VN = vertexNormal(TR)
VN =
0 0 1 % Normals are now pointing in positive z
0 0 1
0 0 1
0 0 1
I am facing the same problem as mentioned in this post, however, I am not facing it with OpenGL, but simply with MATLAB. Depth as distance to camera plane in GLSL
I have a depth image rendered from the Z-Buffer from 3ds max. I was not able to get an orthographic representation of the z-buffer. For a better understanding, I will use the same sketch as made by the previous post:
* |--*
/ |
/ |
C-----* C-----*
\ |
\ |
* |--*
The 3 asterisks are pixels and the C is the camera. The lines from the
asterisks are the "depth". In the first case, I get the distance from the pixel to the camera. In the second, I wish to get the distance from each pixel to the plane.
The settins of my camera are the following:
WIDTH = 512;
HEIGHT = 424;
FOV = 89.971;
aspect_ratio = WIDTH/HEIGHT;
%clipping planes
near = 500;
far = 5000;
I calulate the frustum settings like the following:
%calculate frustums settings
top = tan((FOV/2)*5000)
bottom = -top
right = top*aspect_ratio
left = -top*aspect_ratio
And set the projection matrix like this:
%Generate matrix
O_p = [2/(right-left) 0 0 -((right+left)/(right-left)); ...
0 2/(top-bottom) 0 -((top+bottom)/(top-bottom));...
0 0 -2/(far-near) -(far+near)/(far-near);...
0 0 0 1];
After this I read in the depth image, which was saved as a 48 bit RGB- image, where each channel is the same, thus only one channel has to be used.
%Read in image
img = imread('KinectImage.png');
%Throw away, except one channel (all hold the same information)
c1 = img(:,:,1);
The pixel values have to be inverted, since the closer the values are to the camera, the brigher they were. If a pixel is 0 (no object to render available) it is set to 2^16, so , that after the bit complementation, the value is still 0.
%Inverse bits that are not zero, so that the z-image has the correct values
c1(c1 == 0) = 2^16
c1_cmp = bitcmp(c1);
To apply the matrix, to each z-Buffer value, I lay out the vector one dimensional and build up a vector like this [0 0 z 1] , over every element.
c1_cmp1d = squeeze(reshape(c1_cmp,[512*424,1]));
converted = double([zeros(WIDTH*HEIGHT,1) zeros(WIDTH*HEIGHT,1) c1_cmp1d zeros(WIDTH*HEIGHT,1)]) * double(O_p);
After that, I pick out the 4th element of the result vector and reshape it to a image
img_con = converted(:,4);
img_con = reshape(img_con,[424,512]);
However, the effect, that the Z-Buffer is not orthographic is still there, so did I get sth wrong? Is my calculation flawed ? Or did I make mistake here?
Depth Image coming from 3ds max
After the computation (the white is still "0" , but the color axis has changed)
It would be great to achieve this with 3ds max, which would resolve this issue, however I was not able to find this setting for the z-buffer. Thus, I want to solve this using Matlab.
I am in need of an idea! I want to model the vascular network on the eye in 3D. I have made statistics on the branching behaviour in relation to vessel diameter, length etc. What I am stuck at right now is the visualization:
The eye is approximated as a sphere E with center in origo C = [0, 0, 0] and a radius r.
What I want to achieve is that based on the following input parameters, it should be able to draw a segment on the surface/perimeter of E:
Input:
Cartesian position of previous segment ending: P_0 = [x_0, y_0, z_0]
Segment length: L
Segment diameter: d
Desired angle relative to the previous segment: a (1)
Output:
Cartesian position of resulting segment ending: P_1 = [x_1, y_1, z_1]
What I do now, is the following:
From P_0, generate a sphere with radius L, representing all the points we could possibly draw to with the correct length. This set is called pool.
Limit pool to only include points with a distance to C between r*0.95 and r, so only the points around the perimeter of the eye are included.
Select only the point that would generate a relative angle (2) closest to the desired angle a.
The problem is, that whatever angle a I desire, is actually not what is measured by the dot product. Say I want an angle at 0 (i.e. that the new segment is following the same direction as the previous`, what I actually get is an angle around 30 degrees because of the curvature of the sphere. I guess what I want is more the 2D angle when looking from an angle orthogonal from the sphere to the branching point. Please take a look at the screenshots below for a visualization.
Any ideas?
(1) The reason for this is, that the child node with the greatest diameter is usually follows the path of the previous segment, whereas smaller child nodes tend to angle differently.
(2) Calculated by acos(dot(v1/norm(v1), v2/norm(v2)))
Screenshots explaining the problem:
Yellow line: previous segment
Red line: "new" segment to one of the points (not neccesarily the correct one)
Blue x'es: Pool (text=angle in radians)
I will restate the problem with my own notation:
Given two points P and Q on the surface of a sphere centered at C with radius r, find a new point T such that the angle of the turn from PQ to QT is A and the length of QT is L.
Because the segments are small in relation to the sphere, we will use a locally-planar approximation of the sphere at the pivot point Q. (If this isn't an okay assumption, you need to be more explicit in your question.)
You can then compute T as follows.
// First compute an aligned orthonormal basis {U,V,W}.
// - {U,V} should be a basis for the plane tangent at Q.
// - W should be normal to the plane tangent at Q.
// - U should be in the direction PQ in the plane tangent at Q
W = normalize(Q - C)
U = normalize(Q - P)
U = normalize(U - W * dotprod(W, U))
V = normalize(crossprod(W, U))
// Next compute the next point S in the plane tangent at Q.
// In a regular plane, the parametric equation of a unit circle
// centered at the origin is:
// f(A) = (cos A, sin A) = (1,0) cos A + (0,1) sin A
// We just do the same thing, but with the {U,V} basis instead
// of the standard basis {(1,0),(0,1)}.
S = Q + L * (U cos A + V sin A)
// Finally project S onto the sphere, obtaining the segment QT.
T = C + r * normalize(S - C)
i am working on some camera data. I have some points which consist of azimuth, angle, distance, and of course coordinate field attributes. In postgresql postgis I want to draw shapes like this with functions.
how can i draw this pink range shape?
at first should i draw 360 degree circle then extracting out of my shape... i dont know how?
I would create a circle around the point(x,y) with your radius distance, then use the info below to create a triangle that has a larger height than the radius.
Then using those two polygons do an ST_Intersection between the two geometries.
NOTE: This method only works if the angle is less than 180 degrees.
Note, that if you extend the outer edges and meet it with a 90 degree angle from the midpoint of your arc, you have a an angle, and an adjacent side. Now you can SOH CAH TOA!
Get Points B and C
Let point A = (x,y)
To get the top point:
point B = (x + radius, y + (r * tan(angle)))
to get the bottom point:
point C = (x + radius, y - (r * tan(angle)))
Rotate your triangle to you azimouth
Now that you have the triangle, you need to rotate it to your azimuth, with a pivot point of A. This means you need point A at the origin when you do the rotation. The rotation is the trickiest part. Its used in computer graphics all the time. (Actually, if you know OpenGL you could get it to do the rotation for you.)
NOTE: This method rotates counter-clockwise through an angle (theta) around the origin. You might have to adjust your azimuth accordingly.
First step: translate your triangle so that A (your original x,y) is at 0,0. Whatever you added/subtracted to x and y, do the same for the other two points.
(You need to translate it because you need point A to be at the origin)
Second step: Then rotate points B and C using a rotation matrix. More info here, but I'll give you the formula:
Your new point is (x', y')
Do this for points B and C.
Third step: Translate them back to the original place by adding or subtracting. If you subtracted x last time, add it this time.
Finally, use points {A,B,C} to create a triangle.
And then do a ST_Intersection(geom_circle,geom_triangle);
Because this takes a lot of calculations, it would be best to write a program that does all these calculations and then populates a table.
PostGIS supports curves, so one way to achieve this that might require less math on your behalf would be to do something like:
SELECT ST_GeomFromText('COMPOUNDCURVE((0 0, 0 10), CIRCULARSTRING(0 10, 7.071 7.071, 10 0), (10 0, 0 0))')
This describes a sector with an origin at 0,0, a radius of 10 degrees (geographic coordinates), and an opening angle of 45°.
Wrapping that with additional functions to convert it from a true curve into a LINESTRING, reduce the coordinate precision, and to transform it into WKT:
SELECT ST_AsText(ST_SnapToGrid(ST_CurveToLine(ST_GeomFromText('COMPOUNDCURVE((0 0, 0 10), CIRCULARSTRING(0 10, 7.071 7.071, 10 0), (10 0, 0 0))')), 0.01))
Gives:
This requires a few pieces of pre-computed information (the position of the centre, and the two adjacent vertices, and one other point on the edge of the segment) but it has the distinct advantage of actually producing a truly curved geometry. It also works with segments with opening angles greater than 180°.
A tip: the 7.071 x and y positions used in the example can be computed like this:
x = {radius} cos {angle} = 10 cos 45 ≈ 7.0171
y = {radius} sin {angle} = 10 sin 45 ≈ 7.0171
Corner cases: at the antimeridian, and at the poles.