How to recreate the original vector from tangent space vector in Houdini within Unity Shader? - houdini

Currently, there is a need to convert normals from model space to tangent space in Houdini, and save them to vertex color. Then in Unity Shader, the normals can be converted from tangent space to model space by reading vertex color.
vector T = normalize(v#tangentu);
vector B = normalize(v#tangentv) ;
vector N = normalize(#N);
matrix m = set(#T, #B, #N);
#Cd = m * #Cd;
#Cd = #Cd * {-0.5,0.5,0.5} + 0.5; //houdini is leftHand
In Unity Shader:
float3 bitangent = cross(normalize(v.normal) , normalize(v.tangent.xyz)) * v.tangent.w;
float3x3 TtoO = float3x3(v.tangent.x, bitangent.x, v.normal.x,
v.tangent.y, bitangent.y, v.normal.y,
v.tangent.z, bitangent.z, v.normal.z);
float3 normalOS = v.color.xyz;
normalOS = normalOS * 2 -1;
normalOS = mul(TtoO, normalOS) ;
But the normals generated by this method are incorrect. We conducted the following test: In Houdini, we directly wrote the normals into the vertex color channel.
#Cd = #N;
#Cd = #Cd * 0.5 + 0.5;
#Cd = normalize(#Cd);
#N = #Cd;
In RenderDoc:
enter image description here
When we wrote the same values to the normals and vertices, but in Unity, the vector order was different and the values had some deviations, including decimal points.
Can someone who has access to the Unity source code check what calculations Unity performs when importing FBX files, and how to construct the TBN inverse matrix in Houdini to restore the tangent space vectors written in Houdini?

Related

Depth to world registration hololens2 unity

I'm working on a program on hololens2 research mode on unity. Hololens give us a depth image that is distance from depth sensor to object in front, for every pixel.
What I do is for every pixel I project pixel to image plane, then backproject it according to depth distance captured by depth sensor and it gives me the xyz in depth sensor coordinate frame. now it is needed to transform this coordinate to global coordinate system. to do so I get camera coordinate from unity by cam_pose = Camera.main.transform and in the other hand saved depth sensor extrinsic matrix.
From these two matrices I create a depth_to_world = cam_pose # inv(extrinsic). Now for every xyz on depth I perform global_xyz = depth_to_world # xyz to get point in real world. Problem is it return a point with 10-15 cm error. What am I doing wrong? (code is in python)
x = self.us[Depth_i, Depth_j] # projection from pixels to image plane
y = self.vs[Depth_i, Depth_j] # projection from pixels to image plane
D = distance_img[Depth_i, Depth_j] #distance_img is depth image
distance = 1000*float(D) / np.sqrt(x * x + y * y + 1) #distance according to spherical image plane D is in millimeter
depth_to_world = cam_pose # np.linalg.inv(Constants.camera_extrinsic)
X = (np.array([x * distance, y * distance, 1.0 * distance, 1])).reshape(4, 1)
point = (depth_to_world # X )[0:3, 0]
I got it! according to (https://github.com/petergu684/HoloLens2-ResearchMode-Unity) first I passed unity world origin to a winrt plugin, and depth_to_world was depth_to_world = inv(extrinsic) * cam_pose witch cam_pose is given by TryLocateAtTimeStamp. And other point is that unity coordinate is left handed (surprisingly!) so we should multiply a (-1) to z. (z <- -z)
my depth_to_world transformation was near but not correct.

How to apply transformation using 3x3 rotation matrix and a translation vector?

I am working on an Augmented Reality project using ARCore. Coordinate system of ARCore changes every time you launch the application making the initial position as origin. I have 5 points in another coordinate system and i can find 4 of these positions in Unity world space using ARCore Augmented Image. These points have different values in my other coordinate system of course. I have to find position of a 5th point in Unity world space using its position in other coordinate system.
I have followed this tutorial to achieve this. But since Unity does not support 3x3 matrices i used Accord.NET framework. Using the tutorial and Accord matrices i can calculate a 3x3 Rotation matrix and a Translation vector.
However, when i tried to apply this to my 5th point using TestObject.transform.Translate(AccordtoUnity(Translation),Space.World)i am having trouble. When the initial 4 objects and reference objects are at same orientation my translation works perfect. However, when my reference objects are rotated this translation does not work. This makes sense of course since i have only done a translation. My question is how can i apply rotation and translation to my 5th point. Or is there a way to convert my 3x3 rotation matrix and translation to Unity 4x4Matrix since then i can use Matrix4x4.MultiplyPoint3x4. Or is it possible to convert my 3x3 rotation matrix to a Quaternion which let me use4x4Matrix.SetTRS. I am a bit confused about this conversion because 4x4Matrix includes scaling as well but i am not doing any scaling.
I would be happy if someone can give me some hint or offer a better approach to find a way to find 5th point. Thanks!
EDIT:
I actually solved the problem based on Daveloper's answer. I constructed a Unity 4x4 matrix like this:
[ R00 R01 R02 T.x ]
[ R10 R11 R12 T.y ]
[ R20 R21 R22 T.z ]
[ 0 0 0 1 ]
I tested this creating primitive objects in Unity and apply translation and rotation using the matrix above like this:
TestObject.transform.position = TransformationMatrix.MultiplyPoint3x4(TestObject.transform.position);
TestObject.transform.rotation *= Quaternion.LookRotation(TransformationMatrix.GetColumn(2), TransformationMatrix.GetColumn(1));
to use a 3x3 rotation matrix and a translation vector to set a transform, use
// rotationMatrixCV = your 3x3 rotation matrix; translation = your translation vector
var rotationMatrix = new Matrix4x4();
for (int i = 0; i < 3; i++)
{
for (int j = 0; j < 3; j++)
{
rotationMatrix[i, j] = rotationMatrixCV[i, j];
}
}
rotationMatrix[3, 3] = 1f;
var localToWorldMatrix = Matrix4x4.Translate(translation) * rotationMatrix);
Vector3 scale;
scale.x = new Vector4(localToWorldMatrix.m00, localToWorldMatrix.m10, matrix.m20, localToWorldMatrix.m30).magnitude;
scale.y = new Vector4(localToWorldMatrix.m01, localToWorldMatrix.m11, matrix.m21, localToWorldMatrix.m31).magnitude;
scale.z = new Vector4(localToWorldMatrix.m02, localToWorldMatrix.m12, matrix.m22, localToWorldMatrix.m32).magnitude;
transform.localScale = scale;
Vector3 position;
position.x = localToWorldMatrix.m03;
position.y = localToWorldMatrix.m13;
position.z = localToWorldMatrix.m23;
transform.position = position;
Vector3 forward;
forward.x = localToWorldMatrix.m02;
forward.y = localToWorldMatrix.m12;
forward.z = localToWorldMatrix.m22;
Vector3 upwards;
upwards.x = localToWorldMatrix.m01;
upwards.y = localToWorldMatrix.m11;
upwards.z = localToWorldMatrix.m21;
transform.rotation = Quaternion.LookRotation(forward, upwards);
NOTICE:
This is only useful if this rotation and translation define your 5th point's location in the world in the coordinate system that is actively being used...
If your rotation and translation mean anything else, you'll have to do more. Glad to help further if you can define what this rotation and translation mean exactly.
If I understand your question correctly, you want to be able to create a rotation quaternion from a 3x3 matrix.
You can think of a 3x3 rotation matrix as three vectors of length 1 all at 90 degrees to each other. e.g.:
| forward.x forward.y forward.z |
| up.x up.y up.z |
| right.x right.y right.z |
A pretty reliable way to do the conversion is to take the forward and up vectors out of your matrix and applying them to the Unity method https://docs.unity3d.com/ScriptReference/Quaternion.LookRotation.html
This will create a quaternion that corresponds to your matrix. Depending on your actual situation, you might need to use the inverse of the quaternion, but essentially this is what you need.
Note you only need the forward and up, because the right is always the cross product of those two and adds no information. Also take care that you matrix is a pure rotation matrix (i.e. no scaling or skewing), otherwise you might get unexpected results.

Triangulate set of points on arbitrary plane in 3D space

I have a set of points in 3D space. With maximum error of 10^-5 i can place a plane through them (error is the distance from point to plane).
Is there a way to triangulate these points on this arbitrary plane? I have tried Bowyer-Watson but this only works when the error is 0. Anything else and it wont triangulate or i wont get a good triangulation(overlapping triangles).
Edit
I think i found the problem. At certain angles the bowyer watson algorithm wont work because my calculation of the circumcenter is off. How can i calculate the circumcenter of a triangle in 3D?
Since i know the points on the plane i can calculate a vector. This vector lies on the plane. Next i calculate the center of mass of the points.
Using the vector and center of mass i can create a large triangle on the plane
Vertex p1 = new Vertex(dir * 3000 + center);
Vertex p2 = new Vertex(Quaternion.AngleAxis(120, plane.normal) * dir * 3000 + center);
Vertex p3 = new Vertex(Quaternion.AngleAxis(240, plane.normal) * dir * 3000 + center);
Now that i have the enclosing triangle i can just use Bowyer-Watson.
For circumcenter in 3D i use:
Vector3 ac = p3 - p1;
Vector3 ab = p2 - p1;
Vector3 abXac = Vector3.Cross(ab, ac);
circumceter = p1 + (Vector3.Cross(abXac, ab) * ac.sqrMagnitude + Vector3.Cross(ac, abXac) * ab.sqrMagnitude) / (2 * abXac.sqrMagnitude);
And now i have a triangulated set of points on an arbitrary plane in 3D.

Finding the pixel displacement in fish eye images

I am trying to plot the displacement of a pixel from the original image to the fish eye image based on the radius from the center of the image.
I was successful in producing fish images in MATLAB using maketform
testImg = imread('ship.jpg');
optTra = maketform('custom',2,2,[],#radial,options);
newX = imtransform(testImg,optTra);
imshow(newX);
the radial function here helps me to get the fish eye distorted image.
I need to find the displacement of each pixel in the original image to that of the distorted image.
If the transformation applied (a.k.a "#radial") was angular, the inverse transformation is given by:
u = r cos(phi) + 0.5;
v = r sin(phi) + 0.5;
where
r = atan2(sqrt(x*x+y*y),p.z)/pi;
phi = atan2(y,x);
x,y are assumed to be normalized coordinates (centered and between -1 to 1).

picking in 3D with ray-tracing using NinevehGL or OpenGL i-phone

I couldn't find the correct and understandable expression of picking in 3D with method of ray-tracing. Has anyone implemented this algorithm in any language? Share directly working code, because since pseudocodes can not be compiled, they are genereally written with lacking parts.
What you have is a position in 2D on the screen. The first thing to do is convert that point from pixels to normalized device coordinates — -1 to 1. Then you need to find the line in 3D space that the point represents. For this, you need the transformation matrix/ces that your 3D app uses to create a projection and camera.
Typically you have 3 matrics: projection, view and model. When you specify vertices for an object, they're in "object space". Multiplying by the model matrix gives the vertices in "world space". Multiplying again by the view matrix gives "eye/camera space". Multiplying again by the projection gives "clip space". Clip space has non-linear depth. Adding a Z component to your mouse coordinates puts them in clip space. You can perform the line/object intersection tests in any linear space, so you must at least move the mouse coordinates to eye space, but it's more convenient to perform the intersection tests in world space (or object space depending on your scene graph).
To move the mouse coordinates from clip space to world space, add a Z-component and multiply by the inverse projection matrix and then the inverse camera/view matrix. To create a line, two points along Z will be computed — from and to.
In the following example, I have a list of objects, each with a position and bounding radius. The intersections of course never match perfectly but it works well enough for now. This isn't pseudocode, but it uses my own vector/matrix library. You'll have to substitute your own in places.
vec2f mouse = (vec2f(mousePosition) / vec2f(windowSize)) * 2.0f - 1.0f;
mouse.y = -mouse.y; //origin is top-left and +y mouse is down
mat44 toWorld = (camera.projection * camera.transform).inverse();
//equivalent to camera.transform.inverse() * camera.projection.inverse() but faster
vec4f from = toWorld * vec4f(mouse, -1.0f, 1.0f);
vec4f to = toWorld * vec4f(mouse, 1.0f, 1.0f);
from /= from.w; //perspective divide ("normalize" homogeneous coordinates)
to /= to.w;
int clickedObject = -1;
float minDist = 99999.0f;
for (size_t i = 0; i < objects.size(); ++i)
{
float t1, t2;
vec3f direction = to.xyz() - from.xyz();
if (intersectSphere(from.xyz(), direction, objects[i].position, objects[i].radius, t1, t2))
{
//object i has been clicked. probably best to find the minimum t1 (front-most object)
if (t1 < minDist)
{
minDist = t1;
clickedObject = (int)i;
}
}
}
//clicked object is objects[clickedObject]
Instead of intersectSphere, you could use a bounding box or other implicit geometry, or intersect a mesh's triangles (this may require building a kd-tree for performance reasons).
[EDIT]
Here's an implementation of the line/sphere intersect (based off the link above). It assumes the sphere is at the origin, so instead of passing from.xyz() as p, give from.xyz() - objects[i].position.
//ray at position p with direction d intersects sphere at (0,0,0) with radius r. returns intersection times along ray t1 and t2
bool intersectSphere(const vec3f& p, const vec3f& d, float r, float& t1, float& t2)
{
//http://wiki.cgsociety.org/index.php/Ray_Sphere_Intersection
float A = d.dot(d);
float B = 2.0f * d.dot(p);
float C = p.dot(p) - r * r;
float dis = B * B - 4.0f * A * C;
if (dis < 0.0f)
return false;
float S = sqrt(dis);
t1 = (-B - S) / (2.0f * A);
t2 = (-B + S) / (2.0f * A);
return true;
}
vec4f from = toWorld * vec4f(mouse, -1.0f, 1.0f);
vec4f to = toWorld * vec4f(mouse, 1.0f, 1.0f);
I'm assuming that 'from' is the position of the mouse cursor? If so then why is its z negative one, if we are assuming openGL coordinates.
Also in this way do we assume that the depth at this time is -1 to +1 right? Rather than the depth of our frustrum.