I have 3d points from an obj,i want to be able to select a point say v -0.822220 0.216242 -0.025730 and overlay it with a container and save the point.(I have a vehicle 3d obj ,I want to be able to select say the drivers door and save the point selected maybe a door handle).
sample points:
v 0.822220 0.216242 -0.025730 v -0.822220 0.216242 -0.025730 v 0.811726 0.220845 0.029668 v -0.811726 0.220845 0.029668 v 0.777874 0.214472 0.075458 v -0.777874 0.214472 0.075458 v 0.724172 0.189587 0.073470 v -0.724172 0.189587 0.073470 v 0.704111 0.180226 0.027508
what i have achieved
return new GestureDetector(
onTapDown: (TapDownDetails details) => onTapDown(context, details),
child: new Stack(fit: StackFit.expand, children: <Widget>[
Object3D(...),
new Positioned(
child: new Container(color:Colors.red),
left: posx,
top: posy,
)
]),
);
void onTapDown(BuildContext context, TapDownDetails details) {
print('${details.globalPosition}');
final RenderBox box = context.findRenderObject();
final Offset localOffset = box.globalToLocal(details.globalPosition);
setState(() {
posx = localOffset.dx;
posy = localOffset.dy;
});
}
.i got a suggestion to convert the points to 2d and use the 2d points to overlay the container.How do i convert the 3d points to 2d points?
Is there a better way of doing this?
i'm using this package flutter_3d_obj
(Disclaimer: 3D graphics in Dart/Flutter are extremely experimental. Flutter does not provide any 3D rendering context to draw 3D objects onto, and any 3D rendering packages such as flutter_3d_obj are A) software-based and therefore very slow, and B) extremely limited in feature set [i.e. lacking lighting, shading, normals, texturing, etc.]. As such, it's not recommended to try and draw 3D objects in Flutter directly. The recommendation is to either use something like flare to replicate the 3D effect using 2D animations or to use something like the Unity3D Widget package to draw 3D graphics on a non-Flutter canvas.)
The conversion from a point in a 3D space to a 2D plane is called a Projection Transformation. This transformation is the basis of all "cameras" in 3D software from simple games to 3D animated Hollywood films. There are quite a few excellent write-ups on how a projection transformation works (a Google search brings up this one), but an overly simplified explanation is as follows.
(Like any other transformation, this will require linear algebra and matrix multiplication. Fortunately, Dart has a matrix math package in vector_math.)
There are two general types of projection transformations: perspective and orthogonal.
Orthogonal is the easiest to conceptualize and implement, as it's just a flat conversion from a point in 3D space to the place on the plane that is closest to that point. It's literally just stripping the Z coordinate off of the 3D point and using the X and Y coordinates as your new 2D point:
import 'package:vector_math/vector_math.dart';
Vector2 transformPointOrtho(Vector3 input) {
final ortho = Matrix4(
1, 0, 0, 0,
0, 1, 0, 0,
0, 0, 0, 0,
0, 0, 0, 0,
);
return (ortho * input).xy;
}
Perspective is more complex, as it also takes perspective (a.k.a field of view angles) into account. As such, there are some parameters that go into creating the transformation matrix:
n = Near clipping plane
f = Far clipping plane
S = Representation of the vertical viewing angle, derived by the following equation:
fov = Field of view (FOV) angle in degrees
(To use radians, omit the "* (π/180)" part)
import 'dart:math';
import 'package:vector_math/vector_math.dart';
Vector2 transformPointPersp(Vector3 input, double fovDeg, double nearClip, double farClip) {
final s = 1 / (tan((fovDeg / 2) * (pi / 180)));
final fdfn = -farClip / (farClip - nearClip);
final fndfn = -(farClip * nearClip) / (farClip - nearClip);
final persp = Matrix4(
s, 0, 0, 0,
0, s, 0, 0,
0, 0, fdfn, -1,
0, 0, fndfn, 0,
);
return (input * persp).xy;
}
Obviously, this is an overly simplistic explanation and doesn't take into account factors such as camera position/rotation. This is also the simplest way to form the transformation matrices, and not necessarily the best. For further reading, I highly suggest you look at various linear algebra and low-level 3D rendering tutorials (such as the one linked above).
Related
In Qt Quick project, I derived a custom class from QQuickPaintedItem and mapped screen coordinate system to Cartesian coordinate system by providing a transform matrix, now I want to display a png on the custom class with QPainter->drawImage, however, the y coordinate of the image is inversed, how to fix it, thanks!
Below is the code snippet:
void DrawArea::paint(QPainter *painter)
{
painter->setRenderHint(QPainter::Antialiasing, true);
QTransform transform;
transform.setMatrix(800.0/10.0, 0.0, 0.0,
0.0, -600.0/10.0, 0.0,
400, 300, 1.0);
painter->setWorldTransform(transform);
painter->drawImage(QRectF(0, 0, 3, 3), m_image, QRectF(0, 0, m_image.width(),
m_image.height()));
}
the window size is 800x600, the Cartesian coordinate is from -5 to 5 with both x and y.
The y coord is inversed due to -600.0/10.0, but if I remove the minus sign as 600.0/10.0, the image is correct displayed, but the image extend below y=0 axis in Cartesian coordinate system.
In Unity, say you have a 3D object,
Of course, it's trivial to get the AABB, Unity has direct functions for that,
(You might have to "add up all the bounding boxes of the renderers" in the usual way, no issue.)
So Unity does indeed have a direct function to give you the 3D AABB box instantly, out of the internal mesh/render pipeline every frame.
Now, for the Camera in question, as positioned, that AABB indeed covers a certain 2D bounding box ...
In fact ... is there some sort of built-in direct way to find that orange 2D box in Unity??
Question - does Unity have a function which immediately gives that 2D frustrum box from the pipeline?
(Note that to do it manually you just make rays (or use world to screen space as Draco mentions, same) for the 8 points of the AABB; encapsulate those in 2D to make the orange box.)
I don't need a manual solution, I'm asking if the engine gives this somehow from the pipeline every frame?
Is there a call?
(Indeed, it would be even better to have this ...)
My feeling is that one or all of the
occlusion system in particular
the shaders
the renderer
would surely know the orange box, and perhaps even the blue box inside the pipeline, right off the graphics card, just as it knows the AABB for a given mesh.
We know that Unity lets you tap the AABB 3D box instantly every frame for a given mesh: In fact does Unity give the "2D frustrum bound" as shown here?
As far as I am aware, there is no built in for this.
However, finding the extremes yourself is really pretty easy. Getting the mesh's bounding box (the cuboid shown in the screenshot) is just how this is done, you're just doing it in a transformed space.
Loop through all the verticies of the mesh, doing the following:
Transform the point from local to world space (this handles dealing with scale and rotation)
Transform the point from world space to screen space
Determine if the new point's X and Y are above/below the stored min/max values, if so, update the stored min/max with the new value
After looping over all vertices, you'll have 4 values: min-X, min-Y, max-X, and max-Y. Now you can construct your bounding rectangle
You may also wish to first perform a Gift Wrapping of the model first, and only deal with the resulting convex hull (as no points not part of the convex hull will ever be outside the bounds of the convex hull). If you intend to draw this screen space rectangle while the model moves, scales, or rotates on screen, and have to recompute the bounding box, then you'll want to do this and cache the result.
Note that this does not work if the model animates (e.g. if your humanoid stands up and does jumping jacks). Solving for the animated case is much more difficult, as you would have to treat every frame of every animation as part of the original mesh for the purposes of the convex hull solving (to insure that none of your animations ever move a part of the mesh outside the convex hull), increasing the complexity by a power.
3D bounding box
Get given GameObject 3D bounding box's center and size
Compute 8 corners
Transform positions to GUI space (screen space)
Function GUI3dRectWithObject will return the 3D bounding box of given GameObject on screen.
2D bounding box
Iterate through every vertex in a given GameObject
Transform every vertex's position to world space, and transform to GUI space (screen space)
Find 4 corner value: x1, x2, y1, y2
Function GUI2dRectWithObject will return the 2D bounding box of given GameObject on screen.
Code
public static Rect GUI3dRectWithObject(GameObject go)
{
Vector3 cen = go.GetComponent<Renderer>().bounds.center;
Vector3 ext = go.GetComponent<Renderer>().bounds.extents;
Vector2[] extentPoints = new Vector2[8]
{
WorldToGUIPoint(new Vector3(cen.x-ext.x, cen.y-ext.y, cen.z-ext.z)),
WorldToGUIPoint(new Vector3(cen.x+ext.x, cen.y-ext.y, cen.z-ext.z)),
WorldToGUIPoint(new Vector3(cen.x-ext.x, cen.y-ext.y, cen.z+ext.z)),
WorldToGUIPoint(new Vector3(cen.x+ext.x, cen.y-ext.y, cen.z+ext.z)),
WorldToGUIPoint(new Vector3(cen.x-ext.x, cen.y+ext.y, cen.z-ext.z)),
WorldToGUIPoint(new Vector3(cen.x+ext.x, cen.y+ext.y, cen.z-ext.z)),
WorldToGUIPoint(new Vector3(cen.x-ext.x, cen.y+ext.y, cen.z+ext.z)),
WorldToGUIPoint(new Vector3(cen.x+ext.x, cen.y+ext.y, cen.z+ext.z))
};
Vector2 min = extentPoints[0];
Vector2 max = extentPoints[0];
foreach (Vector2 v in extentPoints)
{
min = Vector2.Min(min, v);
max = Vector2.Max(max, v);
}
return new Rect(min.x, min.y, max.x - min.x, max.y - min.y);
}
public static Rect GUI2dRectWithObject(GameObject go)
{
Vector3[] vertices = go.GetComponent<MeshFilter>().mesh.vertices;
float x1 = float.MaxValue, y1 = float.MaxValue, x2 = 0.0f, y2 = 0.0f;
foreach (Vector3 vert in vertices)
{
Vector2 tmp = WorldToGUIPoint(go.transform.TransformPoint(vert));
if (tmp.x < x1) x1 = tmp.x;
if (tmp.x > x2) x2 = tmp.x;
if (tmp.y < y1) y1 = tmp.y;
if (tmp.y > y2) y2 = tmp.y;
}
Rect bbox = new Rect(x1, y1, x2 - x1, y2 - y1);
Debug.Log(bbox);
return bbox;
}
public static Vector2 WorldToGUIPoint(Vector3 world)
{
Vector2 screenPoint = Camera.main.WorldToScreenPoint(world);
screenPoint.y = (float)Screen.height - screenPoint.y;
return screenPoint;
}
Reference: Is there an easy way to get on-screen render size (bounds)?
refer to this
It needs the game object with skinnedMeshRenderer.
Camera camera = GetComponent();
SkinnedMeshRenderer skinnedMeshRenderer = target.GetComponent();
// Get the real time vertices
Mesh mesh = new Mesh();
skinnedMeshRenderer.BakeMesh(mesh);
Vector3[] vertices = mesh.vertices;
for (int i = 0; i < vertices.Length; i++)
{
// World space
vertices[i] = target.transform.TransformPoint(vertices[i]);
// GUI space
vertices[i] = camera.WorldToScreenPoint(vertices[i]);
vertices[i].y = Screen.height - vertices[i].y;
}
Vector3 min = vertices[0];
Vector3 max = vertices[0];
for (int i = 1; i < vertices.Length; i++)
{
min = Vector3.Min(min, vertices[i]);
max = Vector3.Max(max, vertices[i]);
}
Destroy(mesh);
// Construct a rect of the min and max positions
Rect r = Rect.MinMaxRect(min.x, min.y, max.x, max.y);
GUI.Box(r, "");
I have a Quad whose vertices I'm printing like this:
public MeshFilter quadMeshFilter;
for(var vertex in quadMeshFilter.mesh.vertices)
{
print(vertex);
}
And, the localScale like this:
public GameObject quad;
print(quad.transform.localScale);
Vertices are like this:
(-0.5, -0.5), (0.5, 0.5), (0.5, -0.5), (-0.5, 0.5)
while the localScale is:
(6.4, 4.8, 0)
How is this possible - because the vertices make a square but localScale does not.
How do I use vertices and draw another square in front of the quad?
I am not well versed in the matters of meshes, but I believe I know the answer to this question.
Answer
How is this possible
Scale is a value which your mesh is multiplied in size by in given directions (x, y, z). A scale of 1 is default size. A scale of 2 is double size and so on. Your localSpace coordinates will then be multiplied by this scale.
Say a localSpace coordinate is (1, 0, 2), the scale however, is (3, 1, 3). Meaning that the result is (1*3, 0*1, 2*3).
How do I use vertices and draw another square in front of the quad?
I'd personally just create the object and then move it via Unity's Transform system. Since it allows you to change the worldSpace coordinates using transform.position = new Vector3(1f, 5.4f, 3f);
You might be able to move each individual vertex in WorldSpace too, but I haven't tried that before.
I imagine it is related to this bit of code though: vertices[i] = transform.TransformPoint(vertices[i]); since TransformPoint converts from localSpace to worldSpace based on the Transform using it.
Elaboration
Why do I get lots of 0's and 5's in my space coordinates despite them having other positions in the world?
If I print the vertices of a quad using the script below. I get these results, which have 3 coordinates and can be multiplied as such by localScale.
Print result:
Script:
Mesh mesh = GetComponent<MeshFilter>().mesh;
var vertices = mesh.vertices;
Debug.Log("Local Space.");
foreach (var v in vertices)
{
Debug.Log(v);
}
This first result is what we call local space.
There also exists something called WorldSpace. You can convert between local- and worldSpace.
localSpace is the objects mesh vertices in relation to the object itself while worldSpace is the objects location in the Unity scene.
Then you get the results as seen below, first the localSpace coordinates as in the first image, then the WorldSpace coordinates converted from these local coordinates.
Here is the script I used to print the above result.
Mesh mesh = GetComponent<MeshFilter>().mesh;
var vertices = mesh.vertices;
Debug.Log("Local Space.");
foreach (var v in vertices)
{
Debug.Log(v);
}
Debug.Log("World Space");
for (int i = 0; i < vertices.Length; ++i)
{
vertices[i] = transform.TransformPoint(vertices[i]);
Debug.Log(vertices[i]);
}
Good luck with your future learning process.
This becomes clear once you understand how Transform hierarchies work. Its a tree, in which parent transform [3x3] matrix (position, rotation, scale (rotation is actually a quaternion but lets assume its euler for simplicity so that math works). by extension of this philosophy, the mesh itself can be seen as child to the gameoobject that holds it.
If you imagine a 1x1 quad (which is what is described by your vertexes), parented to a gameobject, and that gameobject's Transform has a non-one localScale, all the vertexes in the mesh get multiplied by that value, and all the positions are added.
now if you parent that object to another gameObject, and give it another localScale, this will again multiply all the vertex positions by that scale, translate by its position etc.
to answer your question - global positions of your vertexes are different than contained in the source mesh, because they are feed through a chain of Transforms all the way up to the scene root.
This is both the reason that we only have localScale and not scale, and this is also the reason why non-uniform scaling of objects which contain rotated children can sometimes give very strange results. Transforms stack.
so the quest is this, I got an ARPointCloud with a bunch of 3d points and I'd like to select them based on a 2d frame from the perspective of the camera / screen.
I was thinking about converting the 2d frame to a 3d frustum and check if the points where the 3d frustum box, not sure if this is the ideal method, and not even sure how to do that.
Would anyone know how to do this or have a better method of achieving this?
Given the size of the ARKit frame W x H and the camera intrinsics we can create planes for the view frustum sides.
For example using C++ / Eigen we can construct our four planes (which pass
through the origin) as
std::vector<Eigen::Vector3d> _frustumPlanes;
frustumPlanes.emplace_back(Eigen::Vector3d( fx, 0, cx - W));
frustumPlanes.emplace_back(Eigen::Vector3d(-fx, 0, -cx));
frustumPlanes.emplace_back(Eigen::Vector3d( 0, fy, cy - H));
frustumPlanes.emplace_back(Eigen::Vector3d( 0, -fy, -cy));
We can then clip a 3D point by checking its position against the z < 0
half-space and the four sides of the frustum:
auto pointIsVisible = [&](const Eigen::Vector3d& P) -> bool {
if (P.z() >= 0) return false; // behind camera
for (auto&& N : frustumPlanes) {
if (P.dot(N) < 0)
return false; // outside frustum plane
}
return true;
};
Note that it is best to perform this clipping in 3D (before the projection) since points behind or near the camera or points far outside
the frustum can have unstable projection values (u,v).
I have loaded an wavefront object in Iphone OpenGL.
It can be rotated around x/y axis, panned around, zoomed in/out.
My task is - when object is tapped, highlight it's 2d center coordinates on screen for example like this: (Imagine that + is in the center of visible object.)
When loading OpenGL object I store it's:
object center position in world,
x,y,z position offset,
x,y,z rotation,
zoom scale.
When user taps on the screen, I can distinguish which object was tapped. But - as user can tap anywhere on object - Tapped point is not center.
When user touches an object, I want to be able to find out corresponding object visible approximate center coordinates.
How can I do that?
Most code in google I could find is meant - to translate 3d coordinates to 2d but without rotation.
Some variables in code:
Vertex3D centerPosition;
Vertex3D currentPosition;
Rotation3D currentRotation;
//centerPosition.x, centerPosition.y, centerPosition.z
//currentPosition.x, currentPosition.y, currentPosition.z
//currentRotation.x, currentRotation.y, currentRotation.z
Thank You in advance.
(To find out which object I tapped - re-color each object in different color, thus I know what color user tapped.)
object drawSelf function:
// Save the current transformation by pushing it on the stack
glPushMatrix();
// Load the identity matrix to restore to origin
glLoadIdentity();
// Translate to the current position
glTranslatef(currentPosition.x, currentPosition.y, currentPosition.z);
// Rotate to the current rotation
glRotatef(currentRotation.x, 1.0, 0.0, 0.0);
glRotatef(currentRotation.y, 0.0, 1.0, 0.0);
glRotatef(currentRotation.z, 0.0, 0.0, 1.0);
// Enable and load the vertex array
glEnableClientState(GL_VERTEX_ARRAY);
glEnableClientState(GL_NORMAL_ARRAY);
glVertexPointer(3, GL_FLOAT, 0, vertices);
glNormalPointer(GL_FLOAT, 0, vertexNormals);
// Loop through each group
if (textureCoords != NULL)
{
glEnableClientState(GL_TEXTURE_COORD_ARRAY);
glTexCoordPointer(valuesPerCoord, GL_FLOAT, 0, textureCoords);
}
for (OpenGLWaveFrontGroup *group in groups)
{
if (textureCoords != NULL && group.material.texture != nil)
[group.material.texture bind];
// Set color and materials based on group's material
Color3D ambient = group.material.ambient;
glMaterialfv(GL_FRONT_AND_BACK, GL_AMBIENT, (const GLfloat *)&ambient);
Color3D diffuse = group.material.diffuse;
glColor4f(diffuse.red, diffuse.green, diffuse.blue, diffuse.alpha);
glMaterialfv(GL_FRONT_AND_BACK, GL_DIFFUSE, (const GLfloat *)&diffuse);
Color3D specular = group.material.specular;
glMaterialfv(GL_FRONT_AND_BACK, GL_SPECULAR, (const GLfloat *)&specular);
glMaterialf(GL_FRONT_AND_BACK, GL_SHININESS, group.material.shininess);
glDrawElements(GL_TRIANGLES, 3*group.numberOfFaces, GL_UNSIGNED_SHORT, &(group.faces[0]));
}
if (textureCoords != NULL)
glDisableClientState(GL_TEXTURE_COORD_ARRAY);
glDisableClientState(GL_VERTEX_ARRAY);
glDisableClientState(GL_NORMAL_ARRAY);
// Restore the current transformation by popping it off
glPopMatrix();
ok, as I said, you'll need to apply the same transformations to the object center that are applied to the object's vertices by the graphics pipeline; only this time, the graphics pipeline won't help you - you'll have to do it yourself. And it involves some matrix calculations, so I'd suggest getting a good maths library like the OpenGL Maths library, which has the advatage that function names etc. are extremly similar to OpenGL.
step 1: transform the center form object coordinates to modelview coordinates
in your code, you set up your 4x4 modelview matrix like this:
// Load the identity matrix to restore to origin
glLoadIdentity();
// Translate to the current position
glTranslatef(currentPosition.x, currentPosition.y, currentPosition.z);
// Rotate to the current rotation
glRotatef(currentRotation.x, 1.0, 0.0, 0.0);
glRotatef(currentRotation.y, 0.0, 1.0, 0.0);
glRotatef(currentRotation.z, 0.0, 0.0, 1.0);
you need to multiply that matrix with the object center, and OpenGL does not help you with that, since it's not a maths library itself. If you use glm, there are functions like rotate(), translate() etc that function similiar to glRotatef() & glTranslatef(), and you can use them to build your modelview matrix. Also, since the matrix is 4x4, you'll have to append 1.f as 4th component to the object center ( called 'w-component' ), otherwise you can't multiply it with a 4x4 matrix.
Alternatively, you could query the current value of th modelview matrix directly from OpenGl:
GLfloat matrix[16];
glGetFloatv (GL_MODELVIEW_MATRIX, matrix);
but then you'll have to write your own code for the multiplication...
step 2: go from modelview coordinates to clip coordinates
from what you posted, I can't tell whether you ever change the projection matrix ( is there a glMatrixMode( GL_PROJECTION ) somewhere? ) - if you never touch the projection matrix, you can omit this step; otherwise you'll now need to multiply the transformed object center with the projection matrix as well.
step 3: perspective division
divide all 4 components of the object center by the 4th - then throw away the 4th component, keeping only xyz.
If you omitted step 2, you can also omit the division.
step 4: map the object center coordinates to window coordinates
the object center is now defined in normalized device coordinates, with x&y components in range [-1.f, 1.f]. the last step is mapping them to your viewport, i.e. to pixel positions. the z-component does not really matter to you anyway, so let's ignore z and call the x & y component obj_x and obj_y, respectively.
the viewport dimensions should be set somewhere in your code with glViewport( viewport_x, viewport_y, width, height ). from the function arguments, you can then calculate the pixel position for the center like this:
pixel_x = width/2 * obj_x + viewport_x + width/2;
pixel_y = height/2 * obj_y + viewport_y + height/2;
and that's basically it.