I have run the samples of ARKit4 with iPhone12. The dense point clouds are great, the mesh ... not too much. It's a low poly mesh (sort of). But it can still be made to look pretty if We could apply High quality texture on it.
If that train of thought is correct, can someone please guide me how to go about it? Any pointers would help.
Previously (pre-Lidar), we would fuse Depth data into a voxels and then do a marching cube. But with these points clouds, what's the way forward? or with the mesh output, whichever is the correct path to take, and what should I be reading on.
Related
I have a point-cloud and a rgb texture that fit together from a depth camera. I procedurally created a mesh from a selected part of the point-cloud implementing the quickhull 3D algorithm for mesh creation.
Now, somehow I need to apply the texture that I have to that mesh. Note that there can be multiple selected parts of the point-cloud thus making multiple objects that need the texture. The texture is just a basic 720p file that should be applied to the mesh material.
Basically I have to do this: https://www.andreasjakl.com/capturing-3d-point-cloud-intel-realsense-converting-mesh-meshlab/ but inside Unity. (I'm also using a RealSense camera)
I tried with a decal shader but the result is not precise. The UV map is completely twisted from the creation process, and I'm not sure how to generate a correct one.
UV and the mesh
I only have two ideas but don't really know if they'll work/how to do them.
Try to create a correct UV and then wrap the texture around somehow
Somehow bake colors to vertices and then use vertex colors to create the desired effect.
What other things could I try?
I'm working on quite a similar problem. But in my case I just want to create a complete mesh from the point cloud. Not just a quickhull, because I don't want to lose any depth information.
I'm nearly done with the mesh algorithm (just need to do some optimizations). Quite challenging now is to match the RGB camera's texture with the depth camera sensor's point cloud, because they of course have a different viewport.
Intel RealSense provides an interesting whitepaper about this problem and as far as I know the SDK corrects these different perspectives with uv mapping and provides a red/green uv map stream for your shader.
Maybe the short report can help you out. Here's the link. I'm also very interested in what you are doing. Please keep us up to date.
Regards
I'm trying to use ARKit's mesh scene reconstruction (with lidar) data to improve detected plane/geometry detection.
Right now, when pointing to a surface, ARKit gives me a very rough rectangle (far from actual surface's dimension). It happens almost instantly, but still far from the actual shape.
I'm trying to use this plane info, hit detection, and mesh data, to actually draw a smoothed rectangle around the detected surface. I don't expect full code, but rather just some hints of what to do.
Note: I'm using SceneKit (not RealityKit).
This is what I have so far for visualization:
Basically, I want the blue rectangle to better adjust to the real world shape by using the already available mesh data.
instead of using plane extents, use anchor.geometry
I am beating my head a little bit here for a while but I still could bot find a way to set up a matrix that projects my Unity game in a Tibianeske like manner:
Reading on tutorials on internet I could figure out how a normal orthographic perspective works, but tibia's one is kind of odd.
Digging over webs I found in here a guy (Clint Bellanger) who describes really well how to get the same perspective in blender's render according to him:
Start with a scene in 45 degree isometric. Video game style, where
the camera angle is Blender (60,0,45).
In Blender if you look at Buttons Window -> Scene -> Render Buttons ->
Format, you can set the render aspect ratio. Set AspY to half of
AspX. This is the same as taking regular rendered output and scaling
X by 50%. If you rendered a cube, the top of the cube will be a
perfect square (though at a 45 degree angle).
We can then use Blender nodes to rotate the result 45 degrees. The
output:
Note this started as a cube, so there's a lot of "vertical"
distortion. So you might have to scale meshes to 50% Z before using
this method. Also notice the Edge seems to be applied after the
Aspect, so the edge isn't distorted.
Blend file: http://clintbellanger.net/images/temp/UltimaVII.blend (I'm
a Nodes noob so there might be a smarter setup).
For kicks, here is that tower again. I pulled it into the above
workflow scene and scaled Z by 50%. Click "Re-render this layer" on
the first node to create the composite.
On his method, he used stuff like rescaling the render and changing the scale of models, Im convinced I could get along just with the 4x4matrix in unity(or in any other 3d environment really).
Hope someone more experienced with perks of 3D maths could help me to figure it out. Thank you! =D
What you ask for is a simple parallel projection. The typical orthographic projection is just a special case where the projection rays are perpendicular to the image plane. However, every parallel projection can be represented by an affine shear transformation followed by a standard orthogonal projection.
Im convinced I could get along just with the 4x4matrix in unity(or in any other 3d environment really).
Yes. Using default GL conventions here, all you have to do is to take the standard ortho matrix, post-multiply it by an appropriate shear matrix and use that as the projection matrix.
I can't understand how create procedural world. I found article about voxel terrain generation, but it's only video or pictures and i found some terrain engine, but i can't understand it. I wanna tutorial step by step how to create simple voxel object. I want to create terrain, but not cube terrain.
There's two very different approaches to this.
Voxel terrain stores solid information - not just the surface of the terrain but the whole volume. The perfect example of this would be Minecraft, where the 'terrain' includes not just the surface but caves and tunnels as well. 3DCoat is a sculpting program that uses voxels, it's a good way to see what can and can't be done with them. The ability to represent any 3d volume is the big advantage of voxels.
Traditional surface terrain stores only the surface: there's nothing underneath. This surface could be an polygon mesh, but most often it's a regular guads (4-sided polygons) that are procedurally generated from a heightmap (which is a bitmap that stores heights instead of colors). Polygon terrains are basically just regular 3d models that happen to look like terrain; heightmap terrains can be easier to work with because they are easier to 'sculpt' quickly, and can be procedurally modified for things like explosion craters or erosion. A good example of a heightmap terrain program is the Unity terrain editor or a standalone tool like Vue
In general, voxels are much more expensive than heightmap or polygon terrain - a 1km by 1km terrain at 1 meter resolution includes 1 million bits of data; a 1 km by 1km voxel terrain that runs 1 km deep would be 1 billion samples (!) That can be cut down be smart enncoding (the hot trend here is sparse voxel octrees) but it's still a lot of data to manage. That's one of the reasons Minecraft has to be so blocky.
You could generate either voxels or heightmaps procedurally or by hand. Vterrain.org is a great resource for different techniques doing terrain.
I have a computer vision set up with two cameras. One of this cameras is a time of flight camera. It gives me the depth of the scene at every pixel. The other camera is standard camera giving me a colour image of the scene.
We would like to use the depth information to remove some areas from the colour image. We plan on object, person and hand tracking in the colour image and want to remove far away background pixel with the help of the time of flight camera. It is not sure yet if the cameras can be aligned in a parallel set up.
We could use OpenCv or Matlab for the calculations.
I read a lot about rectification, Epipolargeometry etc but I still have problems to see the steps I have to take to calculate the correspondence for every pixel.
What approach would you use, which functions can be used. In which steps would you divide the problem? Is there a tutorial or sample code available somewhere?
Update We plan on doing an automatic calibration using known markers placed in the scene
If you want robust correspondences, you should consider SIFT. There are several implementations in MATLAB - I use the Vedaldi-Fulkerson VL Feat library.
If you really need fast performance (and I think you don't), you should think about using OpenCV's SURF detector.
If you have any other questions, do ask. This other answer of mine might be useful.
PS: By correspondences, I'm assuming you want to find the coordinates of a projection of the same 3D point on both your images - i.e. the coordinates (i,j) of a pixel u_A in Image A and u_B in Image B which is a projection of the same point in 3D.