i saw this example and i would like to measure an arc trajectory first and after that get the live object trajectory position accordingly.
The iPhone camera and the object are always mounted/located on the same position/distance. Only the object can only rotate in its Z axis.
Is is possible to get live position/angle values using its known trajectory in AI Vision?
The sample rate would be 1/second, like taking a picture sending to AI Vision engine and get an angle back?
Related
I'm trying to make simple VR HMD that can track user's head movement in 6 DoF(Forward and Backward, Left and Right, Up and Down, Pitch, Yaw, Roll). Applying 6 DoF Accelerometer and Gyroscope(GY-521, MPU-6050) made me can track user's rotation, but I still can not track positional differences.
I have googled if there is a way to calculate its spatial velocity or acceleration with accelerometer, but there weren't. Because output values of accelerometer mean not linear acceleration, but angular acceleration.
I wonder what kind of sensor or model can measure rotation and linear motion at the same time. Also, How standalone HMD can measure user's motion with its IMU?
I am trying to build a smile detector over a real time video (front cam) using Uibezierpath over the screen coordinates by detecting face landmarks using VNDetectFaceLandmarksRequest and "Landmarks.outerlips", calculating Y offset between upper points, without using CoreML ideally - but I seem only able to get the normalised points for the landmark, where these points have their own coordinate system. I'm not sure how to convert each point to the screen coordinate system.
This answer from #Rickster seems to be in the right direction but I'm not able to fully grasp next steps:
How to convert normalized points retrived from VNFaceLandmarkRegion2D
Current output:
Desired result:
ARKit updates many intrinsic (and extrinsic) parameters of the ARCamera from frame to frame. I'd like to know if it also takes Radial Lens Distortion into consideration (like in AVCameraCalibrationData class that ARKit doesn't use), and fix the video frames' distortion appropriately (distort/undistort operations) for back iPhone and iPad cameras?
var intrinsics: simd_float3x3 { get }
As we all know, the Radial Lens Distortion greatly affects the 6 DOF pose estimation accuracy when we place undistorted 3D objects in distorted by a lens real world scene.
var lensDistortionLookupTable: Data? { get }
/* A map of floating-point values describing radial */
/* lens distortions in AVCameraCalibrationData class */
If Lens Distortion's math in ARKit is available in API, where I can find it?
Although it's not explicitly stated, I'm certain that ARKit is correcting for non-linear lens distortion. Lens distortion (and inverse distortion) lookup tables exist in iOS11 and are available via AVCameraCalibrationData, but they are not exposed by ARKit, presumably because there is no need for them since you're already working with rectified coordinates.
Whether or not the distortion model parameters are the same for each device model (i.e. exact same values for each iPhone 7) it's an interesting question. I don't have access to multiple phones of the same model, but this shouldn't be hard to figure out for someone who does.
source
As an exapmple from : https://github.com/verebes1/ARKit-Multiplayer
QR marker detection
With the help of Apple's Vision now it's possible to recognize QR marker in camera's videofeed and track it while it is in the field of view. The framework provides us the coordinates of the QR marker square corners in the screen's coordinate system.
QR marker pose estimation
The next thing you probably want to do after detecting the QR markers is to obtain the camera pose from them.
To perform QR marker pose estimation you need to know the calibration parameters of your camera. This is the camera matrix and distortion coefficients. Each camera lens has unique parameters, such as focal length, principal point, and lens distortion model. The process of finding intrinsic camera parameters is called camera calibration. The camera calibration process is important for Augmented Reality applications because it describes the perspective transformation and lens distortion on an output image. To achieve the best user experience with Augmented Reality, visualization of an augmented object should be done using the same perspective projection.
At the end, what you get after the calibration is the camera matrix: a matrix of 3x3 elements with the focal distances and the camera center coordinates (a.k.a intrinsic parameters), and the distortion coefficients: a vector of 5 elements or more that models the distortion produced by your camera. The calibration parameters are pretty the same for most of iDevices.
With the precise location of marker corners, we can estimate a transformation between our camera and a marker in 3D space. This operation is known as pose estimation from 2D-3D correspondences. The pose estimation process finds an Euclidean transformation (that consists only of rotation and translation components) between the camera and the object.
The C is used to denote the camera center. The P1-P4 points are 3D points in the world coordinate system and the p1-p4 points are their projections on the camera's image plane. Our goal is to find relative transformation between a known marker position in the 3D world (p1-p4) and the camera C using an intrinsic matrix and known point projections on image plane (P1-P4).
OpenCV functions are used to calculate the QR marker transformation in such a way that it minimizes the reprojection error, that is the sum of squared distances between the observed projection's imagePoints and the projected objectPoints. The estimated transformation is defined by rotation (rvec) and translation components (tvec). This is also known as Euclidean transformation or rigid transformation. At the end we get rotation quaternion and a translation matrix of the QR marker.
Integration into Apple's ARKit
The final part is the integration of all the information about QR marker's pose into the 3D scene created by ARKit. ARKit uses Visual Inertial Odometry (VIO) to accurately track the world around it. VIO fuses camera sensor data with CoreMotion data. These two inputs allow the device to sense how it moves within a room with a high degree of accuracy, and without any additional calibration. All the rendering stuff is based on Apple's Metal and Apple's SceneKit above it.
In order to render SceneKit's node on our QR marker in a proper way we need to create a model matrix of our QR marker from the quaternion and translation matrix we've got from OpenCV. The next step is to multiply QR marker's model matrix by SceneKit scene virtual camera's transform matrix. As a result, we can see a custom node (Axes node in our project) that repeats all the QR marker's movements in the real world while it's in the field of view of the iPhone's camera and if it is not - it stays on the last updated position so we can examine it around.
I've set up an animation of a tugboat [from VRML library] using the Virtual Reality Animation objects. but am having trouble with viewing the rotation of the boat.
To be more specific: I have a simulator going, where I calculate from rigid body dynamics the trajectory of it in time. This is, I have x, y, z, phi, theta, psi vs. time. I associate the translations and rotations to the node corresponding to the boat. When pressing play, I can see the translation and rotation which is not as expected.
Not sure what the problem could be. I tried to add one Transform in the .wrl for each of the rotational degrees of freedom, but I found it weird as when I give rotation in one direction I see the object rotating and translating other directions as well.
Any help is most welcome.
I want to use the iPhones's accelerometer to detect motions while driving. I'm a bit confused what the accelerometer actually measures, especially when driving a curve.
As you can see in the picture, a car driving a curve causes two forces. One is the centripetal force and one is the velocity. Imagine the iPhone is placed on the dashboard with +y-axis is pointing to the front, +x-axis to the right and +z-axis to the top.
My Question is now what acceleration will be measured when the car drives this curve. Will it measure g-force on the -x-axis or will the g-force appear on the +y axis?
Thanks for helping!
UPDATE!
For thoses interested, as one of the answers suggested it measures both. The accelerometer is effected by centrifugal force and velocity resulting in an acceleration vector that is a combination of these two.
I think it will measure both. But don't forget that the sensor will measure gravity as well. So when your car is not moving, you will still get accelerometer readings. A nice talk on sensors in smartphones http://www.youtube.com/watch?v=C7JQ7Rpwn2k&feature=results_main&playnext=1&list=PL29AD66D8C4372129 (it's on android, but the same type of sensors are used in iphone).
Accelerometer measures acceleration of resultant force applied to it (velocity is not a force by the way). In this case force is F = g + w + c i.e. vector sum of gravity, centrifugal force (reaction to steering centripetal force, points from the center of the turn) and car acceleration force (a force changing absolute value of instantaneous velocity, points along the velocity vector). Providing Z axis of accelerometer always points along the gravity vector (which is rare case for actual car) values of g, w and c accelerations can be accessed in Z, X and Y coordinates respectively.
Unless you are in free fall the g-force (gravity) is always measured. If I understand your setup correctly, the g-force will appear on the z axis, the axis that is vertical in the Earth frame of reference. I cannot tell whether it will be +z or -z, it is partly convention so you will have to check it for yourself.
UPDATE: If the car is also going up/downhill then you have to take the rotation into account. In other words, there are two frames of reference: the iPhone's frame of reference and the Earth frame of reference. If you would like to deal with this situation, then please ask a new question.