How do I use the gravity vector to correctly transform scene for augmented reality? - iphone

Question

I'm trying figure out how to get an OpenGL specified object to be displayed correctly according to the device orientation (ie. according to the gravity vector from the accelerometer, and heading from compass).
The GLGravity sample project has an example which is almost like this (despite ignoring heading), but it has some glitches. For example, the teapot jumps 180deg as the device viewing angle crosses the horizon, and it also rotates spuriously if you tilt the device from portrait into landscape. This is fine for the context of this app, as it just shows off an object and it doesn't matter that it does these things. But it means that the code just doesn't work when you attempt to emulate real life viewing of an OpenGL object according to the device's orientation. What happens is that it almost works, but the heading rotation you apply from the compass gets "corrupted" by the spurious additional rotations seen in the GLGravity example project.
Can anyone provide sample code that shows how to adjust correctly for the device orientation (ie. gravity vector), or to fix the GLGravity example so that it doesn't include spurious heading changes?
//Clear matrix to be used to rotate from the current referential to one based on the gravity vector
bzero(matrix, sizeof(matrix));
matrix[3][3] = 1.0;
//Setup first matrix column as gravity vector
matrix[0][0] = accel[0] / length;
matrix[0][1] = accel[1] / length;
matrix[0][2] = accel[2] / length;
//Setup second matrix column as an arbitrary vector in the plane perpendicular to the gravity vector {Gx, Gy, Gz} defined by by the equation "Gx * x + Gy * y + Gz * z = 0" in which we arbitrarily set x=0 and y=1
matrix[1][0] = 0.0;
matrix[1][1] = 1.0;
matrix[1][2] = -accel[1] / accel[2];
length = sqrtf(matrix[1][0] * matrix[1][0] + matrix[1][1] * matrix[1][1] + matrix[1][2] * matrix[1][2]);
matrix[1][0] /= length;
matrix[1][1] /= length;
matrix[1][2] /= length;
//Setup third matrix column as the cross product of the first two
matrix[2][0] = matrix[0][1] * matrix[1][2] - matrix[0][2] * matrix[1][1];
matrix[2][1] = matrix[1][0] * matrix[0][2] - matrix[1][2] * matrix[0][0];
matrix[2][2] = matrix[0][0] * matrix[1][1] - matrix[0][1] * matrix[1][0];
//Finally load matrix
glMultMatrixf((GLfloat*)matrix);
Here's a clarification showing how to get the elevation and tilt that are needed for gluLookAt solution as shown in my last answer:
// elevation comes from z component (0 = facing horizon)
elevationRadians = asin(gravityVector.z / Vector3DMagnitude(gravityVector));
// tilt is how far screen is from vertical, looking along z axis
tiltRadians = atan2(-gravityVector.y, -gravityVector.x) - M_PI_2;
Following up on Chris's suggestion: I'm not sure if I've got this all correct due to differing conventions of row/column order and heading cw or ccw. However the following code is what I came up with:
Vector3D forward = Vector3DMake(0.0f, 0.0f, -1.0f);
// Multiply it by current rotation matrix to get teapot direction
Vector3D direction;
direction.x = matrix[0][0] * forward.x + matrix[1][0] * forward.y + matrix[2][0] * forward.z;
direction.y = matrix[0][1] * forward.x + matrix[1][1] * forward.y + matrix[2][1] * forward.z;
direction.z = matrix[0][2] * forward.x + matrix[1][2] * forward.y + matrix[2][2] * forward.z;
heading = atan2(direction.z, direction.x) * 180 / M_PI;
// Use this heading to adjust the teapot direction back to keep it fixed
// Rotate about vertical axis (Y), as it is a heading adjustment
glRotatef(heading, 0.0, 1.0, 0.0);
When I run this code, the teapot behaviour has apparently "improved" eg. heading no longer flips 180deg when device screen (in portrait view) is pitched forward/back through upright. However, it still makes major jumps in heading when device (in landscape view) is pitched forward/back. So something's not right. It suggests that the above calculation of the actual heading is incorrect...

Answer 1

I finally found a solution that works. :-)
I dropped the rotation matrix approach, and instead adopted gluLookAt. To make this work you need to know the device "elevation" (viewing angle relative to horizon ie. 0 on horizon, +90 overhead), and the camera's "tilt" (how far the device is from vertical its x/y plane ie. 0 when vertical/portrait, +/-90 when horizontal/landscape), both of which are obtained from the device gravity vector components.
Vector3D eye, scene, up;
CGFloat distanceFromScene = 0.8;
// Adjust eye position for elevation (y/z)
eye.x = 0;
eye.y = distanceFromScene * -sin(elevationRadians); // eye position goes down as elevation angle goes up
eye.z = distanceFromScene * cos(elevationRadians); // z position is maximum when elevation is zero
// Lookat point is origin
scene = Vector3DMake(0, 0, 0); // Scene is at origin
// Camera tilt - involves x/y plane only - arbitrary vector length
up.x = sin(tiltRadians);
up.y = cos(tiltRadians);
up.z = 0;
Then you just apply the gluLookAt transformation, and also rotate the scene according to the device heading.
// Adjust view for device orientation
gluLookAt(eye.x, eye.y, eye.z, scene.x, scene.y, scene.z, up.x, up.y, up.z);
// Apply device heading to scene
glRotatef(currentHeadingDegrees, 0.0, 1.0, 0.0);

Answer 2

Try rotating the object depending upon iphone acceleration values.
float angle = -atan2(accelX, accelY);
glPushMatrix();
glTranslatef(centerPoint.x, centerPoint.y, 0);
glRotatef(angle, 0, 0, 1);
glTranslatef(-centerPoint.x, -centerPoint.y, 0);
glPopMatrix();
Where centerPoint is the middle point the object.

Answer 3

oo, nice.
GLGravity seems to get everything right except for the yaw. Here's what I would try. Do everything GLGravity does, and then this:
Project a vector in the direction you want the teapot to face, using the compass or whatever you so choose. Then multiply a "forward" vector by the teapot's current rotation matrix, which will give you the direction the teapot is facing. Flatten the two vectors to the horizontal plane and take the angle between them.
This angle is your corrective yaw. Then just glRotatef by it.
Whether or not the 3GS's compass is reliable and robust enough for this to work is another thing. Normal compasses don't work when the north vector is perpendicular to their face. But I just tried the Maps app on my workmate's 3GS and it seems to cope, so maybe they have got a mechanical solution in there. Knowing what the device is actually doing will help interpret the results it gives.
Make sure to test your app at the north and south poles once you're done. :-)

Answer 4

Getting a much more stable gravity-based reference, can now be done using CMMotionManager.
When starting motion updates with startDeviceMotionUpdates(), you can specify a reference frame.
This fuses the accelerometer, gyroscope and optionally (depending on chose reference frame) magnetometer data. Accelerometer data is pretty noisy and bouncy (any sideways motion of the device temporarily tilts the gravity vector by any device acceleration) and alone doesn't make a good reference.
I've been low-pass filtering the accelerometer data, which helps a bit but makes the system slow.