Getting compass-like behavior from quaternion - unity3d

Question

Suppose you have a camera projection matrix, i.e. camera translation vector + rotation quaternion, like every typical camera, it is able to move and rotate in any direction. And independent of it's rotation whether it is looking forward, upward or downward I need to show a compass-like gauge pointing where the camera is targeted at.
The problem is that when the camera is pointed downwards the rotation of camera around it's optical center defines the value of the compass, but when the camera points forward, the rotation of camera around it's center no longer affects the value of compass, in this case the direction of camera defines the value of compass.
It get's more ugly when the camera is tilted downwards only 45 degrees, in this case it is not even clear whether the rotation around camera center affects rotation of compass.
So is there an elegant way of getting the compass value based on arbitrary camera projection matrix / quaternion?
Thank you in advance!

Answer 1

If you want just an arrow pointing at the target its:
Transform camera = Camera.main.transform;
Transform target = Target.transform;
Vector3 relativePosition = target.position - camera.position;
Vector3 targetRelative = Vector3.ProjectOnPlane(relativePosition, camera.forward);
float angle = Angle360(camera.up, targetRelative, camera.forward);
Compass.transform.rotation = Quaternion.Euler(0, 0, angle);
The angle function is:
float Angle360(Vector3 from, Vector3 to, Vector3 normal)
{
float dot = Vector3.Dot(from, to);
float det = Vector3.Dot(normal, Vector3.Cross(from, to));
return Mathf.Atan2(det, dot)*Mathf.Rad2Deg;
}
Here is how you can get the direction of the compass in worldspace:
Project the camera direction and target position on the XZ plane
Transform camera = Camera.main.transform;
Transform target = Target.transform;
Vector3 cameraWorldDirXZ = Vector3.ProjectOnPlane(camera.forward, Vector3.up).normalized;
Vector3 targetWorldDirXZ = Vector3.ProjectOnPlane(target.position, Vector3.up).normalized;
The angle between the cameraWorldDirXZ and targetWorldDirXZ is the angle of your compass needle.
But i don't think this will behave like you think it will. This gives you the angle that you need to rotate the camera.forward vector around the y axis to face the target. If you rotate around camera.forward you don't change either the camera.forward vector or the y axis so the compass wont change.
You might want to try a compass in local space. For that you project onto the camera XZ plane:
Vector3 cameraLocalDirXZ = camera.forward;
Vector3 targetLocalDirXZ = Vector3.ProjectOnPlane(target.position, camera.up).normalized;
Again the angle between the cameraLocalDirXZ and targetLocalDirXZ is the angle of your compass needle. This gives you the angle you need to rotate camera.forward around camera.up to face the target. Note that when you rotate around camera.forward it will change camera.up so it will change the compass direction.

Answer 2

If anyone stumbles upon this problem, the solution (thanks to #Pluto) is very simple, multiply your camera quaternion over three axis vectors (0,0,1), (0,1,0), (1,0,0), you will get three vectors defining coordinate system of your camera, project those three vectors onto your plane, find centroid of your three projected points and voila you have compass direction.
Here's the piece of code for that:
var rotation = /* Your quaternion */;
var cameraOrtX = rotation * new Vector3 (1, 0, 0);
var cameraOrtY = rotation * new Vector3 (0, 1, 0);
var cameraOrtZ = rotation * new Vector3 (0, 0, 1);
var cameraOrtPX = Vector3.ProjectOnPlane(cameraOrtX, new Vector3(0, 1, 0));
var cameraOrtPY = Vector3.ProjectOnPlane(cameraOrtY, new Vector3(0, 1, 0));
var cameraOrtPZ = Vector3.ProjectOnPlane(cameraOrtZ, new Vector3(0, 1, 0));
var centroid = (cameraOrtPX + cameraOrtPY + cameraOrtPZ) / 3.0f;