OpenGL/GLM - Combining(?) Quaterions Accurately from Keyboard/Mouse and other sources ...

[tmason]

Hello,

I would like to combine mouse and keyboard inputs with the Oculus Rift to create a smooth experience for the user. The goals are:

- Positional movement 100% controlled by the keyboard relative to the direction the person is facing.

- Orientation controlled 100% by HMD devices like the Oculus Rift.

- Mouse orbit capabilities adding to the orientation of the person using the Oculus Rift. For example, if I am looking left I can still move my mouse to "move" more leftward.

Now, I have 100% working code for when someone doesn't have an Oculus Rift, I just don't know how to combine the orientation and other elements of the Oculus Rift to my already working code to get it 100%.

Anyway, here is my working code for controlling the keyboard and mouse without the Oculus Rift:

/*
 
Variables
 
*/
 
glm::vec3 DirectionOfWhereCameraIsFacing;
glm::vec3 CenterOfWhatIsBeingLookedAt;
glm::vec3 PositionOfEyesOfPerson;
glm::vec3 CameraAxis;
glm::vec3 DirectionOfUpForPerson;
glm::quat CameraQuatPitch;
float     Pitch;
float	  Yaw;
float	  Roll;
float     MouseDampingRate;
float     PhysicalMovementDampingRate;
glm::quat CameraQuatYaw;
glm::quat CameraQuatRoll;
glm::quat CameraQuatBothPitchAndYaw;
glm::vec3 CameraPositionDelta;
 
/*
 
Inside display update function.
 
*/
 
DirectionOfWhereCameraIsFacing = glm::normalize(CenterOfWhatIsBeingLookedAt - PositionOfEyesOfPerson);
CameraAxis = glm::cross(DirectionOfWhereCameraIsFacing, DirectionOfUpForPerson);
CameraQuatPitch = glm::angleAxis(Pitch, CameraAxis);
CameraQuatYaw = glm::angleAxis(Yaw, DirectionOfUpForPerson);
CameraQuatRoll = glm::angleAxis(Roll, CameraAxis);
CameraQuatBothPitchAndYaw = glm::cross(CameraQuatPitch, CameraQuatYaw);
CameraQuatBothPitchAndYaw = glm::normalize(CameraQuatBothPitchAndYaw);
DirectionOfWhereCameraIsFacing = glm::rotate(CameraQuatBothPitchAndYaw, DirectionOfWhereCameraIsFacing);
PositionOfEyesOfPerson += CameraPositionDelta;
CenterOfWhatIsBeingLookedAt = PositionOfEyesOfPerson + DirectionOfWhereCameraIsFacing * 1.0f;
Yaw *= MouseDampingRate;
Pitch *= MouseDampingRate;
CameraPositionDelta = CameraPositionDelta * PhysicalMovementDampingRate;
View = glm::lookAt(PositionOfEyesOfPerson, CenterOfWhatIsBeingLookedAt, DirectionOfUpForPerson);
ProjectionViewMatrix = Projection * View;

The Oculus Rift provides orientation data via their SDK and can be accessed like so:

/*
 
Variables
 
*/
 
ovrMatrix4f OculusRiftProjection;
glm::mat4	Projection;
OVR::Quatf	OculusRiftOrientation;
glm::quat	CurrentOrientation;
 
/*
 
Partial Code for retrieving projection and orientation data from Oculus SDK
 
*/
 
OculusRiftProjection = ovrMatrix4f_Projection(MainEyeRenderDesc[l_Eye].Desc.Fov, 10.0f, 6000.0f, true);
 
for (int o = 0; o < 4; o++){
	for (int i = 0; i < 4; i++) {
		Projection[o][i] = OculusRiftProjection.M[o][i];
	}
}
 
Projection = glm::transpose(Projection);
 
OculusRiftOrientation = PredictedPose.Orientation.Conj();
 
CurrentOrientation.w = OculusRiftOrientation.w;
CurrentOrientation.x = OculusRiftOrientation.x;
CurrentOrientation.y = OculusRiftOrientation.y;
CurrentOrientation.z = OculusRiftOrientation.z;
 
CurrentOrientation = glm::normalize(CurrentOrientation);

After that last line the glm based quaterion "CurrentOrientation" has the correct information which, if plugged straight into the MVP matrix and sent into OpenGL will allow you to move your head around in the environment without issue.

Now, my problem is how to combine the two parts together successfully.

When I have done this in the past it results in the rotation stuck in place (when you turn your head left you keep rotating left as opposed to just rotating in the amount that you turned) and the fact that I can no longer accurately determine the direction the person is facing so that my position controls work.

How can I successfully achieve my goals?

Thank you for your time.

[tmason]

Hello to All,

So I completely resolved the problem down to a simple issue of getting the correct rotation.

I think a video explains it best: [URL="http://tinypic.com/player.php?v=nyi7g2%3E&s=8"]http://tinypic.com/player.php?v=nyi7g2%3E&s=8[/URL]

The link takes you to a free video hosting service tinypic.com.

What happens is that rotating left or right is OK until after you go past approximately 75 degrees in which case the viewpoint goes crazy.

Additionally, looking upwards in the rift actually makes you look down and looking down makes you look up. It also rotates crazily after 75 degrees.

Any ideas? Thank you for your time.

Anyway, here is the new code:

/*
 
Variables
 
*/
 
glm::vec3 DirectionOfWhereCameraIsFacing;
glm::vec3 RiftDirectionOfWhereCameraIsFacing;
glm::vec3 CenterOfWhatIsBeingLookedAt;
glm::vec3 RiftCenterOfWhatIsBeingLookedAt;
glm::vec3 PositionOfEyesOfPerson;
glm::vec3 CameraAxis;
glm::vec3 DirectionOfUpForPerson;
glm::quat CameraQuatPitch;
float     Pitch;
float	  Yaw;
float	  Roll;
float     MouseDampingRate;
float     PhysicalMovementDampingRate;
glm::quat CameraQuatYaw;
glm::quat CameraQuatRoll;
glm::quat CameraQuatBothPitchAndYaw;
glm::vec3 CameraPositionDelta;
 
/*
 
Inside display update function.
 
*/

DirectionOfWhereCameraIsFacing = glm::normalize(CenterOfWhatIsBeingLookedAt - PositionOfEyesOfPerson);
CameraAxis = glm::cross(DirectionOfWhereCameraIsFacing, DirectionOfUpForPerson);
CameraQuatPitch = glm::angleAxis(CurrentCameraViewingSettings.Pitch, CameraAxis);
CameraQuatYaw = glm::angleAxis(CurrentCameraViewingSettings.Yaw, DirectionOfUpForPerson);
CameraQuatRoll = glm::angleAxis(CurrentCameraViewingSettings.Roll, CameraAxis);
CameraQuatBothPitchAndYaw = glm::cross(CameraQuatPitch, CameraQuatYaw);
CameraQuatBothPitchAndYaw = glm::normalize(CameraQuatBothPitchAndYaw);
DirectionOfWhereCameraIsFacing = glm::rotate(CameraQuatBothPitchAndYaw, DirectionOfWhereCameraIsFacing);
CameraUpdateMutex.lock();
RiftDirectionOfWhereCameraIsFacing = DirectionOfWhereCameraIsFacing;
RiftDirectionOfWhereCameraIsFacing = glm::rotate(CurrentOrientation, DirectionOfWhereCameraIsFacing);
CameraUpdateMutex.unlock();
PositionOfEyesOfPerson += CameraPositionDelta;
CenterOfWhatIsBeingLookedAt = PositionOfEyesOfPerson + DirectionOfWhereCameraIsFacing * 1.0f;
CameraUpdateMutex.lock();
RiftCenterOfWhatIsBeingLookedAt = PositionOfEyesOfPerson + RiftDirectionOfWhereCameraIsFacing * 1.0f;
CameraUpdateMutex.unlock();
CurrentCameraViewingSettings.Yaw *= CurrentCameraViewingSettings.MouseDampingRate;
CurrentCameraViewingSettings.Pitch *= CurrentCameraViewingSettings.MouseDampingRate;
CameraPositionDelta = CameraPositionDelta * CurrentCameraViewingSettings.PhysicalMovementDampingRate;
View = glm::lookAt(PositionOfEyesOfPerson, CenterOfWhatIsBeingLookedAt, DirectionOfUpForPerson);

And here is the updated code within the display function.

Again, partial listing:

/*

Variables

*/

ovrMatrix4f OculusRiftProjection;
glm::mat4	RiftProjection;
OVR::Quatf	OculusRiftOrientation;
glm::quat	CurrentOrientation;
glm::vec3	CurrentEulerAngles;
glm::mat4	RiftView;
glm::mat4	RiftProjectionViewMatrix;

/*

Partial Code for retrieving projection and orientation data from Oculus SDK

*/

ovrEyeType l_Eye = (*MainRiftDeviceDesc).EyeRenderOrder[l_EyeIndex];
ovrPosef l_EyePose = ovrHmd_BeginEyeRender((*MainRiftDevice), l_Eye);
OVR::Posef PredictedPose = ovrHmd_GetSensorState((*MainRiftDevice), m_HmdFrameTiming.ScanoutMidpointSeconds).Predicted.Pose;

glViewport(MainEyeRenderDesc[l_Eye].Desc.RenderViewport.Pos.x,
	MainEyeRenderDesc[l_Eye].Desc.RenderViewport.Pos.y,
	MainEyeRenderDesc[l_Eye].Desc.RenderViewport.Size.w,
	MainEyeRenderDesc[l_Eye].Desc.RenderViewport.Size.h);

OculusRiftProjection = ovrMatrix4f_Projection(MainEyeRenderDesc[l_Eye].Desc.Fov, 10.0f, 6000.0f, true);

for (int o = 0; o < 4; o++){
	for (int i = 0; i < 4; i++) {
		RiftProjection[o][i] = OculusRiftProjection.M[o][i];
	}
}

RiftProjection = glm::transpose(RiftProjection);

OculusRiftOrientation = PredictedPose.Orientation;

OculusRiftOrientation.GetEulerAngles<OVR::Axis_X, OVR::Axis_Y, OVR::Axis_Z, OVR::Rotate_CCW, OVR::Handed_R>
	(&CurrentEulerAngles.x, &CurrentEulerAngles.y, &CurrentEulerAngles.z);

CurrentOrientation = glm::quat(CurrentEulerAngles);

RiftView = glm::lookAt(PositionOfEyesOfPerson, RiftCenterOfWhatIsBeingLookedAt, DirectionOfUpForPerson);
RiftView = glm::translate(RiftView, glm::vec3(l_EyePose.Position.x, l_EyePose.Position.y, l_EyePose.Position.z));
RiftProjectionViewMatrix = RiftProjection * RiftView;