WebXR VR Examples are broken

[tbirdso]

High-level description

The VR and SkyboxViewer examples appear to be nonfunctional. In both examples the stereoscopic camera views react improperly to headset movements and each eye even appears to drift independently while the headset is at rest.

Steps to reproduce

1. Use a compatible HMD browser or following the developer notes in vtk.js WebXR documentation to install the Mozilla WebXR emulator in a compatible desktop browser
2. Navigate to the example
3. Launch in VR

Detailed behavior

• Dual views drift independently
• Camera movement in each eye does not reflect headset movement
• Camera sometimes moves more rapidly when headset is in motion
• Camera sometimes drifts while headset is at rest

Expected behavior

Stereoscopic views move together in response to headset movements

Environment

• vtk.js: master
• OS: Windows 10, Oculus HMD (Android)
• Browser: Chrome 101.0.4951.67, Oculus Browser

Other Notes

Tested with both the Mozilla WebXR emulator and Meta Quest 2 HMD.

I have not made any recent changes in vtk.js so I am having to start from scratch on this investigation. Initial suspicion is that something changed in how we are handling camera parameters or 3D styles since the last time I touched these files.

WebXR AR applications are still fully functional.

[tbirdso]

cc @floryst

[tbirdso]

Verified that last stable commit is b5b1c22, breaks in c84b5a0

[floryst]

FYI @JiayiXuu, since you can provide input on how your codebase may be interacting with the webXR code.

[tbirdso]

For the record, it looks like the error occurs when computeLightTransform from #2415 is called, but I don't see any issues in that method itself. Something strange is going on in terms of how that block interacts with how we are setting the position in VR. Will continue to investigate.

[tbirdso]

Current hypothesis is that #2415 exposed some issues in VR rendering that already existed, such as where vtkCamera.computeViewParametersFromPhysicalMatrix does not explicitly set the view matrix that computeCameraLightTransform expects to find. Adjusting this results in correct behavior where the camera rotates with the HMD, though the camera position appears fixed at the origin.

[JiayiXuu]

For the record, it looks like the error occurs when computeLightTransform from #2415 is called, but I don't see any issues in that method itself. Something strange is going on in terms of how that block interacts with how we are setting the position in VR. Will continue to investigate.

In short - to fix the issue, maybe try removing all the calls to ComputeCameraLightTransform() in Core/Camera. If this doesn't fix it, try removing the calls to ComputeCameraLightTransform() in Core/Renderer.

Long story:
ComputeCameraLightTransform() only effects cameraLightMatrix, which supposedly calculate the transform matrix between camera and light in view space. The one thing I am sure about is that the current implementation of ComputeCameraLightTransform() is wrong, which is mentioned in #2415. We decided to come back the issue later because cameraLightMatrix is used only in one place, updateLightsGeometryToFollowCamera() in Core/Renderer. We thought it won't effect other code - apparently it does.

I can think of one thing that might break VR scene, getViewMatrix() is called in ComputeCameraLightTransform() and that might update view matrix at the wrong time.

[JiayiXuu]

ComputeCameraLightTransform() is only used for camera light, so unless the user really wants a camera light, this function is redundant and shouldn't be called.

If removing the calls fix the issue let me know, I'll open an MR and remove all ComputeCameraLightTransform() calls, just to make sure it doesn't break other code in the future until I have the time to look further into it. Thanks @tbirdso

[tbirdso]

Thanks for the background @JiayiXuu! I agree that the issue probably lies with the view matrix being updated unexpectedly. Removing the single ComputeCameraLightTransform call at https://github.com/Kitware/vtk-js/blob/master/Sources/Rendering/Core/Camera/index.js#L75 seems to fix the issue. There may be a more long-term solution involving how the view matrix is set/updated in computeViewParametersFromPhysicalMatrix and computeViewParametersFromViewMatrix.

[JiayiXuu]

Created #2447 to fix the issue.

[tbirdso]

Closed by #2447.