Add WebXR capturing functionality to Spector.js, and WebXR sample

[dmliao]

This change adds a new initialization option to Spector.js, called enableXRCapture, that when enabled allows Spector.js to hook onto and capture commands from WebXR sessions.

This works by doing a partial replacement of the WebXR API, so that information such as active sessions and the WebGL context used in the session are visible to Spector.js, and requires Spector.js to be initialized and used like:

var spector = new SPECTOR({ enableXRCapture: true });

...
// enter a WebXR Session here
...

// once we have an active session, we can capture the context with the following:
spector.captureXRContext();

Without the initialization option, Spector.js will not attempt to replace any of the WebXR APIs, and getXRContext() will automatically produce an error. Either way, captures of non-WebXR contexts and canvases should be unchanged.

There is also a new sample to demonstrate WebXR captures under samples/js/webxr.js, which produces a default WebXR scene and allows you to take a capture by pressing the trigger button of either controller.

Caveats

• The changes to the some of the helpers (formatHelper, readPixelsHelper, and baseWebGLObject) were to deal with errors that showed up from unexpected inputs from commands that would show up in WebXR sessions; I haven't done very thorough testing to see how they would interact with non-XR contexts, nor did I dig super deep to figure out why those changes were needed.
• Even with the fix to get screencaps working with WebXR, they only operate on sessions that use WebXR Layers; without layers, WebXR produces opaque framebuffers that cannot be read from. This should be documented somewhere, but I haven't put that anywhere yet.
• In general, I haven't touched the documentation regarding this new feature and would welcome feedback on what / where to put it.

[sebavan]

@RaananW would you be able to test it once it will go in ???

[sebavan]

Should it be enabled by default in the extension ? It looks good at a first glance, I will have a deeper look on Friday :-)

[RaananW]

Wow, amazing work!
I am just wondering about emulator(s) support - as the browser-based emulators are providing their own XR implementation - will the order of loading spector and the extension hurt the capture? Or will it work anyhow, since they mainly polyfill the XR object?

[dmliao]

@sebavan - I didn't think it's too relevant for the desktop extension right now since the extension isn't really used with WebXR beyond interacting with browser-based WebXR emulators, and those already work with Spector.js as-is [since the emulators render to a canvas that the existing extension can spy]. The extension doesn't have any ability to actually capture the XR context either, since it isn't added to the available contexts / canvases list. But I mostly know about the WebXR use case for immersive VR experiences; there might be use-cases for MR or AR that I'm overlooking.

@RaananW - Theoretically, if emulators completely replace XR APIs that Spector also replaces, then yes, Spector will have to load after the emulator in order to get access to the XR session. Looking at immersive-web-emulator, it doesn't look like they touch any of the APIs changed here (which are mostly creating sessions and layers), but I can't actually verify. The emulator extensions run before any other client code, so if you load Spector.js into your WebXR experience, it'll always load after the emulator has already polyfilled the WebXR API. I did test the library on experiences using the emulator, and I've seen no issues so far.

[sebavan]

Could you revert your changes to the dist and extensions folders ?

[sebavan]

@dmliao I wonder if #258 (comment) could be integrated in your PR ? it might be a bit more generic with scope and I d like to avoid breaking changes later :-)

[dmliao]

The PR got automatically closed when I tried to rebase...anyway, this now incorporates the API changes that were introduced in #259.

[multun]

I believe this change prevents the library from loading on firefox, even with the website examples: Uncaught ReferenceError: XRWebGLBinding is not defined

[sebavan]

ohhhh :-( @dmliao could you have a look ?

[dmliao]

If you’re willing to wait a week, I can look into this then. I’m currently on holiday and don’t have computer access until then. If the extension / library has the WebXR flag enabled, see first if turning it off fixes things. The code shouldn’t touch WebXR at all in that case.

…

On Sep 4, 2023 at 11:50 AM, <sebavan ***@***.***)> wrote: ohhhh :-( @dmliao (https://github.com/dmliao) could you have a look ? — Reply to this email directly, view it on GitHub (#257 (comment)), or unsubscribe (https://github.com/notifications/unsubscribe-auth/AAF37JK4ACA3HOW5HJTGFOTXYYWKFANCNFSM6AAAAAAWEL6BMA). You are receiving this because you were mentioned.Message ID: ***@***.***>

[multun]

If you’re willing to wait a week, I can look into this then. I’m currently on holiday and don’t have computer access until then.

enjoy your holidays! thanks for offering to help

If the extension / library has the WebXR flag enabled, see first if turning it off fixes things. The code shouldn’t touch WebXR at all in that case.

The issue occurs at the <script type="text/javascript" src="spector.bundle.js"></script> stage. I think that's because a class inherits from XRWebGLBinding, which is an experimental API unavailable in firefox.

[RaananW]

it will probably be possible to "create" these objects (empty objects) if they don't exist, while not expecting WebXR to work in an environment where the objects are not defined. Otherwise there is no way to optionally extend from a different class.

[sebavan]

@dmliao no worries enjoy your holidays and let s fix it once you are back :-)