Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Fix: add world tracking on Apple Vision Pro for drawables to be visible #3326

Merged
merged 1 commit into from
Jul 30, 2024

Conversation

okwasniewski
Copy link
Contributor

@okwasniewski okwasniewski commented Jul 17, 2024

This PR fixes the last issue with Apple Vision Pro rendering on a real device.

Note: While reviewing, look only at this commit: b83ca5e another one has already open PR (#3325)

Currently, when we don't attach the device anchor to the drawable (cp_drawable_set_device_anchor) the drawables are not presented on the device. To retrieve the device anchor we need to do following things:

  1. Initalize a new world tracking provider:
ar_world_tracking_configuration_t config = ar_world_tracking_configuration_create();
world_tracking = ar_world_tracking_provider_create(config);
  1. Start an AR session with this provider
  ar_session = ar_session_create();
  auto data_providers = ar_data_providers_create_with_data_providers((ar_data_provider_t)world_tracking, NULL);
  ar_session_run(ar_session, data_providers);

Important

This is API suggestion that unfortunately changes public-facing API, I'm open to another solution to make it work.

  1. Pass world tracking provider to the bgfx::init:
init.platformData.worldTracking = entry::getWorldTracking();
  1. Inside of bgfx we use this world tracking provider to get data about the headset position and set in on the drawable. This is a design decision issue that Apple made forcing us to mix rendering with retrieving device position.

Another possible solution is to run the ARSession inside BGFX, but this may limit the user's ability to run other providers like Hand Tracking Provider, etc.

You can see a full integration into the bgfx examples here: okwasniewski@337474a (it first requires #3292 to be merged)

@okwasniewski
Copy link
Contributor Author

Hey! I rebased the changes on the latest Apple Vision Pro PR.

include/bgfx/bgfx.h Outdated Show resolved Hide resolved
src/renderer_mtl.mm Outdated Show resolved Hide resolved
@okwasniewski
Copy link
Contributor Author

okwasniewski commented Jul 30, 2024

Hey @bkaradzic,

I've refactored the flow to start up ARSession inside of bgfx when running on AVP. This is the last bit that was required to fix rendering.

Here is a recording of the latest changes:

ScreenRecording_07-30-2024.12-37-56_1.MP4

(imgui may need some adjustments but overall it works good)

@bkaradzic bkaradzic merged commit aa4f8c5 into bkaradzic:master Jul 30, 2024
11 checks passed
@bkaradzic
Copy link
Owner

Hey @okwasniewski I created visionOS channel on Discord so people interested in visionOS can discuss features and integration at one place. Please join here: https://discord.com/channels/712512073522872352/1267923444826112120

okwasniewski added a commit to okwasniewski/bgfx that referenced this pull request Aug 1, 2024
okwasniewski added a commit to okwasniewski/bgfx that referenced this pull request Aug 1, 2024
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants