feat: add visionOS support

[okwasniewski]

Hey,

This PR introduces visionOS windowed rendering for Babylon Native.

Recording

ScreenRecording_08-14-2024.12-57-06_1.MP4

TODO

• Merge bgfx visionOS support: feat: add visionOS support bkaradzic/bgfx#3289
  • Fix: add world tracking on Apple Vision Pro for drawables to be visible bkaradzic/bgfx#3326
  • fix: properly set viewports and render targets for Apple Vision Pro bkaradzic/bgfx#3325
• Sync changes from bkaradzic/bgfx to BabylonJS/bgfx: Update to latest bgfx bgfx#38
• Sync bgfx.cmake repo: feat: add visionOS scripts bgfx.cmake#77

NativeXr plugin integration will be a separate PR.

[okwasniewski]

I'm not sure if the latest BGFX changes make Win32 fail or if it's something else... Probably it would be best to separate bgfx bump

[bghgary]

I'm not sure if the latest BGFX changes make Win32 fail or if it's something else... Probably it would be best to separate bgfx bump

It's probably bgfx. We normally verify that bgfx is working before submitting a PR to the bgfx fork. Here is one of the errors:

##[error]buildUWP_arm64\_deps\bgfx.cmake-src\bgfx\src\shader_spirv.cpp(43,73): Error C2065: 'LiteralNumber': undeclared identifier
    98>D:\a\1\s\buildUWP_arm64\_deps\bgfx.cmake-src\bgfx\src\shader_spirv.cpp(43,73): error C2065: 'LiteralNumber': undeclared identifier [D:\a\1\s\buildUWP_arm64\_deps\bgfx.cmake-build\bgfx.vcxproj]
         (compiling source file 'CMakeFiles/bgfx.dir/Unity/unity_2_cxx.cxx')

[okwasniewski]

Okay looks like this change fixed build issues:

diff --git a/src/shader_spirv.cpp b/src/shader_spirv.cpp
index dc4f12282..185fdfd02 100644
--- a/src/shader_spirv.cpp
+++ b/src/shader_spirv.cpp
@@ -17,12 +17,8 @@ namespace bgfx
 #define SPV_OPERAND_7(_a0, _a1, _a2, _a3, _a4, _a5, _a6) SPV_OPERAND_1(_a0), SPV_OPERAND_6(_a1, _a2, _a3, _a4, _a5, _a6)
 #define SPV_OPERAND_8(_a0, _a1, _a2, _a3, _a4, _a5, _a6, _a7) SPV_OPERAND_1(_a0), SPV_OPERAND_7(_a1, _a2, _a3, _a4, _a5, _a6, _a7)
 #define SPV_OPERAND_9(_a0, _a1, _a2, _a3, _a4, _a5, _a6, _a7, _a8) SPV_OPERAND_1(_a0), SPV_OPERAND_8(_a1, _a2, _a3, _a4, _a5, _a6, _a7, _a8)
-#if BX_COMPILER_MSVC
-// Workaround MSVS bug...
-#	define SPV_OPERAND(...) { BX_MACRO_DISPATCHER(SPV_OPERAND_, __VA_ARGS__) BX_VA_ARGS_PASS(__VA_ARGS__) }
-#else
-#	define SPV_OPERAND(...) { BX_MACRO_DISPATCHER(SPV_OPERAND_, __VA_ARGS__)(__VA_ARGS__) }
-#endif // BX_COMPILER_MSVC
+#define SPV_OPERAND(...) { BX_MACRO_DISPATCHER(SPV_OPERAND_, __VA_ARGS__)(__VA_ARGS__) }
+
 #define _ Count
 
 	bool isDebug(SpvOpcode::Enum _opcode)

Here is a PR bringing this back: BabylonJS/bgfx#39

[okwasniewski]

Hey, FYI we will first make windowed rendering work in this PR and then create a separate one with the NativeXr plugin.

[okwasniewski]

Hey @ryantrem @CedricGuillemet this one is ready for re-review!

[docEdub]

Thanks for this PR. I'm getting ready to test on-device, now. Please add a link in README.md to the new VisionOS section in BUILDING.md.

[docEdub]

I'm getting a bgfx build error

[...]/build/visionOS/_deps/bgfx.cmake-src/bgfx/src/renderer_mtl.mm:3528:5: error: 'MTKView' is unavailable: not available on visionOS
                                MTKView *view = (MTKView *)_nwh;
                                ^

[okwasniewski]

@docEdub

Which version of CMake are you using? Can you try with at least 3.29.0?

And also just to make sure did you use this command to generate project?

cmake -B build/visionOS -G Xcode -D VISIONOS=ON

[ryantrem]

@okwasniewski in the PR description you have a link to a closed bgfx.cmake PR. Will you update that to point to the merged PR(s) so we have a good record of all the related changes across repos?

[okwasniewski]

@okwasniewski in the PR description you have a link to a closed bgfx.cmake PR. Will you update that to point to the merged PR(s) so we have a good record of all the related changes across repos?

Fixed

[docEdub]

@docEdub

Which version of CMake are you using? Can you try with at least 3.29.0?

And also just to make sure did you use this command to generate project?

cmake -B build/visionOS -G Xcode -D VISIONOS=ON

I'm have CMake 3.29.3 installed and I'm using cmake -B build/visionOS -G Xcode -D VISIONOS=ON to generate the Xcode project.

[docEdub]

I'm getting a bgfx build error

[...]/build/visionOS/_deps/bgfx.cmake-src/bgfx/src/renderer_mtl.mm:3528:5: error: 'MTKView' is unavailable: not available on visionOS
                                MTKView *view = (MTKView *)_nwh;
                                ^

Is MTKView expected to be available? ...or is this section of renderer_mtl.mm supposed to be skipped for VisionOS builds?

[okwasniewski]

MTKView *view = (MTKView *)_nwh;

It's supposed to be skipped, in that file (renderer_mtl.mm) from BGFX do you have any signs of BX_PLATFORM_VISIONOS macro? Maybe the dependency got cached somehow?

[okwasniewski]

I just did another test of installing fresh deps:

rm -rf build/visionOS

And then:

cmake -B build/visionOS -G Xcode -D VISIONOS=ON

And everything worked fine:

[docEdub]

I just did another test of installing fresh deps:

rm -rf build/visionOS

And then:

cmake -B build/visionOS -G Xcode -D VISIONOS=ON

And everything worked fine:

Which visionOS SDK are you using? I'm using visionOS SDK 1.2.

[okwasniewski]

Oh okay I got the issue, I was running on visionOS 2.0 and switched to older Xcode for simulator build and got the same issue as you did.

Damn I guess I need to submit one more PR to bgfx. Going to work on this tomorrow

[okwasniewski]

@docEdub Can you try if adding ifdefs to this piece of code make bgfx compile? Im hoping you could test out Babylon Native while we wait for this PR to get merged into bgfx

#if !BX_PLATFORM_VISIONOS
			if (NULL != NSClassFromString(@"MTKView") )
			{
				MTKView *view = (MTKView *)_nwh;
				
				if (NULL != view
					&&  [view isKindOfClass:NSClassFromString(@"MTKView")])
				{
					m_metalLayer = (CAMetalLayer *)view.layer;
				}
			}
#endif

[okwasniewski]

Here is the PR which should fix it: bkaradzic/bgfx#3347 going to do sync one more time once it merges

[docEdub]

@docEdub Can you try if adding ifdefs to this piece of code make bgfx compile? Im hoping you could test out Babylon Native while we wait for this PR to get merged into bgfx

#if !BX_PLATFORM_VISIONOS
			if (NULL != NSClassFromString(@"MTKView") )
			{
				MTKView *view = (MTKView *)_nwh;
				
				if (NULL != view
					&&  [view isKindOfClass:NSClassFromString(@"MTKView")])
				{
					m_metalLayer = (CAMetalLayer *)view.layer;
				}
			}
#endif

Yep, that gets it building and it's working on device. Nice!

[ryantrem]

Looks like we're good to go on this one?

[docEdub]

👍 LGTM. Tested on device with most recent changes and it works as expected.

[bghgary]

Congrats @okwasniewski (and @matthargett) on getting this in! Thanks for the contribution!

[matthargett]

Thanks for your partnership (and patience)! Next up is getting Babylon React Native to support React Native visionOS, then immersive mode and 6DOF head tracking 🥽