Adding Nvidia Streamline bindings & support for DLSS & etc for vulkan

[TomTheFurry]

Discussion

I recently started the journey to get DLSS working in our custom in-house engine, and since we use this lib for Vulkan rendering, I am wondering if you would be potentially interested in merging in bindings and support for Nvidia Streamline in Vulkan? Or do you think this is outside the scope of this project?

Regarding Streamline, I currently was able to get prove-of-concept working code along with hacky-manually-written bindings of streamline alongside with your Vortice.Vulkan lib. Note that Streamline is only available in Windows however.

Currently, my hope is to write an open-source lib/project that allows people to integrate Nvidia's fancy stuff like DLSS, Frame Gen, Reflex, etc, easily into their Vulkan stuff in C#. As a user of your project, I am hoping that I could contribute to the project via adding this part of support. However, I understand that this might potentially be a bit outside of the normal scope of this project? If that's the case, I can instead start a separate project that externally support Vortice instead.

Technical details:

Nvidia Streamline's integration with Vulkan is a bit... non-standard, as in, they don't follow the normal Vulkan extension systems. Instead, they just bypass all of it, and use a sort over 'interposer' way to get integration to work.

Basically, instead of loading the Vulkan DLL, you instead load their sl.interposer.dll, which exports the same vulkan dll ABI. It then intercepts GetXXXProcAddr(...) calls to return redirected function pointers for certain VK functions.

As for their own APIs, they basically have a 'core' part of API and a set of 'plugin' sub-APIs.

The core API is the Streamline API that are directly exposed via DLL ABI, and therefore, we can just do the usual [LibraryImport(...)] to call those functions. Those API are for setting up the Streamline stuff, loading what 'plugins' to use, querying what is supported in the running hardware... etc.

The 'plugin' are the actual separate 'features' that Nvidia provides, such as DLSS (Upscaling tech), DLSS-G (Frame-Gen), Reflex (Nvidia Reflex), DLSS-RR (DLSS Ray Reconstruction)... etc. The plugin APIs are exposed in a similar way to how Vulkan exposes extensions: You call their slGetFeatureFunction(...) to get the function pointers to those plugins.

As for their API definition, one thing to note is that they seem to not have a API reference doc (or one that is publicly-available that I could find). Instead, (most of the) Streamline is open-source at here and that they expect you to bind them through their publicly available C Header files.

Now obviously, we can't really do that in C#, so, instead, I am currently exploring ways to generate the c# code from the C Header files though various techniques. I am admittedly a newbie in this aspect, so I would appreciate it if you have some ideas that I could look into? Currently am checking various c-to-ffi and then ffi-to-c# stuff.

So what do you think? Try to integrate into this project or make a separate one?

[TomTheFurry]

Additional note:

In the future, I do plan on also adding direct support for AMD's FSR. I haven't looked into that yet, so I don't know what technical challenges I will encounter yet.

Tho, an interesting thing is that apparently there are direct drop-in replacement of the 'sl.interposer.dll' file that 'fake' support of DLSS via FSR. Technically I don't see any reason that prevents this from allowing the DLSS support to also thus indirectly "support" FSR.

Now clearly that isn't gonna be a good long term solution, hance wanting to directly get FSR working.

But yeah, for now, I am doing one step at a time and want to get DLSS integrated properly first. (Main reason is, well, I only have Nvidia GPU atm, so...)