[API Proposal]: APIs for source generating interop stubs for unmanaged virtual function tables

[jkoritzinsky]

Background and motivation

As part of our COM interop source generator work, we decided to start with implementing support first for the general concept of virtual function tables. Many native APIs, including JNI, MSQuic, and COM are either implemented or presented to users using tables of function pointers, either explicitly like MSQuic or implicitly through abstract member functions like JNI or COM. Providing this source generator would enable developers to use native APIs like these ones with user-friendly types using the new source-generated marshalling model instead of being forced to use the built-in marshalling model with Marshal.GetDelegateForFunctionPointer to manually marshal back a virtual method table or dropping down to manual marshalling with function pointers to get decent performance.

This source generator would enable generating code to call an unmanaged API projected to a managed interface and call a managed interface projected to an unmanaged table of function pointers.

We plan on providing guidance directing users to use these APIs to override behavior from the COM source generator (they will integrate cleanly)

API Proposal

These APIs fall into two categories:

1. APIs that allow developers to define managed APIs that represent unmanaged virtual method tables (such as COM, JNI, MsQuic).
2. APIs that represent concepts that the COM source generator could use as building blocks that are also designed to be seamlessly integrated with the APIs in the first category.

// Types that are only needed for the VTable source generator or to provide abstract concepts that the COM generator would use under the hood.
// These are types that we can exclude from the API proposals and either inline into the generated code, provide as file-scoped types, or not provide publicly (indicated by comments on each type).

using System.Numerics;

namespace System.Runtime.InteropServices.Marshalling;

/// <summary>
/// A factory to create an unmanaged "this pointer" from a managed object and to get a managed object from an unmanaged "this pointer".
/// </summary>
/// <remarks>
/// This interface would be used by the VTable source generator to enable users to indicate how to get the managed object from the "this pointer".
/// We can hard-code the ComWrappers logic here if we don't want to ship this interface.
/// </remarks>
public unsafe interface IUnmanagedObjectUnwrapper
{
    /// <summary>
    /// Get the object wrapped by <paramref name="ptr"/>.
    /// </summary>
    /// <param name="ptr">A an unmanaged "this pointer".</param>
    /// <returns>The object wrapped by <paramref name="ptr"/>.</returns>
    public static abstract object GetObjectForUnmanagedWrapper(void* ptr);
}

// This attribute provides the mechanism for the VTable source generator to know which type to use to get the managed object
// from the unmanaged "this" pointer. If we decide to not expose VirtualMethodIndexAttribute, we don't need to expose this.
[AttributeUsage(AttributeTargets.Interface)]
public class UnmanagedObjectUnwrapperAttribute<TMapper> : Attribute
    where TMapper : IUnmanagedObjectUnwrapper
{
}

// This type implements the logic to get the managed object from the unmanaged "this" pointer.
// If we decide to not expose the VTable source generator, we don't need to expose this and we can just inline the logic
// into the generated code in the source generator.
public sealed unsafe class ComWrappersUnwrapper : IUnmanagedObjectUnwrapper
{
    public static object GetObjectForUnmanagedWrapper(void* ptr)
    {
        return ComWrappers.ComInterfaceDispatch.GetInstance<object>((ComWrappers.ComInterfaceDispatch*)ptr);
    }
}

/// <summary>
/// Marshals an exception object to the value of its <see cref="Exception.HResult"/> converted to <typeparamref name="T"/>.
/// </summary>
/// <typeparam name="T">The unmanaged type to convert the HResult to.</typeparam>
/// <remarks>
/// This type is used by the COM source generator to enable marshalling exceptions to the HResult of the exception.
/// We can skip the exposing the exception marshallers if we decide to not expose the VTable source generator.
/// In that case, we'd hard-code the implementations of these marshallers into the COM source generator.
/// </remarks>
[CustomMarshaller(typeof(Exception), MarshalMode.UnmanagedToManagedOut, typeof(ExceptionHResultMarshaller<>))]
public static class ExceptionHResultMarshaller<T>
    where T : unmanaged, INumber<T>
{
    /// <summary>
    /// Marshals an exception object to the value of its <see cref="Exception.HResult"/> converted to <typeparamref name="T"/>.
    /// </summary>
    /// <param name="e">The exception.</param>
    /// <returns>The HResult of the exception, converted to <typeparamref name="T"/>.</returns>
    public static T ConvertToUnmanaged(Exception e);
}

[CustomMarshaller(typeof(Exception), MarshalMode.UnmanagedToManagedOut, typeof(ExceptionNaNMarshaller<>))]
public static class ExceptionNaNMarshaller<T>
    where T : unmanaged, IFloatingPointIeee754<T>
{
    /// <summary>
    /// Marshals an exception object to <see cref="T.NaN"/>.
    /// </summary>
    /// <param name="e">The exception.</param>
    /// <returns><typeparamref name="T.NaN"/>.</returns>
    public static T ConvertToUnmanaged(Exception e);
}

[CustomMarshaller(typeof(Exception), MarshalMode.UnmanagedToManagedOut, typeof(ExceptionDefaultMarshaller<>))]
public static class ExceptionDefaultMarshaller<T>
    where T : unmanaged
{
    /// <summary>
    /// Marshals an exception object to the default value of <typeparamref name="T"/>.
    /// </summary>
    /// <param name="e">The exception.</param>
    /// <returns>The default value of <typeparamref name="T"/>.</returns>
    public static T ConvertToUnmanaged(Exception e);
}

[CustomMarshaller(typeof(Exception), MarshalMode.UnmanagedToManagedOut, typeof(SwallowExceptionMarshaller))]
public static class SwallowExceptionMarshaller
{
    /// <summary>
    /// Swallows the exception.
    /// </summary>
    /// <param name="e">The exception.</param>
    public static void ConvertToUnmanaged(Exception e);
}

public enum ExceptionMarshalling
{
    Custom = 0,
    Com = 1
}

    public enum MarshalDirection
    {
        ManagedToUnmanaged = 0,
        UnmanagedToManaged = 1,
        Bidirectional = 2
    }

// This is the trigger attribute for the VTable source generator.
// If we decide we want to only expose the COM source generator, then we would keep this attribute internal.
// The current plan is to use this attribute to provide the "don't use the defaults, use this custom logic" options
// for the COM source generator, so if we decide to not expose this, we should provide a different mechanism.
[AttributeUsage(AttributeTargets.Method, AllowMultiple = false)]
public class VirtualMethodIndexAttribute : Attribute
{
    public VirtualMethodIndexAttribute(int index);

    public int Index { get; }

    public bool ImplicitThisParameter { get; set; } = true;

    /// <summary>
    /// Gets or sets how to marshal string arguments to the method.
    /// </summary>
    /// <remarks>
    /// If this field is set to a value other than <see cref="StringMarshalling.Custom" />,
    /// <see cref="StringMarshallingCustomType" /> must not be specified.
    /// </remarks>
    public StringMarshalling StringMarshalling { get; set; }

    /// <summary>
    /// Gets or sets the <see cref="Type"/> used to control how string arguments to the method are marshalled.
    /// </summary>
    /// <remarks>
    /// If this field is specified, <see cref="StringMarshalling" /> must not be specified
    /// or must be set to <see cref="StringMarshalling.Custom" />.
    /// </remarks>
    public Type? StringMarshallingCustomType { get; set; }

    /// <summary>
    /// Gets or sets whether the callee sets an error (SetLastError on Windows or errno
    /// on other platforms) before returning from the attributed method.
    /// </summary>
    public bool SetLastError { get; set; }
    
    public MarshalDirection Direction { get; set; } = MarshalDirection.Bidirectional;

    public ExceptionMarshalling ExceptionMarshalling { get; set; }

    /// <summary>
    /// Gets or sets the <see cref="Type"/> used to control how an exception is marshalled to the return value.
    /// </summary>
    /// <remarks>
    /// If this field is specified, <see cref="ExceptionMarshalling" /> must not be specified
    /// or must be set to <see cref="ExceptionMarshalling.Custom" />.
    /// </remarks>
    public Type? ExceptionMarshallingType { get; set; }
}

API Usage

[UnmanagedObjectUnwrapper<MyObjectUnwrapper>]
partial interface INativeAPI : IUnmanagedInterfaceType
{
       [VirtualMethodIndex(0)]
        void Method(string param);
}

[UnmanagedObjectUnwrapper<MyObjectUnwrapper>]
unsafe partial interface INativeAPI2 : IUnmanagedInterfaceType
{
		private static void* _table;
		public static void* IUnmanagedInterface.VirtualMethodTableManagedImplementation => _table;

		static INativeAPI2()
       {
              _table = RuntimeHelpers.AllocateTypeAssociatedMemory(typeof(INativeAPI2), sizeof(void*));
			 Native.FillManagedVirtualMethodTableImplementation(_table);
       }

       [VirtualMethodIndex(0)]
        void Method(string param);
}

class NativeAPI : IUnmanagedVirtualMethodTableProvider, INativeAPI.Native
{
		private void* _this;

        public NativeAPI(void* thisPtr) { _this = thisPtr; }

        VirtualMethodTableInfo IUnmanagedVirtualMethodTableProvider.GetVirtualMethodTableInfoForKey(Type type)
        {
				Debug.Assert(type == typeof(INativeAPI));
                return new(_this, *(void***)_this);
        }
}

sealed unsafe class MyObjectUnwrapper : IUnmanagedObjectUnwrapper
{
    public static object GetObjectForUnmanagedWrapper(void* ptr)
    {
        throw new NotImplementedException();
    }
}

Generated code shape:

partial interface INativeAPI
{
		public static void* IUnmanagedInterface.VirtualMethodTableManagedImplementation => /* implementation */ throw null;

        internal partial interface Native : INativeAPI
        {
               // DIM implementations for every method in INativeAPI with a [VirtualMethodIndexAttribute] attribute.
        }
}

partial interface INativeAPI2
{
        internal partial interface Native : INativeAPI2
        {
               // DIM implementations for every method in INativeAPI with a [VirtualMethodIndexAttribute] attribute.

               // Provided only when the user provides their own implementation of IUnmanagedInterface.VirtualMethodTableManagedImplementation.
               // It's extremely annoying to have to know the unmanaged function pointer signatures to take the address of the
              // implemented stubs, so provide this method to fill the table so users don't need to.
               internal static void FillManagedVirtualMethodTableImplementation(void* vtable) {} 
        }
}

Original API Proposal (kept for history to match to comments)

namespace System.Runtime.InteropServices.Marshalling;

/// <summary>
/// Information about a virtual method table and the unmanaged instance pointer.
/// </summary>
public readonly ref struct VirtualMethodTableInfo
{
    /// <summary>
    /// Construct a <see cref="VirtualMethodTableInfo"/> from a given instance pointer and table memory.
    /// </summary>
    /// <param name="thisPointer">The pointer to the instance.</param>
    /// <param name="virtualMethodTable">The block of memory that represents the virtual method table.</param>
    public VirtualMethodTableInfo(IntPtr thisPointer, ReadOnlySpan<IntPtr> virtualMethodTable)
    {
        ThisPointer = thisPointer;
        VirtualMethodTable = virtualMethodTable;
    }

    /// <summary>
    /// The unmanaged instance pointer
    /// </summary>
    public IntPtr ThisPointer { get; }

    /// <summary>
    /// The virtual method table.
    /// </summary>
    public ReadOnlySpan<IntPtr> VirtualMethodTable { get; }

    /// <summary>
    /// Deconstruct this structure into its two fields.
    /// </summary>
    /// <param name="thisPointer">The <see cref="ThisPointer"/> result</param>
    /// <param name="virtualMethodTable">The <see cref="VirtualMethodTable"/> result</param>
    public void Deconstruct(out IntPtr thisPointer, out ReadOnlySpan<IntPtr> virtualMethodTable)
    {
        thisPointer = ThisPointer;
        virtualMethodTable = VirtualMethodTable;
    }
}

/// <summary>
/// This interface allows an object to provide information about a virtual method table for a managed interface that implements <see cref="IUnmanagedInterfaceType{TInterface}"/> to enable invoking methods in the virtual method table.
/// </summary>
/// <typeparam name="T">The type to use to represent the the identity of the unmanaged type.</typeparam>
public unsafe interface IUnmanagedVirtualMethodTableProvider
{
    /// <summary>
    /// Get the information about the virtual method table for a given unmanaged interface type represented by <paramref name="type"/>.
    /// </summary>
    /// <param name="type">The managed type for the unmanaged interface.</param>
    /// <returns>The virtual method table information for the unmanaged interface.</returns>
    protected VirtualMethodTableInfo GetVirtualMethodTableInfoForKey(Type type);

    /// <summary>
    /// Get the information about the virtual method table for the given unmanaged interface type.
    /// </summary>
    /// <typeparam name="TUnmanagedInterfaceType">The managed interface type that represents the unmanaged interface.</typeparam>
    /// <returns>The virtual method table information for the unmanaged interface.</returns>
    public sealed VirtualMethodTableInfo GetVirtualMethodTableInfoForKey<TUnmanagedInterfaceType>()
        where TUnmanagedInterfaceType : IUnmanagedInterfaceType<TUnmanagedInterfaceType>
    {
        return GetVirtualMethodTableInfoForKey(typeof(TUnmanagedInterfaceType));
    }

    /// <summary>
    /// Get the length of the virtual method table for the given unmanaged interface type.
    /// </summary>
    /// <typeparam name="TUnmanagedInterfaceType">The managed interface type that represents the unmanaged interface.</typeparam>
    /// <returns>The length of the virtual method table for the unmanaged interface.</returns>
    public static int GetVirtualMethodTableLength<TUnmanagedInterfaceType>()
        where TUnmanagedInterfaceType : IUnmanagedInterfaceType<TUnmanagedInterfaceType>
    {
        return TUnmanagedInterfaceType.VirtualMethodTableLength;
    }

    /// <summary>
    /// Get a pointer to the virtual method table  of managed implementations of the unmanaged interface type.
    /// </summary>
    /// <typeparam name="TUnmanagedInterfaceType">The managed interface type that represents the unmanaged interface.</typeparam>
    /// <returns>A pointer to the virtual method table  of managed implementations of the unmanaged interface type</returns>
    public static void* GetVirtualMethodTableManagedImplementation<TUnmanagedInterfaceType>()
        where TUnmanagedInterfaceType : IUnmanagedInterfaceType<TUnmanagedInterfaceType>
    {
        return TUnmanagedInterfaceType.VirtualMethodTableManagedImplementation;
    }

    /// <summary>
    /// Get a pointer that wraps a managed implementation of an unmanaged interface that can be passed to unmanaged code.
    /// </summary>
    /// <typeparam name="TUnmanagedInterfaceType">The managed type that represents the unmanaged interface.</typeparam>
    /// <param name="obj">The managed object that implements the unmanaged interface.</param>
    /// <returns>A pointer-sized value that can be passed to unmanaged code that represents <paramref name="obj"/></returns>
    public static void* GetUnmanagedWrapperForObject<TUnmanagedInterfaceType>(TUnmanagedInterfaceType obj)
        where TUnmanagedInterfaceType : IUnmanagedInterfaceType<TUnmanagedInterfaceType>
    {
        return TUnmanagedInterfaceType.GetUnmanagedWrapperForObject(obj);
    }

    /// <summary>
    /// Get the object wrapped by <paramref name="ptr"/>.
    /// </summary>
    /// <typeparam name="TUnmanagedInterfaceType">The managed type that represents the unmanaged interface.</typeparam>
    /// <param name="ptr">A pointer-sized value returned by <see cref="GetUnmanagedWrapperForObject{TUnmanagedInterfaceType}(TUnmanagedInterfaceType)"/> or <see cref="IUnmanagedInterfaceType{TInterface, TKey}.GetUnmanagedWrapperForObject(TInterface)"/>.</param>
    /// <returns>The object wrapped by <paramref name="ptr"/>.</returns>
    public static TUnmanagedInterfaceType GetObjectForUnmanagedWrapper<TUnmanagedInterfaceType>(void* ptr)
        where TUnmanagedInterfaceType : IUnmanagedInterfaceType<TUnmanagedInterfaceType>
    {
        return TUnmanagedInterfaceType.GetObjectForUnmanagedWrapper(ptr);
    }
}

/// <summary>
/// This interface allows another interface to define that it represents a managed projection of an unmanaged interface from some unmanaged type system.
/// </summary>
/// <typeparam name="TInterface">The managed interface.</typeparam>
/// <typeparam name="TKey">The type of a value that can represent types from the corresponding unmanaged type system.</typeparam>
public unsafe interface IUnmanagedInterfaceType<TInterface>
    where TInterface : IUnmanagedInterfaceType<TInterface>
{
    /// <summary>
    /// Get the length of the virtual method table for the given unmanaged interface type.
    /// </summary>
    /// <returns>The length of the virtual method table for the unmanaged interface.</returns>
    public static abstract int VirtualMethodTableLength { get; }

    /// <summary>
    /// Get a pointer to the virtual method table  of managed implementations of the unmanaged interface type.
    /// </summary>
    /// <returns>A pointer to the virtual method table  of managed implementations of the unmanaged interface type</returns>
    public static abstract void* VirtualMethodTableManagedImplementation { get; }

    /// <summary>
    /// Get a pointer that wraps a managed implementation of an unmanaged interface that can be passed to unmanaged code.
    /// </summary>
    /// <param name="obj">The managed object that implements the unmanaged interface.</param>
    /// <returns>A pointer-sized value that can be passed to unmanaged code that represents <paramref name="obj"/></returns>
    public static abstract void* GetUnmanagedWrapperForObject(TInterface obj);

    /// <summary>
    /// Get the object wrapped by <paramref name="ptr"/>.
    /// </summary>
    /// <param name="ptr">A pointer-sized value returned by <see cref="IUnmanagedVirtualMethodTableProvider{TKey}.GetUnmanagedWrapperForObject{IUnmanagedInterfaceType{TInterface, TKey}}(IUnmanagedInterfaceType{TInterface, TKey})"/> or <see cref="GetUnmanagedWrapperForObject(TInterface)"/>.</param>
    /// <returns>The object wrapped by <paramref name="ptr"/>.</returns>
    public static abstract TInterface GetObjectForUnmanagedWrapper(void* ptr);
}

API Usage

partial interface INativeAPI : IUnmanagedInterfaceType<INativeAPI>
{
       static int IUnmanagedInterfaceType<INativeAPI>.VirtualMethodTableLength => 1;

                private static void** s_vtable = (void**)RuntimeHelpers.AllocateTypeAssociatedMemory(typeof(INativeAPI), sizeof(void*) * IUnmanagedVirtualMethodTableProvider.GetVirtualMethodTableLength<INativeAPI>());
                static void* IUnmanagedInterfaceType<INativeAPI>.VirtualMethodTableManagedImplementation
                {
                    get
                    {
                        if (s_vtable[0] == null)
                        {
                            Native.PopulateUnmanagedVirtualMethodTable(new Span<IntPtr>(s_vtable, IUnmanagedVirtualMethodTableProvider.GetVirtualMethodTableLength<INativeAPI>()));
                        }
                        return s_vtable;
                    }
                }

       static void* IUnmanagedInterfaceType<INativeAPI>.GetUnmanagedWrapperForObject(INativeAPI api) => throw new NotImplementedException();

        static INativeAPI IUnmanagedInterfaceType<INativeAPI>.GetObjectForUnmanagedWrapper(void* ptr) => throw new NotImplementedException();

       static INativeAPI()
       {
			
       }

       [VirtualMethodIndex(0)]
        void Method(string param);
}

class NativeAPI : IUnmanagedVirtualMethodTableProvider, INativeAPI.Native
{
		private void* _this;

        public NativeAPI(void* thisPtr) { _this = thisPtr; }

        VirtualMethodTableInfo IUnmanagedVirtualMethodTableProvider.GetVirtualMethodTableInfoForKey(Type type)
        {
				Debug.Assert(type == typeof(INativeAPI));
                return new((IntPtr)_this, new ReadOnlySpan<IntPtr>(*(void**)_this,  IUnmanagedVirtualMethodTableProvider.GetVirtualMethodTableLength<INativeAPI>()));
        }
}

Generated code shape:

partial interface INativeAPI
{
        internal partial interface Native : INativeAPI
        {
               // DIM implementations for every method in INativeAPI with a [VirtualMethodIndexAttribute] attribute.
               internal static PopulateUnmanagedVirtualMethodTable(Span<nint> vtable) { /* fill the vtable with function pointers for the unmanaged->managed stubs */
        }
}

Alternative Designs

Risks

The DIM-implemented methods aren't visible when using the implementing types directly; they're only visible through the interface. As a result, the user experience is a little weird for the cases where only one interface is implemented, as the interface methods can only be called on an object wrapping the native API through the interface, not through the wrapping class.

[ghost]

Tagging subscribers to this area: @dotnet/interop-contrib
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Background and motivation

As part of our COM interop source generator work, we decided to start with implementing support first for the general concept of virtual function tables. Many native APIs, including JNI, MSQuic, and COM are either implemented or presented to users using tables of function pointers, either explicitly like MSQuic or implicitly through abstract member functions like JNI or COM. Providing this source generator would enable developers to use native APIs like these ones with user-friendly types using the new source-generated marshalling model instead of being forced to use the built-in marshalling model with Marshal.GetDelegateForFunctionPointer to manually marshal back a virtual method table or dropping down to manual marshalling with function pointers to get decent performance.

This source generator would enable generating code to call an unmanaged API projected to a managed interface and call a managed interface projected to an unmanaged table of function pointers.

We plan on providing guidance directing users to use these APIs to override behavior from the COM source generator (they will integrate cleanly)

API Proposal

namespace System.Runtime.InteropServices.Marshalling;

/// <summary>
/// Information about a virtual method table and the unmanaged instance pointer.
/// </summary>
public readonly ref struct VirtualMethodTableInfo
{
    /// <summary>
    /// Construct a <see cref="VirtualMethodTableInfo"/> from a given instance pointer and table memory.
    /// </summary>
    /// <param name="thisPointer">The pointer to the instance.</param>
    /// <param name="virtualMethodTable">The block of memory that represents the virtual method table.</param>
    public VirtualMethodTableInfo(IntPtr thisPointer, ReadOnlySpan<IntPtr> virtualMethodTable)
    {
        ThisPointer = thisPointer;
        VirtualMethodTable = virtualMethodTable;
    }

    /// <summary>
    /// The unmanaged instance pointer
    /// </summary>
    public IntPtr ThisPointer { get; }

    /// <summary>
    /// The virtual method table.
    /// </summary>
    public ReadOnlySpan<IntPtr> VirtualMethodTable { get; }

    /// <summary>
    /// Deconstruct this structure into its two fields.
    /// </summary>
    /// <param name="thisPointer">The <see cref="ThisPointer"/> result</param>
    /// <param name="virtualMethodTable">The <see cref="VirtualMethodTable"/> result</param>
    public void Deconstruct(out IntPtr thisPointer, out ReadOnlySpan<IntPtr> virtualMethodTable)
    {
        thisPointer = ThisPointer;
        virtualMethodTable = VirtualMethodTable;
    }
}

/// <summary>
/// This interface allows an object to provide information about a virtual method table for a managed interface that implements <see cref="IUnmanagedInterfaceType{TInterface}"/> to enable invoking methods in the virtual method table.
/// </summary>
/// <typeparam name="T">The type to use to represent the the identity of the unmanaged type.</typeparam>
public unsafe interface IUnmanagedVirtualMethodTableProvider
{
    /// <summary>
    /// Get the information about the virtual method table for a given unmanaged interface type represented by <paramref name="type"/>.
    /// </summary>
    /// <param name="type">The managed type for the unmanaged interface.</param>
    /// <returns>The virtual method table information for the unmanaged interface.</returns>
    protected VirtualMethodTableInfo GetVirtualMethodTableInfoForKey(Type type);

    /// <summary>
    /// Get the information about the virtual method table for the given unmanaged interface type.
    /// </summary>
    /// <typeparam name="TUnmanagedInterfaceType">The managed interface type that represents the unmanaged interface.</typeparam>
    /// <returns>The virtual method table information for the unmanaged interface.</returns>
    public sealed VirtualMethodTableInfo GetVirtualMethodTableInfoForKey<TUnmanagedInterfaceType>()
        where TUnmanagedInterfaceType : IUnmanagedInterfaceType<TUnmanagedInterfaceType>
    {
        return GetVirtualMethodTableInfoForKey(typeof(TUnmanagedInterfaceType));
    }

    /// <summary>
    /// Get the length of the virtual method table for the given unmanaged interface type.
    /// </summary>
    /// <typeparam name="TUnmanagedInterfaceType">The managed interface type that represents the unmanaged interface.</typeparam>
    /// <returns>The length of the virtual method table for the unmanaged interface.</returns>
    public static int GetVirtualMethodTableLength<TUnmanagedInterfaceType>()
        where TUnmanagedInterfaceType : IUnmanagedInterfaceType<TUnmanagedInterfaceType>
    {
        return TUnmanagedInterfaceType.VirtualMethodTableLength;
    }

    /// <summary>
    /// Get a pointer to the virtual method table  of managed implementations of the unmanaged interface type.
    /// </summary>
    /// <typeparam name="TUnmanagedInterfaceType">The managed interface type that represents the unmanaged interface.</typeparam>
    /// <returns>A pointer to the virtual method table  of managed implementations of the unmanaged interface type</returns>
    public static void* GetVirtualMethodTableManagedImplementation<TUnmanagedInterfaceType>()
        where TUnmanagedInterfaceType : IUnmanagedInterfaceType<TUnmanagedInterfaceType>
    {
        return TUnmanagedInterfaceType.VirtualMethodTableManagedImplementation;
    }

    /// <summary>
    /// Get a pointer that wraps a managed implementation of an unmanaged interface that can be passed to unmanaged code.
    /// </summary>
    /// <typeparam name="TUnmanagedInterfaceType">The managed type that represents the unmanaged interface.</typeparam>
    /// <param name="obj">The managed object that implements the unmanaged interface.</param>
    /// <returns>A pointer-sized value that can be passed to unmanaged code that represents <paramref name="obj"/></returns>
    public static void* GetUnmanagedWrapperForObject<TUnmanagedInterfaceType>(TUnmanagedInterfaceType obj)
        where TUnmanagedInterfaceType : IUnmanagedInterfaceType<TUnmanagedInterfaceType>
    {
        return TUnmanagedInterfaceType.GetUnmanagedWrapperForObject(obj);
    }

    /// <summary>
    /// Get the object wrapped by <paramref name="ptr"/>.
    /// </summary>
    /// <typeparam name="TUnmanagedInterfaceType">The managed type that represents the unmanaged interface.</typeparam>
    /// <param name="ptr">A pointer-sized value returned by <see cref="GetUnmanagedWrapperForObject{TUnmanagedInterfaceType}(TUnmanagedInterfaceType)"/> or <see cref="IUnmanagedInterfaceType{TInterface, TKey}.GetUnmanagedWrapperForObject(TInterface)"/>.</param>
    /// <returns>The object wrapped by <paramref name="ptr"/>.</returns>
    public static TUnmanagedInterfaceType GetObjectForUnmanagedWrapper<TUnmanagedInterfaceType>(void* ptr)
        where TUnmanagedInterfaceType : IUnmanagedInterfaceType<TUnmanagedInterfaceType>
    {
        return TUnmanagedInterfaceType.GetObjectForUnmanagedWrapper(ptr);
    }
}

/// <summary>
/// This interface allows another interface to define that it represents a managed projection of an unmanaged interface from some unmanaged type system.
/// </summary>
/// <typeparam name="TInterface">The managed interface.</typeparam>
/// <typeparam name="TKey">The type of a value that can represent types from the corresponding unmanaged type system.</typeparam>
public unsafe interface IUnmanagedInterfaceType<TInterface>
    where TInterface : IUnmanagedInterfaceType<TInterface>
{
    /// <summary>
    /// Get the length of the virtual method table for the given unmanaged interface type.
    /// </summary>
    /// <returns>The length of the virtual method table for the unmanaged interface.</returns>
    public static abstract int VirtualMethodTableLength { get; }

    /// <summary>
    /// Get a pointer to the virtual method table  of managed implementations of the unmanaged interface type.
    /// </summary>
    /// <returns>A pointer to the virtual method table  of managed implementations of the unmanaged interface type</returns>
    public static abstract void* VirtualMethodTableManagedImplementation { get; }

    /// <summary>
    /// Get a pointer that wraps a managed implementation of an unmanaged interface that can be passed to unmanaged code.
    /// </summary>
    /// <param name="obj">The managed object that implements the unmanaged interface.</param>
    /// <returns>A pointer-sized value that can be passed to unmanaged code that represents <paramref name="obj"/></returns>
    public static abstract void* GetUnmanagedWrapperForObject(TInterface obj);

    /// <summary>
    /// Get the object wrapped by <paramref name="ptr"/>.
    /// </summary>
    /// <param name="ptr">A pointer-sized value returned by <see cref="IUnmanagedVirtualMethodTableProvider{TKey}.GetUnmanagedWrapperForObject{IUnmanagedInterfaceType{TInterface, TKey}}(IUnmanagedInterfaceType{TInterface, TKey})"/> or <see cref="GetUnmanagedWrapperForObject(TInterface)"/>.</param>
    /// <returns>The object wrapped by <paramref name="ptr"/>.</returns>
    public static abstract TInterface GetObjectForUnmanagedWrapper(void* ptr);
}

API Usage

partial interface INativeAPI : IUnmanagedInterfaceType<INativeAPI>
{
       static int IUnmanagedInterfaceType<INativeAPI>.VirtualMethodTableLength => 1;

                private static void** s_vtable = (void**)RuntimeHelpers.AllocateTypeAssociatedMemory(typeof(INativeAPI), sizeof(void*) * IUnmanagedVirtualMethodTableProvider.GetVirtualMethodTableLength<INativeAPI>());
                static void* IUnmanagedInterfaceType<INativeAPI>.VirtualMethodTableManagedImplementation
                {
                    get
                    {
                        if (s_vtable[0] == null)
                        {
                            Native.PopulateUnmanagedVirtualMethodTable(new Span<IntPtr>(s_vtable, IUnmanagedVirtualMethodTableProvider.GetVirtualMethodTableLength<INativeAPI>()));
                        }
                        return s_vtable;
                    }
                }

       static void* IUnmanagedInterfaceType<INativeAPI>.GetUnmanagedWrapperForObject(INativeAPI api) => throw new NotImplementedException();

        static INativeAPI IUnmanagedInterfaceType<INativeAPI>.GetObjectForUnmanagedWrapper(void* ptr) => throw new NotImplementedException();

       static INativeAPI()
       {
			
       }

       [VirtualMethodIndex(0)]
        void Method(string param);
}

class NativeAPI : IUnmanagedVirtualMethodTableProvider, INativeAPI.Native
{
		private void* _this;

        public NativeAPI(void* thisPtr) { _this = thisPtr; }

        VirtualMethodTableInfo IUnmanagedVirtualMethodTableProvider.GetVirtualMethodTableInfoForKey(Type type)
        {
				Debug.Assert(type == typeof(INativeAPI));
                return new((IntPtr)_this, new ReadOnlySpan<IntPtr>(*(void**)_this,  IUnmanagedVirtualMethodTableProvider.GetVirtualMethodTableLength<INativeAPI>()));
        }
}

Generated code shape:

partial interface INativeAPI
{
        internal partial interface Native : INativeAPI
        {
               // DIM implementations for every method in INativeAPI with a [VirtualMethodIndexAttribute] attribute.
               internal static PopulateUnmanagedVirtualMethodTable(Span<nint> vtable) { /* fill the vtable with function pointers for the unmanaged->managed stubs */
        }
}

Alternative Designs

We could keep these APIs for testing and only expose the COM source generator APIs for the opinionated COM source generator and provide more options on those APIs (still to be proposed) instead of providing a lower-level tool like these APIs.

Risks

The DIM-implemented methods aren't visible when using the implementing types directly; they're only visible through the interface. As a result, the user experience is a little weird for the cases where only one interface is implemented, as the interface methods can only be called on an object wrapping the native API through the interface, not through the wrapping class.

Author:	jkoritzinsky
Assignees:	-
Labels:	api-suggestion, area-System.Runtime.InteropServices, source-generator
Milestone:	8.0.0

[AaronRobinsonMSFT]

    /// <summary>
    /// The unmanaged instance pointer
    /// </summary>
    public IntPtr ThisPointer { get; }

We should just use void* in these kinds of places. In VS these will all revert to nint which isn't all that helpful as it really is a void*.

[jkoritzinsky]

We can't update the ReadOnlySpan<IntPtr> because you can't put pointers in generics, but we can update the ThisPointer property and the like.

[AaronRobinsonMSFT]

We can't update the ReadOnlySpan because you can't put pointers in generics,

Unsure if using ReadOnlySpan<IntPtr> is all that helpful though. The first thing people will do is check the length and then ignore the niceties the span provides. Instead, we could have a property for the length and return void*. I assume the use case here is going to be all generated so I don't see the span providing a lot of value other than using a type to capture two values - pointer and length. Something to consider as an alternative.

[AaronRobinsonMSFT]

public unsafe interface IUnmanagedInterfaceType

I think this interface needs to be broken apart. The mapping logic is likely to be identical to the vast majority of types for a generator. This pattern seems like it would create a lot of unnecessary metadata that couldn't be removed.

[jkoritzinsky]

I'm fine splitting it, but I'm at a loss on what to name the one that has the unmanaged->managed logic.

[Sergio0694]

Sharing some thoughts here after discussing this with @jkoritzinsky as well. I think this might overlap at least in part with what @AaronRobinsonMSFT said as well. My concern is that the current proposal results in a UX that's rather cumbersome and error prone for developers, specifically around the setup code to initialize vtables. Jeremy explained that the point of initializing a vtable explicitly and not in a static field initializer is that this allows the linker to potentially trim everything if the type is not used. This is fine, but I feel like we could have something that's at least a little bit more ergonomic to use. Consider this:

partial interface INativeAPI : IUnmanagedInterfaceType<INativeAPI>
{
    static int IUnmanagedInterfaceType<INativeAPI>.VirtualMethodTableLength => 1;

    private static void** s_vtable = (void**)RuntimeHelpers.AllocateTypeAssociatedMemory(typeof(INativeAPI), sizeof(void*) * IUnmanagedVirtualMethodTableProvider.GetVirtualMethodTableLength<INativeAPI>());
    static void* IUnmanagedInterfaceType<INativeAPI>.VirtualMethodTableManagedImplementation
    {
        get
        {
            if (s_vtable[0] == null)
            {
                Native.PopulateUnmanagedVirtualMethodTable(new Span<IntPtr>(s_vtable, IUnmanagedVirtualMethodTableProvider.GetVirtualMethodTableLength<INativeAPI>()));
            }
            return s_vtable;
        }
    }

    static void* IUnmanagedInterfaceType<INativeAPI>.GetUnmanagedWrapperForObject(INativeAPI api) => throw new NotImplementedException();

    static INativeAPI IUnmanagedInterfaceType<INativeAPI>.GetObjectForUnmanagedWrapper(void* ptr) => throw new NotImplementedException();

    static INativeAPI()
    {
        
    }

    [VirtualMethodIndex(0)]
    void Method(string param);
}

That's a lot of relatively non obvious code, and writing that check manually every time seems a bit awkward, not to mention it's one more thing developers could get wrong. One possible solution could be to consider the way the vtable is allocated to just be the "de-fact right way" to do it, and as such move that initialization using RuntimeHelpers.AllocateTypeAssociatedMemory in some shared stub (eg. in a DIM), as well as the check, and then only ask developers implementing the interface to write the code to fill in the pre-allocated vtable, and nothing more. Something like:

partial interface INativeAPI : IUnmanagedInterfaceType<INativeAPI>
{
    static int IUnmanagedInterfaceType<INativeAPI>.VirtualMethodTableLength => 1;

    static void IUnmanagedInterfaceType<INativeAPI>.InitializeVirtualMethodTableManagedImplementation(void** vtable)
    {
        // Fill vtable here
    }
}

The allocation + initialization check could all be moved to a static DIM in the base interface. Authors would just indicate the length of the vtable, and provide the logic to fill in an allocated vtable, and nothing else 🙂

[jkoritzinsky]

Here's a revised API proposal. The "create a wrapper" and "retrieve an object from within a wrapper" APIs have been extracted into another interface and the methods on IUnmanagedInterfaceType<TInterfaceType, TUnmanagedObjectWrapperFactory> that deal with wrappers just forward to the wrapper factory implmentation and add nice casts based on @AaronRobinsonMSFT's feedback and splits the interface into two copies (one for consumption and one for consumption + production). It also encapsulates the allocation/initialization pattern for the managed vtable as per @Sergio0694's feedback.

Let me know what you think!

namespace System.Runtime.InteropServices.Marshalling;

/// <summary>
/// Information about a virtual method table and the unmanaged instance pointer.
/// </summary>
public readonly unsafe struct VirtualMethodTableInfo
{
    /// <summary>
    /// Construct a <see cref="VirtualMethodTableInfo"/> from a given instance pointer and table memory.
    /// </summary>
    /// <param name="thisPointer">The pointer to the instance.</param>
    /// <param name="virtualMethodTable">The block of memory that represents the virtual method table.</param>
    public VirtualMethodTableInfo(void* thisPointer, void** virtualMethodTable)
    {
        ThisPointer = thisPointer;
        VirtualMethodTable = virtualMethodTable;
    }

    /// <summary>
    /// The unmanaged instance pointer
    /// </summary>
    public void* ThisPointer { get; }

    /// <summary>
    /// The virtual method table.
    /// </summary>
    public void** VirtualMethodTable { get; }

    /// <summary>
    /// Deconstruct this structure into its two fields.
    /// </summary>
    /// <param name="thisPointer">The <see cref="ThisPointer"/> result</param>
    /// <param name="virtualMethodTable">The <see cref="VirtualMethodTable"/> result</param>
    public void Deconstruct(out void* thisPointer, out void** virtualMethodTable)
    {
        thisPointer = ThisPointer;
        virtualMethodTable = VirtualMethodTable;
    }
}

/// <summary>
/// This interface allows an object to provide information about a virtual method table for a managed interface that implements <see cref="IUnmanagedInterfaceType{TInterface}"/> to enable invoking methods in the virtual method table.
/// </summary>
/// <typeparam name="T">The type to use to represent the the identity of the unmanaged type.</typeparam>
public unsafe interface IUnmanagedVirtualMethodTableProvider
{
    /// <summary>
    /// Get the information about the virtual method table for a given unmanaged interface type represented by <paramref name="type"/>.
    /// </summary>
    /// <param name="type">The managed type for the unmanaged interface.</param>
    /// <returns>The virtual method table information for the unmanaged interface.</returns>
    protected VirtualMethodTableInfo GetVirtualMethodTableInfoForKey(Type type);

    /// <summary>
    /// Get the information about the virtual method table for the given unmanaged interface type.
    /// </summary>
    /// <typeparam name="TUnmanagedInterfaceType">The managed interface type that represents the unmanaged interface.</typeparam>
    /// <returns>The virtual method table information for the unmanaged interface.</returns>
    public sealed VirtualMethodTableInfo GetVirtualMethodTableInfoForKey<TUnmanagedInterfaceType>()
        where TUnmanagedInterfaceType : IUnmanagedInterfaceType
    {
        return GetVirtualMethodTableInfoForKey(typeof(TUnmanagedInterfaceType));
    }
}

/// <summary>
/// This interface allows another interface to define that it represents a managed projection of an unmanaged interface from some unmanaged type system.
/// </summary>
public unsafe interface IUnmanagedInterfaceType
{
    /// <summary>
    /// Get the length of the virtual method table for the given unmanaged interface type.
    /// </summary>
    /// <returns>The length of the virtual method table for the unmanaged interface.</returns>
    public static abstract int VirtualMethodTableLength { get; }
}

/// <summary>
/// A factory to create an unmanaged "this pointer" from a managed object and to get a managed object from an unmanaged "this pointer".
/// </summary>
public unsafe interface IUnmanagedObjectWrapperFactory
{
    /// <summary>
    /// Get a pointer that wraps a managed implementation of an unmanaged interface that can be passed to unmanaged code.
    /// </summary>
    /// <param name="obj">The managed object that implements the unmanaged interface.</param>
    /// <returns>A unmanaged "this pointer" that can be passed to unmanaged code that represents <paramref name="obj"/></returns>
    public static abstract void* GetUnmanagedWrapperForObject(object obj);

    /// <summary>
    /// Get the object wrapped by <paramref name="ptr"/>.
    /// </summary>
    /// <param name="ptr">A an unmanaged "this pointer".</param>
    /// <returns>The object wrapped by <paramref name="ptr"/>.</returns>
    public static abstract object GetObjectForUnmanagedWrapper(void* ptr);
}

/// <summary>
/// This interface allows another interface to define that it represents a managed projection of an unmanaged interface from some unmanaged type system and supports passing managed implementations of unmanaged interfaces to unmanaged code.
/// </summary>
/// <typeparam name="TInterface">The managed interface.</typeparam>
/// <typeparam name="TUnmanagedObjectWrapperFactory">The factory to create an unmanaged "this pointer" from a managed object and to get a managed object from an unmanaged "this pointer".</typeparam>
public unsafe interface IUnmanagedInterfaceType<TInterface, TUnmanagedObjectWrapperFactory> : IUnmanagedInterfaceType
    where TInterface : IUnmanagedInterfaceType<TInterface, TUnmanagedObjectWrapperFactory>
    where TUnmanagedObjectWrapperFactory: IUnmanagedObjectWrapperFactory, new()
{
    /// <summary>
    /// Get a pointer to the virtual method table of managed implementations of the unmanaged interface type.
    /// </summary>
    /// <returns>A pointer to the virtual method table of managed implementations of the unmanaged interface type</returns>
    public static void* VirtualMethodTableManagedImplementation { get; }
    
    /// <summary>
    /// Fill the entries in the virtual method table of managed implementations of the unmanaged interface type
    /// </summary>
    protected static virtual void FillVirtualMethodTableMangedImplementation(void** vtable);

    /// <summary>
    /// Get a pointer that wraps a managed implementation of an unmanaged interface that can be passed to unmanaged code.
    /// </summary>
    /// <param name="obj">The managed object that implements the unmanaged interface.</param>
    /// <returns>A unmanaged "this pointer" that can be passed to unmanaged code that represents <paramref name="obj"/></returns>
    public static void* GetUnmanagedWrapperForObject(TInterface obj);

    /// <summary>
    /// Get the object wrapped by <paramref name="ptr"/>.
    /// </summary>
    /// <param name="ptr">A an unmanaged "this pointer".</param>
    /// <returns>The object wrapped by <paramref name="ptr"/>.</returns>
    public static TInterface GetObjectForUnmanagedWrapper(void* ptr);
}

[Sergio0694]

Looks much better! I love how the vtable setup is way easier for authors now 🙂

A few random notes:

public sealed VirtualMethodTableInfo GetVirtualMethodTableInfoForKey<TUnmanagedInterfaceType>()
    where TUnmanagedInterfaceType : IUnmanagedInterfaceType
{
    return GetVirtualMethodTableInfoForKey(typeof(TUnmanagedInterfaceType));
}

I understand this is meant to be a convenience method, but it being a DIM feels like it might make codegen worse. Couldn't this be an extension method instead targeting IUnmanagedVirtualMethodTableProvider, so the interface would just have the one method that developers are meant to implement?

2. It's not entirely clear to me why the convenience helper is public, but the non-generic one is protected 🤔
3. The name "VirtualMethodTableManagedImplementation" seems a bit confusing to me. I get that it's pointing to managed method implementations, but the function pointers in that vtable are unmanaged (I mean, they're using an unmanaged call conv)... I don't really have a better name suggestion, just figured I'd throw this note out too while at it 😄

[AaronRobinsonMSFT]

We could keep these APIs for testing and only expose the COM source generator APIs for the opinionated COM source generator and provide more options on those APIs (still to be proposed) instead of providing a lower-level tool like these APIs.

I'm inclined to go with this for now. I would prefer to keep these as implementation/testing details until we have a full end to end for our P1 COM scenario. I am thinking of the late changes that occurred around the marshaller design and the annoyance it was to update them all. This design also permits a level of extensibility that isn't obviously beneficial beyond the current goal of the COM source generator. Once we have the COM source generator working for our stakeholders, let's revisit this API proposal and see where we've landed.

[jkoritzinsky]

Looks much better! I love how the vtable setup is way easier for authors now 🙂

A few random notes:

public sealed VirtualMethodTableInfo GetVirtualMethodTableInfoForKey<TUnmanagedInterfaceType>()
    where TUnmanagedInterfaceType : IUnmanagedInterfaceType
{
    return GetVirtualMethodTableInfoForKey(typeof(TUnmanagedInterfaceType));
}

I understand this is meant to be a convenience method, but it being a DIM feels like it might make codegen worse. Couldn't this be an extension method instead targeting IUnmanagedVirtualMethodTableProvider, so the interface would just have the one method that developers are meant to implement?

2. It's not entirely clear to me why the convenience helper is public, but the non-generic one is protected 🤔
3. The name "VirtualMethodTableManagedImplementation" seems a bit confusing to me. I get that it's pointing to managed method implementations, but the function pointers in that vtable are unmanaged (I mean, they're using an unmanaged call conv)... I don't really have a better name suggestion, just figured I'd throw this note out too while at it 😄

Since the method is sealed, it's the same performance as a regular method. In any case, the generated code that will use this type will already have to cast the this pointer to the interface anyway, so the extra cost of having to cast to the interface to see the method doesn't matter. Additionally, having the method on the interface instead of as an extension method ensures that we don't bleed this name into the public API of the implementing types (SharpLab).

The generic method is public as it can have type constraints to ensure that only valid types are passed. The protected method is non-generic to avoid the cost of generic virtual method resolution.

I agree, the name of VirtualMethodTable**Managed**Implementation is a little wordy and odd. I'll try to think of a better name.

We could keep these APIs for testing and only expose the COM source generator APIs for the opinionated COM source generator and provide more options on those APIs (still to be proposed) instead of providing a lower-level tool like these APIs.

I'm inclined to go with this for now. I would prefer to keep these as implementation/testing details until we have a full end to end for our P1 COM scenario. I am thinking of the late changes that occurred around the marshaller design and the annoyance it was to update them all. This design also permits a level of extensibility that isn't obviously beneficial beyond the current goal of the COM source generator. Once we have the COM source generator working for our stakeholders, let's revisit this API proposal and see where we've landed.

I think I may have misstated that alternative. The COM source generator will still use the IUnmanagedVirtualMethodTableProvider and IUnmanagedInterfaceType types, but not the VirtualMethodIndexAttribute. We'll still need either these interfaces or some replacements for the same functionality for the COM source generator (and I'd highly prefer to use the same APIs for the COM case and the general vtable case).

[jkoritzinsky]

@AaronRobinsonMSFT @dotnet/interop-contrib @Sergio0694 I've updated the API proposal based on the interop team's experimenting and feedback. I've split the API proposal section into two in case API review approves the first part and sends the second back for redesign so we can keep moving the COM source generator forward.

[jkoritzinsky]

Moved the "VTable + COM" APIs into #79121 so we hit the APIs in order based on API review's scheduling algorithm.

[teo-tsirpanis]

Was discussed in today's API review.

[jkoritzinsky]

Marking as non-blocking for now as we might not need this for .NET 8, but keeping the milestone as we might still make it blocking again.

[jkoritzinsky]

Video

From #79121,

/// <summary>
/// This interface allows another interface to define that it represents a managed projection of an unmanaged interface from some unmanaged type system and supports passing managed implementations of unmanaged interfaces to unmanaged code.
/// </summary>
public interface IUnmanagedInterfaceType<TSelf> where TSelf: IUnmanagedInterfaceType<TSelf>
{
    /// <summary>
    /// Get a pointer to the virtual method table of managed implementations of the unmanaged interface type.
    /// </summary>
    /// <returns>A pointer to the virtual method table of managed implementations of the unmanaged interface type</returns>
    /// <remarks>
    /// Implementation will be provided by a source generator if not explicitly implemented.
    /// This property can return <c>null</c>. If it does, then the interface is not supported for passing managed implementations to unmanaged code.
    /// </remarks>
    public abstract static unsafe void** ManagedVirtualMethodTable { get; }
}

[terrajobst]

• The default is ExceptionMarshalling.Custom
  • If the developer doesn't specify a behavior, should we give a warning?
  • If the developer doesn't specify a behavior, what's the default? Swallow seems problematic. It seems a clean crash would be preferable.
• Should we have a ExceptionFailFastMarshaller?
  • Or should this be an enum member? If that's the default (0 member) then this would nicely answer "what's the default behavior".
• Should we add an SEH marshaller for Windows?
• We should rename the marshallers by adding As
• @AaronRobinsonMSFT would prefer to wait a bit to see whether we actually need it

namespace System.Runtime.InteropServices.Marshalling;

public interface IUnmanagedObjectUnwrapper
{
    public static abstract object GetObjectForUnmanagedWrapper(void* ptr);
}

public interface IUnmanagedInterfaceType<TSelf>
    where TSelf: IUnmanagedInterfaceType<TSelf>
{
    public abstract static unsafe void** ManagedVirtualMethodTable { get; }
}

[AttributeUsage(AttributeTargets.Interface)]
public class UnmanagedObjectUnwrapperAttribute<TMapper> : Attribute
    where TMapper : IUnmanagedObjectUnwrapper
{
}

public sealed unsafe class ComWrappersUnwrapper : IUnmanagedObjectUnwrapper
{
    public static object GetObjectForUnmanagedWrapper(void* ptr);
}

[CustomMarshaller(typeof(Exception), MarshalMode.UnmanagedToManagedOut, typeof(ExceptionHResultMarshaller<>))]
public static class ExceptionAsHResultMarshaller<T>
    where T : unmanaged, INumber<T>
{
    public static T ConvertToUnmanaged(Exception e);
}

[CustomMarshaller(typeof(Exception), MarshalMode.UnmanagedToManagedOut, typeof(ExceptionNaNMarshaller<>))]
public static class ExceptionAsNaNMarshaller<T>
    where T : unmanaged, IFloatingPointIeee754<T>
{
    public static T ConvertToUnmanaged(Exception e);
}

[CustomMarshaller(typeof(Exception), MarshalMode.UnmanagedToManagedOut, typeof(ExceptionDefaultMarshaller<>))]
public static class ExceptionAsDefaultMarshaller<T>
    where T : unmanaged
{
    public static T ConvertToUnmanaged(Exception e);
}

[CustomMarshaller(typeof(Exception), MarshalMode.UnmanagedToManagedOut, typeof(SwallowExceptionMarshaller))]
public static class ExceptionAsVoidMarshaller
{
    public static void ConvertToUnmanaged(Exception e);
}

public enum ExceptionMarshalling
{
    Custom = 0,
    Com = 1
}

public enum MarshalDirection
{
    ManagedToUnmanaged = 0,
    UnmanagedToManaged = 1,
    Bidirectional = 2
}

[AttributeUsage(AttributeTargets.Method, AllowMultiple = false)]
public sealed class VirtualMethodIndexAttribute : Attribute
{
    public VirtualMethodIndexAttribute(int index);

    public int Index { get; }

    public bool ImplicitThisParameter { get; set; } = true;
    public StringMarshalling StringMarshalling { get; set; }
    public Type? StringMarshallingCustomType { get; set; }
    public bool SetLastError { get; set; }
    
    public MarshalDirection Direction { get; set; } = MarshalDirection.Bidirectional;

    public ExceptionMarshalling ExceptionMarshalling { get; set; }
    public Type? ExceptionMarshallingCustomType { get; set; }
}

[AaronRobinsonMSFT]

• Should we have a ExceptionFailFastMarshaller?
  • Or should this be an enum member? If that's the default (0 member) then this would nicely answer "what's the default behavior".
• Should we add an SEH marshaller for Windows?

Neither of these apply for the COM scenario so we can defer them. We are most interested in the below to satisfy the existing COM interop semantics for now.

• @AaronRobinsonMSFT would prefer to wait a bit to see whether we actually need it

@terrajobst Had to step out, sorry. The below subset from above makes perfect sense for .NET 8. Since this subset was approved, we can remove it from above and revisit the remaining APIs once we get more experience from the community in .NET 9.

[CustomMarshaller(typeof(Exception), MarshalMode.UnmanagedToManagedOut, typeof(ExceptionHResultMarshaller<>))]
public static class ExceptionAsHResultMarshaller<T>
    where T : unmanaged, INumber<T>
{
    public static T ConvertToUnmanaged(Exception e);
}

[CustomMarshaller(typeof(Exception), MarshalMode.UnmanagedToManagedOut, typeof(ExceptionNaNMarshaller<>))]
public static class ExceptionAsNaNMarshaller<T>
    where T : unmanaged, IFloatingPointIeee754<T>
{
    public static T ConvertToUnmanaged(Exception e);
}

[CustomMarshaller(typeof(Exception), MarshalMode.UnmanagedToManagedOut, typeof(ExceptionDefaultMarshaller<>))]
public static class ExceptionAsDefaultMarshaller<T>
    where T : unmanaged
{
    public static T ConvertToUnmanaged(Exception e);
}

[CustomMarshaller(typeof(Exception), MarshalMode.UnmanagedToManagedOut, typeof(SwallowExceptionMarshaller))]
public static class ExceptionAsVoidMarshaller
{
    public static void ConvertToUnmanaged(Exception e);
}