[.NET 9] DllImport with Custom Marshaler Passes non-null address when null value for field is specified.

[Sewer56]

Description

In .NET 9 (Preview 2 and newer), passing null to a parameter which uses a CustomMarshaler results in a non-zero value passed to native code. This can lead to memory corruption or invalid memory reads, depending on what the native code does with the value.

Reproduction Steps

Attached reproduction: cimgui-stub.zip

To Reproduce

Run the csharp project (csharp.csproj) provided.

When ran with .NET 9 Preview 1 or older, the result is:

igDragInt called with:
  label address: 2941681900960
  v address: 750695999576
  format address: 0
  label value: label
  v value: 0
  v_speed: 0.1
  v_min: 0
  v_max: 100
  format: null
  flags: 0

When ran with .NET 9 Preview 2 or newer the result is:

igDragInt called with:
  label address: 1824962101440
  v address: 114923399832
  format address: 14829735431805717965
  label value: label
  v value: 0
  v_speed: 0.1
  v_min: 0
  v_max: 100
Fatal error. 0xC0000005
   at ImGui.DragInt(System.String, Int32 ByRef, Single, Int32, Int32, System.String, Int32)
   at Program.<Main>$(System.String[])

When null is passed to a string parameter with any CustomMarshaler specified, the generated DllImport stub passes a non-zero address. (Note: Marshaler is skipped, so implementation of marshaler used with CustomMarshaler is not relevant)

---

To Compile the Rust Library/ Native Code

[Note: I've included an x64 precompiled DLL in the csharp folder out of the box, this is only needed if you want to change the native DllImport target, e.g. reduce param count for debugging]

1. Install rustup: https://static.rust-lang.org/rustup/dist/x86_64-pc-windows-msvc/rustup-init.exe
2. cd into cimgui_export
3. cargo build --release
4. Result is in target/release/cimgui_export.dll
5. Paste into csharp folder, replacing the existing file.

Expected behavior

format address should be 0 (null). Instead an unexpected address is used.

Actual behavior

format address is not 0 (null)

Regression?

.NET 9 (Preview 1) and older work fine.
Regression is introduced in Preview 2.

Known Workarounds

No response

Configuration

N/A

This reproduces in both x86 and x64 on Windows.

Other information

The reproduction is small enough that it can be included here for convenience:

var v = 0;
while (true)
{
    ImGui.DragInt("label", ref v, 0.1f, 0, 100, null!, 0);
    Thread.Sleep(1000);
}

static class ImGui
{
    [DllImport("cimgui_export", CallingConvention = CallingConvention.Cdecl, EntryPoint = "igDragInt")]
    [SuppressUnmanagedCodeSecurity]
    [return: MarshalAs(UnmanagedType.I1)]
    public static extern bool DragInt(
        [MarshalAs(UnmanagedType.CustomMarshaler, MarshalType = "csharp.UTF8Marshaller, csharp, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null")]
        string label,
        ref int v,
        float v_speed,
        int v_min,
        int v_max,
        [MarshalAs(UnmanagedType.CustomMarshaler, MarshalType = "csharp.UTF8Marshaller, csharp, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null")]
        string format,
        int flags
    );
}

Rust part:

use std::ffi::CStr;
use std::os::raw::{c_char, c_int, c_float, c_uchar};

#[no_mangle]
pub extern "C" fn igDragInt(
    label: *const c_char,
    v: *mut c_int,
    v_speed: c_float,
    v_min: c_int,
    v_max: c_int,
    format: *const c_char,
    flags: c_int,
) -> c_uchar {
    // Print addresses of all pointers
    println!("igDragInt called with:");
    println!("  label address: {:?}", label as usize);
    println!("  v address: {:?}", v as usize);
    println!("  format address: {:?}", format as usize);

    // Handle `label`
    if label.is_null() {
        println!("  label: null");
    } else {
        // Convert C string to Rust string
        let c_label = unsafe { CStr::from_ptr(label) };
        match c_label.to_str() {
            Ok(s) => println!("  label value: {}", s),
            Err(_) => println!("  label: Invalid UTF-8 string"),
        }
    }

    // Handle `v`
    if v.is_null() {
        println!("  v: null");
    } else {
        // Safely read the integer value
        let value = unsafe { *v };
        println!("  v value: {}", value);
    }

    // Print other parameters
    println!("  v_speed: {}", v_speed);
    println!("  v_min: {}", v_min);
    println!("  v_max: {}", v_max);

    // Handle `format`
    if format.is_null() {
        println!("  format: null");
    } else {
        // Convert C string to Rust string
        let c_format = unsafe { CStr::from_ptr(format) };
        match c_format.to_str() {
            Ok(s) => println!("  format value: {}", s),
            Err(_) => println!("  format: Invalid UTF-8 string"),
        }
    }

    println!("  flags: {}", flags);

    // For demonstration, return true (1) to indicate success
    1
}

I can across this issue when trying older code with .NET 9 that used dear imgui bindings made with CppSharp. So in the repro I tried to model that.

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/interop-contrib
See info in area-owners.md if you want to be subscribed.

[huoyaoyuan]

You are using the wrong marshalling mechanism. System.Runtime.InteropServices.ICustomMarshaler is for COM interop. See the documentation at https://learn.microsoft.com/dotnet/fundamentals/runtime-libraries/system-runtime-interopservices-icustommarshaler and https://learn.microsoft.com/dotnet/standard/native-interop/customize-parameter-marshalling#custom-marshallers .

To marshal a string as UTF8 in .NET Standard, you can either specify [MarshalAs((UnmanagedType)48)] to force cast the value of LPUTF8Str, or use the new LibraryImport attribute with StringMarshalling = StringMarshalling.Utf8. See the documentation at https://learn.microsoft.com/dotnet/standard/native-interop/pinvoke-source-generation .

[Sewer56]

That is right. I am also aware that there are better ways to express such a requirement in modern .NET.

I used this code specifically as an example because this is the wrapping code that mono/CppSharp produces by default when a C/C++ header lists a char* as parameter. I.e. all projects (libraries) deriving from CppSharp in last 10 years or so are affected.

I simply ran across the issue while working (using) one of these projects.

Given there didn't seem to be a relevant entry in list of breaking changes for .NET 9, I assumed this was unintentional, or not noted.

[huoyaoyuan]

The format address value is 0xCDCDCDCDCDCDCDCD, which is used for uninitialized heap memory. It looks like some memory is not correctly filled.

[AaronRobinsonMSFT]

runtime/src/coreclr/System.Private.CoreLib/src/System/StubHelpers.cs

Lines 800 to 806 in dfc2b85

	internal static unsafe void ConvertContentsToNative(ICustomMarshaler marshaler, in object pManagedHome, IntPtr* pNativeHome)
	{
	// COMPAT: We never pass null to MarshalManagedToNative.
	if (pManagedHome is null)
	{
	return;
	}

Unclear if this is in .NET 9, but the above branch should also have *pNativeHome = IntPtr.Zero;, unless this is coming from a guaranteed initialized context.

[AaronRobinsonMSFT]

@Sewer56 Can you try this against an RC of .NET 9? Preview 1 and preview 2 are from very early in the year and lots has changed. Is this behavior in the latest .NET 9 RC?

[Sewer56]

@AaronRobinsonMSFT Yes, I originally ran into this bug in RC2 last weekend.

I stated this as a regression since Preview 2 in the issue description because I've gone through the effort to narrow it down to the first binary release where the issue is reproducible.

i.e. it's a commit between Preview 1 and Preview 2

[AaronRobinsonMSFT]

@Sewer56 Ah. Thanks for tracking down the general area. This is likely related to some of our many conversions from native to managed code - these kinds of bugs are too easy to introduce in this conversion. We'll get it addressed in .NET 9, but it will probably go out in an update and not GA for .NET 9.

[Sewer56]

That's alright, thanks for a timely response.

I imagine the timeline for GA would be very tight here, since dotNETConf is 3 weeks away; so that's within expectation.

Hopefully all goes well for another smooth launch 👍

[Sewer56]

Oh, it did actually ship in release; unexpected and very welcome, thank you!