Warn in CustomMarshallerAnalyzer if the MarshaMode is not valid (#87198)

I ran into an issue when making some tests where I tried to use MarshalMode as a flags enum, which lead to the generated code not being correct, but no warning was showing up. This PR creates a warning if you try to do an operation on the MarshalMode argument or if a casted int value is not defined in the MarshalMode enum.

Fixes #87139

src/libraries/System.Runtime.InteropServices/gen/LibraryImportGenerator/Analyzers/CustomMarshallerAttributeAnalyzer.cs

@@ -4,13 +4,9 @@
 using System;
 using System.Collections.Generic;
 using System.Collections.Immutable;
-using System.Diagnostics;
 using System.Diagnostics.CodeAnalysis;
 using System.Linq;
-using System.Runtime.InteropServices;
-using System.Text;
 using Microsoft.CodeAnalysis;
-using Microsoft.CodeAnalysis.CSharp;
 using Microsoft.CodeAnalysis.CSharp.Syntax;
 using Microsoft.CodeAnalysis.Diagnostics;
 using Microsoft.CodeAnalysis.DotnetRuntime.Extensions;
@@ -395,6 +391,17 @@ public static ImmutableDictionary<string, string> CreateDiagnosticPropertiesForM
                 isEnabledByDefault: true,
                 description: GetResourceString(nameof(SR.ManagedTypeMustBeNonNullDescription)));
 
+        /// <inheritdoc cref="SR.MarshalModeMustBeValidEnumValue" />
+        public static readonly DiagnosticDescriptor MarshalModeMustBeValidValue =
+            new DiagnosticDescriptor(
+                Ids.InvalidCustomMarshallerAttributeUsage,
+                GetResourceString(nameof(SR.InvalidMarshalModeTitle)),
+                GetResourceString(nameof(SR.MarshalModeMustBeValidEnumValue)),
+                Category,
+                DiagnosticSeverity.Error,
+                isEnabledByDefault: true,
+                description: GetResourceString(nameof(SR.MarshalModeMustBeValidEnumValue)));
+
         // We are intentionally using the same diagnostic IDs as the parent type.
         // These diagnostics are the same diagnostics, but with a different severity,
         // as the Default marshaller shape can have support for the managed-to-unmanaged shape
@@ -676,11 +683,19 @@ public void AnalyzeAttribute(OperationAnalysisContext context)
                             {
                                 return;
                             }
+                            var marshalModeArgument = attrCreation.GetArgumentByOrdinal(1);
+                            if (marshalModeArgument.Value is not IFieldReferenceOperation { ConstantValue.Value: var marshalMode }
+                                || !Enum.IsDefined(typeof(MarshalMode), (MarshalMode)marshalMode))
+                            {
+                                DiagnosticReporter marshalModeReporter = DiagnosticReporter.CreateForLocation(marshalModeArgument.Syntax.GetLocation(), context.ReportDiagnostic);
+                                marshalModeReporter.CreateAndReportDiagnostic(MarshalModeMustBeValidValue);
+                                return;
+                            }
 
                             AnalyzeMarshallerType(
                                 marshallerTypeReporter,
                                 managedType,
-                                (MarshalMode)attrCreation.GetArgumentByOrdinal(1).Value.ConstantValue.Value,
+                                (MarshalMode)marshalMode,
                                 (INamedTypeSymbol)marshallerType,
                                 ManualTypeMarshallingHelper.IsLinearCollectionEntryPoint(entryType));
                         }

src/libraries/System.Runtime.InteropServices/gen/LibraryImportGenerator/Resources/Strings.resx

@@ -1,17 +1,17 @@
 <?xml version="1.0" encoding="utf-8"?>
 <root>
-  <!-- 
-    Microsoft ResX Schema 
-    
+  <!--
+    Microsoft ResX Schema
+
     Version 2.0
-    
-    The primary goals of this format is to allow a simple XML format 
-    that is mostly human readable. The generation and parsing of the 
-    various data types are done through the TypeConverter classes 
+
+    The primary goals of this format is to allow a simple XML format
+    that is mostly human readable. The generation and parsing of the
+    various data types are done through the TypeConverter classes
     associated with the data types.
-    
+
     Example:
-    
+
     ... ado.net/XML headers & schema ...
     <resheader name="resmimetype">text/microsoft-resx</resheader>
     <resheader name="version">2.0</resheader>
@@ -26,36 +26,36 @@
         <value>[base64 mime encoded string representing a byte array form of the .NET Framework object]</value>
         <comment>This is a comment</comment>
     </data>
-                
-    There are any number of "resheader" rows that contain simple 
+
+    There are any number of "resheader" rows that contain simple
     name/value pairs.
-    
-    Each data row contains a name, and value. The row also contains a 
-    type or mimetype. Type corresponds to a .NET class that support 
-    text/value conversion through the TypeConverter architecture. 
-    Classes that don't support this are serialized and stored with the 
+
+    Each data row contains a name, and value. The row also contains a
+    type or mimetype. Type corresponds to a .NET class that support
+    text/value conversion through the TypeConverter architecture.
+    Classes that don't support this are serialized and stored with the
     mimetype set.
-    
-    The mimetype is used for serialized objects, and tells the 
-    ResXResourceReader how to depersist the object. This is currently not 
+
+    The mimetype is used for serialized objects, and tells the
+    ResXResourceReader how to depersist the object. This is currently not
     extensible. For a given mimetype the value must be set accordingly:
-    
-    Note - application/x-microsoft.net.object.binary.base64 is the format 
-    that the ResXResourceWriter will generate, however the reader can 
+
+    Note - application/x-microsoft.net.object.binary.base64 is the format
+    that the ResXResourceWriter will generate, however the reader can
     read any of the formats listed below.
-    
+
     mimetype: application/x-microsoft.net.object.binary.base64
-    value   : The object must be serialized with 
+    value   : The object must be serialized with
             : System.Runtime.Serialization.Formatters.Binary.BinaryFormatter
             : and then encoded with base64 encoding.
-    
+
     mimetype: application/x-microsoft.net.object.soap.base64
-    value   : The object must be serialized with 
+    value   : The object must be serialized with
             : System.Runtime.Serialization.Formatters.Soap.SoapFormatter
             : and then encoded with base64 encoding.
 
     mimetype: application/x-microsoft.net.object.bytearray.base64
-    value   : The object must be serialized into a byte array 
+    value   : The object must be serialized into a byte array
             : using a System.ComponentModel.TypeConverter
             : and then encoded with base64 encoding.
     -->
@@ -497,4 +497,10 @@
   <data name="ConvertToLibraryImportWithSuffixAddUnsafe" xml:space="preserve">
     <value>Convert to 'LibraryImport' with '{0}' suffix and enable unsafe code</value>
   </data>
+  <data name="InvalidMarshalModeTitle" xml:space="preserve">
+    <value>Invalid 'MarshalMode' value.</value>
+  </data>
+  <data name="MarshalModeMustBeValidEnumValue" xml:space="preserve">
+    <value>The 'marshalMode' argument of 'CustomMarshallerAttribute' must be a valid enum value of 'MarshalMode'.</value>
+  </data>
 </root>

src/libraries/System.Runtime.InteropServices/gen/LibraryImportGenerator/Resources/xlf/Strings.cs.xlf

@@ -217,6 +217,11 @@
         <target state="translated">Zadaný spravovaný typ je neplatný</target>
         <note />
       </trans-unit>
+      <trans-unit id="InvalidMarshalModeTitle">
+        <source>Invalid 'MarshalMode' value.</source>
+        <target state="new">Invalid 'MarshalMode' value.</target>
+        <note />
+      </trans-unit>
       <trans-unit id="InvalidMarshallerTypeTitle">
         <source>Specified marshaller type is invalid</source>
         <target state="translated">Zadaný zařazovací typ je neplatný</target>
@@ -312,6 +317,11 @@
         <target state="translated">Spravovaný typ pro zařazovací typ vstupního bodu {0} nesmí nabývat hodnoty null.</target>
         <note />
       </trans-unit>
+      <trans-unit id="MarshalModeMustBeValidEnumValue">
+        <source>The 'marshalMode' argument of 'CustomMarshallerAttribute' must be a valid enum value of 'MarshalMode'.</source>
+        <target state="new">The 'marshalMode' argument of 'CustomMarshallerAttribute' must be a valid enum value of 'MarshalMode'.</target>
+        <note />
+      </trans-unit>
       <trans-unit id="MarshallerTypeMustBeClosedOrMatchArityDescription">
         <source>The marshaller type must be a closed generic or have the same number of generic parameters as the managed type so the emitted code can use a specific instantiation.</source>
         <target state="translated">Zařazovací typ musí být uzavřený obecný typ nebo mít stejný počet obecných parametrů jako spravovaný typ, aby mohl vygenerovaný kód použít konkrétní vytvoření instance.</target>

src/libraries/System.Runtime.InteropServices/gen/LibraryImportGenerator/Resources/xlf/Strings.de.xlf

@@ -217,6 +217,11 @@
         <target state="translated">Der angegebene verwaltete Typ ist ungültig.</target>
         <note />
       </trans-unit>
+      <trans-unit id="InvalidMarshalModeTitle">
+        <source>Invalid 'MarshalMode' value.</source>
+        <target state="new">Invalid 'MarshalMode' value.</target>
+        <note />
+      </trans-unit>
       <trans-unit id="InvalidMarshallerTypeTitle">
         <source>Specified marshaller type is invalid</source>
         <target state="translated">Der angegebene Marshallertyp ist ungültig.</target>
@@ -312,6 +317,11 @@
         <target state="translated">Der verwaltete Typ für den Einstiegspunkt-Marshallertyp "{0}" darf nicht "NULL" sein</target>
         <note />
       </trans-unit>
+      <trans-unit id="MarshalModeMustBeValidEnumValue">
+        <source>The 'marshalMode' argument of 'CustomMarshallerAttribute' must be a valid enum value of 'MarshalMode'.</source>
+        <target state="new">The 'marshalMode' argument of 'CustomMarshallerAttribute' must be a valid enum value of 'MarshalMode'.</target>
+        <note />
+      </trans-unit>
       <trans-unit id="MarshallerTypeMustBeClosedOrMatchArityDescription">
         <source>The marshaller type must be a closed generic or have the same number of generic parameters as the managed type so the emitted code can use a specific instantiation.</source>
         <target state="translated">Der Marshaller-Typ muss ein geschlossener generischer Typ sein oder dieselbe Anzahl generischer Parameter wie der verwaltete Typ aufweisen, damit der ausgegebene Code eine bestimmte Instanziierung verwenden kann.</target>

src/libraries/System.Runtime.InteropServices/gen/LibraryImportGenerator/Resources/xlf/Strings.es.xlf

@@ -217,6 +217,11 @@
         <target state="translated">El tipo administrado especificado no es válido</target>
         <note />
       </trans-unit>
+      <trans-unit id="InvalidMarshalModeTitle">
+        <source>Invalid 'MarshalMode' value.</source>
+        <target state="new">Invalid 'MarshalMode' value.</target>
+        <note />
+      </trans-unit>
       <trans-unit id="InvalidMarshallerTypeTitle">
         <source>Specified marshaller type is invalid</source>
         <target state="translated">El tipo de serializador especificado no es válido</target>
@@ -312,6 +317,11 @@
         <target state="translated">El tipo administrado para el tipo de serializador de punto de entrada "{0}" no debe ser "null"</target>
         <note />
       </trans-unit>
+      <trans-unit id="MarshalModeMustBeValidEnumValue">
+        <source>The 'marshalMode' argument of 'CustomMarshallerAttribute' must be a valid enum value of 'MarshalMode'.</source>
+        <target state="new">The 'marshalMode' argument of 'CustomMarshallerAttribute' must be a valid enum value of 'MarshalMode'.</target>
+        <note />
+      </trans-unit>
       <trans-unit id="MarshallerTypeMustBeClosedOrMatchArityDescription">
         <source>The marshaller type must be a closed generic or have the same number of generic parameters as the managed type so the emitted code can use a specific instantiation.</source>
         <target state="translated">El tipo serializador debe ser un genérico cerrado o tener el mismo número de parámetros genéricos que el tipo administrado para que el código emitido pueda usar una creación de instancia específica.</target>

src/libraries/System.Runtime.InteropServices/gen/LibraryImportGenerator/Resources/xlf/Strings.fr.xlf

@@ -217,6 +217,11 @@
         <target state="translated">Le type managé spécifié n’est pas valide</target>
         <note />
       </trans-unit>
+      <trans-unit id="InvalidMarshalModeTitle">
+        <source>Invalid 'MarshalMode' value.</source>
+        <target state="new">Invalid 'MarshalMode' value.</target>
+        <note />
+      </trans-unit>
       <trans-unit id="InvalidMarshallerTypeTitle">
         <source>Specified marshaller type is invalid</source>
         <target state="translated">Le type de marshaleur spécifié n’est pas valide</target>
@@ -312,6 +317,11 @@
         <target state="translated">Le type managé du type marshaleur de point d’entrée « {0} » ne doit pas avoir la valeur 'null'</target>
         <note />
       </trans-unit>
+      <trans-unit id="MarshalModeMustBeValidEnumValue">
+        <source>The 'marshalMode' argument of 'CustomMarshallerAttribute' must be a valid enum value of 'MarshalMode'.</source>
+        <target state="new">The 'marshalMode' argument of 'CustomMarshallerAttribute' must be a valid enum value of 'MarshalMode'.</target>
+        <note />
+      </trans-unit>
       <trans-unit id="MarshallerTypeMustBeClosedOrMatchArityDescription">
         <source>The marshaller type must be a closed generic or have the same number of generic parameters as the managed type so the emitted code can use a specific instantiation.</source>
         <target state="translated">Le type marshaler doit être un générique fermé ou avoir le même nombre de paramètres génériques que le type managé pour que le code émis puisse utiliser une instanciation spécifique.</target>

src/libraries/System.Runtime.InteropServices/gen/LibraryImportGenerator/Resources/xlf/Strings.it.xlf

@@ -217,6 +217,11 @@
         <target state="translated">Il tipo gestito specificato non è valido</target>
         <note />
       </trans-unit>
+      <trans-unit id="InvalidMarshalModeTitle">
+        <source>Invalid 'MarshalMode' value.</source>
+        <target state="new">Invalid 'MarshalMode' value.</target>
+        <note />
+      </trans-unit>
       <trans-unit id="InvalidMarshallerTypeTitle">
         <source>Specified marshaller type is invalid</source>
         <target state="translated">Il tipo di marshaller specificato non è valido</target>
@@ -312,6 +317,11 @@
         <target state="translated">Il tipo gestito per il tipo di marshaller del punto di ingresso '{0}' non deve essere 'null'</target>
         <note />
       </trans-unit>
+      <trans-unit id="MarshalModeMustBeValidEnumValue">
+        <source>The 'marshalMode' argument of 'CustomMarshallerAttribute' must be a valid enum value of 'MarshalMode'.</source>
+        <target state="new">The 'marshalMode' argument of 'CustomMarshallerAttribute' must be a valid enum value of 'MarshalMode'.</target>
+        <note />
+      </trans-unit>
       <trans-unit id="MarshallerTypeMustBeClosedOrMatchArityDescription">
         <source>The marshaller type must be a closed generic or have the same number of generic parameters as the managed type so the emitted code can use a specific instantiation.</source>
         <target state="translated">Il tipo di marshaller deve essere un generico chiuso o avere lo stesso numero di parametri generici del tipo gestito, in modo che il codice generato possa usare una creazione di istanza specifica.</target>

src/libraries/System.Runtime.InteropServices/gen/LibraryImportGenerator/Resources/xlf/Strings.ja.xlf

@@ -217,6 +217,11 @@
         <target state="translated">指定されたマネージド型が無効です</target>
         <note />
       </trans-unit>
+      <trans-unit id="InvalidMarshalModeTitle">
+        <source>Invalid 'MarshalMode' value.</source>
+        <target state="new">Invalid 'MarshalMode' value.</target>
+        <note />
+      </trans-unit>
       <trans-unit id="InvalidMarshallerTypeTitle">
         <source>Specified marshaller type is invalid</source>
         <target state="translated">指定されたマーシャラー型が無効です</target>
@@ -312,6 +317,11 @@
         <target state="translated">エントリ ポイント マーシャラー型 '{0}' のマネージド型を 'null' にすることはできません</target>
         <note />
       </trans-unit>
+      <trans-unit id="MarshalModeMustBeValidEnumValue">
+        <source>The 'marshalMode' argument of 'CustomMarshallerAttribute' must be a valid enum value of 'MarshalMode'.</source>
+        <target state="new">The 'marshalMode' argument of 'CustomMarshallerAttribute' must be a valid enum value of 'MarshalMode'.</target>
+        <note />
+      </trans-unit>
       <trans-unit id="MarshallerTypeMustBeClosedOrMatchArityDescription">
         <source>The marshaller type must be a closed generic or have the same number of generic parameters as the managed type so the emitted code can use a specific instantiation.</source>
         <target state="translated">マーシャラー型は、クローズ ジェネリックであるか、マネージド型と同じ数のジェネリック パラメーターを持つ必要があります。これにより、生成されたコードが特定のインスタンス化を使用できるようになります。</target>

src/libraries/System.Runtime.InteropServices/gen/LibraryImportGenerator/Resources/xlf/Strings.ko.xlf

@@ -217,6 +217,11 @@
         <target state="translated">지정된 관리 유형이 잘못되었습니다.</target>
         <note />
       </trans-unit>
+      <trans-unit id="InvalidMarshalModeTitle">
+        <source>Invalid 'MarshalMode' value.</source>
+        <target state="new">Invalid 'MarshalMode' value.</target>
+        <note />
+      </trans-unit>
       <trans-unit id="InvalidMarshallerTypeTitle">
         <source>Specified marshaller type is invalid</source>
         <target state="translated">지정된 마샬러 유형이 잘못되었습니다.</target>
@@ -312,6 +317,11 @@
         <target state="translated">진입점 마샬러 유형 '{0}'의 관리 유형은 'null'이 아니어야 합니다.</target>
         <note />
       </trans-unit>
+      <trans-unit id="MarshalModeMustBeValidEnumValue">
+        <source>The 'marshalMode' argument of 'CustomMarshallerAttribute' must be a valid enum value of 'MarshalMode'.</source>
+        <target state="new">The 'marshalMode' argument of 'CustomMarshallerAttribute' must be a valid enum value of 'MarshalMode'.</target>
+        <note />
+      </trans-unit>
       <trans-unit id="MarshallerTypeMustBeClosedOrMatchArityDescription">
         <source>The marshaller type must be a closed generic or have the same number of generic parameters as the managed type so the emitted code can use a specific instantiation.</source>
         <target state="translated">마샬러 형식은 닫힌 제네릭이거나 관리되는 형식과 동일한 수의 제네릭 매개 변수가 있어야 내보낸 코드에서 특정 인스턴스화를 사용할 수 있습니다.</target>

src/libraries/System.Runtime.InteropServices/gen/LibraryImportGenerator/Resources/xlf/Strings.pl.xlf

@@ -217,6 +217,11 @@
         <target state="translated">Określony typ zarządzany jest nieprawidłowy</target>
         <note />
       </trans-unit>
+      <trans-unit id="InvalidMarshalModeTitle">
+        <source>Invalid 'MarshalMode' value.</source>
+        <target state="new">Invalid 'MarshalMode' value.</target>
+        <note />
+      </trans-unit>
       <trans-unit id="InvalidMarshallerTypeTitle">
         <source>Specified marshaller type is invalid</source>
         <target state="translated">Określony typ marshallera jest nieprawidłowy</target>
@@ -312,6 +317,11 @@
         <target state="translated">Typ zarządzany dla typu marshaller punktu wejścia „{0}” nie może mieć wartości „null”</target>
         <note />
       </trans-unit>
+      <trans-unit id="MarshalModeMustBeValidEnumValue">
+        <source>The 'marshalMode' argument of 'CustomMarshallerAttribute' must be a valid enum value of 'MarshalMode'.</source>
+        <target state="new">The 'marshalMode' argument of 'CustomMarshallerAttribute' must be a valid enum value of 'MarshalMode'.</target>
+        <note />
+      </trans-unit>
       <trans-unit id="MarshallerTypeMustBeClosedOrMatchArityDescription">
         <source>The marshaller type must be a closed generic or have the same number of generic parameters as the managed type so the emitted code can use a specific instantiation.</source>
         <target state="translated">Typ marshallera musi być zamkniętym typem ogólnym lub mieć taką samą liczbę parametrów ogólnych jak typ zarządzany, aby emitowany kod mógł używać określonego wystąpienia.</target>