Duplicate EDM types in ClientEdmModel due to race conditions cause sporadic ODataWriter validation errors

[habbes]

We have received reports of a sporadic error that affect some high-traffic services with a lot of nodes that cause the following exception to be thrown sporadically. The issue affects a small percentage of the nodes and occurs unpredictably, usually a short time after the service has been booted up. Once affected, the service will continue throwing the exception for any request that references the affected EDM type:

Microsoft.OData.Client.DataServiceRequestException:
An error occurred while processing this request. --->
Microsoft.OData.Client.DataServiceClientException:
The type 'NS.SomeType' of a resource in an expanded link is not compatible with the element type 'NS.SomeType' of the expanded link. Entries in an expanded link must have entity types that are assignable to the element type of the expanded link. --->
Microsoft.OData.ODataException:
The type 'NS.SomeType' of a resource in an expanded link is not compatible with the element type 'NS.SomeType' of the expanded link. Entries in an expanded link must have entity types that are assignable to the element type of the expanded link.
at Microsoft.OData.WriterValidationUtils.ValidateNestedResource(IEdmStructuredType resourceType, IEdmStructuredType pa rentNavigationPropertyType)
at Microsoft.OData.WriterValidator.ValidateResourceInNestedResourceInfo(IEdmStructuredType resourceType, IEdmStructuredType parentNavigationPropertyType)
at Microsoft.OData.ODataWriterCore.ValidateResourceForResourceSet(ODataResourceBase resource, ResourceBaseScope resourceScope)
at Microsoft.OData.ODataWriterCore.<>c.<WriteStartResourceImplementation>b__147_0(ODataWriterCore thisParam, ODataResource resourceParam)
at Microsoft.OData.ODataWriterCore.InterceptException[TArg0](Action`2 action, TArg0 arg0)
at Microsoft.OData.ODataWriterCore.WriteStartResourceImplementation(ODataResource resource)
at Microsoft.OData.Client.Serializer.WriteResource(ODataWriterWrapper writer, ODataResourceWrapper resourceWrapper)
at Microsoft.OData.Client.Serializer.WriteResourceSet(ODataWriterWrapper writer, ODataResourceSetWrapper resourceSetWrapper)
at Microsoft.OData.Client.Serializer.WriteItem(ODataWriterWrapper writer, ODataItemWrapper odataItemWrapper)
at Microsoft.OData.Client.Serializer.WriteNestedResourceInfo(ODataWriterWrapper writer, ODataNestedResourceInfoWrapper nestedResourceInfo)
at Microsoft.OData.Client.Serializer.WriteNestedComplexProperties(Object entity, String serverTypeName, IEnumerable`1 properties, ODataWriterWrapper odataWriter)
at Microsoft.OData.Client.Serializer.WriteEntry(EntityDescriptor entityDescriptor, IEnumerable`1 relatedLinks, ODataRequestMessageWrapper requestMessage)
at Microsoft.OData.Client.BaseSaveResult.CreateRequestData(EntityDescriptor entityDescriptor, ODataRequestMessageWrapper requestMessage)
at Microsoft.OData.Client.SaveResult.CreateNonBatchChangeData(Int32 index, ODataRequestMessageWrapper requestMessage)
at Microsoft.OData.Client.SaveResult.BeginCreateNextChange()
--- End of inner exception stack trace ---
--- End of inner exception stack trace ---
at Microsoft.OData.Client.SaveResult.HandleResponse()
at Microsoft.OData.Client.BaseSaveResult.EndRequest()
at Microsoft.OData.Client.DataServiceContext.EndSaveChanges(IAsyncResult asyncResult)

The validator tries to ensure that when you're writing a resource as the value of a property, that the EDM type of the resource is the same or derived from the EDM type of the property or container. As you can see, the exception message says that NS.SomeType and NS.SomeType do not match, even though they are presumably the same type (same fully qualified name).

Assemblies affected

Microsoft.OData.Client 7.x (reproduce in 7.12.4, but also affects earlier versions)

Reproduce steps

You may be able to reproduce the issue using the sample and instructions here: https://github.com/habbes/experiments/tree/master/ODataClientDuplicateEdmTypeRaceCondition

Expected result

The exception should not be thrown if the type of the resource being serialized matches that of the target property.

Actual result

The above exception is thrown even when the type of the resource matches that of the target property.

Additional detail

The WriteValidationUtils.ValidateNestedResource method eventually uses IsEquivalentTo extension method to compare two IEdmType objects. For complex and entity types it simply does a ReferenceEquals check against the two EDM types. This suggests that the issue may occur when we have two separate instances that represent the same EDM type.

By analyzing memory dumps obtained from an affected node, we did in fact find 2 instances of EdmComplexTypeWithDelayedProperties representing the same complex type (NS.SomeType in this example). We also found that one of the instances was stored in the ClientEdmModel.clrToEdmTypeCache and the other instance in ClientEdmModel.typeNameToClientTypeAnnotationCache. Both were in the same ClientEdmModel instance.

The EdmComplexTypeWithDelayedProperties are used by the ClientEdmModel as the IEdmTypes during serialization. The method responsible for creating them is ClientEdmModel.GetOrCreateEdmType, which in turn calls ClientEdmModel.GetOrCreateEdmTypeInternal. The latter is the one that actually creates these edm types and store them in the two caches if they don't exist. Access to either cache by this method is synchronized using a lock on the respective cache. But the two caches are synchronized independently, one thread can be first to acquire the lock to one cache, insert the edm type, release the lock and be the last to acquire the lock for the second cache after another thread has already inserted an item.

My hunch is that we have a scenario where two threads, T1 and T2, are racing to insert values into two independently synchronized caches. T1 manages to enter the first cache first and inserts an instance representing the complex type. T2 fails to insert the instance in the first cache because by the time it gets there, it already exists. However, T2 manages to enter the second cache first and inserts its instance of the complex type. T1 fails to insert its instance. The result is that we two caches end up with separate instances representing the same complex type.