You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I'm using reflection to write a IEnumerable<POCO> to Parquet. In setting up data types for each property, I can see on debug that the writerInstance that calls the Write method has a correct List<ChoParquetRecordConfiguration> object (Nullable Integers, etc).
However when the Write method is called, all Nullable types that are Numeric are just written as Strings. Don't know if reflection is getting in the way.
Example of code:
TypedtoType=GetDtoType(initReqBodyData.PathRoot,initReqBodyData.PathName);// returns Type Root.Sales.Data.SalesItemdynamicrestResponseDto=JsonConvert.DeserializeObject(restResponse.Content,typeof(IList<>).MakeGenericType(dtoType));IEnumerable<PropertyInfo>dtoProperties=dtoType.GetAllProperties();Dictionary<string,Type>fieldsAndTypes=SetFieldTypes(dtoProperties);TypegenericParquetWriter=typeof(ChoParquetWriter<>).MakeGenericType(dtoType);dynamicwriterInstance=ChoActivator.CreateInstance(genericParquetWriter,newobject[]{localPath,newChoParquetRecordConfiguration(){TypeConverterFormatSpec=newChoTypeConverterFormatSpec{EnumFormat=ChoEnumFormatSpec.Name},TreatDateTimeAsString=false,CultureName="en-001",AutoDiscoverFieldTypes=false,CompressionLevel=CompressionLevel.Fastest,CompressionMethod=Parquet.CompressionMethod.Snappy,Append=File.Exists(localPath),FieldValueTrimOption=ChoFieldValueTrimOption.Trim,}});foreach(stringpropertyNameindtoProperties.Select(c =>c.Name)){writerInstance.WithField(propertyName,fieldType:fieldsAndTypes[propertyName]);}writerInstance.Write(restResponseDto);writerInstance.Close();writerInstance.Dispose();
From the above, the detailAmount is a Nullable Double in the ChoParquetRecordConfiguration but when I examine the column in the Parquet file it's a VARCHAR / String.
The text was updated successfully, but these errors were encountered:
Hi,
I'm using reflection to write a
IEnumerable<POCO>
to Parquet. In setting up data types for each property, I can see on debug that the writerInstance that calls theWrite
method has a correctList<ChoParquetRecordConfiguration>
object (Nullable Integers, etc).However when the
Write
method is called, all Nullable types that are Numeric are just written as Strings. Don't know if reflection is getting in the way.Example of code:
Example of JSON:
From the above, the
detailAmount
is a Nullable Double in theChoParquetRecordConfiguration
but when I examine the column in the Parquet file it's a VARCHAR / String.The text was updated successfully, but these errors were encountered: