Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

MemoryPackSerializationException: Length header size is larger than buffer size when serializing large collection of metrics #356

Open
proxusiiotplatform opened this issue Dec 24, 2024 · 2 comments

Comments

@proxusiiotplatform
Copy link

I'm experiencing an issue when trying to serialize/deserialize a collection of metrics data using MemoryPack with NATS.

When attempting to serialize and send a large collection of metrics via NATS, I'm getting this error:
"MemoryPack.MemoryPackSerializationException: Length header size is larger than buffer size, length: 33554432"

Code Example

[MemoryPackable]
public partial record ProxusActorMetrics {
    public string Device { get; set; }
    public long TotalRequests { get; set; }
    public long SuccessfulRequests { get; set; }
    // ... other metrics properties
}

// NATS Serializer
public sealed class NatsMemoryPackContextSerializer<T> : INatsSerializer<T> {
    public static readonly INatsSerializer<T> Default = new NatsMemoryPackContextSerializer<T>();

    public void Serialize(IBufferWriter<byte> bufferWriter, T value) {
        MemoryPackSerializer.Serialize(bufferWriter, value);
    }

    public T? Deserialize(in ReadOnlySequence<byte> buffer) =>
        MemoryPackSerializer.Deserialize<T>(buffer);
}

// Publishing code
var metricsCollection = new List<ProxusActorMetrics>();
// ... populate metrics
await fJs.PublishAsync("metrics.topic", metricsCollection, cancellationToken: fCts.Token);
@proxusiiotplatform
Copy link
Author

Any update?

@neuecc
Copy link
Member

neuecc commented Feb 5, 2025

Thank you.
The phenomenon occurs when the ReadOnlySequence is less than the byte string to be deserialised.
It is difficult to tell whether this is a problem with NATS or MemoryPack.
If there is a code that can reproduce it easily, we can investigate it.

# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants