Scalability and Concurrency in spatie/laravel-event-sourcing

[b-barry]

I'm exploring spatie/laravel-event-sourcing for a potentially high-volume project and have questions about its scalability:

• How does the package handle large volumes of events? Are there known limitations?
• What strategies are recommended for vertical and horizontal scaling?
• How is concurrency managed to prevent race conditions and ensure consistency? (I saw some questions but I don't if everything is implemented)
• Are there performance considerations or best practices for managing growing event stores?
• How do projections perform with large event streams? Any optimization techniques?
• Are there benchmarks or case studies demonstrating the package's performance at scale?

Any insights or real-world experiences would be valuable. Thanks!

[inmanturbo]

Spatie maintains it for a huge app. They can't share details but they have shared the size in an interview or something or other. Multi tenant, millions of rows per tenant.

Concurrency is an issue I don't think was ever completely solved. If you need it I recommend keeping your projectors synchronous and manually dispatching your own jobs with some sort of locking mechanism.

Projections (read model) performance in entirely dependent on you as they are not read from the "stream", they are just regular pieces of state, like database entries, that written on event occurrence.

Aggregate reads and retrieval is your "keep up model" and retrieval can slow down over time if you use the keep up method. Snapshot functionality is provided to mitigate this and is effective when applied correctly. However huge aggregates are a bit of an anti-pattern. You usually should only use this technique for smaller and short lived aggregates (think shopping cart).

The package doesn't provide any functionality to help with horizontal scaling, but you could easily divide your app onto many micro services, say one per aggregate.

I'm not aware of any benchmarks being documented by anyone regarding this package.

[b-barry]

Hey!
Thanks a ton for this awesome response. You've really helped me out!

I'm particularly interested in the concurrency part you mentioned. Could you explain a bit more about "manually dispatching your own jobs with some sort of locking mechanism"? I'm not sure how to implement this, especially considering our setup:

We're plugging into Kafka for our events
We're looking at about 18,000 events per day, and that number's growing

How would you recommend handling concurrency in this scenario? Any tips or examples for implementing the locking mechanism you mentioned?
Really appreciate your expertise here.
Thanks again!

[inmanturbo]

Much has been written on the subject here: #214.

In the past I've used a fairly simple technique, that is manually dispatching and batching jobs on the queue. I kept the aggregate->recordThat => projector->onHappened flow synchronous and simply called the command or aggregate from within an asynchronous job using laravel's standard batching and retry, which you can read about in laravel's docs: https://laravel.com/docs/11.x/queues#main-content.

[b-barry]

Thanks @inmanturbo I will check it