Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Write to cache entries read but not served to Consumers #102

Closed
wants to merge 2 commits into from

Conversation

eolivelli
Copy link
Collaborator

@eolivelli eolivelli commented Jul 7, 2022

This is a POC PR to solve apache#16421

the problem that it is trying to solve is that when you are using batching the consumer receives N entries, containing M messages (with M >> N), then it exhaust quickly its permits and at the next dispatcher round the dispatcher loads too many messages from storage (BK or TieredStorage).
there is a bunch of entries that are read but cannot be dispatched, so we are wasting precious reads.
my latest change adds back to the "cache" the entries that have not been dispatched, so that at the second round the entries are still in the cache and we save from reading from tiered storage (or even BK!) again the same data.
This will be a good improvement also for non offloaded shared subscription.
the change applies only to Shared Subscriptions

@eolivelli
Copy link
Collaborator Author

this patch is based on #101

@eolivelli eolivelli closed this Aug 24, 2022
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant