Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

[improve][fn] Make producer cache bounded and expiring in Functions/Connectors #22945

Merged
merged 4 commits into from
Jun 25, 2024

Conversation

lhotari
Copy link
Member

@lhotari lhotari commented Jun 19, 2024

Motivation

In Pulsar Functions and Pulsar IO Connectors, there's a Pulsar Producer cache, that
doesn't expire the producers and doesn't have a maximum size.
This is a problem for functions/connectors that produce to a large number of topics.

Modifications

  • Extract common ProducerCache
    • common Producer cache with separate cache areas for
      • producers created by calling Context, SinkContext, SourceContext methods
      • producers created in Pulsar Sources
        • multiple topics are possible by returning destination topics by SinkRecord.getDestinationTopic call
  • Keep backwards compatibility on the cache key
    • useThreadLocalProducers will use the Thread's ID as part of the key so that each thread has it's own isolated producer
    • In Sources, the SinkRecord.getPartitionId() is used as part of the cache key
  • Cache size is limited by the number of partitions in the producer
    • Default limit of the cache is 10000 partitions in total. Older entries will get removed from the cache if the cache overflows
  • Default cache entry expiration in 300 seconds after last access

Documentation

  • doc
  • doc-required
  • doc-not-needed
  • doc-complete

@lhotari lhotari added this to the 3.4.0 milestone Jun 19, 2024
@lhotari lhotari self-assigned this Jun 19, 2024
@github-actions github-actions bot added the doc-not-needed Your PR changes do not impact docs label Jun 19, 2024
@lhotari lhotari requested a review from nlu90 June 20, 2024 05:34
@lhotari
Copy link
Member Author

lhotari commented Jun 20, 2024

There's also #22950 improving code and fixing bugs in this area.

@lhotari lhotari force-pushed the lh-producer-cache-expiration branch 3 times, most recently from 2992fb6 to 27d42b7 Compare June 20, 2024 13:33
@lhotari lhotari force-pushed the lh-producer-cache-expiration branch from 27d42b7 to 6b36081 Compare June 20, 2024 13:47
@lhotari lhotari merged commit 6fe8100 into apache:master Jun 25, 2024
47 of 51 checks passed
lhotari added a commit that referenced this pull request Jun 25, 2024
lhotari added a commit that referenced this pull request Jun 25, 2024
…onnectors (#22945)

(cherry picked from commit 6fe8100)
(cherry picked from commit f10708f)
lhotari added a commit that referenced this pull request Jun 25, 2024
lhotari added a commit that referenced this pull request Jun 25, 2024
nikhil-ctds pushed a commit to datastax/pulsar that referenced this pull request Jun 27, 2024
…onnectors (apache#22945)

(cherry picked from commit 6fe8100)
(cherry picked from commit f10708f)
(cherry picked from commit 9f5432a)
nikhil-ctds pushed a commit to datastax/pulsar that referenced this pull request Jun 27, 2024
…onnectors (apache#22945)

(cherry picked from commit 6fe8100)
(cherry picked from commit f10708f)
(cherry picked from commit 9f5432a)
srinath-ctds pushed a commit to datastax/pulsar that referenced this pull request Jul 1, 2024
…onnectors (apache#22945)

(cherry picked from commit 6fe8100)
(cherry picked from commit f10708f)
(cherry picked from commit 9f5432a)
hanmz pushed a commit to hanmz/pulsar that referenced this pull request Feb 12, 2025
# for free to join this conversation on GitHub. Already have an account? # to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants