Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

add a Processor Pool Service in logisland core #589

Open
oalam opened this issue Aug 31, 2021 · 0 comments
Open

add a Processor Pool Service in logisland core #589

oalam opened this issue Aug 31, 2021 · 0 comments
Labels
core framework, processor, doc, build, ... feature

Comments

@oalam
Copy link
Member

oalam commented Aug 31, 2021

pipeline objects (aka processors) are intanciated at each micro-batch by spark executors. This leads to too many objects creation and GC purge.
we could instead lazily create a pool of processors into a CoreControllerService that handles a pool of objects needed for record processing.

may be with https://commons.apache.org/proper/commons-pool/

@oalam oalam added feature core framework, processor, doc, build, ... labels Aug 31, 2021
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
core framework, processor, doc, build, ... feature
Projects
None yet
Development

No branches or pull requests

1 participant