Skip to content

[Enhancement] Optimise memory performance in Parallel Processing #45

New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom
Closed
caffeine-addictt opened this issue Mar 4, 2024 · 0 comments · Fixed by #46
Closed

[Enhancement] Optimise memory performance in Parallel Processing #45

caffeine-addictt opened this issue Mar 4, 2024 · 0 comments · Fixed by #46
Assignees
@caffeine-addictt
Labels
Type: Enhancement Suggest an improvement for an existing feature.

Comments

@caffeine-addictt
Copy link
Member

Enhancement Request

Your issue may already be reported!
Please check out our active issues before creating one.

Is Your Enhancement Request Related to an Issue?

Currently, multiple new lists are created from the dataset and are passed into each child thread.
This is not memory efficient and could limit performance in large datasets.

Describe the Solution You'd Like

A more memory-efficient solution using generators or indexes.
@caffeine-addictt caffeine-addictt added the Type: Enhancement Suggest an improvement for an existing feature. label Mar 4, 2024
@caffeine-addictt caffeine-addictt self-assigned this Mar 4, 2024
@caffeine-addictt caffeine-addictt moved this from Backlog to In progress in Thread Overview Mar 4, 2024
caffeine-addictt added a commit that referenced this issue Mar 4, 2024
@caffeine-addictt
perf: Create and parse generator
09e4817 
Improves memory efficiency and runtime

Fixes #45
@caffeine-addictt caffeine-addictt mentioned this issue Mar 4, 2024
Merged
8 tasks
@caffeine-addictt caffeine-addictt linked a pull request Mar 4, 2024 that will close this issue
Perf: memory optimisation for ParallelProcessing #46
Merged
8 tasks
@github-project-automation github-project-automation bot moved this from In progress to Done in Thread Overview Mar 4, 2024
# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees

@caffeine-addictt caffeine-addictt

Labels
Type: Enhancement Suggest an improvement for an existing feature.
Projects
No open projects
Thread Overview
Status: Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Perf: memory optimisation for ParallelProcessing python-thread/thread
1 participant
@caffeine-addictt