Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

[Enhancement]: Batching data when forwarding stream delete in load procedure #40558

Open
1 task done
congqixia opened this issue Mar 11, 2025 · 0 comments
Open
1 task done

[Enhancement]: Batching data when forwarding stream delete in load procedure #40558

congqixia opened this issue Mar 11, 2025 · 0 comments
Labels
kind/enhancement Issues or changes related to enhancement

Comments

@congqixia
Copy link
Contributor

Is there an existing issue for this?

  • I have searched the existing issues

What would you like to be added?

Add buffered batch util function to forward streaming delete

Why is this needed?

When stream delete size is large, loading segment shall forward these deletion with cherry-picking hit entries by segment BF.

  • If the hit data set is large or
  • Loading segment is growing one(use always true candidate by mistake) or skip bf load switch is on

The memory usage will be multiplied and could cause potential OOM

Anything else?

No response
@congqixia congqixia added the kind/enhancement Issues or changes related to enhancement label Mar 11, 2025
congqixia added a commit to congqixia/milvus that referenced this issue Mar 11, 2025
@congqixia
enhance: Add buffer forwarder for stream delta loading
6e45cb9 
See also milvus-io#40558
Related to milvus-io#35303 & milvus-io#38066 as well

This PR:
- Add `BufferedForward` to limit memory usage forwarding stream delete
- Add `UseLoad` flag to determine `Delete` shall use `segment.Delete` or
  `segment.LoadDelta`
- Fix delegator accidentally use always true candidate while load
  streaming delta

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
@congqixia congqixia mentioned this issue Mar 11, 2025
Open
congqixia added a commit to congqixia/milvus that referenced this issue Mar 12, 2025
@congqixia
enhance: Add buffer forwarder for stream delta loading
7cb186d 
See also milvus-io#40558
Related to milvus-io#35303 & milvus-io#38066 as well

This PR:
- Add `BufferedForward` to limit memory usage forwarding stream delete
- Add `UseLoad` flag to determine `Delete` shall use `segment.Delete` or
  `segment.LoadDelta`
- Fix delegator accidentally use always true candidate while load
  streaming delta

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees
No one assigned
Labels
kind/enhancement Issues or changes related to enhancement
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@congqixia