Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Optimize transferring small objects for storage #3058

Draft
wants to merge 5 commits into
base: master
Choose a base branch
from

Conversation

hekaisheng
Copy link
Contributor

What do these changes do?

Related issue number

This PR implements a method that send small objects directly to receiver without split them into small blocks.

Fixes #xxxx

Check code requirements

  • tests added / passed (if needed)
  • Ensure all linting tests pass, see here for how to run them

@hekaisheng hekaisheng force-pushed the enh/simple-transfer branch from e62540e to 302553c Compare June 7, 2022 09:46
@hekaisheng hekaisheng force-pushed the enh/simple-transfer branch from 302553c to 99d086e Compare June 7, 2022 10:10
@qinxuye qinxuye changed the title Add _send_small_objects for sender to optimize sending small objects Optimize transferring small objects for storage Jun 8, 2022
@qinxuye qinxuye added the to be backported Indicate that the PR need to be backported to stable branch label Jun 8, 2022
@qinxuye
Copy link
Collaborator

qinxuye commented Jun 9, 2022

Seems vineyard cannot pass the ut for this PR, @sighingnow do you have to time to help to see what's going on?

@sighingnow
Copy link
Contributor

Seems vineyard cannot pass the ut for this PR, @sighingnow do you have to time to help to see what's going on?

I would take a look later today.

@qinxuye qinxuye modified the milestones: v0.10.0a1, v0.10.0a2 Jun 12, 2022
@sighingnow
Copy link
Contributor

Seems vineyard cannot pass the ut for this PR, @sighingnow do you have to time to help to see what's going on?

I would take a look later today.

Sorry for the delay... Will be fixed today.

@sighingnow
Copy link
Contributor

I failed to reproduce the failure if only run the single test case. But I found the test case test_execute_describe failed both for default storage and vineyard storage under this PR.

I have tried to debugging but failed to address the root cause. I think it is not a vineyard-only issue, @hekaisheng could you please help to try the test_execute_describe in your environment?

You could update the params to test the default storage only. The failre could be reproduced by

pytest -s -vvv mars/deploy/oscar/tests/test_local.py -k test_execute_describe

# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
mod: storage to be backported Indicate that the PR need to be backported to stable branch type: enhancement request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants