Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

[FEA] support barrier mode for mapInPandas/mapInArrow #10344

Closed
wbo4958 opened this issue Jan 31, 2024 · 3 comments · Fixed by #10343 or #10375
Closed

[FEA] support barrier mode for mapInPandas/mapInArrow #10344

wbo4958 opened this issue Jan 31, 2024 · 3 comments · Fixed by #10343 or #10375
Labels
feature request New feature or request

Comments

@wbo4958
Copy link
Collaborator

wbo4958 commented Jan 31, 2024

Is your feature request related to a problem? Please describe.

apache/spark#40520 has introduced barrier mode for mapInPandas/mapInArrow, I hope spark-rapids can support this feature.

How to test?

spark.range(1, 10, 1, 100).mapInPandas(lambda x: x, "id long", True)

to check if all tasks are launched at the same time.
@wbo4958 wbo4958 added feature request New feature or request ? - Needs Triage Need team to review and classify labels Jan 31, 2024
@wbo4958 wbo4958 mentioned this issue Jan 31, 2024
Merged
@sameerz sameerz mentioned this issue Feb 2, 2024
Merged
@wbo4958 wbo4958 mentioned this issue Feb 2, 2024
Merged
@wbo4958 wbo4958 reopened this Feb 2, 2024
@wbo4958
Copy link
Collaborator Author

wbo4958 commented Feb 2, 2024

PR got merged for branch 24.04. Close this issue.

@wbo4958 wbo4958 closed this as completed Feb 2, 2024
@mattahrens mattahrens removed the ? - Needs Triage Need team to review and classify label Feb 2, 2024
@revans2 revans2 reopened this Feb 2, 2024
@revans2
Copy link
Collaborator

revans2 commented Feb 2, 2024

reopening because it is being reverted again in 24.04 because it does not build on databricks

@wbo4958 wbo4958 mentioned this issue Feb 5, 2024
Merged
@wbo4958
Copy link
Collaborator Author

wbo4958 commented Feb 7, 2024

PR got merged. close it

@wbo4958 wbo4958 closed this as completed Feb 7, 2024
# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees
No one assigned
Labels
feature request New feature or request
Projects
None yet
Milestone
No milestone
3 participants
@wbo4958 @revans2 @mattahrens