Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

[FEA] Disable Spark UI for integration tests #10227

Closed
revans2 opened this issue Jan 19, 2024 · 2 comments
Closed

[FEA] Disable Spark UI for integration tests #10227

revans2 opened this issue Jan 19, 2024 · 2 comments
Assignees
Labels
build Related to CI / CD or cleanly building

Comments

@revans2
Copy link
Collaborator

revans2 commented Jan 19, 2024

Is your feature request related to a problem? Please describe.
I recently had a CI job fail because the UI could not bind to the desired port. We don't need a UI when we run the integration tests, especially in CI. It is also a potential security problem if the environment the tests are running in are not set up properly. We should probably just set spark.ui.enabled to false for our integration test runs by default. Especially if we know that they are running in local mode.

@revans2 revans2 added ? - Needs Triage Need team to review and classify task Work required that improves the product but is not user facing labels Jan 19, 2024
@abellina
Copy link
Collaborator

I think the standalone cluster we run for the shuffle tests should also take this config change.

@GaryShen2008
Copy link
Collaborator

ok, I think we need to config spark.ui.enabled=false.

# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
build Related to CI / CD or cleanly building
Projects
None yet
Development

No branches or pull requests

6 participants