Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Enable custom cluster managers #133

Merged
merged 1 commit into from
May 8, 2022
Merged

Enable custom cluster managers #133

merged 1 commit into from
May 8, 2022

Conversation

MilesCranmer
Copy link
Owner

This let's PySR launch jobs over a cluster. If you are on a slurm cluster (on the head node of a job), you simply pass cluster_manager="slurm" to PySRRegressor, and it will set up a bunch of workers over the slurm job. Likewise, the other cluster managers listed on https://github.com/JuliaParallel/ClusterManagers.jl can be used.

The one unfortunate thing is I don't think this functionality can be unit-tested; it will just be manual tests on clusters.

cc @kazewong!

@MilesCranmer
Copy link
Owner Author

MilesCranmer commented May 8, 2022

This model:

model = PySRRegressor(
     unary_operators=["cos"],
     procs=40 * 4,
     populations=40 * 4 * 2,
     population_size=1000,
     ncyclesperiteration=3000,
     cluster_manager="slurm",  # < Only change required
)

will launch 160 workers over a slurm job (4 nodes in this case), when ran from the head node.

Here's what the usage looks like:

Screen Shot 2022-05-07 at 10 30 40 PM

Keep in mind this was launched from within IPython on the head node!

@MilesCranmer MilesCranmer merged commit fc75036 into master May 8, 2022
@MilesCranmer MilesCranmer deleted the cluster-management branch May 8, 2022 12:55
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant