Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

[GCP DWS] Fix None issue when no provision timeout is provided #3835

Merged
merged 3 commits into from
Aug 16, 2024
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions sky/provision/gcp/instance_utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -1081,16 +1081,17 @@ def create_instances(
run_duration=managed_instance_group_config['run_duration'])
cls.wait_for_operation(operation, project_id, zone=zone)

provision_timeout = managed_instance_group_config.get('provision_timeout')
if provision_timeout is None:
provision_timeout = constants.DEFAULT_MANAGED_INSTANCE_GROUP_PROVISION_TIMEOUT
# This will block the provisioning until the nodes are ready, which
# makes the failover not effective. We rely on the request timeout set
# by user to trigger failover.
mig_utils.wait_for_managed_group_to_be_stable(
project_id,
zone,
managed_instance_group_name,
timeout=managed_instance_group_config.get(
'provision_timeout',
constants.DEFAULT_MANAGED_INSTANCE_GROUP_PROVISION_TIMEOUT))
timeout=provision_timeout)

pending_running_instance_names = cls._add_labels_and_find_head(
cluster_name, project_id, zone, labels, potential_head_instances)
Expand Down
1 change: 1 addition & 0 deletions sky/provision/gcp/mig_utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -207,3 +207,4 @@ def wait_for_managed_group_to_be_stable(project_id: str, zone: str,
except subprocess.CalledProcessError as e:
stderr = e.stderr.decode('ascii')
logger.info(stderr)
raise
Loading