[bug] Failing to run as docker runtime #4005

merryHunter · 2024-09-26T20:37:00Z

Hi, I am facing a strange error when running Container as Runtime environment. This is the output:

I 09-26 10:09:00 common.py:288] --------------------End:   initialize_docker --------------------
I 09-26 10:09:00 common.py:288] 
D 09-26 10:09:00 provisioner.py:468] Docker user: �[1;31m
D 09-26 10:09:00 provisioner.py:468] ________                               _______________
D 09-26 10:09:00 provisioner.py:468] ___  __/__________________________________  ____/__  /________      __
D 09-26 10:09:00 provisioner.py:468] __  /  _  _ \_  __ \_  ___/  __ \_  ___/_  /_   __  /_  __ \_ | /| / /
D 09-26 10:09:00 provisioner.py:468] _  /   /  __/  / / /(__  )/ /_/ /  /   _  __/   _  / / /_/ /_ |/ |/ /
D 09-26 10:09:00 provisioner.py:468] /_/    \___//_/ /_//____/ \____//_/    /_/      /_/  \____/____/|__/
D 09-26 10:09:00 provisioner.py:468] 
D 09-26 10:09:00 provisioner.py:468] �[0;33m
D 09-26 10:09:00 provisioner.py:468] WARNING: You are running this container as root, which can cause new files in
D 09-26 10:09:00 provisioner.py:468] mounted volumes to be created as the root user on your host machine.
D 09-26 10:09:00 provisioner.py:468] 
D 09-26 10:09:00 provisioner.py:468] To avoid this, run the container by specifying your user's userid:
D 09-26 10:09:00 provisioner.py:468] 
D 09-26 10:09:00 provisioner.py:468] $ docker run -u $(id -u):$(id -g) args...
D 09-26 10:09:00 provisioner.py:468] �[m
D 09-26 10:09:00 provisioner.py:468] root
I 09-26 10:09:00 common.py:284] 
I 09-26 10:09:00 common.py:284] --------------------Start: internal_file_mounts --------------------
D 09-26 10:09:00 instance_setup.py:485] Using 31 workers for file mounts.
hostname contains invalid characters
/bin/sh: 1: 31m: not found
/bin/sh: 2: ________: not found
/bin/sh: 3: ___: not found
/bin/sh: 4: /: Permission denied
/bin/sh: 4: /: Permission denied
/bin/sh: 4: __: not found
/bin/sh: 5: Syntax error: "(" unexpected
I 09-26 10:09:00 instance_setup.py:90] _internal_file_mounts: Retrying in 1.2 seconds, due to Command mkdir -p ~/.sky/.runtime_files failed with return code 2.
I 09-26 10:09:00 instance_setup.py:90] �[31mFailed to run command before rsync /tmp/tmpxh55gfof -> ~/.sky/.runtime_files.�[0m
hostname contains invalid characters
/bin/sh: 1: 31m: not found
/bin/sh: 2: ________: not found
/bin/sh: 3: ___: not found
/bin/sh: 4: /: Permission denied
/bin/sh: 4: /: Permission denied
/bin/sh: 4: __: not found
/bin/sh: 5: Syntax error: "(" unexpected
I 09-26 10:09:01 instance_setup.py:90] _int

this is my yaml file:

# train.yaml

name: automl-train-spot

resources:
  accelerators: T4:1
  use_spot: true
  region: us-east-1
  instance_type: g4dn.xlarge
  image_id: docker:XX.dkr.ecr.us-east-1.amazonaws.com/train-image:latest

envs:
  SKYPILOT_DOCKER_USERNAME: AWS
  SKYPILOT_DOCKER_PASSWORD: ${SKYPILOT_DOCKER_PASSWORD}
  SKYPILOT_DOCKER_SERVER: XX.dkr.ecr.us-east-1.amazonaws.com

file_mounts:
   /app/src/data:
    source: s3://data  
    mode: COPY  

run: |
  python main.py

Version & Commit info:

sky -v: skypilot, version 0.6.1
sky -c: commit bc30c0b

Do you have any idea what may cause this strange output? the command I use:

sky jobs launch -c mycluster train.yaml --env SKYPILOT_DOCKER_PASSWORD

The text was updated successfully, but these errors were encountered:

Michaelvll · 2024-09-26T20:41:24Z

Hey @merryHunter , thanks for raising this issue! This should be a resolved issue in our recent PR #3867. Would you like to have a try with the latest nightly build? You could install it with pip uninstall skypilot; pip install skypilot-nightly

merryHunter · 2024-09-26T20:43:00Z

@Michaelvll wow, no way ! let me try, will let you know

merryHunter · 2024-09-26T22:05:36Z

hmm, I will check tomorrow with another docker image, maybe it's problem with this one. The error changed just a bit, but essentially same module and internal mounts / rsync

Michaelvll · 2024-09-26T22:12:16Z

Oh, if this happens again with the latest nightly build, there might be some corner case we have missed. Is it possible to share the base image you used for building this custom docker image, so we can try to reproduce?

merryHunter · 2024-09-28T02:45:50Z

Hey @Michaelvll , the problem still occurs, used nightly build with:

sky --version
skypilot, version 1.0.0.dev20240926

Instructions to repeat below.

Dockerfile used for image

FROM tensorflow/tensorflow:2.14.0-gpu-jupyter

RUN apt-get update && apt-get install -y \
    libgl1-mesa-glx \
    libglib2.0-0 \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/*
WORKDIR /app

COPY . .

ENTRYPOINT ["python", "main.py"]

main.py

import tensorflow as tf
print(tf.__version__)
print(tf.config.experimental.list_physical_devices('GPU'))

Task YAML: docker.yaml

name: test-spot
resources:
  accelerators: T4:1
  use_spot: true
  region: us-east-1
  instance_type: g4dn.xlarge
  image_id: docker:783764614904.dkr.ecr.us-east-1.amazonaws.com/tf14:latest
envs:
  SKYPILOT_DOCKER_USERNAME: AWS
  SKYPILOT_DOCKER_PASSWORD: ${SKYPILOT_DOCKER_PASSWORD}
  SKYPILOT_DOCKER_SERVER: 783764614904.dkr.ecr.us-east-1.amazonaws.com
run: |
  nvidia-smi

Command to run:

sky launch  --cloud aws -c automl docker.yaml  --yes --env SKYPILOT_DOCKER_PASSWORD

I exported SKYPILOT_DOCKER_PASSWORD in the terminal before. When I used command sky jobs launch --cloud aws -c automl docker.yaml --yes --env SKYPILOT_DOCKER_PASSWORD the output was the same.
The full stacktrace output:

sky launch  --cloud aws -c automl docker.yaml  --yes --env SKYPILOT_DOCKER_PASSWORD
Task from YAML spec: docker.yaml
I 09-27 21:36:02 optimizer.py:719] == Optimizer ==
I 09-27 21:36:02 optimizer.py:730] Target: minimizing cost
I 09-27 21:36:02 optimizer.py:742] Estimated cost: $0.2 / hour
I 09-27 21:36:02 optimizer.py:742] 
I 09-27 21:36:02 optimizer.py:867] Considered resources (1 node):
I 09-27 21:36:02 optimizer.py:937] ------------------------------------------------------------------------------------------------
I 09-27 21:36:02 optimizer.py:937]  CLOUD   INSTANCE            vCPUs   Mem(GB)   ACCELERATORS   REGION/ZONE   COST ($)   CHOSEN   
I 09-27 21:36:02 optimizer.py:937] ------------------------------------------------------------------------------------------------
I 09-27 21:36:02 optimizer.py:937]  AWS     g4dn.xlarge[Spot]   4       16        T4:1           us-east-1f    0.15          ✔     
I 09-27 21:36:02 optimizer.py:937] ------------------------------------------------------------------------------------------------
I 09-27 21:36:02 optimizer.py:937] 
Running task on cluster automl...
I 09-27 21:36:02 execution.py:252] Launching an unmanaged spot task, which does not automatically recover from preemptions.
I 09-27 21:36:02 execution.py:252] To get automatic recovery, use managed job instead: sky jobs launch or sky.jobs.launch().
I 09-27 21:36:02 cloud_vm_ray_backend.py:4415] Creating a new cluster: 'automl' [1x AWS(g4dn.xlarge[Spot], {'T4': 1}, image_id={'us-east-1': 'docker:783764614904.dkr.ecr.us-east-1.amazonaws.com/tf14:latest'})].
I 09-27 21:36:02 cloud_vm_ray_backend.py:4415] Tip: to reuse an existing cluster, specify --cluster (-c). Run `sky status` to see existing clusters.
I 09-27 21:36:03 cloud_vm_ray_backend.py:1315] To view detailed progress: tail -n100 -f /home/ivan/sky_logs/sky-2024-09-27-21-36-02-464901/provision.log
I 09-27 21:36:04 provisioner.py:65] Launching on AWS us-east-1 (us-east-1f)
I 09-27 21:37:34 provisioner.py:450] Successfully provisioned or found existing instance.
E 09-27 21:42:53 subprocess_utils.py:84] hostname contains invalid characters
E 09-27 21:42:53 subprocess_utils.py:84] /bin/sh: 1: 31m: not found
E 09-27 21:42:53 subprocess_utils.py:84] /bin/sh: 2: ________: not found
E 09-27 21:42:53 subprocess_utils.py:84] /bin/sh: 3: ___: not found
E 09-27 21:42:53 subprocess_utils.py:84] /bin/sh: 4: /bin/sh: 4: /: Permission denied
E 09-27 21:42:53 subprocess_utils.py:84] /: Permission denied
E 09-27 21:42:53 subprocess_utils.py:84] /bin/sh: 4: __: not found
E 09-27 21:42:53 subprocess_utils.py:84] /bin/sh: 5: Syntax error: "(" unexpected
E 09-27 21:42:53 subprocess_utils.py:84] 
E 09-27 21:42:54 subprocess_utils.py:84] hostname contains invalid characters
E 09-27 21:42:54 subprocess_utils.py:84] /bin/sh: 1: 31m: not found
E 09-27 21:42:54 subprocess_utils.py:84] /bin/sh: 2: ________: not found
E 09-27 21:42:54 subprocess_utils.py:84] /bin/sh: 3: ___: not found
E 09-27 21:42:54 subprocess_utils.py:84] /bin/sh: 4: /: Permission denied
E 09-27 21:42:54 subprocess_utils.py:84] /bin/sh: 4: /: Permission denied
E 09-27 21:42:54 subprocess_utils.py:84] /bin/sh: 4: __: not found
E 09-27 21:42:54 subprocess_utils.py:84] /bin/sh: 5: Syntax error: "(" unexpected
E 09-27 21:42:54 subprocess_utils.py:84] 
E 09-27 21:42:56 subprocess_utils.py:84] hostname contains invalid characters
E 09-27 21:42:56 subprocess_utils.py:84] /bin/sh: 1: 31m: not found
E 09-27 21:42:56 subprocess_utils.py:84] /bin/sh: 2: ________: not found
E 09-27 21:42:56 subprocess_utils.py:84] /bin/sh: 3: ___: not found
E 09-27 21:42:56 subprocess_utils.py:84] /bin/sh: 4: /: Permission denied
E 09-27 21:42:56 subprocess_utils.py:84] /bin/sh: 4: /: Permission denied
E 09-27 21:42:56 subprocess_utils.py:84] /bin/sh: 4: __: not found
E 09-27 21:42:56 subprocess_utils.py:84] /bin/sh: 5: Syntax error: "(" unexpected
E 09-27 21:42:56 subprocess_utils.py:84] 
E 09-27 21:42:59 subprocess_utils.py:84] hostname contains invalid characters
E 09-27 21:42:59 subprocess_utils.py:84] /bin/sh: 1: 31m: not found
E 09-27 21:42:59 subprocess_utils.py:84] /bin/sh: 2: ________: not found
E 09-27 21:42:59 subprocess_utils.py:84] /bin/sh: 3: ___: not found
E 09-27 21:42:59 subprocess_utils.py:84] /bin/sh: 4: /: Permission denied/bin/sh: 4: 
E 09-27 21:42:59 subprocess_utils.py:84] /: Permission denied
E 09-27 21:42:59 subprocess_utils.py:84] /bin/sh: 4: __: not found
E 09-27 21:42:59 subprocess_utils.py:84] /bin/sh: 5: Syntax error: "(" unexpected
E 09-27 21:42:59 subprocess_utils.py:84] 
E 09-27 21:43:04 subprocess_utils.py:84] hostname contains invalid characters
E 09-27 21:43:04 subprocess_utils.py:84] /bin/sh: 1: 31m: not found
E 09-27 21:43:04 subprocess_utils.py:84] /bin/sh: 2: ________: not found
E 09-27 21:43:04 subprocess_utils.py:84] /bin/sh: 3: ___: not found
E 09-27 21:43:04 subprocess_utils.py:84] /bin/sh: 4: /: Permission denied
E 09-27 21:43:04 subprocess_utils.py:84] /bin/sh: 4: /: Permission denied
E 09-27 21:43:04 subprocess_utils.py:84] /bin/sh: 4: __: not found
E 09-27 21:43:04 subprocess_utils.py:84] /bin/sh: 5: Syntax error: "(" unexpected
E 09-27 21:43:04 subprocess_utils.py:84] 
E 09-27 21:43:08 subprocess_utils.py:84] hostname contains invalid characters
E 09-27 21:43:08 subprocess_utils.py:84] /bin/sh: 1: 31m: not found
E 09-27 21:43:08 subprocess_utils.py:84] /bin/sh: 2: ________: not found
E 09-27 21:43:08 subprocess_utils.py:84] /bin/sh: 3: ___: not found
E 09-27 21:43:08 subprocess_utils.py:84] /bin/sh: 4: /: Permission denied
E 09-27 21:43:08 subprocess_utils.py:84] /bin/sh: 4: /: Permission denied
E 09-27 21:43:08 subprocess_utils.py:84] /bin/sh: 4: __: not found
E 09-27 21:43:08 subprocess_utils.py:84] /bin/sh: 5: Syntax error: "(" unexpected
E 09-27 21:43:08 subprocess_utils.py:84] 
E 09-27 21:43:08 provisioner.py:583] *** Failed setting up cluster. ***
Clusters
NAME    LAUNCHED    RESOURCES                                                                  STATUS  AUTOSTOP  COMMAND                       
automl  7 mins ago  1x AWS(g4dn.xlarge[Spot], {'T4': 1}, image_id={'us-east-1': 'docker:78...  INIT    -         sky launch --cloud aws -c...  

sky.exceptions.CommandError: Command mkdir -p ~/.sky/.runtime_files failed with return code 2.
Failed to run command before rsync /tmp/tmp6ng1mu8o -> ~/.sky/.runtime_files.

merryHunter · 2024-09-28T02:49:08Z

Posting full provision.log separately:

D 09-27 21:36:04 provisioner.py:159] SkyPilot version: 1.0.0.dev20240926; commit: e95332b9eb8de4cdcac464ff704bf64f3285e776
D 09-27 21:36:04 provisioner.py:161] 
D 09-27 21:36:04 provisioner.py:161] 
D 09-27 21:36:04 provisioner.py:161] ==================== Provisioning ====================
D 09-27 21:36:04 provisioner.py:161] 
D 09-27 21:36:04 provisioner.py:162] Provision config:
D 09-27 21:36:04 provisioner.py:162] {
D 09-27 21:36:04 provisioner.py:162]   "provider_config": {
D 09-27 21:36:04 provisioner.py:162]     "type": "external",
D 09-27 21:36:04 provisioner.py:162]     "module": "sky.provision.aws",
D 09-27 21:36:04 provisioner.py:162]     "region": "us-east-1",
D 09-27 21:36:04 provisioner.py:162]     "availability_zone": "us-east-1f",
D 09-27 21:36:04 provisioner.py:162]     "cache_stopped_nodes": true,
D 09-27 21:36:04 provisioner.py:162]     "security_group": {
D 09-27 21:36:04 provisioner.py:162]       "GroupName": "sky-sg-ivan-6410",
D 09-27 21:36:04 provisioner.py:162]       "ManagedBySkyPilot": true
D 09-27 21:36:04 provisioner.py:162]     },
D 09-27 21:36:04 provisioner.py:162]     "use_internal_ips": false,
D 09-27 21:36:04 provisioner.py:162]     "disable_launch_config_check": true
D 09-27 21:36:04 provisioner.py:162]   },
D 09-27 21:36:04 provisioner.py:162]   "authentication_config": {
D 09-27 21:36:04 provisioner.py:162]     "ssh_user": "ubuntu",
D 09-27 21:36:04 provisioner.py:162]     "ssh_private_key": "~/.ssh/sky-key"
D 09-27 21:36:04 provisioner.py:162]   },
D 09-27 21:36:04 provisioner.py:162]   "docker_config": {
D 09-27 21:36:04 provisioner.py:162]     "image": "783764614904.dkr.ecr.us-east-1.amazonaws.com/tf14:latest",
D 09-27 21:36:04 provisioner.py:162]     "container_name": "sky_container",
D 09-27 21:36:04 provisioner.py:162]     "run_options": [
D 09-27 21:36:04 provisioner.py:162]       "--ulimit nofile=1048576:1048576"
D 09-27 21:36:04 provisioner.py:162]     ],
D 09-27 21:36:04 provisioner.py:162]     "docker_login_config": {
D 09-27 21:36:04 provisioner.py:162]       "username": "AWS",
D 09-27 21:36:04 provisioner.py:162]       "password": "<REDUCTED_MANUALLY>",
D 09-27 21:36:04 provisioner.py:162]       "server": "783764614904.dkr.ecr.us-east-1.amazonaws.com"
D 09-27 21:36:04 provisioner.py:162]     }
D 09-27 21:36:04 provisioner.py:162]   },
D 09-27 21:36:04 provisioner.py:162]   "node_config": {
D 09-27 21:36:04 provisioner.py:162]     "InstanceType": "g4dn.xlarge",
D 09-27 21:36:04 provisioner.py:162]     "ImageId": "ami-0ac1f653c5b6af751",
D 09-27 21:36:04 provisioner.py:162]     "BlockDeviceMappings": [
D 09-27 21:36:04 provisioner.py:162]       {
D 09-27 21:36:04 provisioner.py:162]         "DeviceName": "/dev/sda1",
D 09-27 21:36:04 provisioner.py:162]         "Ebs": {
D 09-27 21:36:04 provisioner.py:162]           "VolumeSize": 256,
D 09-27 21:36:04 provisioner.py:162]           "VolumeType": "gp3",
D 09-27 21:36:04 provisioner.py:162]           "Encrypted": false,
D 09-27 21:36:04 provisioner.py:162]           "Iops": 3500,
D 09-27 21:36:04 provisioner.py:162]           "Throughput": 218
D 09-27 21:36:04 provisioner.py:162]         }
D 09-27 21:36:04 provisioner.py:162]       }
D 09-27 21:36:04 provisioner.py:162]     ],
D 09-27 21:36:04 provisioner.py:162]     "InstanceMarketOptions": {
D 09-27 21:36:04 provisioner.py:162]       "MarketType": "spot"
D 09-27 21:36:04 provisioner.py:162]     },
D 09-27 21:36:04 provisioner.py:162]     "CapacityReservationSpecification": {
D 09-27 21:36:04 provisioner.py:162]       "CapacityReservationPreference": "open"
D 09-27 21:36:04 provisioner.py:162]     },
D 09-27 21:36:04 provisioner.py:162]     "UserData": "#cloud-config\nusers:\n  - name: ubuntu\n    shell: /bin/bash\n    sudo: ALL=(ALL) NOPASSWD:ALL\n    ssh_authorized_keys:\n      - |-\n        ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCeCrhh0eDNAWoPinefahDcx+2rLXFIybEPReU+QtCp8JaX78c4NM97kq7q2lEtRW+7sHua2+OJfwPf5RipQ2CNC+iWZtWEzcntmQaatGCV5RAGiUi5g9aygID1zD1AOuPNyOruh35kd5zLPC2hqJ7qCXkDxPiPnTTpvtzJKZXyhxNDiZOjm3JmAPlZCZi7g29Xd0vmHqDA4s/nllsDYx4FsGJ7Z/WghSTGAAKLRrndFpcs8Kk7+/QSyUPJUBXai14NfJJM91k7G9wn6hIjxoHOCe0yCTiizlZS4OAp1OUHtRqUm4cunQtxuLoNNMKAtdiw5QVYtkfxOdqjfJb6PX4X\nwrite_files:\n  - path: /etc/apt/apt.conf.d/20auto-upgrades\n    content: |\n      APT::Periodic::Update-Package-Lists \"0\";\n      APT::Periodic::Download-Upgradeable-Packages \"0\";\n      APT::Periodic::AutocleanInterval \"0\";\n      APT::Periodic::Unattended-Upgrade \"0\";\n  - path: /etc/apt/apt.conf.d/10cloudinit-disable\n    content: |\n      APT::Periodic::Enable \"0\";\n",
D 09-27 21:36:04 provisioner.py:162]     "TagSpecifications": [
D 09-27 21:36:04 provisioner.py:162]       {
D 09-27 21:36:04 provisioner.py:162]         "ResourceType": "instance",
D 09-27 21:36:04 provisioner.py:162]         "Tags": [
D 09-27 21:36:04 provisioner.py:162]           {
D 09-27 21:36:04 provisioner.py:162]             "Key": "skypilot-user",
D 09-27 21:36:04 provisioner.py:162]             "Value": "ivan"
D 09-27 21:36:04 provisioner.py:162]           }
D 09-27 21:36:04 provisioner.py:162]         ]
D 09-27 21:36:04 provisioner.py:162]       }
D 09-27 21:36:04 provisioner.py:162]     ]
D 09-27 21:36:04 provisioner.py:162]   },
D 09-27 21:36:04 provisioner.py:162]   "count": 1,
D 09-27 21:36:04 provisioner.py:162]   "tags": {},
D 09-27 21:36:04 provisioner.py:162]   "resume_stopped_nodes": true,
D 09-27 21:36:04 provisioner.py:162]   "ports_to_open_on_launch": null
D 09-27 21:36:04 provisioner.py:162] }
I 09-27 21:36:04 provisioner.py:65] �[1mLaunching on AWS us-east-1�[0m (us-east-1f)
D 09-27 21:36:06 config.py:258] subnet subnet-06151bd9e871d0c89 route tables: []
D 09-27 21:36:06 config.py:267] Checking main route table
D 09-27 21:36:06 config.py:258] subnet subnet-08eb80dc1d4d48bc0 route tables: []
D 09-27 21:36:06 config.py:267] Checking main route table
D 09-27 21:36:06 config.py:258] subnet subnet-01f54f6fdf2fbe4eb route tables: []
D 09-27 21:36:06 config.py:267] Checking main route table
D 09-27 21:36:06 config.py:258] subnet subnet-0dc14af6f9b65a368 route tables: []
D 09-27 21:36:06 config.py:267] Checking main route table
D 09-27 21:36:06 config.py:258] subnet subnet-0aac8f5336a23e50b route tables: []
D 09-27 21:36:06 config.py:267] Checking main route table
D 09-27 21:36:06 config.py:258] subnet subnet-0d6fc9158bdd6ba23 route tables: []
D 09-27 21:36:06 config.py:267] Checking main route table
D 09-27 21:36:06 config.py:356] use_internal_ips: False
D 09-27 21:36:06 config.py:357] subnets: [ec2.Subnet(id='subnet-0dc14af6f9b65a368'), ec2.Subnet(id='subnet-0d6fc9158bdd6ba23'), ec2.Subnet(id='subnet-08eb80dc1d4d48bc0'), ec2.Subnet(id='subnet-01f54f6fdf2fbe4eb'), ec2.Subnet(id='subnet-06151bd9e871d0c89'), ec2.Subnet(id='subnet-0aac8f5336a23e50b')]
D 09-27 21:36:06 config.py:85] 
D 09-27 21:36:06 config.py:85] Creating or updating security groups...
I 09-27 21:36:06 config.py:100] Security groups created or updated in 0.19095 seconds.
D 09-27 21:36:07 instance.py:212] Creating 1 instances with config: 
D 09-27 21:36:07 instance.py:212] {'InstanceType': 'g4dn.xlarge', 'ImageId': 'ami-0ac1f653c5b6af751', 'BlockDeviceMappings': [{'DeviceName': '/dev/sda1', 'Ebs': {'VolumeSize': 256, 'VolumeType': 'gp3', 'Encrypted': False, 'Iops': 3500, 'Throughput': 218}}], 'InstanceMarketOptions': {'MarketType': 'spot'}, 'CapacityReservationSpecification': {'CapacityReservationPreference': 'open'}, 'UserData': '#cloud-config\nusers:\n  - name: ubuntu\n    shell: /bin/bash\n    sudo: ALL=(ALL) NOPASSWD:ALL\n    ssh_authorized_keys:\n      - |-\n        ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCeCrhh0eDNAWoPinefahDcx+2rLXFIybEPReU+QtCp8JaX78c4NM97kq7q2lEtRW+7sHua2+OJfwPf5RipQ2CNC+iWZtWEzcntmQaatGCV5RAGiUi5g9aygID1zD1AOuPNyOruh35kd5zLPC2hqJ7qCXkDxPiPnTTpvtzJKZXyhxNDiZOjm3JmAPlZCZi7g29Xd0vmHqDA4s/nllsDYx4FsGJ7Z/WghSTGAAKLRrndFpcs8Kk7+/QSyUPJUBXai14NfJJM91k7G9wn6hIjxoHOCe0yCTiizlZS4OAp1OUHtRqUm4cunQtxuLoNNMKAtdiw5QVYtkfxOdqjfJb6PX4X\nwrite_files:\n  - path: /etc/apt/apt.conf.d/20auto-upgrades\n    content: |\n      APT::Periodic::Update-Package-Lists "0";\n      APT::Periodic::Download-Upgradeable-Packages "0";\n      APT::Periodic::AutocleanInterval "0";\n      APT::Periodic::Unattended-Upgrade "0";\n  - path: /etc/apt/apt.conf.d/10cloudinit-disable\n    content: |\n      APT::Periodic::Enable "0";\n', 'TagSpecifications': [{'ResourceType': 'instance', 'Tags': [{'Key': 'Name', 'Value': 'automl-6ae8'}, {'Key': 'ray-cluster-name', 'Value': 'automl-6ae8'}, {'Key': 'skypilot-cluster-name', 'Value': 'automl-6ae8'}, {'Key': 'skypilot-user', 'Value': 'ivan'}]}], 'IamInstanceProfile': {'Arn': 'arn:aws:iam::783764614904:instance-profile/skypilot-v1'}, 'MinCount': 1, 'MaxCount': 1}
D 09-27 21:36:09 provisioner.py:93] 
D 09-27 21:36:09 provisioner.py:93] Waiting for instances of 'automl' to be ready...
D 09-27 21:36:11 instance.py:900] [ec2.Instance(id='i-0fe91783c5384fde7')]
D 09-27 21:36:16 provisioner.py:112] Instances of 'automl' are ready after 0 retries.
D 09-27 21:36:16 provisioner.py:116] 
D 09-27 21:36:16 provisioner.py:116] Provisioning 'automl' took 12.35 seconds.
D 09-27 21:36:16 provisioner.py:576] 
D 09-27 21:36:16 provisioner.py:576] 
D 09-27 21:36:16 provisioner.py:576] ==================== System Setup After Provision ====================
D 09-27 21:36:16 provisioner.py:576] 
D 09-27 21:36:17 provisioner.py:423] Provision record:
D 09-27 21:36:17 provisioner.py:423] {
D 09-27 21:36:17 provisioner.py:423]   "provider_name": "aws",
D 09-27 21:36:17 provisioner.py:423]   "region": "us-east-1",
D 09-27 21:36:17 provisioner.py:423]   "zone": "us-east-1f",
D 09-27 21:36:17 provisioner.py:423]   "cluster_name": "automl-6ae8",
D 09-27 21:36:17 provisioner.py:423]   "head_instance_id": "i-0fe91783c5384fde7",
D 09-27 21:36:17 provisioner.py:423]   "resumed_instance_ids": [],
D 09-27 21:36:17 provisioner.py:423]   "created_instance_ids": [
D 09-27 21:36:17 provisioner.py:423]     "i-0fe91783c5384fde7"
D 09-27 21:36:17 provisioner.py:423]   ]
D 09-27 21:36:17 provisioner.py:423] }
D 09-27 21:36:17 provisioner.py:423] Cluster info:
D 09-27 21:36:17 provisioner.py:423] {
D 09-27 21:36:17 provisioner.py:423]   "instances": {
D 09-27 21:36:17 provisioner.py:423]     "i-0fe91783c5384fde7": [
D 09-27 21:36:17 provisioner.py:423]       {
D 09-27 21:36:17 provisioner.py:423]         "instance_id": "i-0fe91783c5384fde7",
D 09-27 21:36:17 provisioner.py:423]         "internal_ip": "172.31.77.158",
D 09-27 21:36:17 provisioner.py:423]         "external_ip": "3.238.25.127",
D 09-27 21:36:17 provisioner.py:423]         "tags": {
D 09-27 21:36:17 provisioner.py:423]           "Name": "sky-automl-6ae8-head",
D 09-27 21:36:17 provisioner.py:423]           "ray-cluster-name": "automl-6ae8",
D 09-27 21:36:17 provisioner.py:423]           "ray-node-type": "head",
D 09-27 21:36:17 provisioner.py:423]           "skypilot-cluster-name": "automl-6ae8",
D 09-27 21:36:17 provisioner.py:423]           "skypilot-head-node": "1",
D 09-27 21:36:17 provisioner.py:423]           "skypilot-user": "ivan"
D 09-27 21:36:17 provisioner.py:423]         },
D 09-27 21:36:17 provisioner.py:423]         "ssh_port": 22
D 09-27 21:36:17 provisioner.py:423]       }
D 09-27 21:36:17 provisioner.py:423]     ]
D 09-27 21:36:17 provisioner.py:423]   },
D 09-27 21:36:17 provisioner.py:423]   "head_instance_id": "i-0fe91783c5384fde7",
D 09-27 21:36:17 provisioner.py:423]   "provider_name": "aws",
D 09-27 21:36:17 provisioner.py:423]   "provider_config": {
D 09-27 21:36:17 provisioner.py:423]     "type": "external",
D 09-27 21:36:17 provisioner.py:423]     "module": "sky.provision.aws",
D 09-27 21:36:17 provisioner.py:423]     "region": "us-east-1",
D 09-27 21:36:17 provisioner.py:423]     "availability_zone": "us-east-1f",
D 09-27 21:36:17 provisioner.py:423]     "cache_stopped_nodes": true,
D 09-27 21:36:17 provisioner.py:423]     "security_group": {
D 09-27 21:36:17 provisioner.py:423]       "GroupName": "sky-sg-ivan-6410",
D 09-27 21:36:17 provisioner.py:423]       "ManagedBySkyPilot": true
D 09-27 21:36:17 provisioner.py:423]     },
D 09-27 21:36:17 provisioner.py:423]     "use_internal_ips": false,
D 09-27 21:36:17 provisioner.py:423]     "disable_launch_config_check": true
D 09-27 21:36:17 provisioner.py:423]   },
D 09-27 21:36:17 provisioner.py:423]   "docker_user": null,
D 09-27 21:36:17 provisioner.py:423]   "ssh_user": null,
D 09-27 21:36:17 provisioner.py:423]   "custom_ray_options": null
D 09-27 21:36:17 provisioner.py:423] }
D 09-27 21:36:17 provisioner.py:445] 
D 09-27 21:36:17 provisioner.py:445] Waiting for SSH to be available for 'automl' ...
D 09-27 21:36:18 provisioner.py:323] Waiting for SSH to 3.238.25.127. Try: ssh -T -i '~/.ssh/sky-key' ubuntu@3.238.25.127 -p 22 -o StrictHostKeyChecking=no -o PasswordAuthentication=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Timeout: SSH connection to 3.238.25.127 is not ready.
D 09-27 21:36:18 provisioner.py:400] Retrying in 1 second...
D 09-27 21:36:20 provisioner.py:323] Waiting for SSH to 3.238.25.127. Try: ssh -T -i '~/.ssh/sky-key' ubuntu@3.238.25.127 -p 22 -o StrictHostKeyChecking=no -o PasswordAuthentication=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Timeout: SSH connection to 3.238.25.127 is not ready.
D 09-27 21:36:20 provisioner.py:400] Retrying in 1 second...
D 09-27 21:36:22 provisioner.py:323] Waiting for SSH to 3.238.25.127. Try: ssh -T -i '~/.ssh/sky-key' ubuntu@3.238.25.127 -p 22 -o StrictHostKeyChecking=no -o PasswordAuthentication=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Timeout: SSH connection to 3.238.25.127 is not ready.
D 09-27 21:36:22 provisioner.py:400] Retrying in 1 second...
D 09-27 21:36:24 provisioner.py:323] Waiting for SSH to 3.238.25.127. Try: ssh -T -i '~/.ssh/sky-key' ubuntu@3.238.25.127 -p 22 -o StrictHostKeyChecking=no -o PasswordAuthentication=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Timeout: SSH connection to 3.238.25.127 is not ready.
D 09-27 21:36:24 provisioner.py:400] Retrying in 1 second...
D 09-27 21:36:26 provisioner.py:323] Waiting for SSH to 3.238.25.127. Try: ssh -T -i '~/.ssh/sky-key' ubuntu@3.238.25.127 -p 22 -o StrictHostKeyChecking=no -o PasswordAuthentication=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Timeout: SSH connection to 3.238.25.127 is not ready.
D 09-27 21:36:26 provisioner.py:400] Retrying in 1 second...
D 09-27 21:36:28 provisioner.py:323] Waiting for SSH to 3.238.25.127. Try: ssh -T -i '~/.ssh/sky-key' ubuntu@3.238.25.127 -p 22 -o StrictHostKeyChecking=no -o PasswordAuthentication=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Timeout: SSH connection to 3.238.25.127 is not ready.
D 09-27 21:36:28 provisioner.py:400] Retrying in 1 second...
D 09-27 21:36:30 provisioner.py:323] Waiting for SSH to 3.238.25.127. Try: ssh -T -i '~/.ssh/sky-key' ubuntu@3.238.25.127 -p 22 -o StrictHostKeyChecking=no -o PasswordAuthentication=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Timeout: SSH connection to 3.238.25.127 is not ready.
D 09-27 21:36:30 provisioner.py:400] Retrying in 1 second...
D 09-27 21:36:32 provisioner.py:323] Waiting for SSH to 3.238.25.127. Try: ssh -T -i '~/.ssh/sky-key' ubuntu@3.238.25.127 -p 22 -o StrictHostKeyChecking=no -o PasswordAuthentication=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Timeout: SSH connection to 3.238.25.127 is not ready.
D 09-27 21:36:32 provisioner.py:400] Retrying in 1 second...
D 09-27 21:36:34 provisioner.py:323] Waiting for SSH to 3.238.25.127. Try: ssh -T -i '~/.ssh/sky-key' ubuntu@3.238.25.127 -p 22 -o StrictHostKeyChecking=no -o PasswordAuthentication=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Timeout: SSH connection to 3.238.25.127 is not ready.
D 09-27 21:36:34 provisioner.py:400] Retrying in 1 second...
D 09-27 21:36:36 provisioner.py:323] Waiting for SSH to 3.238.25.127. Try: ssh -T -i '~/.ssh/sky-key' ubuntu@3.238.25.127 -p 22 -o StrictHostKeyChecking=no -o PasswordAuthentication=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Timeout: SSH connection to 3.238.25.127 is not ready.
D 09-27 21:36:36 provisioner.py:400] Retrying in 1 second...
D 09-27 21:36:38 provisioner.py:323] Waiting for SSH to 3.238.25.127. Try: ssh -T -i '~/.ssh/sky-key' ubuntu@3.238.25.127 -p 22 -o StrictHostKeyChecking=no -o PasswordAuthentication=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Timeout: SSH connection to 3.238.25.127 is not ready.
D 09-27 21:36:38 provisioner.py:400] Retrying in 1 second...
D 09-27 21:36:40 provisioner.py:323] Waiting for SSH to 3.238.25.127. Try: ssh -T -i '~/.ssh/sky-key' ubuntu@3.238.25.127 -p 22 -o StrictHostKeyChecking=no -o PasswordAuthentication=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Timeout: SSH connection to 3.238.25.127 is not ready.
D 09-27 21:36:40 provisioner.py:400] Retrying in 1 second...
D 09-27 21:36:42 provisioner.py:323] Waiting for SSH to 3.238.25.127. Try: ssh -T -i '~/.ssh/sky-key' ubuntu@3.238.25.127 -p 22 -o StrictHostKeyChecking=no -o PasswordAuthentication=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Timeout: SSH connection to 3.238.25.127 is not ready.
D 09-27 21:36:42 provisioner.py:400] Retrying in 1 second...
D 09-27 21:36:44 provisioner.py:323] Waiting for SSH to 3.238.25.127. Try: ssh -T -i '~/.ssh/sky-key' ubuntu@3.238.25.127 -p 22 -o StrictHostKeyChecking=no -o PasswordAuthentication=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Timeout: SSH connection to 3.238.25.127 is not ready.
D 09-27 21:36:44 provisioner.py:400] Retrying in 1 second...
D 09-27 21:36:46 provisioner.py:323] Waiting for SSH to 3.238.25.127. Try: ssh -T -i '~/.ssh/sky-key' ubuntu@3.238.25.127 -p 22 -o StrictHostKeyChecking=no -o PasswordAuthentication=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Timeout: SSH connection to 3.238.25.127 is not ready.
D 09-27 21:36:46 provisioner.py:400] Retrying in 1 second...
D 09-27 21:36:48 provisioner.py:323] Waiting for SSH to 3.238.25.127. Try: ssh -T -i '~/.ssh/sky-key' ubuntu@3.238.25.127 -p 22 -o StrictHostKeyChecking=no -o PasswordAuthentication=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Timeout: SSH connection to 3.238.25.127 is not ready.
D 09-27 21:36:48 provisioner.py:400] Retrying in 1 second...
D 09-27 21:36:50 provisioner.py:323] Waiting for SSH to 3.238.25.127. Try: ssh -T -i '~/.ssh/sky-key' ubuntu@3.238.25.127 -p 22 -o StrictHostKeyChecking=no -o PasswordAuthentication=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Timeout: SSH connection to 3.238.25.127 is not ready.
D 09-27 21:36:50 provisioner.py:400] Retrying in 1 second...
D 09-27 21:36:52 provisioner.py:323] Waiting for SSH to 3.238.25.127. Try: ssh -T -i '~/.ssh/sky-key' ubuntu@3.238.25.127 -p 22 -o StrictHostKeyChecking=no -o PasswordAuthentication=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Timeout: SSH connection to 3.238.25.127 is not ready.
D 09-27 21:36:52 provisioner.py:400] Retrying in 1 second...
D 09-27 21:36:54 provisioner.py:323] Waiting for SSH to 3.238.25.127. Try: ssh -T -i '~/.ssh/sky-key' ubuntu@3.238.25.127 -p 22 -o StrictHostKeyChecking=no -o PasswordAuthentication=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Timeout: SSH connection to 3.238.25.127 is not ready.
D 09-27 21:36:54 provisioner.py:400] Retrying in 1 second...
D 09-27 21:36:56 provisioner.py:323] Waiting for SSH to 3.238.25.127. Try: ssh -T -i '~/.ssh/sky-key' ubuntu@3.238.25.127 -p 22 -o StrictHostKeyChecking=no -o PasswordAuthentication=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Timeout: SSH connection to 3.238.25.127 is not ready.
D 09-27 21:36:56 provisioner.py:400] Retrying in 1 second...
D 09-27 21:36:58 provisioner.py:323] Waiting for SSH to 3.238.25.127. Try: ssh -T -i '~/.ssh/sky-key' ubuntu@3.238.25.127 -p 22 -o StrictHostKeyChecking=no -o PasswordAuthentication=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Timeout: SSH connection to 3.238.25.127 is not ready.
D 09-27 21:36:58 provisioner.py:400] Retrying in 1 second...
D 09-27 21:37:00 provisioner.py:323] Waiting for SSH to 3.238.25.127. Try: ssh -T -i '~/.ssh/sky-key' ubuntu@3.238.25.127 -p 22 -o StrictHostKeyChecking=no -o PasswordAuthentication=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Timeout: SSH connection to 3.238.25.127 is not ready.
D 09-27 21:37:00 provisioner.py:400] Retrying in 1 second...
D 09-27 21:37:02 provisioner.py:323] Waiting for SSH to 3.238.25.127. Try: ssh -T -i '~/.ssh/sky-key' ubuntu@3.238.25.127 -p 22 -o StrictHostKeyChecking=no -o PasswordAuthentication=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Timeout: SSH connection to 3.238.25.127 is not ready.
D 09-27 21:37:02 provisioner.py:400] Retrying in 1 second...
D 09-27 21:37:04 provisioner.py:323] Waiting for SSH to 3.238.25.127. Try: ssh -T -i '~/.ssh/sky-key' ubuntu@3.238.25.127 -p 22 -o StrictHostKeyChecking=no -o PasswordAuthentication=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Timeout: SSH connection to 3.238.25.127 is not ready.
D 09-27 21:37:04 provisioner.py:400] Retrying in 1 second...
D 09-27 21:37:06 provisioner.py:323] Waiting for SSH to 3.238.25.127. Try: ssh -T -i '~/.ssh/sky-key' ubuntu@3.238.25.127 -p 22 -o StrictHostKeyChecking=no -o PasswordAuthentication=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Timeout: SSH connection to 3.238.25.127 is not ready.
D 09-27 21:37:06 provisioner.py:400] Retrying in 1 second...
D 09-27 21:37:08 provisioner.py:323] Waiting for SSH to 3.238.25.127. Try: ssh -T -i '~/.ssh/sky-key' ubuntu@3.238.25.127 -p 22 -o StrictHostKeyChecking=no -o PasswordAuthentication=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Timeout: SSH connection to 3.238.25.127 is not ready.
D 09-27 21:37:08 provisioner.py:400] Retrying in 1 second...
D 09-27 21:37:10 provisioner.py:323] Waiting for SSH to 3.238.25.127. Try: ssh -T -i '~/.ssh/sky-key' ubuntu@3.238.25.127 -p 22 -o StrictHostKeyChecking=no -o PasswordAuthentication=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Timeout: SSH connection to 3.238.25.127 is not ready.
D 09-27 21:37:10 provisioner.py:400] Retrying in 1 second...
D 09-27 21:37:12 provisioner.py:323] Waiting for SSH to 3.238.25.127. Try: ssh -T -i '~/.ssh/sky-key' ubuntu@3.238.25.127 -p 22 -o StrictHostKeyChecking=no -o PasswordAuthentication=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Timeout: SSH connection to 3.238.25.127 is not ready.
D 09-27 21:37:12 provisioner.py:400] Retrying in 1 second...
D 09-27 21:37:14 provisioner.py:323] Waiting for SSH to 3.238.25.127. Try: ssh -T -i '~/.ssh/sky-key' ubuntu@3.238.25.127 -p 22 -o StrictHostKeyChecking=no -o PasswordAuthentication=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Timeout: SSH connection to 3.238.25.127 is not ready.
D 09-27 21:37:14 provisioner.py:400] Retrying in 1 second...
D 09-27 21:37:15 provisioner.py:323] Waiting for SSH to 3.238.25.127. Try: ssh -T -i '~/.ssh/sky-key' ubuntu@3.238.25.127 -p 22 -o StrictHostKeyChecking=no -o PasswordAuthentication=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Error: �[1mConnectionRefusedError:�[0m [Errno 111] Connection refused
D 09-27 21:37:15 provisioner.py:400] Retrying in 1 second...
D 09-27 21:37:16 provisioner.py:323] Waiting for SSH to 3.238.25.127. Try: ssh -T -i '~/.ssh/sky-key' ubuntu@3.238.25.127 -p 22 -o StrictHostKeyChecking=no -o PasswordAuthentication=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Error: �[1mConnectionRefusedError:�[0m [Errno 111] Connection refused
D 09-27 21:37:16 provisioner.py:400] Retrying in 1 second...
D 09-27 21:37:17 provisioner.py:323] Waiting for SSH to 3.238.25.127. Try: ssh -T -i '~/.ssh/sky-key' ubuntu@3.238.25.127 -p 22 -o StrictHostKeyChecking=no -o PasswordAuthentication=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Error: �[1mConnectionRefusedError:�[0m [Errno 111] Connection refused
D 09-27 21:37:17 provisioner.py:400] Retrying in 1 second...
D 09-27 21:37:19 provisioner.py:323] Waiting for SSH to 3.238.25.127. Try: ssh -T -i '~/.ssh/sky-key' ubuntu@3.238.25.127 -p 22 -o StrictHostKeyChecking=no -o PasswordAuthentication=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Error: �[1mConnectionRefusedError:�[0m [Errno 111] Connection refused
D 09-27 21:37:19 provisioner.py:400] Retrying in 1 second...
D 09-27 21:37:20 provisioner.py:323] Waiting for SSH to 3.238.25.127. Try: ssh -T -i '~/.ssh/sky-key' ubuntu@3.238.25.127 -p 22 -o StrictHostKeyChecking=no -o PasswordAuthentication=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Error: �[1mConnectionRefusedError:�[0m [Errno 111] Connection refused
D 09-27 21:37:20 provisioner.py:400] Retrying in 1 second...
D 09-27 21:37:21 provisioner.py:323] Waiting for SSH to 3.238.25.127. Try: ssh -T -i '~/.ssh/sky-key' ubuntu@3.238.25.127 -p 22 -o StrictHostKeyChecking=no -o PasswordAuthentication=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Error: �[1mConnectionRefusedError:�[0m [Errno 111] Connection refused
D 09-27 21:37:21 provisioner.py:400] Retrying in 1 second...
D 09-27 21:37:22 provisioner.py:323] Waiting for SSH to 3.238.25.127. Try: ssh -T -i '~/.ssh/sky-key' ubuntu@3.238.25.127 -p 22 -o StrictHostKeyChecking=no -o PasswordAuthentication=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Error: �[1mConnectionRefusedError:�[0m [Errno 111] Connection refused
D 09-27 21:37:22 provisioner.py:400] Retrying in 1 second...
D 09-27 21:37:23 provisioner.py:323] Waiting for SSH to 3.238.25.127. Try: ssh -T -i '~/.ssh/sky-key' ubuntu@3.238.25.127 -p 22 -o StrictHostKeyChecking=no -o PasswordAuthentication=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Error: �[1mConnectionRefusedError:�[0m [Errno 111] Connection refused
D 09-27 21:37:23 provisioner.py:400] Retrying in 1 second...
D 09-27 21:37:24 provisioner.py:345] Waiting for SSH using command: ssh -T -i '~/.ssh/sky-key' ubuntu@3.238.25.127 -p 22 -o StrictHostKeyChecking=no -o PasswordAuthentication=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime
D 09-27 21:37:25 provisioner.py:356] Waiting for SSH using command: ssh -T -i '~/.ssh/sky-key' ubuntu@3.238.25.127 -p 22 -o StrictHostKeyChecking=no -o PasswordAuthentication=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptimeError: Warning: Permanently added '3.238.25.127' (ED25519) to the list of known hosts.

D 09-27 21:37:25 provisioner.py:356] "System is booting up. Unprivileged users are not permitted to log in yet. Please come back later. For technical details, see pam_nologin(8)."
D 09-27 21:37:25 provisioner.py:356] Connection closed by 3.238.25.127 port 22

D 09-27 21:37:25 provisioner.py:356] 
D 09-27 21:37:25 provisioner.py:400] Retrying in 1 second...
D 09-27 21:37:26 provisioner.py:345] Waiting for SSH using command: ssh -T -i '~/.ssh/sky-key' ubuntu@3.238.25.127 -p 22 -o StrictHostKeyChecking=no -o PasswordAuthentication=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime
D 09-27 21:37:26 provisioner.py:356] Waiting for SSH using command: ssh -T -i '~/.ssh/sky-key' ubuntu@3.238.25.127 -p 22 -o StrictHostKeyChecking=no -o PasswordAuthentication=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptimeError: Warning: Permanently added '3.238.25.127' (ED25519) to the list of known hosts.

D 09-27 21:37:26 provisioner.py:356] "System is booting up. Unprivileged users are not permitted to log in yet. Please come back later. For technical details, see pam_nologin(8)."
D 09-27 21:37:26 provisioner.py:356] Connection closed by 3.238.25.127 port 22

D 09-27 21:37:26 provisioner.py:356] 
D 09-27 21:37:26 provisioner.py:400] Retrying in 1 second...
D 09-27 21:37:27 provisioner.py:345] Waiting for SSH using command: ssh -T -i '~/.ssh/sky-key' ubuntu@3.238.25.127 -p 22 -o StrictHostKeyChecking=no -o PasswordAuthentication=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime
D 09-27 21:37:28 provisioner.py:356] Waiting for SSH using command: ssh -T -i '~/.ssh/sky-key' ubuntu@3.238.25.127 -p 22 -o StrictHostKeyChecking=no -o PasswordAuthentication=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptimeError: Warning: Permanently added '3.238.25.127' (ED25519) to the list of known hosts.

D 09-27 21:37:28 provisioner.py:356] "System is booting up. Unprivileged users are not permitted to log in yet. Please come back later. For technical details, see pam_nologin(8)."
D 09-27 21:37:28 provisioner.py:356] Connection closed by 3.238.25.127 port 22

D 09-27 21:37:28 provisioner.py:356] 
D 09-27 21:37:28 provisioner.py:400] Retrying in 1 second...
D 09-27 21:37:29 provisioner.py:345] Waiting for SSH using command: ssh -T -i '~/.ssh/sky-key' ubuntu@3.238.25.127 -p 22 -o StrictHostKeyChecking=no -o PasswordAuthentication=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime
D 09-27 21:37:34 provisioner.py:448] SSH Conection ready for 'automl'
I 09-27 21:37:34 provisioner.py:450] �[32mSuccessfully provisioned or found existing instance.�[0m
I 09-27 21:37:34 common.py:292] 
I 09-27 21:37:34 common.py:292] --------------------Start: initialize_docker --------------------
D 09-27 21:37:34 docker_utils.py:160] + command -v docker || echo 'NoExist'
/usr/bin/docker
D 09-27 21:37:36 docker_utils.py:160] + docker inspect -f "{{.State.Running}}" sky_container || true

Error: No such object: sky_container
D 09-27 21:37:40 docker_utils.py:160] + docker login --username AWS --password eyJwYXlsb2FkIjoicjNZemNLSzljY1RnQ2F4bDZhRS9wcGpGb2d2ejVjSFhvT2dLdFZXS1d6cTdQek1MeWMwUVZaRUdBczBMR2dTMFhSVnJKTEpIVDZSS0F6VkZ0b2Jab29EcVNCdHNHWkdLL2ZUTUdvL3VDWEtEelZUWEMzb056a1FwUTlwcFNlSENJMkI2VUNqM01XaXBZZ0dZL0JreXgxeFE0SkVrUnZ3QmRvV0tRRzJhaXc0Q2dmNHZhZFRrSFVzajJqMHZKc3UxWEZBSVI2emJHS2hhVU5lZit3UU44bVRaRDQvNTBKZWM5aU84b3VuWXFtWTdTYko5VmtPalhFWE5IVTRjc2Y4U0xaK2ZiaGd2ZlNtQUFWNi9iSzcyWk9ab1FNWG8xc1pWNWZKNUttY2dWVXJKNloyL0Jvb2dWRjBWcnF0U2JyMTIwZE43S3NLZHNTTVlqeVMvTHNOK3VDT3pWVFdjZjJsOGdpOEl6dVBiNlRid1JHaGFZaFhuNitlZG5yMGJGQXVaOFU2YnByZDk3UXJpdFQyTlR6MkdLaE9UNnp2ZnlzT2wxcUIwYXE2a3ZER0hBNHQvM0tkOUVZeXVYbGkxbUdmckUrU3N4aEU3KzY0eWZVQ2t1NThuZFBQQjFYd2paK0x0dXJ3Uml3aGd6eWRmN09TSWh5RjdoUUlVL1ZvMzhNQndHQU44MnEzdTRaN2pCS2JEVVpiMWFRV2s1MEtuZmdXaEQrYndySEdObEFxWDZlOUh2NE94MmMvR056eVpQcHFEZlE3eW5uRXF5OXdCQXh4VUR4QWtYUVBrWEo3dU55eGRVS0RuT05JYlRYT2xQY3k0Z1pzMU5rd29FODJaTHYvUHZhU01Bei9Gd3pWUnl6VWlNU2p5S1EwRDFaR29hYVBTNmpvVFQvdGEzV0pIMUhSU2NGd2tzNEZpNWpVWUVuZkx0V0svdnUvWUUxK1Bod2tQaUtCUUY4Z0JERnZOY3FBWUh5L01La0xwZ0pyTmY3R1hTNUtjMkxwZmIrbHlab0ZvMEpYWkhzQzg4N0tFZE5mdjVyZXdKQWMxRXh1U3ZENUJwQWVQbnZaTitXRXlaTGJIQ0I0Um92dVJUYVgvYlluUjFObDZvSDMvblNZN3hqYnhNMFo1bkd6emJxemwxQUYyMERnQnQ1YUs1c1BJeEd4dHNVU2FmVEhzdnBaNmt4VEFRT3dKeG1HemJzK3ZTN3h6NjV3SEN1eGpZRXBXT3M4eXAva0ViNExEVDNEQUtmTE54VXhSb0kzTWIzK0lpcFFWK1hkVzUyTG0zK0V2WXlKMkVianppcTRUcklVQm1UZFhpMzJvOHVIK3pPc2R0Mmd6S3ozNjRMOWZEeXFlUWxjNnoydmV6a21YVk4yTDlkNUk3TDFIVStSakJ5VUhORDl5U1dRZ2FFZEhLdDZrR3BDOU9USURhU3VDbFc2czN4cWJIVExIbllIRzJQbjhudFJVMHd4RHZLTUtjYkU0ay81RWlJTHptaDY0ZDk1VTdBV2VLNlRjdEVhdHN1ZXYvekw5MUR4cEpLQUMxNXdia0tEUjV6MkhSeHZtZURzK3lCZUVZa1NXVXF1WGhTQ3ZLanl1OWdxdllPRDR1S1F5cGV2N2JrVWRTUlpEOFVrcDcyVWRDT3NkOWhGRWFCaDlVUzJETG1VT0xMMmhpOUE2cmN2YUlZU2I3OWszcnNNdkZYUmdSUTBxSkpaelZESDc2MEUyR05DaVNoVzNNZjFRTjdPTDA4M2VYVEJDSzduc3Q4YXVWc2c1QjJaYjc3blBIb2hmTFBLNk4rLzBacm1qMC9oWFdMZ1p2YzVwZ2k1eXdUNkU2V3daTnN5NDR4dmM4WHZNdGdPWGdzVT0iLCJkYXRha2V5IjoiQVFFQkFIaHdtMFlhSVNKZVJ0Sm01bjFHNnVxZWVrWHVvWFhQZTVVRmNlOVJxOC8xNHdBQUFINHdmQVlKS29aSWh2Y05BUWNHb0c4d2JRSUJBREJvQmdrcWhraUc5dzBCQndFd0hnWUpZSVpJQVdVREJBRXVNQkVFRE0yQlozeGthMm9WRDFPK01RSUJFSUE3UE43bjVGQ1d1VXpodHhyUDQ2aWVZQzRScHgvbjRwckxiWWhQTHNIUjErYkZ4SzJnMy9pVTVoeEo4WGlSbWhsUEw1eUlBSGFHSmZuVzd1UT0iLCJ2ZXJzaW9uIjoiMiIsInR5cGUiOiJEQVRBX0tFWSIsImV4cGlyYXRpb24iOjE3Mjc1MzA3NTN9 783764614904.dkr.ecr.us-east-1.amazonaws.com
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in /home/ubuntu/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/#/#credentials-store

Login Succeeded
D 09-27 21:37:41 docker_utils.py:160] + docker pull 783764614904.dkr.ecr.us-east-1.amazonaws.com/tf14:latest
latest: Pulling from tf14
3f94e4e483ea: Pulling fs layer
4586c00479c6: Pulling fs layer
4304fa233a80: Pulling fs layer
afa3f70b397f: Pulling fs layer
d963a42bc712: Pulling fs layer
01d85bc616ac: Pulling fs layer
7d32cc707b9d: Pulling fs layer
4cea12640dc8: Pulling fs layer
076ff2aa313a: Pulling fs layer
5a520d59fe5e: Pulling fs layer
22d5103f16e8: Pulling fs layer
74669e8faa15: Pulling fs layer
400aa4b0a962: Pulling fs layer
81960391dc73: Pulling fs layer
45d81655ae23: Pulling fs layer
4db7c9771a19: Pulling fs layer
91c6784a9bf6: Pulling fs layer
2ef3e831f865: Pulling fs layer
e173899c2eb7: Pulling fs layer
85b495785842: Pulling fs layer
257d67e46fcc: Pulling fs layer
d6842126d25b: Pulling fs layer
de81a1a4e647: Pulling fs layer
4f4fb700ef54: Pulling fs layer
37168bd2d506: Pulling fs layer
35beabccd47f: Pulling fs layer
37c649e84e08: Pulling fs layer
afa3f70b397f: Waiting
d963a42bc712: Waiting
01d85bc616ac: Waiting
7d32cc707b9d: Waiting
4cea12640dc8: Waiting
076ff2aa313a: Waiting
5a520d59fe5e: Waiting
22d5103f16e8: Waiting
74669e8faa15: Waiting
400aa4b0a962: Waiting
81960391dc73: Waiting
45d81655ae23: Waiting
4db7c9771a19: Waiting
91c6784a9bf6: Waiting
2ef3e831f865: Waiting
e173899c2eb7: Waiting
85b495785842: Waiting
257d67e46fcc: Waiting
d6842126d25b: Waiting
de81a1a4e647: Waiting
4f4fb700ef54: Waiting
37168bd2d506: Waiting
35beabccd47f: Waiting
37c649e84e08: Waiting
4586c00479c6: Verifying Checksum
4586c00479c6: Download complete
afa3f70b397f: Verifying Checksum
afa3f70b397f: Download complete
d963a42bc712: Verifying Checksum
d963a42bc712: Download complete
01d85bc616ac: Verifying Checksum
01d85bc616ac: Download complete
7d32cc707b9d: Verifying Checksum
7d32cc707b9d: Download complete
4cea12640dc8: Verifying Checksum
4cea12640dc8: Download complete
3f94e4e483ea: Verifying Checksum
3f94e4e483ea: Download complete
4304fa233a80: Download complete
076ff2aa313a: Verifying Checksum
076ff2aa313a: Download complete
22d5103f16e8: Verifying Checksum
22d5103f16e8: Download complete
74669e8faa15: Download complete
3f94e4e483ea: Pull complete
4586c00479c6: Pull complete
400aa4b0a962: Verifying Checksum
400aa4b0a962: Download complete
45d81655ae23: Verifying Checksum
45d81655ae23: Download complete
4db7c9771a19: Verifying Checksum
4db7c9771a19: Download complete
91c6784a9bf6: Verifying Checksum
91c6784a9bf6: Download complete
2ef3e831f865: Verifying Checksum
2ef3e831f865: Download complete
e173899c2eb7: Verifying Checksum
e173899c2eb7: Download complete
85b495785842: Verifying Checksum
85b495785842: Download complete
4304fa233a80: Pull complete
afa3f70b397f: Pull complete
d963a42bc712: Pull complete
01d85bc616ac: Pull complete
7d32cc707b9d: Pull complete
4cea12640dc8: Pull complete
076ff2aa313a: Pull complete
257d67e46fcc: Verifying Checksum
257d67e46fcc: Download complete
d6842126d25b: Verifying Checksum
d6842126d25b: Download complete
de81a1a4e647: Download complete
4f4fb700ef54: Verifying Checksum
4f4fb700ef54: Download complete
37168bd2d506: Download complete
35beabccd47f: Download complete
37c649e84e08: Download complete
81960391dc73: Verifying Checksum
81960391dc73: Download complete
5a520d59fe5e: Verifying Checksum
5a520d59fe5e: Download complete
5a520d59fe5e: Pull complete
22d5103f16e8: Pull complete
74669e8faa15: Pull complete
400aa4b0a962: Pull complete
81960391dc73: Pull complete
45d81655ae23: Pull complete
4db7c9771a19: Pull complete
91c6784a9bf6: Pull complete
2ef3e831f865: Pull complete
e173899c2eb7: Pull complete
85b495785842: Pull complete
257d67e46fcc: Pull complete
d6842126d25b: Pull complete
de81a1a4e647: Pull complete
4f4fb700ef54: Pull complete
37168bd2d506: Pull complete
35beabccd47f: Pull complete
37c649e84e08: Pull complete
Digest: sha256:2be03bb243950d3116f12deee4269b44061f8e8a6259cbe56a66fd5a232d62cb
Status: Downloaded newer image for 783764614904.dkr.ecr.us-east-1.amazonaws.com/tf14:latest
783764614904.dkr.ecr.us-east-1.amazonaws.com/tf14:latest
I 09-27 21:42:18 docker_utils.py:239] Starting container sky_container with image 783764614904.dkr.ecr.us-east-1.amazonaws.com/tf14:latest
D 09-27 21:42:18 docker_utils.py:160] + docker inspect -f "{{.State.Running}}" sky_container || true

Error: No such object: sky_container
D 09-27 21:42:19 docker_utils.py:160] + [ -f /etc/docker/daemon.json ] || echo "{}" | sudo tee /etc/docker/daemon.json;sudo jq '.["exec-opts"] = ["native.cgroupdriver=cgroupfs"]' /etc/docker/daemon.json > /tmp/daemon.json;sudo mv /tmp/daemon.json /etc/docker/daemon.json;sudo systemctl restart docker
D 09-27 21:42:25 docker_utils.py:160] + cat /proc/meminfo || true
MemTotal:       16070200 kB
MemFree:         6407552 kB
MemAvailable:   15334100 kB
Buffers:          100588 kB
Cached:          8789712 kB
SwapCached:            0 kB
Active:           270352 kB
Inactive:        8745276 kB
Active(anon):        812 kB
Inactive(anon):   127736 kB
Active(file):     269540 kB
Inactive(file):  8617540 kB
Unevictable:       25580 kB
Mlocked:           18472 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:               228 kB
Writeback:             0 kB
AnonPages:        151136 kB
Mapped:           181260 kB
Shmem:               916 kB
KReclaimable:     375740 kB
Slab:             458684 kB
SReclaimable:     375740 kB
SUnreclaim:        82944 kB
KernelStack:        3572 kB
PageTables:         3516 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     8035100 kB
Committed_AS:    1232548 kB
VmallocTotal:   34359738367 kB
VmallocUsed:       80312 kB
VmallocChunk:          0 kB
Percpu:             3696 kB
HardwareCorrupted:     0 kB
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
FileHugePages:         0 kB
FilePmdMapped:         0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB
DirectMap4k:      262056 kB
DirectMap2M:     7819264 kB
DirectMap1G:     8388608 kB
D 09-27 21:42:25 docker_utils.py:160] + docker info -f "{{.Runtimes}}"
map[io.containerd.runc.v2:{runc []  map[] <nil>} nvidia:{nvidia-container-runtime []  map[] <nil>} runc:{runc []  map[] <nil>}]
D 09-27 21:42:30 docker_utils.py:160] + nvidia-smi
Sat Sep 28 02:42:31 2024       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.12             Driver Version: 535.104.12   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Tesla T4                       On  | 00000000:00:1E.0 Off |                    0 |
| N/A   31C    P8               9W /  70W |      0MiB / 15360MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+
D 09-27 21:42:30 docker_utils.py:160] + docker run --name sky_container -d -it -e LC_ALL=C.UTF-8 -e LANG=C.UTF-8 --ulimit nofile=1048576:1048576 --shm-size="5181699072.0b" --runtime=nvidia --gpus all --net=host --cap-add=SYS_ADMIN --device=/dev/fuse --security-opt=apparmor:unconfined --entrypoint=/bin/bash 783764614904.dkr.ecr.us-east-1.amazonaws.com/tf14:latest
749c295bd5c29f92d59731ab66d3a912acb69e3410f7f1acc7a0c5265cb7f242
D 09-27 21:42:34 docker_utils.py:160] + docker exec sky_container printenv HOME
/root
D 09-27 21:42:34 docker_utils.py:160] + docker exec sky_container /bin/bash -c 'bash --login -c -i '"'"'source ~/.bashrc; export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && (echo '"'"'"'"'"'"'"'"'{ [ "$(whoami)" == "root" ] && function sudo() { "$@"; } || true; }'"'"'"'"'"'"'"'"' >> /root/.bashrc;echo "export DEBIAN_FRONTEND=noninteractive" >> /root/.bashrc;)'"'"'' 
�[1;31m
________                               _______________
___  __/__________________________________  ____/__  /________      __
__  /  _  _ \_  __ \_  ___/  __ \_  ___/_  /_   __  /_  __ \_ | /| / /
_  /   /  __/  / / /(__  )/ /_/ /  /   _  __/   _  / / /_/ /_ |/ |/ /
/_/    \___//_/ /_//____/ \____//_/    /_/      /_/  \____/____/|__/

�[0;33m
WARNING: You are running this container as root, which can cause new files in
mounted volumes to be created as the root user on your host machine.

To avoid this, run the container by specifying your user's userid:

$ docker run -u $(id -u):$(id -g) args...
�[m
D 09-27 21:42:34 docker_utils.py:160] + docker exec sky_container /bin/bash -c 'bash --login -c -i '"'"'source ~/.bashrc; export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && (sudo apt-get update; sudo apt-get -o DPkg::Options::="--force-confnew" install -y rsync curl wget patch openssh-server python3-pip fuse;)'"'"'' 
�[1;31m
________                               _______________
___  __/__________________________________  ____/__  /________      __
__  /  _  _ \_  __ \_  ___/  __ \_  ___/_  /_   __  /_  __ \_ | /| / /
_  /   /  __/  / / /(__  )/ /_/ /  /   _  __/   _  / / /_/ /_ |/ |/ /
/_/    \___//_/ /_//____/ \____//_/    /_/      /_/  \____/____/|__/

�[0;33m
WARNING: You are running this container as root, which can cause new files in
mounted volumes to be created as the root user on your host machine.

To avoid this, run the container by specifying your user's userid:

$ docker run -u $(id -u):$(id -g) args...
�[m
Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB]
Get:2 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  InRelease [1581 B]
Get:3 http://archive.ubuntu.com/ubuntu jammy InRelease [270 kB]
Get:4 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [1156 kB]
Get:5 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB]
Get:6 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages [1001 kB]
Get:7 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB]
Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [3113 kB]
Get:9 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [2318 kB]
Get:10 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [44.7 kB]
Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 Packages [1792 kB]
Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 Packages [17.5 MB]
Get:13 http://archive.ubuntu.com/ubuntu jammy/multiverse amd64 Packages [266 kB]
Get:14 http://archive.ubuntu.com/ubuntu jammy/restricted amd64 Packages [164 kB]
Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1445 kB]
Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [2595 kB]
Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [3191 kB]
Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [51.8 kB]
Get:19 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [81.4 kB]
Get:20 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [33.7 kB]
Fetched 35.4 MB in 2s (16.5 MB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
patch is already the newest version (2.7.6-7build2).
patch set to manually installed.
The following additional packages will be installed:
  javascript-common libcbor0.8 libcurl4 libfido2-1 libfuse2 libjs-jquery
  libjs-sphinxdoc libjs-underscore libpopt0 libpython3-dev libpython3-stdlib
  libpython3.10 libpython3.10-dev libpython3.10-minimal libpython3.10-stdlib
  libwrap0 libxmuu1 ncurses-term openssh-client openssh-sftp-server python3
  python3-dev python3-minimal python3-pkg-resources python3-setuptools
  python3-wheel python3.10 python3.10-dev python3.10-minimal ssh-import-id ucf
  xauth
Suggested packages:
  apache2 | lighttpd | httpd keychain libpam-ssh monkeysphere ssh-askpass
  molly-guard ufw python3-doc python3-tk python3-venv python-setuptools-doc
  python3.10-venv python3.10-doc binfmt-support python3-braceexpand
The following NEW packages will be installed:
  fuse javascript-common libcbor0.8 libfido2-1 libfuse2 libjs-jquery
  libjs-sphinxdoc libjs-underscore libpopt0 libpython3-dev libpython3.10
  libpython3.10-dev libwrap0 libxmuu1 ncurses-term openssh-client
  openssh-server openssh-sftp-server python3-dev python3-pip
  python3-setuptools python3-wheel python3.10-dev rsync ssh-import-id ucf wget
  xauth
The following packages will be upgraded:
  curl libcurl4 libpython3-stdlib libpython3.10-minimal libpython3.10-stdlib
  python3 python3-minimal python3-pkg-resources python3.10 python3.10-minimal
10 upgraded, 28 newly installed, 0 to remove and 86 not upgraded.
Need to get 18.4 MB of archives.
After this operation, 50.2 MB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-minimal amd64 3.10.6-1~22.04.1 [24.3 kB]
Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3 amd64 3.10.6-1~22.04.1 [22.8 kB]
Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3.10 amd64 3.10.12-1~22.04.6 [509 kB]
Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpython3.10-stdlib amd64 3.10.12-1~22.04.6 [1850 kB]
Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3.10-minimal amd64 3.10.12-1~22.04.6 [2258 kB]
Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpython3.10-minimal amd64 3.10.12-1~22.04.6 [814 kB]
Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpython3-stdlib amd64 3.10.6-1~22.04.1 [6812 B]
Get:8 http://archive.ubuntu.com/ubuntu jammy/main amd64 libpopt0 amd64 1.18-3build1 [28.2 kB]
Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 rsync amd64 3.2.7-0ubuntu0.22.04.2 [436 kB]
Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 libcbor0.8 amd64 0.8.0-2ubuntu1 [24.6 kB]
Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfido2-1 amd64 1.10.0-1 [82.8 kB]
Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 openssh-client amd64 1:8.9p1-3ubuntu0.10 [906 kB]
Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 openssh-sftp-server amd64 1:8.9p1-3ubuntu0.10 [38.9 kB]
Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 ucf all 3.0043 [56.1 kB]
Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 libwrap0 amd64 7.6.q-31build2 [47.9 kB]
Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 openssh-server amd64 1:8.9p1-3ubuntu0.10 [435 kB]
Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-pkg-resources all 59.6.0-1.2ubuntu0.22.04.2 [133 kB]
Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 libxmuu1 amd64 2:1.1.3-3 [10.2 kB]
Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 wget amd64 1.21.2-2ubuntu1.1 [339 kB]
Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 xauth amd64 1:1.1-1build2 [27.5 kB]
Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 curl amd64 7.81.0-1ubuntu1.18 [194 kB]
Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libcurl4 amd64 7.81.0-1ubuntu1.18 [289 kB]
Get:23 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libfuse2 amd64 2.9.9-5ubuntu3 [90.3 kB]
Get:24 http://archive.ubuntu.com/ubuntu jammy/universe amd64 fuse amd64 2.9.9-5ubuntu3 [27.0 kB]
Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 javascript-common all 11+nmu1 [5936 B]
Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjs-jquery all 3.6.0+dfsg+~3.5.13-1 [321 kB]
Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjs-underscore all 1.13.2~dfsg-2 [118 kB]
Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjs-sphinxdoc all 4.3.2-1 [139 kB]
Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpython3.10 amd64 3.10.12-1~22.04.6 [1949 kB]
Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpython3.10-dev amd64 3.10.12-1~22.04.6 [4764 kB]
Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpython3-dev amd64 3.10.6-1~22.04.1 [7064 B]
Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 ncurses-term all 6.3-2ubuntu0.1 [267 kB]
Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3.10-dev amd64 3.10.12-1~22.04.6 [508 kB]
Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-dev amd64 3.10.6-1~22.04.1 [26.0 kB]
Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-setuptools all 59.6.0-1.2ubuntu0.22.04.2 [340 kB]
Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 python3-wheel all 0.37.1-2ubuntu0.22.04.1 [32.0 kB]
Get:37 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 python3-pip all 22.0.2+dfsg-1ubuntu0.4 [1305 kB]
Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 ssh-import-id all 5.11-0ubuntu1 [10.1 kB]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 18.4 MB in 1s (23.2 MB/s)
(Reading database ... 
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 20106 files and directories currently installed.)
Preparing to unpack .../python3-minimal_3.10.6-1~22.04.1_amd64.deb ...
Unpacking python3-minimal (3.10.6-1~22.04.1) over (3.10.6-1~22.04) ...
Setting up python3-minimal (3.10.6-1~22.04.1) ...
(Reading database ... 
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 20106 files and directories currently installed.)
Preparing to unpack .../00-python3_3.10.6-1~22.04.1_amd64.deb ...
Unpacking python3 (3.10.6-1~22.04.1) over (3.10.6-1~22.04) ...
Preparing to unpack .../01-python3.10_3.10.12-1~22.04.6_amd64.deb ...
Unpacking python3.10 (3.10.12-1~22.04.6) over (3.10.12-1~22.04.2) ...
Preparing to unpack .../02-libpython3.10-stdlib_3.10.12-1~22.04.6_amd64.deb ...
Unpacking libpython3.10-stdlib:amd64 (3.10.12-1~22.04.6) over (3.10.12-1~22.04.2) ...
Preparing to unpack .../03-python3.10-minimal_3.10.12-1~22.04.6_amd64.deb ...
Unpacking python3.10-minimal (3.10.12-1~22.04.6) over (3.10.12-1~22.04.2) ...
Preparing to unpack .../04-libpython3.10-minimal_3.10.12-1~22.04.6_amd64.deb ...
Unpacking libpython3.10-minimal:amd64 (3.10.12-1~22.04.6) over (3.10.12-1~22.04.2) ...
Preparing to unpack .../05-libpython3-stdlib_3.10.6-1~22.04.1_amd64.deb ...
Unpacking libpython3-stdlib:amd64 (3.10.6-1~22.04.1) over (3.10.6-1~22.04) ...
Selecting previously unselected package libpopt0:amd64.
Preparing to unpack .../06-libpopt0_1.18-3build1_amd64.deb ...
Unpacking libpopt0:amd64 (1.18-3build1) ...
Selecting previously unselected package rsync.
Preparing to unpack .../07-rsync_3.2.7-0ubuntu0.22.04.2_amd64.deb ...
Unpacking rsync (3.2.7-0ubuntu0.22.04.2) ...
Selecting previously unselected package libcbor0.8:amd64.
Preparing to unpack .../08-libcbor0.8_0.8.0-2ubuntu1_amd64.deb ...
Unpacking libcbor0.8:amd64 (0.8.0-2ubuntu1) ...
Selecting previously unselected package libfido2-1:amd64.
Preparing to unpack .../09-libfido2-1_1.10.0-1_amd64.deb ...
Unpacking libfido2-1:amd64 (1.10.0-1) ...
Selecting previously unselected package openssh-client.
Preparing to unpack .../10-openssh-client_1%3a8.9p1-3ubuntu0.10_amd64.deb ...
Unpacking openssh-client (1:8.9p1-3ubuntu0.10) ...
Selecting previously unselected package openssh-sftp-server.
Preparing to unpack .../11-openssh-sftp-server_1%3a8.9p1-3ubuntu0.10_amd64.deb ...
Unpacking openssh-sftp-server (1:8.9p1-3ubuntu0.10) ...
Selecting previously unselected package ucf.
Preparing to unpack .../12-ucf_3.0043_all.deb ...
Moving old data out of the way
Unpacking ucf (3.0043) ...
Selecting previously unselected package libwrap0:amd64.
Preparing to unpack .../13-libwrap0_7.6.q-31build2_amd64.deb ...
Unpacking libwrap0:amd64 (7.6.q-31build2) ...
Selecting previously unselected package openssh-server.
Preparing to unpack .../14-openssh-server_1%3a8.9p1-3ubuntu0.10_amd64.deb ...
Unpacking openssh-server (1:8.9p1-3ubuntu0.10) ...
Preparing to unpack .../15-python3-pkg-resources_59.6.0-1.2ubuntu0.22.04.2_all.deb ...
Unpacking python3-pkg-resources (59.6.0-1.2ubuntu0.22.04.2) over (59.6.0-1.2ubuntu0.22.04.1) ...
Selecting previously unselected package libxmuu1:amd64.
Preparing to unpack .../16-libxmuu1_2%3a1.1.3-3_amd64.deb ...
Unpacking libxmuu1:amd64 (2:1.1.3-3) ...
Selecting previously unselected package wget.
Preparing to unpack .../17-wget_1.21.2-2ubuntu1.1_amd64.deb ...
Unpacking wget (1.21.2-2ubuntu1.1) ...
Selecting previously unselected package xauth.
Preparing to unpack .../18-xauth_1%3a1.1-1build2_amd64.deb ...
Unpacking xauth (1:1.1-1build2) ...
Preparing to unpack .../19-curl_7.81.0-1ubuntu1.18_amd64.deb ...
Unpacking curl (7.81.0-1ubuntu1.18) over (7.81.0-1ubuntu1.13) ...
Preparing to unpack .../20-libcurl4_7.81.0-1ubuntu1.18_amd64.deb ...
Unpacking libcurl4:amd64 (7.81.0-1ubuntu1.18) over (7.81.0-1ubuntu1.13) ...
Selecting previously unselected package libfuse2:amd64.
Preparing to unpack .../21-libfuse2_2.9.9-5ubuntu3_amd64.deb ...
Unpacking libfuse2:amd64 (2.9.9-5ubuntu3) ...
Selecting previously unselected package fuse.
Preparing to unpack .../22-fuse_2.9.9-5ubuntu3_amd64.deb ...
Unpacking fuse (2.9.9-5ubuntu3) ...
Selecting previously unselected package javascript-common.
Preparing to unpack .../23-javascript-common_11+nmu1_all.deb ...
Unpacking javascript-common (11+nmu1) ...
Selecting previously unselected package libjs-jquery.
Preparing to unpack .../24-libjs-jquery_3.6.0+dfsg+~3.5.13-1_all.deb ...
Unpacking libjs-jquery (3.6.0+dfsg+~3.5.13-1) ...
Selecting previously unselected package libjs-underscore.
Preparing to unpack .../25-libjs-underscore_1.13.2~dfsg-2_all.deb ...
Unpacking libjs-underscore (1.13.2~dfsg-2) ...
Selecting previously unselected package libjs-sphinxdoc.
Preparing to unpack .../26-libjs-sphinxdoc_4.3.2-1_all.deb ...
Unpacking libjs-sphinxdoc (4.3.2-1) ...
Selecting previously unselected package libpython3.10:amd64.
Preparing to unpack .../27-libpython3.10_3.10.12-1~22.04.6_amd64.deb ...
Unpacking libpython3.10:amd64 (3.10.12-1~22.04.6) ...
Selecting previously unselected package libpython3.10-dev:amd64.
Preparing to unpack .../28-libpython3.10-dev_3.10.12-1~22.04.6_amd64.deb ...
Unpacking libpython3.10-dev:amd64 (3.10.12-1~22.04.6) ...
Selecting previously unselected package libpython3-dev:amd64.
Preparing to unpack .../29-libpython3-dev_3.10.6-1~22.04.1_amd64.deb ...
Unpacking libpython3-dev:amd64 (3.10.6-1~22.04.1) ...
Selecting previously unselected package ncurses-term.
Preparing to unpack .../30-ncurses-term_6.3-2ubuntu0.1_all.deb ...
Unpacking ncurses-term (6.3-2ubuntu0.1) ...
Selecting previously unselected package python3.10-dev.
Preparing to unpack .../31-python3.10-dev_3.10.12-1~22.04.6_amd64.deb ...
Unpacking python3.10-dev (3.10.12-1~22.04.6) ...
Selecting previously unselected package python3-dev.
Preparing to unpack .../32-python3-dev_3.10.6-1~22.04.1_amd64.deb ...
Unpacking python3-dev (3.10.6-1~22.04.1) ...
Selecting previously unselected package python3-setuptools.
Preparing to unpack .../33-python3-setuptools_59.6.0-1.2ubuntu0.22.04.2_all.deb ...
Unpacking python3-setuptools (59.6.0-1.2ubuntu0.22.04.2) ...
Selecting previously unselected package python3-wheel.
Preparing to unpack .../34-python3-wheel_0.37.1-2ubuntu0.22.04.1_all.deb ...
Unpacking python3-wheel (0.37.1-2ubuntu0.22.04.1) ...
Selecting previously unselected package python3-pip.
Preparing to unpack .../35-python3-pip_22.0.2+dfsg-1ubuntu0.4_all.deb ...
Unpacking python3-pip (22.0.2+dfsg-1ubuntu0.4) ...
Selecting previously unselected package ssh-import-id.
Preparing to unpack .../36-ssh-import-id_5.11-0ubuntu1_all.deb ...
Unpacking ssh-import-id (5.11-0ubuntu1) ...
Setting up javascript-common (11+nmu1) ...
Setting up wget (1.21.2-2ubuntu1.1) ...
Setting up libcbor0.8:amd64 (0.8.0-2ubuntu1) ...
Setting up libfuse2:amd64 (2.9.9-5ubuntu3) ...
Setting up libwrap0:amd64 (7.6.q-31build2) ...
Setting up libpython3.10-minimal:amd64 (3.10.12-1~22.04.6) ...
Setting up ucf (3.0043) ...
Setting up libcurl4:amd64 (7.81.0-1ubuntu1.18) ...
Setting up curl (7.81.0-1ubuntu1.18) ...
Setting up libjs-jquery (3.6.0+dfsg+~3.5.13-1) ...
Setting up libfido2-1:amd64 (1.10.0-1) ...
Setting up libxmuu1:amd64 (2:1.1.3-3) ...
Setting up libjs-underscore (1.13.2~dfsg-2) ...
Setting up libpopt0:amd64 (1.18-3build1) ...
Setting up ncurses-term (6.3-2ubuntu0.1) ...
Setting up fuse (2.9.9-5ubuntu3) ...
Setting up openssh-client (1:8.9p1-3ubuntu0.10) ...
update-alternatives: using /usr/bin/ssh to provide /usr/bin/rsh (rsh) in auto mode
update-alternatives: warning: skip creation of /usr/share/man/man1/rsh.1.gz because associated file /usr/share/man/man1/ssh.1.gz (of link group rsh) doesn't exist
update-alternatives: using /usr/bin/slogin to provide /usr/bin/rlogin (rlogin) in auto mode
update-alternatives: warning: skip creation of /usr/share/man/man1/rlogin.1.gz because associated file /usr/share/man/man1/slogin.1.gz (of link group rlogin) doesn't exist
update-alternatives: using /usr/bin/scp to provide /usr/bin/rcp (rcp) in auto mode
update-alternatives: warning: skip creation of /usr/share/man/man1/rcp.1.gz because associated file /usr/share/man/man1/scp.1.gz (of link group rcp) doesn't exist
Setting up python3.10-minimal (3.10.12-1~22.04.6) ...
Setting up libpython3.10-stdlib:amd64 (3.10.12-1~22.04.6) ...
Setting up libjs-sphinxdoc (4.3.2-1) ...
Setting up xauth (1:1.1-1build2) ...
Setting up rsync (3.2.7-0ubuntu0.22.04.2) ...
invoke-rc.d: could not determine current runlevel
invoke-rc.d: policy-rc.d denied execution of start.
Setting up libpython3-stdlib:amd64 (3.10.6-1~22.04.1) ...
Setting up openssh-sftp-server (1:8.9p1-3ubuntu0.10) ...
Setting up libpython3.10:amd64 (3.10.12-1~22.04.6) ...
Setting up python3.10 (3.10.12-1~22.04.6) ...
Setting up openssh-server (1:8.9p1-3ubuntu0.10) ...

Creating config file /etc/ssh/sshd_config with new version
Creating SSH2 RSA key; this may take some time ...
3072 SHA256:5mwt8Dl1JnE6xFcK3kqUB7m1mMEWo/2rkoQEErxCJwE root@ip-172-31-77-158 (RSA)
Creating SSH2 ECDSA key; this may take some time ...
256 SHA256:3+jOkesZw5iLwa2GWwYw9UQGW/8BRHxSEku5b1Yq4ts root@ip-172-31-77-158 (ECDSA)
Creating SSH2 ED25519 key; this may take some time ...
256 SHA256:T4szgoxRZS1YVyUp2NhboG0WOnzkh4a7LshyF/kjyWM root@ip-172-31-77-158 (ED25519)
invoke-rc.d: could not determine current runlevel
invoke-rc.d: policy-rc.d denied execution of start.
Created symlink /etc/systemd/system/sshd.service → /lib/systemd/system/ssh.service.
Created symlink /etc/systemd/system/multi-user.target.wants/ssh.service → /lib/systemd/system/ssh.service.
Setting up python3 (3.10.6-1~22.04.1) ...
Setting up python3-wheel (0.37.1-2ubuntu0.22.04.1) ...
Setting up libpython3.10-dev:amd64 (3.10.12-1~22.04.6) ...
Setting up python3.10-dev (3.10.12-1~22.04.6) ...
Setting up python3-pkg-resources (59.6.0-1.2ubuntu0.22.04.2) ...
Setting up libpython3-dev:amd64 (3.10.6-1~22.04.1) ...
Setting up python3-setuptools (59.6.0-1.2ubuntu0.22.04.2) ...
Setting up ssh-import-id (5.11-0ubuntu1) ...
Setting up python3-dev (3.10.6-1~22.04.1) ...
Setting up python3-pip (22.0.2+dfsg-1ubuntu0.4) ...
Processing triggers for libc-bin (2.35-0ubuntu3.1) ...
D 09-27 21:42:52 docker_utils.py:160] + rsync -e "docker exec -i" -avz ~/.ssh/authorized_keys sky_container:/tmp/host_ssh_authorized_keys;sudo systemctl stop jupyter > /dev/null 2>&1 || true;sudo systemctl disable jupyter > /dev/null 2>&1 || true;sudo systemctl stop jupyterhub > /dev/null 2>&1 || true;sudo systemctl disable jupyterhub > /dev/null 2>&1 || true;
sending incremental file list
authorized_keys

sent 431 bytes  received 35 bytes  932.00 bytes/sec
total size is 381  speedup is 0.82
D 09-27 21:42:52 docker_utils.py:160] + docker exec sky_container /bin/bash -c 'bash --login -c -i '"'"'source ~/.bashrc; export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && (sudo sed -i "s/#Port 22/Port 10022/" /etc/ssh/sshd_config;mkdir -p /root/.ssh;cat /tmp/host_ssh_authorized_keys >> /root/.ssh/authorized_keys;sudo service ssh start;sudo sed -i "s/mesg n/tty -s \&\& mesg n/" /root/.profile;prefix_cmd() { if [ $(id -u) -ne 0 ]; then echo "sudo"; else echo ""; fi; } && printenv | while IFS='"'"'"'"'"'"'"'"'='"'"'"'"'"'"'"'"' read -r key value; do echo "export $key=\"$value\""; done > /root/container_env_var.sh && $(prefix_cmd) mv /root/container_env_var.sh /etc/profile.d/container_env_var.sh)'"'"'' 
�[1;31m
________                               _______________
___  __/__________________________________  ____/__  /________      __
__  /  _  _ \_  __ \_  ___/  __ \_  ___/_  /_   __  /_  __ \_ | /| / /
_  /   /  __/  / / /(__  )/ /_/ /  /   _  __/   _  / / /_/ /_ |/ |/ /
/_/    \___//_/ /_//____/ \____//_/    /_/      /_/  \____/____/|__/

�[0;33m
WARNING: You are running this container as root, which can cause new files in
mounted volumes to be created as the root user on your host machine.

To avoid this, run the container by specifying your user's userid:

$ docker run -u $(id -u):$(id -g) args...
�[m
 * Starting OpenBSD Secure Shell server sshd
   ...done.
D 09-27 21:42:52 docker_utils.py:160] + docker exec sky_container /bin/bash -c 'bash --login -c -i '"'"'source ~/.bashrc; export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && (whoami)'"'"'' 
�[1;31m
________                               _______________
___  __/__________________________________  ____/__  /________      __
__  /  _  _ \_  __ \_  ___/  __ \_  ___/_  /_   __  /_  __ \_ | /| / /
_  /   /  __/  / / /(__  )/ /_/ /  /   _  __/   _  / / /_/ /_ |/ |/ /
/_/    \___//_/ /_//____/ \____//_/    /_/      /_/  \____/____/|__/

�[0;33m
WARNING: You are running this container as root, which can cause new files in
mounted volumes to be created as the root user on your host machine.

To avoid this, run the container by specifying your user's userid:

$ docker run -u $(id -u):$(id -g) args...
�[m
root
D 09-27 21:42:53 instance_setup.py:155] Initialized docker user: �[1;31m
D 09-27 21:42:53 instance_setup.py:155] ________                               _______________
D 09-27 21:42:53 instance_setup.py:155] ___  __/__________________________________  ____/__  /________      __
D 09-27 21:42:53 instance_setup.py:155] __  /  _  _ \_  __ \_  ___/  __ \_  ___/_  /_   __  /_  __ \_ | /| / /
D 09-27 21:42:53 instance_setup.py:155] _  /   /  __/  / / /(__  )/ /_/ /  /   _  __/   _  / / /_/ /_ |/ |/ /
D 09-27 21:42:53 instance_setup.py:155] /_/    \___//_/ /_//____/ \____//_/    /_/      /_/  \____/____/|__/
D 09-27 21:42:53 instance_setup.py:155] 
D 09-27 21:42:53 instance_setup.py:155] �[0;33m
D 09-27 21:42:53 instance_setup.py:155] WARNING: You are running this container as root, which can cause new files in
D 09-27 21:42:53 instance_setup.py:155] mounted volumes to be created as the root user on your host machine.
D 09-27 21:42:53 instance_setup.py:155] 
D 09-27 21:42:53 instance_setup.py:155] To avoid this, run the container by specifying your user's userid:
D 09-27 21:42:53 instance_setup.py:155] 
D 09-27 21:42:53 instance_setup.py:155] $ docker run -u $(id -u):$(id -g) args...
D 09-27 21:42:53 instance_setup.py:155] �[m
D 09-27 21:42:53 instance_setup.py:155] root
D 09-27 21:42:53 instance_setup.py:167] All docker users: ["\x1b[1;31m\n________                               _______________\n___  __/__________________________________  ____/__  /________      __\n__  /  _  _ \\_  __ \\_  ___/  __ \\_  ___/_  /_   __  /_  __ \\_ | /| / /\n_  /   /  __/  / / /(__  )/ /_/ /  /   _  __/   _  / / /_/ /_ |/ |/ /\n/_/    \\___//_/ /_//____/ \\____//_/    /_/      /_/  \\____/____/|__/\n\n\x1b[0;33m\nWARNING: You are running this container as root, which can cause new files in\nmounted volumes to be created as the root user on your host machine.\n\nTo avoid this, run the container by specifying your user's userid:\n\n$ docker run -u $(id -u):$(id -g) args...\n\x1b[m\nroot"]
I 09-27 21:42:53 common.py:296] --------------------End:   initialize_docker --------------------
I 09-27 21:42:53 common.py:296] 
D 09-27 21:42:53 provisioner.py:470] Docker user: �[1;31m
D 09-27 21:42:53 provisioner.py:470] ________                               _______________
D 09-27 21:42:53 provisioner.py:470] ___  __/__________________________________  ____/__  /________      __
D 09-27 21:42:53 provisioner.py:470] __  /  _  _ \_  __ \_  ___/  __ \_  ___/_  /_   __  /_  __ \_ | /| / /
D 09-27 21:42:53 provisioner.py:470] _  /   /  __/  / / /(__  )/ /_/ /  /   _  __/   _  / / /_/ /_ |/ |/ /
D 09-27 21:42:53 provisioner.py:470] /_/    \___//_/ /_//____/ \____//_/    /_/      /_/  \____/____/|__/
D 09-27 21:42:53 provisioner.py:470] 
D 09-27 21:42:53 provisioner.py:470] �[0;33m
D 09-27 21:42:53 provisioner.py:470] WARNING: You are running this container as root, which can cause new files in
D 09-27 21:42:53 provisioner.py:470] mounted volumes to be created as the root user on your host machine.
D 09-27 21:42:53 provisioner.py:470] 
D 09-27 21:42:53 provisioner.py:470] To avoid this, run the container by specifying your user's userid:
D 09-27 21:42:53 provisioner.py:470] 
D 09-27 21:42:53 provisioner.py:470] $ docker run -u $(id -u):$(id -g) args...
D 09-27 21:42:53 provisioner.py:470] �[m
D 09-27 21:42:53 provisioner.py:470] root
I 09-27 21:42:53 common.py:292] 
I 09-27 21:42:53 common.py:292] --------------------Start: internal_file_mounts --------------------
D 09-27 21:42:53 instance_setup.py:485] Using 31 workers for file mounts.
hostname contains invalid characters
/bin/sh: 1: 31m: not found
/bin/sh: 2: ________: not found
/bin/sh: 3: ___: not found
/bin/sh: 4: /bin/sh: 4: /: Permission denied
/: Permission denied
/bin/sh: 4: __: not found
/bin/sh: 5: Syntax error: "(" unexpected
I 09-27 21:42:53 instance_setup.py:90] _internal_file_mounts: Retrying in 1.2 seconds, due to Command mkdir -p ~/.sky/.runtime_files failed with return code 2.
I 09-27 21:42:53 instance_setup.py:90] �[31mFailed to run command before rsync /tmp/tmp6ng1mu8o -> ~/.sky/.runtime_files.�[0m
hostname contains invalid characters
/bin/sh: 1: 31m: not found
/bin/sh: 2: ________: not found
/bin/sh: 3: ___: not found
/bin/sh: 4: /: Permission denied
/bin/sh: 4: /: Permission denied
/bin/sh: 4: __: not found
/bin/sh: 5: Syntax error: "(" unexpected
I 09-27 21:42:54 instance_setup.py:90] _internal_file_mounts: Retrying in 1.9 seconds, due to Command mkdir -p ~/.sky/.runtime_files failed with return code 2.
I 09-27 21:42:54 instance_setup.py:90] �[31mFailed to run command before rsync /tmp/tmp6ng1mu8o -> ~/.sky/.runtime_files.�[0m
hostname contains invalid characters
/bin/sh: 1: 31m: not found
/bin/sh: 2: ________: not found
/bin/sh: 3: ___: not found
/bin/sh: 4: /: Permission denied
/bin/sh: 4: /: Permission denied
/bin/sh: 4: __: not found
/bin/sh: 5: Syntax error: "(" unexpected
I 09-27 21:42:56 instance_setup.py:90] _internal_file_mounts: Retrying in 2.7 seconds, due to Command mkdir -p ~/.sky/.runtime_files failed with return code 2.
I 09-27 21:42:56 instance_setup.py:90] �[31mFailed to run command before rsync /tmp/tmp6ng1mu8o -> ~/.sky/.runtime_files.�[0m
hostname contains invalid characters
/bin/sh: 1: 31m: not found
/bin/sh: 2: ________: not found
/bin/sh: 3: ___: not found
/bin/sh: 4: /: Permission denied/bin/sh: 4: 
/: Permission denied
/bin/sh: 4: __: not found
/bin/sh: 5: Syntax error: "(" unexpected
I 09-27 21:42:59 instance_setup.py:90] _internal_file_mounts: Retrying in 5.1 seconds, due to Command mkdir -p ~/.sky/.runtime_files failed with return code 2.
I 09-27 21:42:59 instance_setup.py:90] �[31mFailed to run command before rsync /tmp/tmp6ng1mu8o -> ~/.sky/.runtime_files.�[0m
hostname contains invalid characters
/bin/sh: 1: 31m: not found
/bin/sh: 2: ________: not found
/bin/sh: 3: ___: not found
/bin/sh: 4: /: Permission denied
/bin/sh: 4: /: Permission denied
/bin/sh: 4: __: not found
/bin/sh: 5: Syntax error: "(" unexpected
I 09-27 21:43:04 instance_setup.py:90] _internal_file_mounts: Retrying in 4.1 seconds, due to Command mkdir -p ~/.sky/.runtime_files failed with return code 2.
I 09-27 21:43:04 instance_setup.py:90] �[31mFailed to run command before rsync /tmp/tmp6ng1mu8o -> ~/.sky/.runtime_files.�[0m
hostname contains invalid characters
/bin/sh: 1: 31m: not found
/bin/sh: 2: ________: not found
/bin/sh: 3: ___: not found
/bin/sh: 4: /: Permission denied
/bin/sh: 4: /: Permission denied
/bin/sh: 4: __: not found
/bin/sh: 5: Syntax error: "(" unexpected
I 09-27 21:43:08 common.py:296] --------------------End:   internal_file_mounts --------------------
I 09-27 21:43:08 common.py:296] 
E 09-27 21:43:08 provisioner.py:583] *** Failed setting up cluster. ***
D 09-27 21:43:08 provisioner.py:584] Stacktrace:
D 09-27 21:43:08 provisioner.py:584] sky.exceptions.CommandError: Command mkdir -p ~/.sky/.runtime_files failed with return code 2.
D 09-27 21:43:08 provisioner.py:584] �[31mFailed to run command before rsync /tmp/tmp6ng1mu8o -> ~/.sky/.runtime_files.�[0m
D 09-27 21:43:08 provisioner.py:584]

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[bug] Failing to run as docker runtime #4005

[bug] Failing to run as docker runtime #4005

merryHunter commented Sep 26, 2024

Michaelvll commented Sep 26, 2024

merryHunter commented Sep 26, 2024

merryHunter commented Sep 26, 2024

Michaelvll commented Sep 26, 2024

merryHunter commented Sep 28, 2024

merryHunter commented Sep 28, 2024

[bug] Failing to run as docker runtime #4005

[bug] Failing to run as docker runtime #4005

Comments

merryHunter commented Sep 26, 2024

Michaelvll commented Sep 26, 2024

merryHunter commented Sep 26, 2024

merryHunter commented Sep 26, 2024

Michaelvll commented Sep 26, 2024

merryHunter commented Sep 28, 2024

merryHunter commented Sep 28, 2024