Propagate kubernetes custom metadata annotations to sub-services

[fozziethebeat]

This forwards kubernetes annotation values in the ~/.sky/config to sub services such as the load balancer. This ensures users can set custom annotations, such as for AWS, to ensure load balancers are internet facing.

Tested (run the relevant ones):

• Code formatting: bash format.sh
• Any manual or new tests for this PR (please specify below)
• All smoke tests: pytest tests/test_smoke.py
• Relevant individual smoke tests: pytest tests/test_smoke.py::test_fill_in_the_name
• Backward compatibility tests: conda deactivate; bash -i tests/backward_compatibility_tests.sh

Manually update my ~/.sky/config to have the following contents:

kubernetes:
  ports: loadbalancer
  custom_metadata:
    annotations:
      service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing

And ran sky launch with this config:

service:
  readiness_probe: /v1/models

resources:
  # Can change to use more via `--gpus A100:N`.  N can be 1 to 8.
  accelerators: A100:2
  cpus: 22
  memory: 500
  # Note: Big models need LOTS of disk space, especially if saved in float32.
  # So specify a lot of disk.
  disk_size: 400
  # Keep fixed.
  cloud: kubernetes
  ports: 8000
  image_id: docker:vllm/vllm-openai:latest

envs:
  # Specify the training config via `--env MODEL=collinear-ai/model-repo-name`
  MODEL: ""

setup: |
  conda deactivate
  python3 -c "import huggingface_hub; huggingface_hub.login('${HUGGINGFACE_TOKEN}')"

run: |
  conda deactivate
  python3 -u -m vllm.entrypoints.openai.api_server \
    --host 0.0.0.0 \
    --port 8000 \
    --tensor-parallel-size $SKYPILOT_NUM_GPUS_PER_NODE \
    --trust-remote-code \
    --model $MODEL

I verified that my EKS cluster in AWS launched a standard network load balancer that was internet facing.

[romilbhardwaj]

Thanks @fozziethebeat! Left a quick comment about doing the same for labels. LGTM to otherwise!

[romilbhardwaj]

This is awesome, thanks for the fix @fozziethebeat!