A4 support for prod #412
A4 support for prod #412
Conversation
gcie
requested review from
Obliviour,
44past4,
sharabiani,
pawloch00,
BluValor and
RoshaniN
as code owners
|
This PR is blocked by #416 |
| - name: "cpu" | ||
| nominalQuota: 10000 | ||
| - name: "memory" | ||
| nominalQuota: 10000Gi |
There was a problem hiding this comment.
Not sure if we should remove these resources. We need them to schedule workloads requesting CPUs and Memory. I think maybe removing Gi from nominalQuota: 10000Gi will solve the error?
nominalQuota: 10000
| spec: | ||
| namespaceSelector: {} # match all. | ||
| resourceGroups: | ||
| - coveredResources: ["nvidia.com/gpu"] |
There was a problem hiding this comment.
same here, we need cpu and memory resources.
src/xpk/commands/kjob_common.py
Outdated
| @@ -39,6 +39,6 @@ def add_gpu_networking_annotations_to_command(args, cmd: str) -> str: | |||
|
|
|||
| if gpu_type == H100_MEGA_DEVICE_TYPE: | |||
| return add_tcpxo_annotations(args, cmd) | |||
| if gpu_type == H200_DEVICE_TYPE: | |||
| if gpu_type == H200_DEVICE_TYPE or gpu_type == B200_DEVICE_TYPE: | |||
There was a problem hiding this comment.
maybe if gpu_type in [H200_DEVICE_TYPE, B200_DEVICE_TYPE] ?
src/xpk/commands/kjob_common.py
Outdated
| @@ -39,6 +39,6 @@ def add_gpu_networking_annotations_to_command(args, cmd: str) -> str: | |||
|
|
|||
| if gpu_type == H100_MEGA_DEVICE_TYPE: | |||
| return add_tcpxo_annotations(args, cmd) | |||
| if gpu_type == H200_DEVICE_TYPE: | |||
| if gpu_type == H200_DEVICE_TYPE or gpu_type == B200_DEVICE_TYPE: | |||
| return add_rdma_annotations(args, cmd) | |||
There was a problem hiding this comment.
I would change this method to accept device_type: add_rdma_annotations(args, cmd, gpu_type)
because it's possible the subnetwork_names for A3Ultra and A4 will be the same; also mitigates the confusion that why we add a3ultra subnetworks for A4.
There was a problem hiding this comment.
I think this was a mistake by me, we should add A4 networks for A4...
I tweaked this part a bit to not overcomplicate things. Let me know what you think!
they are required and most likely not the cause of the error
Fixes / Features