[Core] Add env var to force using NvidiaGPUAcceleratorManager

[HollowMan6]

Why are these changes needed?

(I'm using MI250x)

PR #42434 leads to breaks of Ray using AMD GPUs starting from v2.10.0, as an alternative,
RAY_EXPERIMENTAL_NOSET_ROCR_VISIBLE_DEVICES is needed to fix:

1. vLLM: https://github.com/vllm-project/vllm/blob/v0.6.4.post1/Dockerfile.rocm#L131-L132
2. DeepSpeed: Add support when RAY_EXPERIMENTAL_NOSET_*_VISIBLE_DEVICES is set OpenRLHF/OpenRLHF#524 (comment)
3. One failure of Ray Collective Communication Lib example: https://github.com/ray-project/ray/blob/master/python/ray/util/collective/examples/nccl_allreduce_example.py

Traceback (most recent call last):
  File "nccl_allreduce_example.py", line 33, in <module>
    send = cp.ones((4,), dtype=cp.float32)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "cupy/cupy/_creation/basic.py", line 183, in ones
    a = cupy.ndarray(shape, dtype, order=order)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "cupy/_core/core.pyx", line 154, in cupy._core.core.ndarray.__new__
    x._init(*args, **kwargs)
  File "cupy/_core/core.pyx", line 241, in cupy._core.core._ndarray_base._init
    self.data = memory.alloc(self.size * itemsize)
  File "cupy/cuda/memory.pyx", line 875, in cupy.cuda.memory.alloc
    return get_allocator()(size)
  File "cupy/cuda/memory.pyx", line 1579, in cupy.cuda.memory.MemoryPool.malloc
    cpdef MemoryPointer malloc(self, size_t size):
  File "cupy/cuda/memory.pyx", line 1599, in cupy.cuda.memory.MemoryPool.malloc
    mp = <SingleDeviceMemoryPool>self._pools[device.get_device_id()]
  File "cupy/cuda/device.pyx", line 40, in cupy.cuda.device.get_device_id
    return runtime.getDevice()
  File "cupy_backends/cuda/api/runtime.pyx", line 202, in cupy_backends.cuda.api.runtime.getDevice
    check_status(status)
  File "cupy_backends/cuda/api/runtime.pyx", line 146, in cupy_backends.cuda.api.runtime.check_status
    raise CUDARuntimeError(status)
cupy_backends.cuda.api.runtime.CUDARuntimeError: hipErrorNoDevice: no ROCm-capable device is detected

However, for Ray Collective Communication Lib example, setting RAY_EXPERIMENTAL_NOSET_ROCR_VISIBLE_DEVICES is not enough, as further errors will still happen:

Traceback (most recent call last):
  File "nccl_allreduce_example.py", line 43, in <module>
    results = ray.get([w.compute.remote() for w in workers])
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "ray/_private/auto_init_hook.py", line 21, in auto_init_wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "ray/_private/client_mode_hook.py", line 103, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "ray/_private/worker.py", line 2755, in get
    values, debugger_breakpoint = worker.get_objects(object_refs, timeout=timeout)
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "ray/_private/worker.py", line 906, in get_objects
    raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(NcclError): ray::Worker.compute() (repr=<nccl_allreduce_example.Worker object at 0x1474faf38dd0>)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "nccl_allreduce_example.py", line 22, in compute
    collective.allreduce(self.send, "default")
  File "ray/util/collective/collective.py", line 273, in allreduce
    g.allreduce([tensor], opts)
  File "ray/util/collective/collective_group/nccl_collective_group.py", line 197, in allreduce
    self._collective(tensors, tensors, collective_fn)
  File "ray/util/collective/collective_group/nccl_collective_group.py", line 604, in _collective
    comms = self._get_nccl_collective_communicator(key, devices)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "ray/util/collective/collective_group/nccl_collective_group.py", line 451, in _get_nccl_collective_communicator
    nccl_util.groupEnd()
  File "cupy_backends/cuda/libs/nccl.pyx", line 210, in cupy_backends.cuda.libs.nccl.groupEnd
  File "cupy_backends/cuda/libs/nccl.pyx", line 243, in cupy_backends.cuda.libs.nccl.groupEnd
  File "cupy_backends/cuda/libs/nccl.pyx", line 129, in cupy_backends.cuda.libs.nccl.check_status
cupy_backends.cuda.libs.nccl.NcclError: NCCL_ERROR_INVALID_USAGE: invalid usage (run with NCCL_DEBUG=WARN for details)

So let’s use an env var to determine if we do want to use AMDGPUAcceleratorManager, so that users can still revert to the behavior we have in v2.9.3 (before #42434) to get everything running again on AMD GPUs.

Related issue number

Caused by #42434

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[stale]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

• If you'd like to keep this open, just leave any comment, and the stale label will be removed.

[HollowMan6]

This is necessary to get ray working on AMD GPUs, so I would hope to get this reviewed and merged soon.