Machine intelligence for your business

Problem

After moving to a newer vLLM image, the inference pod enters a crash loop. It fails during startup, while NCCL is setting up communication between the GPUs — so the model never finishes loading and the server never accepts a request.

Every tensor-parallel worker dies with the same error. The failure of the engine core then brings down the API server process, so the pod exits rather than degrading.

[<timestamp>] NCCL version 2.27.5+cuda12.9, cudaDriverVersion 13000
[<timestamp>] init.cc:426 NCCL WARN Cuda failure 'operation not supported'
[<timestamp>] ncclCommInitRank failed: unhandled cuda error
[<timestamp>] RuntimeError: NCCL error: unhandled cuda error
[<timestamp>] EngineCore failed to start
[<timestamp>] APIServer exiting

The block above is abridged and reformatted — the exact wording varies with the vLLM and NCCL versions in your image. The line that identifies this issue is Cuda failure 'operation not supported' raised at NCCL initialisation (ncclCommInitRank).

Affected versions

Cause

A mediated vGPU does not expose the same memory and peer-to-peer capabilities as a physical GPU. By default, the unified memory and peer addressing that NCCL relies on are not presented to the guest at all.

Running nvidia-smi -q inside the container on an affected host shows that plumbing stubbed out rather than merely restricted:

Addressing Mode : None
Peer Type       : N/A
Atomic Caps     : N/A

Newer NCCL releases require that capability when they build a communicator. When the driver reports it as unavailable, NCCL raises Cuda failure 'operation not supported' and aborts, instead of falling back to a transport that would work on this platform. Older NCCL releases did not take that path, which is why an older image continues to run on the identical host and driver.

The precise code path inside NCCL has not been isolated, and this article does not claim one. What is established is narrower and sufficient to act on: the capability is missing from the guest, the newer NCCL requires it, and supplying it resolves the crash.

What this is not

Three plausible explanations have been tested and ruled out. They are listed because each one costs time to chase:

Workaround

Solution

pciPassthru0.cfg.enable_uvm = 1
pciPassthru1.cfg.enable_uvm = 1

Where this is documented

The hypervisor vendor documents this parameter for deep-learning virtual machine classes — for example VMs running retrieval-augmented generation workloads or an inference server. It is not published as a blanket requirement for every AI workload, so you may not find it by searching for your own use case, and the failure it prevents is not named on that page.

Be aware of a wrinkle in the vendor's own text: it instructs you to set pciPassthru<vgpu-id>.cfg.enable_uvm, then shows an example written as pciPassthru0.cfg.parameter=1. The first form is the correct one. Use enable_uvm as written above.

How to verify

1. Confirm this is a mediated vGPU

Run inside the container, or in a pod on the affected node. If the mode is not a virtualised one, this article does not apply:

nvidia-smi -q | grep -i "Virtualization Mode"

2. Check the peer capability, before and after

This is the quickest confirmation that the change took effect. Capture it before applying the parameter and again after the VM is back up:

nvidia-smi -q | grep -Ei "Addressing Mode|Peer Type|Atomic Caps"

Values of None and N/A on an affected host are the signature described under Cause.

3. Test NCCL directly, before launching vLLM

A minimal collective is far faster to iterate on than a full model load, and it isolates the failure to NCCL rather than to vLLM. Save this inside the container as nccl_check.py:

import os
import torch
import torch.distributed as dist

dist.init_process_group("nccl")
torch.cuda.set_device(int(os.environ["LOCAL_RANK"]))
t = torch.ones(1024, device="cuda")
dist.all_reduce(t)
torch.cuda.synchronize()
print("rank", dist.get_rank(), "ok", t[0].item())

Run it across two GPUs:

torchrun --nproc_per_node=2 nccl_check.py

Before the fix this fails with the same Cuda failure 'operation not supported'. After the fix each rank prints ok 2.0.

4. Start the workload

1. Redeploy vLLM on the newer image with the tensor-parallel setting you originally wanted.
2. Confirm the pod reaches a ready state rather than crash-looping, and that the log shows the API server started.
3. Send a real inference request and confirm a response. A pod that starts is necessary but not sufficient — this failure previously killed the server before it ever served one.

If this doesn't resolve it

Re-run the workload with NCCL_DEBUG=INFO set in the container. This produces the transport selection and initialisation detail needed to tell a remaining capability problem apart from an unrelated failure that shares the wording.

Then open a support ticket including:

• The vLLM and NCCL versions in the image, and the versions of the last image that worked
• Hypervisor and version, the vGPU profile in use, and the number of vGPUs on the VM
• GPU driver and CUDA versions, in the guest and on the hypervisor host
• Output of nvidia-smi, nvidia-smi -q and nvidia-smi topo -m from inside the container
• Confirmation that enable_uvm is set for every vGPU on the VM, and that the VM was power-cycled afterwards
• The NCCL_DEBUG=INFO log from a failing start
• Whether the minimal NCCL test in step 3 passes or fails

Related

• Broadcom — Configure vGPU-based VM classes for AI workloads — the vendor reference for the UVM parameter and the vMotion consequence.
• NVIDIA NCCL issue #2101 — upstream discussion of this failure on vGPU, including what did and did not work.
• vLLM issue #19188 — the same error reported against vLLM on a virtualised multi-GPU host.
• vLLM troubleshooting guide — general guidance on diagnosing NCCL and multi-GPU startup problems.
• NVIDIA developer forums — the identical error in a different NVIDIA workload on vGPU.