Quick-lookup table of NCCL log patterns → code → root cause → fix. Used by the diagnostic script to map log lines to a remediation section in debugging-guide.md.
| Log pattern | Code | Root cause | Fix |
|---|---|---|---|
| Rendezvous / connection | |||
Timeout waiting for |
TIMEOUT_RENDEZVOUS |
Peers not joining init | SG self-ref, NetworkPolicy, MASTER_ADDR |
Connection refused |
CONN_REFUSED |
Rank-0 not listening | Fix MASTER_ADDR + headless service |
Address already in use |
PORT_CONFLICT |
Port 29500 bound | Change MASTER_PORT to 29501 |
NCCL WARN Connect to |
CONNECT_FAIL |
NCCL peer blocked | SG self-ref + NetworkPolicy |
network is unreachable |
NET_UNREACHABLE |
No route to MASTER_ADDR | DNS + VPC routing + SG |
Error in Store / DistStoreError |
STORE_ERR |
c10d rendezvous timeout | Fix network first |
RendezvousConnectionError |
RDZV_CONN_ERR |
Elastic rendezvous failed | MASTER_ADDR DNS + SG |
RendezvousTimeout |
RDZV_TIMEOUT |
Elastic rendezvous timed out | Peers not reachable |
Name or service not known |
DNS_FAIL |
DNS resolution failed | Create headless service |
getaddrinfo failed |
DNS_FAIL |
DNS resolution failed | CoreDNS + headless service |
| Runtime / AllReduce | |||
Watchdog timeout |
WATCHDOG_TIMEOUT |
AllReduce timed out | Extend PyTorch init_process_group(timeout=...); find straggler |
unhandled system error |
SYSTEM_ERROR |
GPU/EFA hardware | SSM: dmesg XID errors; reboot node |
unhandled cuda error |
CUDA_ERROR |
CUDA runtime error | GPU driver crash or hardware fault |
peer access is not supported |
P2P_FAIL |
GPU P2P blocked by ACS/IOMMU | Disable ACS; check IOMMU |
NCCL WARN Cuda failure |
CUDA_ERROR |
CUDA failure inside NCCL | GPU hardware or driver |
Call to ncclCommAbort |
NCCL_COMM_ABORT |
Communicator aborted | Check for straggler or hardware fault |
| EFA / libfabric | |||
fi_getinfo failed |
EFA_INIT_FAIL |
EFA not available | Fix EFA; use gloo on non-EFA |
NCCL_OFI_RDMA |
OFI_ERROR |
aws-ofi-nccl broken | Check plugin + EFA version |
Call to ibv_reg_mr failed |
RDMA_REG_FAIL |
memlock=0 blocks EFA RDMA | ulimit -l 8388608 |
NET/OFI Using TCP |
EFA_TCP_FALLBACK |
Fell back to TCP | Fix EFA device plugin + env |
Failed to load NCCL |
NCCL_LOAD_FAIL |
libnccl.so missing | Check LD_LIBRARY_PATH |
libnccl-net.so |
OFI_LOAD_FAIL |
OFI plugin missing | Install aws-ofi-nccl |
| OOM / resource limits | |||
OOMKilled |
OOM_KILL |
Pod out of memory | Reduce batch size; increase limits |
CUDA out of memory / cudaMalloc failed |
CUDA_OOM |
GPU VRAM exhausted | Reduce batch size, enable ZeRO |
failed to extend /dev/shm / Bus error |
SHM_FULL |
/dev/shm too small | emptyDir medium:Memory 10Gi |
ENOMEM |
ENOMEM |
Memory alloc/registration failure | Check memlock + GPU memory |
| Version / config | |||
NCCL function not found |
NCCL_VERSION_MISMATCH |
Mixed NCCL versions | Use identical container images |
Incompatible NCCL version |
NCCL_VERSION_MISMATCH |
Mixed NCCL versions | Use identical container images |
Could not find interface |
IFACE_NOT_FOUND |
Bad NCCL_SOCKET_IFNAME | Set ^lo,docker,efa,veth,virbr |
world_size mismatch |
WORLD_SIZE_MISMATCH |
WORLD_SIZE ≠ ranks | WORLD_SIZE = pods × GPUs/pod |
doesn't have NCCL built in |
NCCL_NOT_BUILT |
PyTorch without NCCL | Use AWS DLC image |
CUDA_VISIBLE_DEVICES |
CUDA_VIS_DEV |
GPUs hidden from training | Remove CUDA_VISIBLE_DEVICES |
invalid alignment |
CUDA_ALIGN_ERR |
CUDA alignment error | Check driver/NCCL version compat |
| Stale state / topology | |||
unlink shared memory |
SHM_STALE |
Stale /dev/shm/nccl-* files | Set RemoveIPC=no; clean up |
MNNVL topology |
MNNVL_TOPO_FAIL |
NCCL topology search failure | Try fixed memlock (e.g. ulimit -l 8388608) — field-observed workaround, not NCCL-documented; see debugging-guide.md § 17 |