4x Arc Pro B60: multi-GPU DDP training fails with UR_RESULT_ERROR_OUT_OF_RESOURCES at first kernel (large models only; single-GPU and small-model DDP both work)

[apt-team-018]

One-line summary

On a 4x Intel Arc Pro B60 server, any single PyTorch job that spans more than one GPU
(torch.distributed / DDP) crashes at the first real-model compute kernel with
UR backend failed. UR backend returns:40 (UR_RESULT_ERROR_OUT_OF_RESOURCES), while
each GPU individually has ~12 GB of its 24 GB free. Single-GPU training and small
synthetic multi-GPU collectives both work perfectly. The failure is independent of
PyTorch version, oneCCL version, kernel, IOMMU mode, and every documented workaround.

Hardware

• 4x Intel Arc Pro B60 24GB (Battlemage BMG-G21), no XeLink, P2P over PCIe 5.0 x8
• Intel Xeon 698X (86 cores / 172 threads), 60 GB RAM
• Server: single-node, bare metal

Software (all combinations tried — same failure in every one)

• OS: Ubuntu 25.10
• Kernels tested: 6.17.0-40-generic, and mainline 7.1.3 (Intel's validation kernel)
• GPU stack: intel compute-runtime 26.22.38646.6, Level-Zero 1.28.6, GuC firmware 70.49.4
• PyTorch XPU: 2.9.1+xpu (oneCCL 2021.16.1), 2.12.1+xpu (oneCCL 2021.17.2),
  2.13.0+xpu (oneCCL 2022.0.0 — the Arc-B-series scale-up release)
• HuggingFace: accelerate 1.13.0, transformers 5.x, peft 0.19.1
• Attention: PyTorch SDPA (flash-attn unavailable on XPU)

Reproduction

Minimal: 4-rank torchrun/accelerate launch, load any multi-billion-parameter model
in bf16 (one full copy per rank via device_map), wrap in DistributedDataParallel, run one
training step at sequence length 4096. Fails at the first real forward/backward kernel.

Observed memory at the crash (instrumented in the trainer's compute_loss):
pre-first-batch alloc=9.76GB reserved=11.42GB per rank — i.e. ~12 GB used of 24 GB.
It is NOT an out-of-memory-by-size condition.

What works vs what fails (the discriminating evidence)

WORKS:

• Single-process training on any one GPU (full run, 5h17m, clean).
• 4 concurrent INDEPENDENT single-GPU jobs (one process per GPU, no torch.distributed).
• 4-rank synthetic DDP with a SMALL model (few hundred tensors): all-reduce + broadcast
  of 0.5-2 GiB buffers, 200 iterations, zero errors.
  FAILS:
• Any single DDP job with a real (large, many-kernel) model across 2, 3, or 4 ranks.
  The DDP weight-broadcast completes; the crash is at the first real compute kernel after.

Configurations tried, all producing the identical UR_RESULT_ERROR_OUT_OF_RESOURCES

1. Platform: intel_iommu=off; iommu=pt (passthrough); (both).
2. Runtime adapter: Level-Zero v1 and v2 (SYCL_UR_USE_LEVEL_ZERO_V2=0).
3. Collectives transport: OFI/libfabric (shm) and Intel MPI (CCL_ATL_TRANSPORT=mpi).
4. Copy path: CCL_TOPO_P2P_ACCESS=0 (host-staged), CCL_ZE_COPY_ENGINE=link/none,
   UR_L0_V2_FORCE_DISABLE_COPY_OFFLOAD=1, NEO EnableBlitterOperationsSupport=0.
5. Allocator: field-proven USM pool set (EnableHostUsmAllocationPool=0,
   EnableUsmPoolLazyInit=1, RenderCompressedBuffersEnabled=0, CCL_ZE_TMP_BUF_SIZE,
   UR_L0_USM_ALLOCATOR pool string).
6. OS limits: file-descriptor limit raised 1024 -> 65536.
7. oneCCL algo: CCL_ALLREDUCE=ring; CCL_ENABLE_SYCL_KERNELS=0;
   CCL_TOPO_FABRIC_VERTEX_CONNECTION_CHECK=0; CCL_ZE_IPC_EXCHANGE=sockets/drmfd/pidfd.
8. PyTorch stack: 2.9.1, 2.12.1, 2.13.0 (with their bundled oneCCL 2021.16/2021.17/2022.0).
9. Kernel: 6.17 and mainline 7.1.3.

Every reboot was clean-state (fresh boot before each trial) to avoid accumulated xe
driver state from prior crashes.

Note on a secondary symptom (not the primary bug)

At larger sequence lengths, before the platform fixes, cross-GPU copies also triggered
"GT0: Engine reset: engine_class=bcs" blitter resets. Those specific resets were fixed by
intel_iommu=off + GuC 70.49.4 + compute-runtime 26.22 (a 200-iteration synthetic P2P
stress test then passes cleanly). The UR_RESULT_ERROR_OUT_OF_RESOURCES failure above is a
SEPARATE, still-open issue that survives all of those fixes.

Suspected relation to open trackers

Matches the profile of #948 (GSD-13010) — engine-reset / wedged
L0 context under sustained multi-GPU load on BMG — and #935 (GSD-12874, BMG cross-root-port
P2P). This 4-GPU topology crosses PCIe root ports.

What we need

Guidance on the correct multi-GPU configuration for 4x Arc Pro B60 for a single DDP/FSDP
training job, or confirmation this is a known driver limitation with an ETA. We can provide
/sys/class/drm/cardN/device/devcoredump/data, full dmesg, and the minimal repro on request.