[BUG] Source installation with PyTorch + CUDA backend fails due to undocumented build-system dependencies and incomplete CUDA path detection

[qchempku2017]

Bug summary

I am trying to build DeepMD-kit from source with the PyTorch backend and CUDA enabled. The installation process is currently fragile and significantly under-documented.

The build fails at multiple stages unless several environment variables and Python build dependencies are manually set or installed. These requirements are not clearly documented in the source installation instructions, and some of them appear to be required before the custom PEP 517 build backend can even be imported.

This creates a poor installation experience for users who need to build DeepMD-kit from source for PyTorch/CUDA workflows.

DeePMD-kit Version

Unreleased main branch at 2026/7/4 in the morning

Backend and its version

pytorch 2.11.0 cuda 128 wheel

How did you download the software?

Built from source

Input Files, Running Commands, Error Log, etc.

-DeepMD-kit source tree: 3.2.0b1.dev94+gd279173a1
-Python: 3.12.3

• pip: 26.1.2
• PyTorch: 2.11.0+cu128
• PyTorch CUDA: 12.8
• CUDA Toolkit: 12.8.2, installed via run file.
• CUDA path: /opt/cuda_128
• NVIDIA driver: 580
• GPU: NVIDIA Tesla T4
• OS/container: Bohrium, Ubuntu 24.04 base image, root inside container

Problem 1: CUDA path detection is incomplete for PyTorch/Caffe2

With only CUDAToolkit_ROOT set, DeepMD-kit’s own CMake configuration appears to find CUDA:

-- Found CUDAToolkit: /opt/cuda_128/targets/x86_64-linux/include (found version "12.8.93")
-- Found CUDA in /opt/cuda_128/bin, build nv GPU support

However, PyTorch/Caffe2 then fails:

CUDA_TOOLKIT_ROOT_DIR not found or specified
-- Could NOT find CUDA (missing: CUDA_TOOLKIT_ROOT_DIR CUDA_NVCC_EXECUTABLE CUDA_INCLUDE_DIRS CUDA_CUDART_LIBRARY)

CMake Error at .../torch/share/cmake/Caffe2/Caffe2Config.cmake:90 (message):
  Your installed Caffe2 version uses CUDA but I cannot find the CUDA
  libraries. Please set the proper CUDA prefixes and / or install CUDA.

This means that setting only CUDAToolkit_ROOT is insufficient. Users also need to manually set the older CUDA variables expected indirectly by PyTorch/Caffe2:

export CUDA_HOME=/opt/cuda_128
export CUDA_ROOT=/opt/cuda_128
export CUDA_PATH=/opt/cuda_128
export CUDA_TOOLKIT_ROOT_DIR=/opt/cuda_128
export CUDA_NVCC_EXECUTABLE=/opt/cuda_128/bin/nvcc
export CUDA_INCLUDE_DIRS=/opt/cuda_128/include
export CUDA_CUDART_LIBRARY=/opt/cuda_128/lib64/libcudart.so
export PATH=/opt/cuda_128/bin:$PATH
export LD_LIBRARY_PATH=/opt/cuda_128/lib64:${LD_LIBRARY_PATH:-}

It would be useful if the documentation explicitly stated this, or if the build system propagated CUDAToolkit_ROOT to the legacy variables when building with PyTorch + CUDA.

Problem 2: Custom PEP 517 backend cannot be imported unless source root is added to PYTHONPATH

After trying to work around CUDA detection and using source installation, the build fails with:

BackendUnavailable: Cannot import 'backend.dp_backend'

This appears to require manually setting:

export PYTHONPATH="$(pwd):${PYTHONPATH:-}"

Otherwise, pip cannot import the custom build backend from the source tree.

If this is expected, it should be documented. If not, the pyproject.toml configuration may need to be adjusted, for example with the appropriate backend-path.

Problem 3: Build backend has undeclared Python dependencies

After setting PYTHONPATH, importing the backend fails again because required build-time Python packages are missing.

First failure:

File "/opt/deepmd-kit/backend/dp_backend.py", line 6, in <module>
  from scikit_build_core import build as _orig
ModuleNotFoundError: No module named 'scikit_build_core'

After manually installing scikit-build-core, another failure occurs:

File "/opt/deepmd-kit/backend/utils.py", line 7, in <module>
  from dependency_groups import (
ModuleNotFoundError: No module named 'dependency_groups'

This suggests that the custom build backend imports packages that are not guaranteed to exist before the backend itself is loaded.

The user currently has to manually install build dependencies such as:

python -m pip install -U \
  pip setuptools wheel \
  scikit-build-core cmake ninja build \
  pyproject-metadata pathspec dependency-groups

These dependencies should either be declared in [build-system].requires or documented clearly before source installation instructions.

Current workaround

The following workaround gets further than the documented/simple installation path:

#!/usr/bin/env bash
set -euo pipefail

python -m pip install -U \
  pip setuptools wheel \
  scikit-build-core cmake ninja build \
  pyproject-metadata pathspec dependency-groups

export DP_ENABLE_PYTORCH=1
export DP_ENABLE_TENSORFLOW=0
export DP_VARIANT=cuda

CUDA_DIR="/opt/cuda_128"

export CUDA_HOME="${CUDA_DIR}"
export CUDA_ROOT="${CUDA_DIR}"
export CUDA_PATH="${CUDA_DIR}"
export CUDAToolkit_ROOT="${CUDA_DIR}"
export CUDA_TOOLKIT_ROOT_DIR="${CUDA_DIR}"
export CUDA_NVCC_EXECUTABLE="${CUDA_DIR}/bin/nvcc"
export CUDA_INCLUDE_DIRS="${CUDA_DIR}/include"
export CUDA_CUDART_LIBRARY="${CUDA_DIR}/lib64/libcudart.so"

export PATH="${CUDA_DIR}/bin:${PATH}"
export LD_LIBRARY_PATH="${CUDA_DIR}/lib64:${LD_LIBRARY_PATH:-}"
export PYTHONPATH="$(pwd):${PYTHONPATH:-}"

export CMAKE_ARGS="-DDP_ENABLE_PYTORCH=ON -DDP_ENABLE_TENSORFLOW=OFF -DDP_VARIANT=cuda -DCUDAToolkit_ROOT=${CUDA_DIR} -DCUDA_TOOLKIT_ROOT_DIR=${CUDA_DIR} -DCUDA_NVCC_EXECUTABLE=${CUDA_DIR}/bin/nvcc -DCUDA_INCLUDE_DIRS=${CUDA_DIR}/include -DCUDA_CUDART_LIBRARY=${CUDA_DIR}/lib64/libcudart.so"

pip install . -v

This is a large amount of undocumented manual configuration for what appears to be a supported PyTorch + CUDA source build.

Expected behavior

For a source build with PyTorch + CUDA, the documentation should clearly specify:

Required Python build dependencies before invoking pip install ..
Required environment variables for CUDA Toolkit discovery.
Whether CUDAToolkit_ROOT alone is sufficient, or whether legacy variables such as CUDA_TOOLKIT_ROOT_DIR, CUDA_NVCC_EXECUTABLE, CUDA_INCLUDE_DIRS, and CUDA_CUDART_LIBRARY must also be set.
Whether PYTHONPATH=$(pwd) is required for the custom build backend, or whether pyproject.toml should handle this automatically.
Whether Python 3.12 is officially supported for source builds with PyTorch + CUDA.
Actual behavior

A straightforward source installation fails repeatedly at different stages:

PyTorch/Caffe2 cannot find CUDA unless several legacy CUDA variables are manually set.
pip cannot import backend.dp_backend unless the source root is added to PYTHONPATH.
The custom backend then fails due to missing build dependencies such as scikit_build_core and dependency_groups.
Request

Please consider improving the source installation documentation and/or build configuration for the PyTorch + CUDA backend.

In particular, it would be very helpful to:

Add a complete documented installation recipe for source builds with DP_ENABLE_PYTORCH=1, DP_ENABLE_TENSORFLOW=0, and DP_VARIANT=cuda.
Add all required build-backend dependencies to [build-system].requires, if missing.
Ensure the custom backend can be imported by pip without requiring users to manually set PYTHONPATH.
Clarify CUDA Toolkit detection behavior for PyTorch/Caffe2 and document the required variables.
Provide a minimal tested Dockerfile or shell script for PyTorch + CUDA source builds.

This is a serious usability issue for users who need to build DeepMD-kit from source in GPU-enabled PyTorch environments.

Steps to Reproduce

Intended configurations:

export DP_ENABLE_PYTORCH=1
export DP_ENABLE_TENSORFLOW=0 
export DP_VARIANT=cuda 
export CUDAToolkit_ROOT=/opt/cuda_128 

pip install .

Further Information, Files, and Links

No response

[dosubot]

Thanks for the detailed report. I looked into the current state of the build system and can clarify some of the issues you encountered:

Problem 2 & 3 (build backend import failures) — likely a pip version or invocation issue, not a code bug:

The pyproject.toml on the main branch already has:

[build-system]
requires = [
    "scikit-build-core>=0.5,!=0.6.0,<0.12",
    "packaging",
    'tomli >= 1.1.0 ; python_version < "3.11"',
    "dependency_groups",
]
build-backend = "backend.dp_backend"
backend-path = ["."]

Both scikit-build-core and dependency_groups are declared in [build-system].requires, and backend-path = ["."] is set — meaning pip should be able to discover the backend without manually setting PYTHONPATH [1].

If you're seeing BackendUnavailable: Cannot import 'backend.dp_backend' and ModuleNotFoundError: No module named 'scikit_build_core', the most likely causes are:

• You're running pip install . from a directory that is not the source root (where pyproject.toml lives).
• Your pip version is too old to properly support backend-path. pip 23.1+ should handle this correctly; you mentioned pip 26.1.2, so that should be fine.
• There's a stale build artifact or virtual environment issue causing pip's build isolation to malfunction. Try pip install . -v --no-build-isolation after manually installing the build deps, or conversely, make sure build isolation is working (don't set PYTHONPATH manually, as that can interfere with isolated builds).

tl;dr: Setting PYTHONPATH manually should not be required. If pip's build isolation is working, it will install the [build-system].requires packages into an isolated environment and resolve backend-path relative to the source directory automatically.

Problem 1 (CUDA path detection for PyTorch/Caffe2):

This is a genuine pain point. DeePMD-kit's own CMake uses the modern find_package(CUDAToolkit) which respects CUDAToolkit_ROOT. However, PyTorch's bundled Caffe2 CMake config uses the older/legacy FindCUDA module which looks for CUDA_TOOLKIT_ROOT_DIR and related variables [2]. This is ultimately a PyTorch packaging issue — their CMake config depends on legacy CUDA discovery paths.

The documentation does mention CUDA_ROOT as an environment variable for pip builds [2], but it doesn't explicitly call out the additional legacy variables needed by PyTorch's Caffe2. For a non-standard CUDA install path (like /opt/cuda_128 rather than /usr/local/cuda), the simplest reliable workaround is:

export CUDA_HOME=/opt/cuda_128
export PATH=/opt/cuda_128/bin:$PATH
export LD_LIBRARY_PATH=/opt/cuda_128/lib64:${LD_LIBRARY_PATH:-}

Setting CUDA_HOME and having nvcc on PATH is typically sufficient for PyTorch's Caffe2 detection — you shouldn't need all the individual variables (CUDA_NVCC_EXECUTABLE, CUDA_INCLUDE_DIRS, CUDA_CUDART_LIBRARY, etc.) unless something else is wrong.

Summary:

• Problems 2 & 3 appear to be environment/invocation issues rather than bugs in pyproject.toml — the dependencies and backend-path are correctly declared.
• Problem 1 (CUDA legacy variable detection for PyTorch/Caffe2) is a documentation gap worth addressing. A note in the source install docs about setting CUDA_HOME and ensuring nvcc is on PATH for PyTorch+CUDA builds would be helpful.

If you can confirm that running pip install . -v from the source root (without manual PYTHONPATH) still fails with the backend import error, that would indicate a genuine bug worth investigating further — it would help to see the full pip output with -v in that case.

To reply, just mention @dosu.

---

Share context across your team and agents. Try Dosu.

[njzjz-bot]

Some parts of this report do not match the current main branch.

For the PEP 517 backend import / missing Python build dependencies:

[build-system]
requires = [
    "scikit-build-core>=0.11,!=0.6.0,<0.13",
    "packaging",
    'tomli >= 1.1.0 ; python_version < "3.11"',
    "dependency_groups",
]
build-backend = "backend.dp_backend"
backend-path = ["."]

So scikit-build-core and dependency_groups are already declared in [build-system].requires, and backend-path = ["."] should make backend.dp_backend importable without manually setting PYTHONPATH=$(pwd).

If pip install . reports:

BackendUnavailable: Cannot import 'backend.dp_backend'
ModuleNotFoundError: No module named 'scikit_build_core'
ModuleNotFoundError: No module named 'dependency_groups'

then the first thing to check is whether pip is actually using the current source tree's pyproject.toml with build isolation enabled. Possible explanations include running from the wrong directory, using a modified/stale source tree, disabling or breaking build isolation, or otherwise invoking pip in a way that bypasses the normal PEP 517 dependency installation. Those errors do not by themselves show that the build-system dependencies or backend-path are missing from DeepMD-kit.

The CUDA part is more plausible as a documentation/usability issue. DeepMD-kit may find CUDA via modern CUDAToolkit_ROOT, while PyTorch/Caffe2's CMake config can still go through legacy CUDA discovery paths. For a non-standard CUDA prefix, documenting CUDA_HOME/PATH/LD_LIBRARY_PATH more explicitly for PyTorch+CUDA source builds would be useful. But that is separate from the PEP 517 backend-dependency claims above.

In short: the CUDA discovery complaint may be worth improving in docs, but the claims that the custom build backend lacks scikit-build-core, lacks dependency_groups, or requires manual PYTHONPATH do not match the current pyproject.toml.

Authored by OpenClaw (model: custom-chat-jinzhezeng-group/gpt-5.5)

[qchempku2017]

Thanks for the clarification. I agree that the PEP 517 / build-system dependency part of my original report should be corrected.

After checking the current pyproject.toml, I see that:

[build-system]
requires = [
    "scikit-build-core>=0.11,!=0.6.0,<0.13",
    "packaging",
    'tomli >= 1.1.0 ; python_version < "3.11"',
    "dependency_groups",
]
build-backend = "backend.dp_backend"
backend-path = ["."]

So the claims that scikit-build-core and dependency_groups are not declared, or that backend-path is missing, do not apply to the current main branch.

Those errors likely came from my troubleshooting attempts involving --no-build-isolation, manual backend import checks, or a stale/modified local build state. They should not be used as evidence that the current pyproject.toml lacks the required build-system metadata.

I would like to revise the issue scope accordingly.

The remaining problem is not that the PEP 517 build dependencies are missing. The remaining problem is installation-path discoverability and runtime usability for PyTorch + CUDA + DPA4/SeZM workflows.

In particular:

1. For PyTorch runtime use, the intended command appears to be pip install '.[torch]', not plain pip install .. This distinction is easy to miss, especially when DP_ENABLE_PYTORCH=1 is already set during source build.

2. DP_ENABLE_PYTORCH=1 is a build-time option, while .[torch] is a runtime dependency extra. The documentation should make this distinction explicit.

3. The dynamically generated torch extra may correctly include e3nn, vesin[torch], and nvalchemi-toolkit-ops, but if users install with plain pip install ., the PyTorch backend can still be built while the runtime dependencies are absent. The resulting failures occur later as runtime import or neighbor-list errors.

4. CUDA discovery remains a separate documentation/usability issue. DeepMD-kit may find CUDA through CUDAToolkit_ROOT, while PyTorch/Caffe2 or PyTorch Inductor may still require CUDA_HOME, PATH, LD_LIBRARY_PATH, or legacy CUDA variables in non-standard CUDA installations.

5. A dp doctor / dp --check-env command or an early runtime preflight check would help catch these problems before a user reaches a deep stack trace inside PyTorch, SeZM, or neighbor-list construction.

So I agree that the PEP 517 dependency claim should be withdrawn. The issue I still want to raise is that the user-facing installation workflow for PyTorch + CUDA + DPA4/SeZM stated in documentation is not sufficiently explicit or self-diagnostic.