Skip to content

repro_deps_pinned does not recognize pyproject.toml or conda environment files #446

Description

@mlieberman85

repro_deps_pinned recognizes five loose manifests (handlers.py):

loose_manifests = {
    "requirements.txt": "pip requirements",
    "setup.py": "setuptools",
    "package.json": "npm package",
    "Cargo.toml": "Cargo manifest",
    "go.mod": "Go module",
}

pyproject.toml is not among them, and neither is conda's environment.yml / environment.yaml.

Observed

Auditing pypa/pip -- a repository whose only dependency declaration is pyproject.toml, with no lock file:

RE-01.01  WARN  Could not automatically verify - manual verification required

The repository declares dependencies and has no lock file. That is exactly the situation the control exists to detect, and the tool had every file it needed in front of it. Instead the operator is told the tool could not determine anything, which is indistinguishable from a control that genuinely lacks evidence.

Auditing CompVis/stable-diffusion, whose environment is declared in environment.yaml:

RE-01.01  FAIL  Dependency manifests found but no lock files: setup.py (setuptools)

The verdict happens to be right, but for the wrong file. The conda environment -- the thing that actually determines what gets installed -- was never looked at.

Why each gap matters

pyproject.toml has been the standard Python dependency declaration since PEP 621 (2020). setup.py is already on the list, and the two play the same role; including one and not the other is not a deliberate distinction. Modern Python projects that have moved off setup.py entirely are invisible to this control.

Conda is the dominant environment manager in computational biology, climate science, and adjacent fields. darnit-reproducibility exists for scientific reproducibility (#414, #415, #416), so this is not a peripheral ecosystem for this plugin -- it is arguably the central one. environment.yml is a loose manifest; conda-lock.yml is its lock file and is likewise unrecognized.

Also missing

Lock files not recognized: conda-lock.yml, pdm.lock, Pipfile (the manifest; Pipfile.lock is recognized), bun.lockb, pnpm-lock.yaml.

Note on scope

Adding a manifest to the list changes verdicts for repositories that currently return INCONCLUSIVE, which is a behavior change rather than a pure bug fix and should be called out in whatever PR does it. Feature 037 (#429) deliberately did not widen this list for that reason.

Content inspection for these new manifests is a separate question -- #429 covers only requirements.txt, and each ecosystem needs its own handling.

Found by running the reproducibility framework against nine real repositories.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions