feat: add optional Slurm package

[andreatnvidia]

📋 Summary

Adds the optional data-designer-slurm distribution and exposes it through the exact-version data-designer[slurm] extra. This establishes package isolation and shared-namespace behavior before Slurm runtime features are added.

🔗 Related Issue

Closes #852

Part of #850

🔄 Changes

• Add data-designer-slurm as a publishable workspace package under data_designer.slurm.
• Generate exact same-version dependencies in both directions while keeping the leaf optional for base installs.
• Extend dependency auditing to evaluate explicitly selected static and dynamic extras.
• Add clean built-wheel checks for base-only, extra-selected, direct-leaf, missing-counterpart, metadata, and namespace behavior.
• Wire the package into development, test, lint, build, and CI workflows.

🧪 Testing

• .venv/bin/pytest packages/data-designer/tests packages/data-designer-slurm/tests - 1121 passed, 1 skipped
• make test-slurm-wheel-install
• .venv/bin/ruff check --fix .
• .venv/bin/ruff format .
• .venv/bin/python scripts/update_license_headers.py --check
• uv lock --check
• Unit tests added/updated
• Built-wheel installation tests added

✅ Checklist

• Follows commit message conventions
• Commits are signed off (DCO)
• Architecture docs updated

[greptile-apps]

Greptile Summary

The PR adds the optional data-designer-slurm workspace distribution and exposes it through the exact-version data-designer[slurm] extra.

• Adds package metadata and shared PEP 420 namespace coverage.
• Extends dependency auditing to selected static and dynamic extras.
• Adds clean-wheel installation checks and integrates Slurm into development, CI, build, and publication workflows.

Confidence Score: 4/5

The PR is not yet safe to merge because a failed Slurm upload can leave the already-published main package’s exact-version Slurm extra unresolvable.

The release script uploads data-designer before data-designer-slurm and aborts on the first failed upload, so the previously reported partial-release state remains reachable.

Files Needing Attention: scripts/publish.sh

Important Files Changed

Filename	Overview
packages/data-designer/pyproject.toml	Adds the dynamically generated exact-version Slurm extra while preserving the base installation as Slurm-free.
packages/data-designer-slurm/pyproject.toml	Defines the optional Slurm distribution with an exact-version dependency on the main package.
scripts/audit_package_dependencies.py	Extends dependency auditing to explicitly selected extras and prevents workspace cycles from inflating transitive guarantees.
scripts/test_slurm_package_install.py	Builds clean wheels and verifies metadata, base-only, extra-selected, direct-leaf, missing-counterpart, and namespace behavior.
scripts/publish.sh	Adds the Slurm artifact to publication, but the previously reported partial-release failure remains possible because uploads are sequential.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  User["pip install data-designer[slurm]"] --> Base["data-designer X"]
  Base -->|optional exact-version dependency| Slurm["data-designer-slurm X"]
  Slurm -->|exact-version dependency| Base
  Base --> Namespace["data_designer namespace"]
  Slurm --> Namespace

Loading

_{Reviews (3): Last reviewed commit: "fix: scope workspace dependency extras" | Re-trigger Greptile}

[greptile-apps]

Slurm release artifact is omitted

When a maintainer releases version X through make publish, this extra requires data-designer-slurm==X, but the publishing script uploads only the original three distributions, causing pip install data-designer[slurm]==X from PyPI to fail dependency resolution.

Knowledge Base Used: Testing and Tooling

Prompt To Fix With AI

This is a comment left during a code review.
Path: packages/data-designer/pyproject.toml
Line: 59

Comment:
**Slurm release artifact is omitted**

When a maintainer releases version X through `make publish`, this extra requires `data-designer-slurm==X`, but the publishing script uploads only the original three distributions, causing `pip install data-designer[slurm]==X` from PyPI to fail dependency resolution.

**Knowledge Base Used:** [Testing and Tooling](https://app.greptile.com/nvidia-public-github/-/custom-context/knowledge-base/nvidia-nemo/datadesigner/-/docs/testing-and-tooling.md)

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[github-actions]

Thanks for putting this together, @andreatnvidia — this is a clean, well-scoped foundation for the Slurm work.

Summary

This PR adds the optional data-designer-slurm workspace package (namespace data_designer.slurm) and exposes it via the exact-version data-designer[slurm] extra, establishing package isolation and shared-namespace behavior ahead of any Slurm runtime features. It wires the new package into the build/test/lint/CI tooling and extends the dependency auditor to evaluate selected extras. The implementation matches the stated intent: the package is intentionally a near-empty shell, with the substance being packaging correctness, which is validated by an impressively thorough built-wheel test.

Findings

Warnings — Worth addressing

scripts/publish.sh:31 — New package is not wired into the publish pipeline

• What: PACKAGE_DIRS still lists only data-designer-config, data-designer-engine, and data-designer. data-designer-slurm is added to install, build, test, lint, and CI, but not to the publisher. The post-publish summary URLs (publish.sh:544-546, 564-566) are also hardcoded to the three existing packages.
• Why: The base data-designer wheel now declares data-designer-slurm=={{ version }} under the slurm extra, and the README advertises pip install "data-designer[slurm]". Once a release goes out, data-designer[slurm] will resolve to a data-designer-slurm version that was never uploaded to PyPI, so the extra install will fail for real users. The excellent missing-counterpart case in test_slurm_package_install.py:799-819 actually demonstrates exactly this failure mode.
• Suggestion: Add packages/data-designer-slurm to PACKAGE_DIRS (and a corresponding project URL in both summary blocks). If publishing is intentionally deferred to a later PR in the Epic: Slurm batch execution v1 #850 series, a one-line note in the PR description or a # TODO(#850): publish slurm marker would make that explicit so it isn't forgotten at release time.

Suggestions — Take it or leave it

Makefile:209 — test-slurm uses .venv/bin/pytest while sibling targets use uv run --group dev

• What: test-config/test-engine/test-interface all invoke uv run --group dev pytest ..., which auto-syncs the environment. The new test-slurm (and test-slurm-wheel-install, verify-imports, format-slurm, etc.) shell out to .venv/bin/... directly.
• Why: make test now depends on test-slurm, so running make test on a machine without a populated .venv would fail on the Slurm leg where the other legs would have self-provisioned. It's a minor consistency wrinkle rather than a bug (CI runs make install-dev first).
• Suggestion: Consider uv run --group dev pytest $(SLURM_TESTS) for test-slurm to match the neighbors, unless the .venv/bin form is a deliberate choice worth a short comment.

Makefile:220 — test-isolated doesn't cover the Slurm package

• What: The isolated-test suite (test-isolated → config/engine/interface) verifies dependency boundaries per package but omits slurm.
• Why: The new package's whole reason to exist is its packaging/dependency boundary, and the built-wheel test already covers isolation well — so this is low priority. But an isolated target would keep the pattern uniform if the package later gains code.
• Suggestion: Optional — add test-slurm-isolated if/when the package has importable logic worth isolating. Fine to leave as-is for now given the wheel test.

What Looks Good

• scripts/test_slurm_package_install.py is genuinely excellent — it validates version pinning in both directions, the extra marker, Provides-Extra, the absence of a top-level __init__.py in the leaf wheel, and all four install permutations (base-only, extra, direct-leaf, missing-counterpart). That last case proving the install fails without the counterpart is exactly the right kind of negative test.
• The audit extension (audit_package_dependencies.py) is a clean, backward-compatible addition: selected_extras defaults to None, the PACKAGE:EXTRA CLI parsing validates its format, and both static and dynamic optional-dependency sources are handled symmetrically. The accompanying test_includes_selected_dynamic_optional_dependencies test exercises the real dynamic-hook path.
• Docs and invariants stay honest: AGENTS.md is updated to four packages and, nicely, reframes the layering as "packaging-only reverse edge; no code may import against this flow" — which correctly captures the subtlety that the [slurm] extra points "backward" without violating import direction.

Structural Impact

(graphify, 3.5s)

Risk: LOW (localized change)

• 5 Python files, 0 AST entities, 0/78 clusters

• Note: changes touch unknown package(s) (data-designer-slurm); analysis may be incomplete - update _PACKAGE_SUBDIRS in structural_impact.py.

Verdict

Needs changes — The publish-pipeline gap (publish.sh) is the one item worth resolving (or explicitly deferring in the PR description) before this ships, since the advertised data-designer[slurm] extra would otherwise break at install time once released. The two suggestions are optional polish.

---

This review was generated by an AI assistant.

[greptile-apps]

Partial release breaks Slurm extra

When the data-designer upload succeeds but the subsequent data-designer-slurm upload fails, version X remains published with an exact dependency on an unavailable data-designer-slurm==X, causing data-designer[slurm]==X installations to fail until the missing artifact is manually uploaded.

Knowledge Base Used: Testing and Tooling

Prompt To Fix With AI

This is a comment left during a code review.
Path: scripts/publish.sh
Line: 35

Comment:
**Partial release breaks Slurm extra**

When the `data-designer` upload succeeds but the subsequent `data-designer-slurm` upload fails, version X remains published with an exact dependency on an unavailable `data-designer-slurm==X`, causing `data-designer[slurm]==X` installations to fail until the missing artifact is manually uploaded.

**Knowledge Base Used:** [Testing and Tooling](https://app.greptile.com/nvidia-public-github/-/custom-context/knowledge-base/nvidia-nemo/datadesigner/-/docs/testing-and-tooling.md)

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[nabinchha]

Thanks for putting this together, @andreatnvidia — the built-wheel coverage gives the new package boundary a strong foundation.

Summary

This PR adds the optional data-designer-slurm distribution, exposes it through the exact-version data-designer[slurm] extra, and integrates it with the workspace, CI, dependency audit, build, and release tooling. The packaging behavior matches the stated scope, but the extra-aware audit currently misclassifies some dependency gaps and the operational guidance still assumes a three-package repository in several important places.

Findings

Warnings — Worth addressing

scripts/audit_package_dependencies.py:173 — Selected project extras leak into unrelated dependency closures

• What: Each workspace project's declarations are precomputed with selected_extras_by_project, then requirements_for() returns that expanded declaration set regardless of the extras requested on the current dependency edge. I reproduced this with a consumer of plain base (not base[leaf]): selecting base:leaf globally made an import from leaf appear guaranteed_by: ["base"] and downgraded it from high to low severity. The current repository output shows the same pollution: adding --extra data-designer:slurm makes data-designer-slurm appear to guarantee click for data-designer through the base↔leaf cycle.
• Why: The dependency recipe explicitly treats low-severity entries as safely guaranteed by mandatory dependencies. This false downgrade can therefore hide a real standalone-installation gap from both the report and automated fix prioritization.
• Suggestion: Cache workspace declarations by (project_name, frozenset(edge_extras)) and call declared_dependencies() with the extras carried by that specific dependency edge. Use selected_extras_by_project only when constructing the selected root project's own declarations. Please add regression cases for a consumer of plain base and a consumer of base[leaf].

.github/workflows/agentic-ci-daily.yml:135 — The four-package change is not reflected in downstream audit instructions and canonical guides

• What: This workflow now generates a four-package inventory, but .agents/recipes/dependencies/recipe.md:14-46 still instructs the audit agent to inspect “all three packages” and explicitly reads only the config, engine, and interface manifests. architecture/overview.md:3-22,53 likewise documents three distributions and the old dependency direction, while DEVELOPMENT.md:108-117 says make test runs only three suites. Even make help still describes installation and isolated testing as config → engine → interface at Makefile:47,63.
• Why: The scheduled audit may omit manual inspection of the leaf manifest, and maintainers now receive contradictory guidance about the repository's package graph and standard test surface.
• Suggestion: Update the dependency recipe, architecture/overview.md, DEVELOPMENT.md, and the remaining Makefile help text to include the Slurm leaf and distinguish the import direction from the packaging-only reverse extra.

What Looks Good

• The integration script validates actual wheel metadata and clean environments for base-only, extra-selected, direct-leaf, and missing-counterpart installs.
• Namespace composition is tested from both source and built wheels, including the absence of a top-level data_designer/__init__.py.
• The exact-version metadata, workspace lock, publishing list, CI wiring, formatting, and focused tests all validate successfully; the full GitHub check suite is green.

Verdict

Needs changes — preserve extra selection per dependency edge and align the scheduled audit/canonical package guidance with the new four-package structure before merge.

---

This review was generated by an AI assistant.

[andreatnvidia]

@nabinchha Thanks, good catches. Addressed both in 9290cd12:

• Workspace dependency closures now respect extras selected on each dependency edge and stop cycles through the package under audit. I added regressions for base vs. base[leaf] and cyclic workspace dependencies.
• Updated the dependency, structure, and test-health recipes, architecture overview, development guide, and Makefile help for all four packages.

The live audit no longer lists data-designer-slurm as guaranteeing core dependencies. All affected tests and package installation checks pass. Could you take another look?