Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 6 additions & 5 deletions .agents/recipes/dependencies/recipe.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ permissions:

# Dependency Audit

Audit the dependency graph across all three packages. Write findings to
Audit the dependency graph across all four packages. Write findings to
`/tmp/audit-{{suite}}.md`.

Dependabot handles version bump PRs. This recipe focuses on what Dependabot
Expand Down Expand Up @@ -39,11 +39,12 @@ Read the `pyproject.toml` for each package:
cat packages/data-designer-config/pyproject.toml
cat packages/data-designer-engine/pyproject.toml
cat packages/data-designer/pyproject.toml
cat packages/data-designer-slurm/pyproject.toml
```

Note: engine and interface packages use `uv-dynamic-versioning` to inject
dependencies. Check both static declarations and the dynamic versioning
config.
Note: engine, interface, and Slurm packages use `uv-dynamic-versioning` to
inject dependencies. Check both static declarations and the dynamic
versioning config, including selected optional dependencies.

### 2. Direct dependency declaration gaps

Expand Down Expand Up @@ -79,7 +80,7 @@ intentionally deferred but still need to be declared as dependencies.

Check that shared dependencies use consistent version constraints:
```bash
# Extract dependency specs from all three pyproject.toml files
# Extract dependency specs from all four pyproject.toml files
grep -E "^\s+\"[a-zA-Z]" packages/*/pyproject.toml
```

Expand Down
13 changes: 8 additions & 5 deletions .agents/recipes/structure/recipe.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,21 +19,24 @@ to `/tmp/audit-{{suite}}.md`.
Before starting, read the authoritative sources for the structural rules:

1. **`AGENTS.md`** (repo root) - "The Layering Is Structural" section defines
the three packages, their ownership, and the dependency direction rule.
the four packages, their ownership, and the dependency direction rule.
2. **`architecture/overview.md`** - system architecture diagram, package layout,
and the "no reverse imports" rule.

The canonical rules:

```
data-designer (interface) -> data-designer-engine -> data-designer-config
data-designer-slurm -> data-designer (interface) -> data-designer-engine -> data-designer-config
```

- `packages/data-designer-config/` must NOT import from `data_designer.engine`,
`data_designer.interface`, or `data_designer.cli`
`data_designer.interface`, `data_designer.cli`, or `data_designer.slurm`
- `packages/data-designer-engine/` must NOT import from
`data_designer.interface` or `data_designer.cli`
- `packages/data-designer/` CAN import from both engine and config
`data_designer.interface`, `data_designer.cli`, or `data_designer.slurm`
- `packages/data-designer/` CAN import from engine and config but must NOT
import from `data_designer.slurm`
- `packages/data-designer-slurm/` uses public interface/config APIs and must
NOT import engine internals

**What CI already enforces**: ruff rule `TID` catches relative imports. The
CI test matrix runs config, engine, and interface tests in isolation (separate
Expand Down
2 changes: 1 addition & 1 deletion .agents/recipes/test-health/recipe.md
Original file line number Diff line number Diff line change
Expand Up @@ -306,7 +306,7 @@ Write the report to `/tmp/audit-{{suite}}.md`:

| Check | Status | Detail |
|-------|--------|--------|
| Package imports | OK/FAIL | All three packages import cleanly |
| Package imports | OK/FAIL | All four packages import cleanly |
| Import timing | OK/FAIL | X.XXs (budget: 3s) |
| Registry completeness | OK/WARN | Column types resolve to config classes |

Expand Down
1 change: 1 addition & 0 deletions .github/workflows/agentic-ci-daily.yml
Original file line number Diff line number Diff line change
Expand Up @@ -135,6 +135,7 @@ jobs:
if: matrix.suite == 'dependencies'
run: |
.venv/bin/python scripts/audit_package_dependencies.py \
--extra data-designer:slurm \
--output /tmp/dependency-inventory.json
jq '.packages[] | {package, missing, unresolved_modules}' \
/tmp/dependency-inventory.json
Expand Down
31 changes: 29 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -129,6 +129,31 @@ jobs:
uv run --with pytest --with pytest-asyncio --with pytest-httpx --with pytest-env \
pytest packages/data-designer/tests

test-slurm-package:
name: Test Slurm package wheels
needs: validate-dispatch
runs-on: ubuntu-latest

steps:
- name: Checkout code
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1

- name: Install uv
uses: astral-sh/setup-uv@c771a70e6277c0a99b617c7a806ffedaca235ff9 # v9.0.0
with:
version: "latest"
python-version: "3.11"
enable-cache: true

- name: Install development dependencies
run: make install-dev

- name: Run package tests
run: .venv/bin/pytest packages/data-designer-slurm/tests

- name: Run built-wheel installation tests
run: make test-slurm-wheel-install

# ===========================================================================
# Combined Coverage Check
# Runs all tests together to verify overall coverage threshold
Expand Down Expand Up @@ -281,7 +306,7 @@ jobs:
test-summary:
name: Test (Python ${{ matrix.python-version }} on ${{ matrix.os }})
runs-on: ubuntu-latest
needs: [validate-dispatch, test-config, test-engine, test-interface]
needs: [validate-dispatch, test-config, test-engine, test-interface, test-slurm-package]
if: always()
strategy:
matrix:
Expand All @@ -294,12 +319,14 @@ jobs:
if [[ "${{ needs.validate-dispatch.result }}" != "success" ]] || \
[[ "${{ needs.test-config.result }}" != "success" ]] || \
[[ "${{ needs.test-engine.result }}" != "success" ]] || \
[[ "${{ needs.test-interface.result }}" != "success" ]]; then
[[ "${{ needs.test-interface.result }}" != "success" ]] || \
[[ "${{ needs.test-slurm-package.result }}" != "success" ]]; then
echo "One or more test jobs failed"
echo "validate-dispatch: ${{ needs.validate-dispatch.result }}"
echo "test-config: ${{ needs.test-config.result }}"
echo "test-engine: ${{ needs.test-engine.result }}"
echo "test-interface: ${{ needs.test-interface.result }}"
echo "test-slurm-package: ${{ needs.test-slurm-package.result }}"
exit 1
fi
echo "All test jobs passed successfully"
7 changes: 4 additions & 3 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,15 +7,16 @@ If you are an agent helping a user **build a dataset**, use the [`data-designer`

## The Layering Is Structural

The `data_designer` namespace is split across three installable packages that merge at runtime via PEP 420 implicit namespace packages (no top-level `__init__.py`).
The `data_designer` namespace is split across four installable packages that merge at runtime via PEP 420 implicit namespace packages (no top-level `__init__.py`).

| Package | Path | Owns |
|---------|------|------|
| `data-designer-config` | `packages/data-designer-config/` | `data_designer.config` β€” column configs, model configs, sampler params, builder API, plugin system, lazy imports |
| `data-designer-engine` | `packages/data-designer-engine/` | `data_designer.engine` β€” column generators, dataset builders, DAG execution, model facade, validators, sampling |
| `data-designer` | `packages/data-designer/` | `data_designer.interface` β€” public `DataDesigner` class, results, errors; `data_designer.cli` β€” CLI entry point; `data_designer.integrations` |
| `data-designer-slurm` | `packages/data-designer-slurm/` | `data_designer.slurm` β€” optional Slurm batch execution |

**Dependency direction (left depends on right):** interface β†’ engine β†’ config. Never import against this flow.
**Import direction (left imports right):** Slurm β†’ interface β†’ engine β†’ config. The `data-designer[slurm]` extra creates a packaging-only reverse edge; no code may import against this flow.

## Core Concepts

Expand All @@ -34,7 +35,7 @@ The `data_designer` namespace is split across three installable packages that me

## Structural Invariants

- **Import direction** β€” interface β†’ engine β†’ config (left depends on right). No reverse imports.
- **Import direction** β€” Slurm β†’ interface β†’ engine β†’ config (left imports right). No reverse imports.
- **Fast imports** β€” heavy third-party libraries are lazy-loaded via `data_designer.lazy_heavy_imports`. See [STYLEGUIDE.md](STYLEGUIDE.md) for the pattern.
- **No relative imports** β€” absolute imports only, enforced by ruff rule `TID`.
- **Typed code** β€” all functions, methods, and class attributes require type annotations. Modern syntax: `list[str]`, `str | None`.
Expand Down
5 changes: 3 additions & 2 deletions DEVELOPMENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -105,16 +105,17 @@ uv run ruff format --check # Check formatting

### Running Tests

`make test` runs all three package test suites in sequence (config, engine, interface). When iterating on a single package, run its tests directly:
`make test` runs all four package test suites in sequence (config, engine, interface, Slurm). When iterating on a single package, run its tests directly:

```bash
# Run all tests (config + engine + interface)
# Run all tests (config + engine + interface + Slurm)
make test

# Run a single package's tests
make test-config # data-designer-config
make test-engine # data-designer-engine
make test-interface # data-designer (interface)
make test-slurm # data-designer-slurm

# Run a specific test file
uv run pytest tests/config/test_sampler_constraints.py
Expand Down
Loading
Loading