diff --git a/dev/specs/infp-504-artifact-composition/contracts/filter-interfaces.md b/dev/specs/infp-504-artifact-composition/contracts/filter-interfaces.md new file mode 100644 index 00000000..e034bbba --- /dev/null +++ b/dev/specs/infp-504-artifact-composition/contracts/filter-interfaces.md @@ -0,0 +1,107 @@ +# Filter Interface Contracts + +**Feature**: INFP-504 | **Date**: 2026-03-20 + +## Jinja2 Filter Signatures + +### artifact_content + +```python +async def artifact_content(storage_id: str) -> str +``` + +| Input | Output | Error | +| ----- | ------ | ----- | +| Valid storage_id string | Raw artifact content (text) | — | +| `None` | — | `JinjaFilterError("artifact_content", "storage_id is null", hint="...")` | +| `""` (empty) | — | `JinjaFilterError("artifact_content", "storage_id is empty", hint="...")` | +| Non-existent storage_id | — | `JinjaFilterError("artifact_content", "content not found: {id}")` | +| Permission denied (401/403) | — | `JinjaFilterError("artifact_content", "permission denied for storage_id: {id}")` | +| No client provided | — | `JinjaFilterError("artifact_content", "requires InfrahubClient", hint="pass client via Jinja2Template(client=...)")` | + +**Validation**: Blocked in `CORE` context. Allowed in `WORKER` context. + +### file_object_content + +```python +async def file_object_content(storage_id: str) -> str +``` + +| Input | Output | Error | +| ----- | ------ | ----- | +| Valid storage_id (text file) | Raw file content (text) | — | +| Valid storage_id (binary file) | — | `JinjaFilterError("file_object_content", "binary content not supported for storage_id: {id}")` | +| `None` | — | `JinjaFilterError("file_object_content", "storage_id is null", hint="...")` | +| `""` (empty) | — | `JinjaFilterError("file_object_content", "storage_id is empty", hint="...")` | +| Non-existent storage_id | — | `JinjaFilterError("file_object_content", "content not found: {id}")` | +| Permission denied (401/403) | — | `JinjaFilterError("file_object_content", "permission denied for storage_id: {id}")` | +| No client provided | — | `JinjaFilterError("file_object_content", "requires InfrahubClient", hint="pass client via Jinja2Template(client=...)")` | + +**Validation**: Blocked in `CORE` context. Allowed in `WORKER` context. + +### from_json + +```python +def from_json(value: str) -> dict | list +``` + +| Input | Output | Error | +| ----- | ------ | ----- | +| Valid JSON string | Parsed dict or list | — | +| `""` (empty) | `{}` | — | +| Malformed JSON | — | `JinjaFilterError("from_json", "invalid JSON: {error_detail}")` | + +**Validation**: Allowed in all contexts (`ALL`). + +### from_yaml + +```python +def from_yaml(value: str) -> dict | list +``` + +| Input | Output | Error | +| ----- | ------ | ----- | +| Valid YAML string | Parsed dict, list, or scalar | — | +| `""` (empty) | `{}` | — | +| Malformed YAML | — | `JinjaFilterError("from_yaml", "invalid YAML: {error_detail}")` | + +**Validation**: Allowed in all contexts (`ALL`). + +## ObjectStore API Contract + +### GET /api/storage/object/{identifier} (existing) + +Used by `artifact_content`. Returns plain text content. + +### GET /api/files/by-storage-id/{storage_id} (new) + +Used by `file_object_content`. Returns file content with appropriate content-type header. + +**Accepted content-types** (text-based): + +- `text/*` +- `application/json` +- `application/yaml` +- `application/x-yaml` + +**Rejected**: All other content-types → `JinjaFilterError` with binary content message. + +## Validation Contract + +### validate() method + +```python +def validate( + self, + restricted: bool = True, + context: ExecutionContext | None = None, +) -> None +``` + +| Context | Trusted filters | Worker filters | Untrusted filters | +| ------- | :-: | :-: | :-: | +| `CORE` | allowed | blocked | blocked | +| `WORKER` | allowed | allowed | blocked | +| `LOCAL` | allowed | allowed | allowed | + +**Backward compat**: `restricted=True` → `CORE`, `restricted=False` → `LOCAL`. diff --git a/dev/specs/infp-504-artifact-composition/data-model.md b/dev/specs/infp-504-artifact-composition/data-model.md new file mode 100644 index 00000000..ad1e0d29 --- /dev/null +++ b/dev/specs/infp-504-artifact-composition/data-model.md @@ -0,0 +1,166 @@ +# Data Model: Artifact Content Composition + +**Feature**: INFP-504 | **Date**: 2026-03-20 + +## New Entities + +### ExecutionContext (Flag enum) + +**Location**: `infrahub_sdk/template/filters.py` + +```python +class ExecutionContext(Flag): + CORE = auto() # API server computed attributes — most restrictive + WORKER = auto() # Prefect background workers + LOCAL = auto() # Local CLI / unrestricted rendering + ALL = CORE | WORKER | LOCAL +``` + +**Semantics**: Represents where template code executes. A filter's `allowed_contexts` flags are an allowlist — fewer flags means less trusted. + +### FilterDefinition (modified) + +**Location**: `infrahub_sdk/template/filters.py` + +```python +@dataclass +class FilterDefinition: + name: str + allowed_contexts: ExecutionContext + source: str + + @property + def trusted(self) -> bool: + """Backward compatibility: trusted means allowed in all contexts.""" + return self.allowed_contexts == ExecutionContext.ALL +``` + +**Migration**: + +| Current | New | +| ------- | --- | +| `FilterDefinition("abs", trusted=True, source="jinja2")` | `FilterDefinition("abs", allowed_contexts=ExecutionContext.ALL, source="jinja2")` | +| `FilterDefinition("safe", trusted=False, source="jinja2")` | `FilterDefinition("safe", allowed_contexts=ExecutionContext.LOCAL, source="jinja2")` | + +### JinjaFilterError (new exception) + +**Location**: `infrahub_sdk/template/exceptions.py` + +```python +class JinjaFilterError(JinjaTemplateError): + def __init__(self, filter_name: str, message: str, hint: str | None = None) -> None: + self.filter_name = filter_name + self.hint = hint + full_message = f"Filter '{filter_name}': {message}" + if hint: + full_message += f" — {hint}" + super().__init__(full_message) +``` + +**Inheritance**: `Error` → `JinjaTemplateError` → `JinjaFilterError` + +### InfrahubFilters (new class) + +**Location**: `infrahub_sdk/template/infrahub_filters.py` (new file) + +```python +class InfrahubFilters: + def __init__(self, client: InfrahubClient) -> None: + self.client = client + + async def artifact_content(self, storage_id: str) -> str: + """Retrieve artifact content by storage_id.""" + ... + + async def file_object_content(self, storage_id: str) -> str: + """Retrieve file object content by storage_id.""" + ... +``` + +**Key design decisions**: + +- Methods are `async` — Jinja2's `auto_await` handles them in async rendering mode +- Holds an `InfrahubClient` (async only), not `InfrahubClientSync` +- Each method validates inputs and catches `AuthenticationError` to wrap in `JinjaFilterError` + +## Modified Entities + +### Jinja2Template (modified constructor) + +**Location**: `infrahub_sdk/template/__init__.py` + +```python +def __init__( + self, + template: str | Path, + template_directory: Path | None = None, + filters: dict[str, Callable] | None = None, + client: InfrahubClient | None = None, # NEW +) -> None: +``` + +**Changes**: + +- New optional `client` parameter +- When `client` provided: instantiate `InfrahubFilters`, register `artifact_content` and `file_object_content` +- Always register `from_json` and `from_yaml` (no client needed) +- File-based environment must add `enable_async=True` for async filter support + +### Jinja2Template.validate() (modified signature) + +```python +def validate(self, restricted: bool = True, context: ExecutionContext | None = None) -> None: +``` + +**Changes**: + +- New optional `context` parameter (takes precedence over `restricted` when provided) +- Backward compat: `restricted=True` → `ExecutionContext.CORE`, `restricted=False` → `ExecutionContext.LOCAL` +- Validation logic: filter allowed if `filter.allowed_contexts & context` is truthy + +### ObjectStore (new method) + +**Location**: `infrahub_sdk/object_store.py` + +```python +async def get_file_by_storage_id(self, storage_id: str, tracker: str | None = None) -> str: + """Retrieve file object content by storage_id. + + Raises error if content-type is not text-based. + """ + ... +``` + +**API endpoint**: `GET /api/files/by-storage-id/{storage_id}` + +**Content-type check**: Allow `text/*`, `application/json`, `application/yaml`, `application/x-yaml`. Reject all others. + +## New Filter Registrations + +```python +# In AVAILABLE_FILTERS: + +# Infrahub client-dependent filters (worker context only) +FilterDefinition("artifact_content", allowed_contexts=ExecutionContext.WORKER, source="infrahub"), +FilterDefinition("file_object_content", allowed_contexts=ExecutionContext.WORKER, source="infrahub"), + +# Parsing filters (trusted, all contexts) +FilterDefinition("from_json", allowed_contexts=ExecutionContext.ALL, source="infrahub"), +FilterDefinition("from_yaml", allowed_contexts=ExecutionContext.ALL, source="infrahub"), +``` + +## Relationships + +```text +Jinja2Template + ├── has-a → InfrahubFilters (when client provided) + ├── uses → FilterDefinition registry (for validation) + └── uses → ExecutionContext (for context-aware validation) + +InfrahubFilters + ├── has-a → InfrahubClient + └── uses → ObjectStore (for content retrieval) + +JinjaFilterError + └── extends → JinjaTemplateError → Error +``` diff --git a/dev/specs/infp-504-artifact-composition/plan.md b/dev/specs/infp-504-artifact-composition/plan.md new file mode 100644 index 00000000..58808d78 --- /dev/null +++ b/dev/specs/infp-504-artifact-composition/plan.md @@ -0,0 +1,108 @@ +# Implementation Plan: Artifact Content Composition + +**Branch**: `infp-504-artifact-composition` | **Date**: 2026-03-20 | **Spec**: [spec.md](spec.md) +**Jira**: INFP-504 | **Epic**: IFC-2275 + +## Summary + +Enable Jinja2 templates to reference and inline rendered content from other artifacts and file objects via new filters (`artifact_content`, `file_object_content`, `from_json`, `from_yaml`). Requires evolving the filter trust model from a binary boolean to a flag-based execution context system, creating a new `InfrahubFilters` class to hold client-dependent filter logic, and extending `Jinja2Template` with an optional client parameter. + +## Technical Context + +**Language/Version**: Python 3.10-3.13 +**Primary Dependencies**: jinja2, httpx, pydantic >=2.0, PyYAML (already available via netutils) +**Storage**: Infrahub object store (REST API) +**Testing**: pytest (`uv run pytest tests/unit/`) +**Target Platform**: SDK library consumed by Prefect workers, CLI, and API server +**Project Type**: Single Python package +**Constraints**: No new external dependencies. Must maintain async/sync dual pattern. Must not break existing filter behavior. + +## Key Technical Decisions + +### 1. Async Filters via Jinja2 native support (R-001) + +The `SandboxedEnvironment` already uses `enable_async=True`. Jinja2's `auto_await` automatically awaits filter return values during `render_async()`. The new content-fetching filters can be `async def` — no bridging needed. + +**Required change**: Add `enable_async=True` to the file-based environment (`_get_file_based_environment()`) so async filters work for file-based templates too. + +### 2. Flag-based trust model (R-004) + +Replace `FilterDefinition.trusted: bool` with `allowed_contexts: ExecutionContext` using Python's `Flag` enum. Three contexts: `CORE` (most restrictive), `WORKER`, `LOCAL` (least restrictive). A backward-compatible `trusted` property preserves existing API. + +### 3. Content-type checking for file objects (R-003) + +New `ObjectStore.get_file_by_storage_id()` method checks response `content-type` header. Text-based types are allowed; binary types are rejected with a descriptive error. + +## Project Structure + +### Documentation (this feature) + +```text +dev/specs/infp-504-artifact-composition/ +├── spec.md # Feature specification +├── plan.md # This file +├── research.md # Phase 0 research findings +├── data-model.md # Entity definitions +├── quickstart.md # Usage examples +├── contracts/ +│ └── filter-interfaces.md # Filter I/O contracts +└── checklists/ + └── requirements.md # Quality checklist +``` + +### Source Code (files to create or modify) + +```text +infrahub_sdk/ +├── template/ +│ ├── __init__.py # MODIFY: Jinja2Template (client param, validate context) +│ ├── filters.py # MODIFY: ExecutionContext enum, FilterDefinition migration +│ ├── exceptions.py # MODIFY: Add JinjaFilterError +│ └── infrahub_filters.py # CREATE: InfrahubFilters class +├── object_store.py # MODIFY: Add get_file_by_storage_id() +``` + +```text +tests/unit/ +├── template/ +│ ├── test_filters.py # MODIFY: Tests for new filters and trust model +│ └── test_infrahub_filters.py # CREATE: Tests for InfrahubFilters +``` + +## Implementation Order + +The 13 Jira tasks under IFC-2275 follow this dependency graph: + +```text +Phase 1 (Foundation — no dependencies, can be parallel): + IFC-2367: JinjaFilterError exception + IFC-2368: Flag-based trust model (ExecutionContext + FilterDefinition migration) + IFC-2373: ObjectStore.get_file_by_storage_id() + +Phase 2 (Filters — depend on Phase 1): + IFC-2369: from_json filter (depends on IFC-2367) + IFC-2370: from_yaml filter (depends on IFC-2367) + IFC-2371: InfrahubFilters class (depends on IFC-2367) + +Phase 3 (Content filters — depend on Phase 2): + IFC-2372: artifact_content filter (depends on IFC-2371) + IFC-2374: file_object_content filter (depends on IFC-2371, IFC-2373) + +Phase 4 (Integration — depend on Phase 3): + IFC-2375: Jinja2Template client param + wiring (depends on IFC-2368, IFC-2371, IFC-2372) + IFC-2376: Filter registration with correct contexts (depends on IFC-2368, IFC-2369, IFC-2370, IFC-2372, IFC-2374) + +Phase 5 (Documentation + Server — depend on Phase 4): + IFC-2377: Documentation (depends on IFC-2376) + IFC-2378: integrator.py threading [Infrahub server] (depends on IFC-2375) + IFC-2379: Schema validation [Infrahub server] (depends on IFC-2368) +``` + +## Risk Register + +| Risk | Likelihood | Impact | Mitigation | +| ---- | --------- | ------ | ---------- | +| Jinja2 `auto_await` doesn't work as expected for filters | Low | High | Verify with a minimal test before building on the assumption. Fallback: sync wrapper with thread executor. | +| File-based environment breaks with `enable_async=True` | Low | Medium | File-based env change is isolated and testable. Existing tests will catch regressions. | +| ObjectStore API returns incorrect content-type for file objects | Medium | Low | Already flagged by @wvandeun. The filter will use best-effort content-type checking; can be refined when API is fixed. | +| `validate()` backward compat breaks existing callers | Low | High | Keep `restricted` param with deprecation path. Test all existing call sites. | diff --git a/dev/specs/infp-504-artifact-composition/quickstart.md b/dev/specs/infp-504-artifact-composition/quickstart.md new file mode 100644 index 00000000..1e1a6dc7 --- /dev/null +++ b/dev/specs/infp-504-artifact-composition/quickstart.md @@ -0,0 +1,90 @@ +# Quickstart: Artifact Content Composition + +**Feature**: INFP-504 | **Date**: 2026-03-20 + +## Jinja2 Templates + +### Inline artifact content + +Query artifacts via GraphQL and use the `artifact_content` filter to include their content: + +```jinja2 +{% set device = data.NetworkDevice.edges[0].node %} +hostname {{ device.hostname.value }} + +{% for artifact in device.artifacts.edges %} +{% set content = artifact.node.storage_id.value | artifact_content %} +{% if content %} +{{ content }} +{% endif %} +{% endfor %} +``` + +### Inline file object content + +```jinja2 +{% set file_content = file_object.storage_id.value | file_object_content %} +{{ file_content }} +``` + +### Parse structured content + +Chain `artifact_content` with `from_json` or `from_yaml` to access structured data: + +```jinja2 +{% set config = artifact.node.storage_id.value | artifact_content | from_json %} +interface {{ config.interface_name }} + ip address {{ config.ip_address }} +``` + +```jinja2 +{% set config = artifact.node.storage_id.value | artifact_content | from_yaml %} +{% for route in config.static_routes %} +ip route {{ route.prefix }} {{ route.next_hop }} +{% endfor %} +``` + +## Python Transforms + +For Python transforms, use the SDK's object store directly: + +```python +async def transform(self, data: dict, client: InfrahubClient) -> str: + storage_id = ( + data["NetworkDevice"]["edges"][0]["node"] + ["artifacts"]["edges"][0]["node"] + ["storage_id"]["value"] + ) + content = await client.object_store.get(identifier=storage_id) + return content +``` + +## GraphQL Query Pattern + +Reference artifacts in your query via the `artifacts` relationship: + +```graphql +query StartupConfig($name: String!) { + NetworkDevice(hostname__value: $name) { + edges { + node { + hostname { value } + artifacts { + edges { + node(name__value: "base_config") { + id + storage_id { value } + } + } + } + } + } + } +} +``` + +## Known Limitations + +- **No ordering guarantee**: Artifacts may be generated in parallel. A composite artifact template may render before its dependencies are ready. Future event-driven pipeline work (INFP-227) will address this. +- **Worker context only**: `artifact_content` and `file_object_content` are only available on Prefect workers, not in computed attributes or local CLI. +- **Text content only**: `file_object_content` rejects binary file objects. `artifact_content` always returns text (artifacts are text-only). diff --git a/dev/specs/infp-504-artifact-composition/research.md b/dev/specs/infp-504-artifact-composition/research.md new file mode 100644 index 00000000..92f590a7 --- /dev/null +++ b/dev/specs/infp-504-artifact-composition/research.md @@ -0,0 +1,75 @@ +# Research: Artifact Content Composition + +**Feature**: INFP-504 | **Date**: 2026-03-20 + +## Research Findings + +### R-001: Async-to-Sync Bridge for Jinja2 Filters + +**Decision**: Use Jinja2's native async filter support (`auto_await`) — no bridging needed. + +**Rationale**: The `SandboxedEnvironment` is already created with `enable_async=True` (`template/__init__.py`:137), and rendering uses `template.render_async()` (`template/__init__.py`:122). In Jinja2 async mode, filter call results are wrapped in `auto_await()`, which detects awaitables and awaits them automatically. This means we can register async functions directly as filters. + +**Caveat**: The file-based environment (`_get_file_based_environment()` at line 140) does NOT currently set `enable_async=True`. This must be added for async filters to work with file-based templates. + +**Alternatives considered**: + +- `asyncio.run()`: Cannot be used — we're already inside a running event loop during `render_async()`. Would raise `RuntimeError: This event loop is already running`. +- Thread-based executor: Overly complex, introduces thread safety concerns, and is unnecessary given Jinja2's built-in async support. +- `nest_asyncio`: External dependency, fragile, not needed. + +**Evidence**: Jinja2 source code confirms `auto_await` wrapping of filter results in async mode. SDK's existing pytest plugin already uses `asyncio.run()` for a different scenario (sync test runner calling async render), which is a distinct pattern. + +### R-002: File Object Content API Path + +**Decision**: Use `/api/files/by-storage-id/{storage_id}` endpoint. + +**Rationale**: Confirmed by product owner. The `storage_id` alone is sufficient for retrieval. Future endpoints for by-hfid and by-node are anticipated but not in scope. + +**Implementation note**: The existing `ObjectStore.get()` uses the path `/api/storage/object/{identifier}`. The file object endpoint is completely different, so a new method is needed rather than parameterizing the existing one. + +### R-003: Binary Content Detection for File Objects + +**Decision**: Check the `content-type` response header from the API response. Reject non-text content types. + +**Rationale**: Artifacts are always plain text (no detection needed). File objects can be any type, but the response `content-type` header reliably indicates the type. The current `ObjectStore.get()` returns `response.text` directly without checking the content type — the new file object method must inspect the header first. + +**Text types to allow**: `text/*`, `application/json`, `application/yaml`, `application/x-yaml`. Everything else should be rejected with `JinjaFilterError`. + +### R-004: Filter Trust Model Design + +**Decision**: Flag-based `ExecutionContext` using Python's `Flag` enum. + +**Rationale**: The requirements don't form a clean hierarchy. `artifact_content` must be allowed in WORKER but not LOCAL (no client), while `safe` must be allowed in LOCAL but not WORKER. A flag-based system with an allowlist per filter is the only model that handles all cases without implicit ordering assumptions. + +**Design**: + +```python +class ExecutionContext(Flag): + CORE = auto() # API server computed attributes (most restrictive) + WORKER = auto() # Prefect background workers + LOCAL = auto() # Local CLI / unrestricted rendering + ALL = CORE | WORKER | LOCAL +``` + +```python +@dataclass +class FilterDefinition: + name: str + allowed_contexts: ExecutionContext + source: str +``` + +**Migration**: `trusted=True` → `allowed_contexts=ALL`, `trusted=False` → `allowed_contexts=LOCAL`. A `trusted` property can be preserved for backward compatibility: `return bool(self.allowed_contexts & ExecutionContext.CORE)`. + +### R-005: Existing Netutils Filter Inventory + +**Decision**: `from_json` and `from_yaml` do NOT exist in the current filter set. + +**Rationale**: Searched all 51 builtin filters and 87 netutils filters in `infrahub_sdk/template/filters.py`. No `from_json`, `from_yaml`, `parse_json`, or `parse_yaml` entries. `tojson` exists (builtin, untrusted) but is the reverse operation. Safe to add without de-duplication concerns. + +### R-006: ObjectStore Authentication Error Handling + +**Decision**: Reuse the existing pattern from `ObjectStore.get()`. + +**Rationale**: `ObjectStore.get()` (object_store.py:34-40) already handles 401/403 by raising `AuthenticationError`. The new filters should catch `AuthenticationError` and wrap it in `JinjaFilterError` with a permission-specific message. No new auth handling logic needed in ObjectStore itself. diff --git a/dev/specs/infp-504-artifact-composition/spec.md b/dev/specs/infp-504-artifact-composition/spec.md index 18b8b5f2..fa39a004 100644 --- a/dev/specs/infp-504-artifact-composition/spec.md +++ b/dev/specs/infp-504-artifact-composition/spec.md @@ -97,12 +97,12 @@ The existing single `restricted: bool` parameter on `validate()` is insufficient - **FR-002**: A dedicated class (for example, `InfrahubFilters`) MUST be introduced to hold the client reference and expose the Infrahub-specific filter callable methods. `Jinja2Template` instantiates this class when a client is provided and registers its filters into the Jinja2 environment. - **FR-003**: The system MUST provide an `artifact_content` Jinja2 filter that accepts a `storage_id` string and returns the raw string content of the referenced artifact, using the artifact-specific API path. - **FR-004**: The system MUST provide a `file_object_content` Jinja2 filter that accepts a `storage_id` string and returns the raw string content of the referenced file object, using the file-object-specific API path or metadata handling — this implementation is distinct from `artifact_content`. -- **FR-005**: Both `artifact_content` and `file_object_content` MUST raise `JinjaFilterError` when the input `storage_id` is null or empty, or when the object store cannot retrieve the content for any reason (not found, network failure, auth failure). +- **FR-005**: Both `artifact_content` and `file_object_content` MUST raise `JinjaFilterError` when the input `storage_id` is null or empty, or when the object store cannot retrieve the content for any reason (not found, network failure, auth failure). Additionally, `file_object_content` MUST raise `JinjaFilterError` when the retrieved content has a non-text content type (i.e., not `text/*`, `application/json`, or `application/yaml`). - **FR-006**: Both `artifact_content` and `file_object_content` MUST raise `JinjaFilterError` when invoked and no `InfrahubClient` was supplied to `Jinja2Template` at construction time. The error message MUST name the filter and explain that an `InfrahubClient` is required. -- **FR-007**: Both `artifact_content` and `file_object_content` MUST be registered with `trusted=False` in the `FilterDefinition` registry so that `validate(restricted=True)` blocks them in the computed attributes execution context (Infrahub API server). They are only permitted to execute on Prefect workers, where an `InfrahubClient` is available. Within Infrahub any Jinja2 based computed attributes that use these new filters should cause a schema violation when loading the schema. +- **FR-007**: Both `artifact_content` and `file_object_content` MUST be registered with `allowed_contexts=ExecutionContext.WORKER` in the `FilterDefinition` registry. The `validate()` method accepts an `ExecutionContext` flag; these filters are blocked in the `CORE` context (API server computed attributes) and permitted in the `WORKER` context (Prefect workers, where an `InfrahubClient` is available). Within Infrahub, any Jinja2-based computed attributes that use these new filters should cause a schema violation when loading the schema. - **FR-008**: The system MUST provide `from_json` and `from_yaml` Jinja2 filters (adding them only if not already present in the environment) that parse a string into a Python dict/list. Applying them to an empty string MUST return an empty dict without raising. Applying them to malformed content MUST raise `JinjaFilterError`. - **FR-009**: `from_json` and `from_yaml` MUST be registered as trusted filters (`trusted=True`) since they perform no external I/O. -- **FR-010**: All new filters MUST work correctly with `InfrahubClient` (async). `InfrahubClientSync` is not a supported client type for `Jinja2Template`. +- **FR-010**: All new filters MUST work correctly with `InfrahubClient` (async). `InfrahubClientSync` is not a supported client type for `Jinja2Template`. Both the sandboxed environment (string-based templates) and the file-based environment MUST have `enable_async=True` to support async filter callables via Jinja2's `auto_await`. - **FR-011**: All `JinjaFilterError` instances MUST carry an actionable error message that identifies the filter name, the cause of failure, and any remediation hint (for example: "artifact_content requires an InfrahubClient — pass one via Jinja2Template(client=...)"). - **FR-012**: A new `JinjaFilterError` exception class MUST be added to `infrahub_sdk/template/exceptions.py` as a subclass of `JinjaTemplateError`. - **FR-013**: Documentation MUST include a Python transform example demonstrating artifact content retrieval via `client.object_store.get(identifier=storage_id)`. No new SDK convenience method will be added. @@ -113,7 +113,7 @@ The existing single `restricted: bool` parameter on `validate()` is insufficient - **`Jinja2Template`**: Gains an optional `client` constructor parameter; delegates client-bound filter registration to `InfrahubFilters`. - **`InfrahubFilters`**: New class that holds an `InfrahubClient` reference and exposes `artifact_content`, `file_object_content`, and any other client-dependent filter methods. Registered into the Jinja2 filter map when a client is provided. - **`FilterDefinition`**: Existing dataclass used to declare filter `name`, `trusted` flag, and `source`. New entries are added here for all new filters. -- **`ObjectStore` / `ObjectStoreSync`**: Existing async/sync storage clients used by `InfrahubFilters` to perform `get(identifier=storage_id)` calls. +- **`ObjectStore`**: Existing async storage client used by `InfrahubFilters` to perform `get(identifier=storage_id)` calls. (`ObjectStoreSync` is not used; `InfrahubClientSync` is explicitly out of scope — see FR-001, FR-010.) - **`JinjaFilterError`**: New exception class, subclass of `JinjaTemplateError`, raised by `InfrahubFilters` methods on all filter-level failures (no client, null/empty storage_id, retrieval error). ## Success criteria *(mandatory)* @@ -134,7 +134,7 @@ The existing single `restricted: bool` parameter on `validate()` is insufficient - `from_json` and `from_yaml` are not currently present in the builtin or netutils filter sets; they will be added as part of this feature. If they already exist, the implementation de-duplicates rather than overrides. - All failure modes from the filters (null storage_id, empty storage_id, object not found, network error, auth error) raise exceptions. There is no silent fallback to an empty string. - The permitted execution context for `artifact_content` and `file_object_content` is Prefect workers only. The computed attributes path in the Infrahub API server always runs `validate(restricted=True)`, which blocks these filters before rendering begins. -- The `InfrahubFilters` class provides synchronous callables to Jinja2's filter map; the underlying client is always `InfrahubClient` (async). Async I/O calls are handled consistently with the SDK's existing pattern. +- The `InfrahubFilters` class provides `async def` callables to Jinja2's filter map; the underlying client is always `InfrahubClient` (async). Jinja2's `auto_await` mechanism (enabled via `enable_async=True` on the environment) automatically awaits filter return values during `render_async()`, so no explicit sync-to-async bridging is needed. ## Dependencies & constraints diff --git a/dev/specs/infp-504-artifact-composition/tasks.md b/dev/specs/infp-504-artifact-composition/tasks.md new file mode 100644 index 00000000..47e372bb --- /dev/null +++ b/dev/specs/infp-504-artifact-composition/tasks.md @@ -0,0 +1,188 @@ +# Tasks: Artifact Content Composition via Jinja2 Filters + +**Input**: Design documents from `dev/specs/infp-504-artifact-composition/` +**Prerequisites**: plan.md, spec.md, research.md, data-model.md, contracts/ +**Jira Epic**: IFC-2275 + +**Tests**: Included with each implementation task (per project convention). + +**Organization**: Tasks grouped by user story. US4 (security gate) is foundational and combined with US1 as both are P1. + +## Format: `[ID] [P?] [Story] Description` + +- **[P]**: Can run in parallel (different files, no dependencies) +- **[Story]**: Which user story this task belongs to (e.g., US1, US2, US3, US4) +- Include exact file paths in descriptions + +--- + +## Phase 1: Foundational (Blocking Prerequisites) + +**Purpose**: Exception class and trust model that ALL user stories depend on + +**CRITICAL**: No user story work can begin until this phase is complete + +- [ ] T001 [P] Create `JinjaFilterError` exception class in `infrahub_sdk/template/exceptions.py` — subclass of `JinjaTemplateError` with `filter_name`, `message`, and optional `hint` attributes. Include unit tests for instantiation, inheritance chain, and message formatting. (IFC-2367) +- [ ] T002 [P] Implement `ExecutionContext` flag enum and migrate `FilterDefinition` in `infrahub_sdk/template/filters.py` — add `ExecutionContext(Flag)` with `CORE`, `WORKER`, `LOCAL`, `ALL` values. Replace `FilterDefinition.trusted: bool` with `allowed_contexts: ExecutionContext`. Add backward-compat `trusted` property. Migrate all 138 existing filter entries (`trusted=True` → `ALL`, `trusted=False` → `LOCAL`). Update `validate()` in `infrahub_sdk/template/__init__.py` to accept optional `context: ExecutionContext` parameter (takes precedence over `restricted`; `restricted=True` → `CORE`, `restricted=False` → `LOCAL`). Include unit tests for all 3 contexts with existing filters, backward compat path, and no regressions. (IFC-2368) + +**Checkpoint**: Foundation ready — JinjaFilterError and ExecutionContext available for all stories + +--- + +## Phase 2: US1 — Inline Artifact Content + US4 — Security Gate (Priority: P1) MVP + +**Goal**: A Jinja2 template can use `storage_id | artifact_content` to inline rendered sub-artifact content. Validation blocks this filter in CORE context but allows it in WORKER context. + +**Independent Test**: Render a template calling `artifact_content` with a mocked `InfrahubClient` and verify output matches expected content. Validate the same template in CORE context raises `JinjaTemplateOperationViolationError`, and in WORKER context passes. + +### Implementation for US1 + US4 + +- [ ] T003 [US1] Create `InfrahubFilters` class in `infrahub_sdk/template/infrahub_filters.py` — new file. Class holds `InfrahubClient` reference, exposes async filter methods. Methods are `async def` (Jinja2 `auto_await` handles them in async render mode per R-001). Raises `JinjaFilterError` when called without a client. Include unit tests for instantiation with/without client. (IFC-2371) +- [ ] T004 [US1] Implement `artifact_content` async method on `InfrahubFilters` in `infrahub_sdk/template/infrahub_filters.py` — uses `self.client.object_store.get(identifier=storage_id)`. Raises `JinjaFilterError` on: null/empty storage_id, retrieval failure, permission denied (catch `AuthenticationError` per R-006). Artifacts are always text (no binary check needed per R-003). Include unit tests: happy path (mocked ObjectStore), null, empty, not-found, network error, permission denied, no-client error with descriptive message. (IFC-2372) +- [ ] T005 [US1] [US4] Add `client` parameter to `Jinja2Template.__init__` and wire up filter registration in `infrahub_sdk/template/__init__.py` — add `client: InfrahubClient | None = None` param. When client provided: instantiate `InfrahubFilters`, register `artifact_content` into Jinja2 env filter map. Add `enable_async=True` to `_get_file_based_environment()` (per R-001 caveat). Register `artifact_content` in `FilterDefinition` registry with `allowed_contexts=ExecutionContext.WORKER`. Include unit tests: render with client (mocked), render without client (error), validation in CORE (blocked), WORKER (allowed), LOCAL (allowed). Verify existing untrusted filters like `safe` remain blocked in WORKER context (US4 AC3). (IFC-2375 partial + IFC-2376 partial) + +**Checkpoint**: US1 + US4 fully functional. `artifact_content` renders in WORKER context, blocked in CORE. MVP complete. + +--- + +## Phase 3: US2 — Inline File Object Content (Priority: P2) + +**Goal**: A Jinja2 template can use `storage_id | file_object_content` to inline file object content with binary rejection. + +**Independent Test**: Render a template calling `file_object_content` with a mocked client. Verify text content returned, binary content rejected. Validation blocks in CORE, allows in WORKER. + +### Implementation for US2 + +- [ ] T006 [P] [US2] Add `get_file_by_storage_id()` method to `ObjectStore` in `infrahub_sdk/object_store.py` — async method using endpoint `GET /api/files/by-storage-id/{storage_id}`. Check `content-type` response header: allow `text/*`, `application/json`, `application/yaml`, `application/x-yaml`; reject all others with descriptive error. Handle 401/403 as `AuthenticationError`. Include unit tests: text response, binary rejection, 404, auth failure, network error. (IFC-2373) +- [ ] T007 [US2] Implement `file_object_content` async method on `InfrahubFilters` in `infrahub_sdk/template/infrahub_filters.py` — uses new `self.client.object_store.get_file_by_storage_id(storage_id)`. Same error handling as `artifact_content` plus binary content error (delegated to ObjectStore). Include unit tests: happy path, all error conditions, binary content rejection. (IFC-2374) +- [ ] T008 [US2] Register `file_object_content` filter in `Jinja2Template` and `FilterDefinition` in `infrahub_sdk/template/__init__.py` and `infrahub_sdk/template/filters.py` — register when client provided. `allowed_contexts=ExecutionContext.WORKER`. Include unit tests: render with client, validation in CORE (blocked), WORKER (allowed). (IFC-2375 partial + IFC-2376 partial) + +**Checkpoint**: US2 complete. `file_object_content` works alongside `artifact_content`. + +--- + +## Phase 4: US3 — Parse Structured Artifact Content (Priority: P3) + +**Goal**: Templates can chain `artifact_content | from_json` or `artifact_content | from_yaml` to access structured data. + +**Independent Test**: Render a template chaining `artifact_content | from_json` and verify parsed fields accessible. `from_json("")` and `from_yaml("")` return `{}`. + +### Implementation for US3 + +- [ ] T009 [P] [US3] Implement `from_json` filter function in `infrahub_sdk/template/infrahub_filters.py` — pure sync function (no client needed). Empty string → `{}` (explicit special-case since `json.loads("")` raises). Malformed JSON → `JinjaFilterError`. Register in `FilterDefinition` with `allowed_contexts=ExecutionContext.ALL`. Register unconditionally in `Jinja2Template._set_filters()`. Include unit tests: valid JSON dict, valid JSON list, empty string → `{}`, malformed → error, render through template. (IFC-2369) +- [ ] T010 [P] [US3] Implement `from_yaml` filter function in `infrahub_sdk/template/infrahub_filters.py` — pure sync function. Empty string → `{}` (explicit special-case since `yaml.safe_load("")` returns `None`). Malformed YAML → `JinjaFilterError`. Register in `FilterDefinition` with `allowed_contexts=ExecutionContext.ALL`. Register unconditionally in `Jinja2Template._set_filters()`. Include unit tests: valid YAML mapping, valid YAML list, empty string → `{}`, malformed → error, render through template. (IFC-2370) +- [ ] T011 [US3] Integration test for filter chaining in `tests/unit/template/test_infrahub_filters.py` — test `artifact_content | from_json` and `artifact_content | from_yaml` end-to-end with mocked ObjectStore returning JSON/YAML content. Verify template can access parsed fields. (IFC-2376 partial, SC-006) + +**Checkpoint**: US3 complete. All 4 filters work, chain correctly, and are validated per context. + +--- + +## Phase 5: Polish & Cross-Cutting Concerns + +**Purpose**: Documentation, full regression, and server-side tasks + +- [ ] T012 Run full unit test suite (`uv run pytest tests/unit/`) and verify zero regressions (SC-005) +- [ ] T013 Run `uv run invoke format lint-code` and fix any issues +- [ ] T014 Documentation: artifact composition usage guide — create or update docs with Jinja2 filter examples, Python transform example using `client.object_store.get(identifier=storage_id)`, GraphQL query patterns, known limitations (no ordering guarantee). Run `uv run invoke docs-generate` and `uv run invoke docs-validate`. (IFC-2377) +- [ ] T015 [Infrahub server] Thread SDK client into `Jinja2Template` in `integrator.py` — pass `self.sdk` from `render_jinja2_template` as `Jinja2Template(client=...)` on Prefect workers. Integration test with composite template. (IFC-2378) +- [ ] T016 [Infrahub server] Schema validation: block new filters in computed attributes — validate with `context=ExecutionContext.CORE` at schema load time. Templates using `artifact_content`/`file_object_content` must be rejected. (IFC-2379) + +--- + +## Dependencies & Execution Order + +### Phase Dependencies + +- **Foundational (Phase 1)**: No dependencies — start immediately +- **US1 + US4 (Phase 2)**: Depends on Phase 1 completion — BLOCKS remaining stories +- **US2 (Phase 3)**: Depends on Phase 2 (uses InfrahubFilters + Jinja2Template wiring) +- **US3 (Phase 4)**: Depends on Phase 1 only (from_json/from_yaml need JinjaFilterError). Can start in parallel with Phase 2 if desired. +- **Polish (Phase 5)**: Depends on all user stories being complete + +### User Story Dependencies + +- **US4 + US1 (P1)**: Can start after Phase 1 — No dependencies on other stories. This is the MVP. +- **US2 (P2)**: Depends on US1 (reuses InfrahubFilters and Jinja2Template wiring from Phase 2) +- **US3 (P3)**: Depends on Phase 1 only for `from_json`/`from_yaml`. Chaining test (T011) depends on US1. + +### Within Each Phase + +- Tasks marked [P] can run in parallel (different files) +- Tests are included within each implementation task +- Core implementation before wiring/registration + +### Parallel Opportunities + +- T001 and T002 can run in parallel (different files: exceptions.py vs filters.py) +- T006 can run in parallel with T003/T004 (ObjectStore vs InfrahubFilters) +- T009 and T010 can run in parallel (from_json and from_yaml are independent) +- US3 Phase 4 (T009, T010) can start in parallel with Phase 2 after Phase 1 completes + +--- + +## Parallel Example: Phase 1 + +```text +# Launch both foundational tasks together (different files): +Task T001: "JinjaFilterError in infrahub_sdk/template/exceptions.py" +Task T002: "ExecutionContext + FilterDefinition in infrahub_sdk/template/filters.py" +``` + +## Parallel Example: Phase 4 + +```text +# Launch both parsing filters together (same file but independent functions): +Task T009: "from_json filter in infrahub_sdk/template/infrahub_filters.py" +Task T010: "from_yaml filter in infrahub_sdk/template/infrahub_filters.py" +``` + +--- + +## Implementation Strategy + +### MVP First (US1 + US4 Only) + +1. Complete Phase 1: Foundational (T001, T002) +2. Complete Phase 2: US1 + US4 (T003, T004, T005) +3. **STOP and VALIDATE**: artifact_content renders, validation blocks in CORE, allows in WORKER +4. This alone delivers the primary value proposition + +### Incremental Delivery + +1. Phase 1 → Foundation ready +2. Phase 2 → artifact_content + security gate → **MVP deployed** +3. Phase 3 → file_object_content extends to file objects +4. Phase 4 → from_json/from_yaml enable structured composition +5. Phase 5 → Documentation + server integration +6. Each phase adds value without breaking previous phases + +### Jira Task Mapping + +| Task | Jira | Phase | +| ---- | ---- | ----- | +| T001 | IFC-2367 | 1 | +| T002 | IFC-2368 | 1 | +| T003 | IFC-2371 | 2 | +| T004 | IFC-2372 | 2 | +| T005 | IFC-2375 + IFC-2376 (partial) | 2 | +| T006 | IFC-2373 | 3 | +| T007 | IFC-2374 | 3 | +| T008 | IFC-2375 + IFC-2376 (partial) | 3 | +| T009 | IFC-2369 | 4 | +| T010 | IFC-2370 | 4 | +| T011 | IFC-2376 (partial) | 4 | +| T012-T013 | — | 5 | +| T014 | IFC-2377 | 5 | +| T015 | IFC-2378 | 5 | +| T016 | IFC-2379 | 5 | + +--- + +## Notes + +- [P] tasks = different files, no dependencies +- [Story] label maps task to specific user story for traceability +- Tests are included in each implementation task (not separate) +- All error paths must produce actionable messages with filter name, cause, and remediation hint +- Commit after each task or logical group +- Stop at any checkpoint to validate story independently