feat(workflows): workflow functions - project-installed custom pipe filters

design/workflow-step.md

@@ -82,6 +82,66 @@ The registry holds one shared instance per type. Concurrent `fan-out` can
 invoke that instance from multiple threads: keep execution stateless and
 thread-safe, with per-run data in `config` and `context`, not on `self`.
 
+## Workflow functions
+
+Workflow functions are a separate project-installed extension point for
+expression pipe filters. A package under
+`.specify/workflows/functions/<id>/` contains `function.yml` and `__init__.py`;
+the manifest selects callable attributes explicitly and declares argument
+bounds. Function names share a flat namespace with the immutable built-in
+filter snapshot and other loaded packages. Built-in and duplicate exports are
+installation and load errors; built-ins are never mutated.
+
+The function loader clears the previous project's custom filter registrations,
+validates all manifests and collisions before importing code, then imports each
+package under a synthetic package name so relative imports work. A package's
+exports are registered all-or-nothing and failed imports are cleaned from
+`sys.modules`. Installation validates and stages files without importing them;
+run, resume, workflow add, and workflow update load functions before workflow
+validation. Loading executable Python is not a sandbox or trust endorsement.
+
+Function handlers are wrapped at registration to normalize ordinary
+`Exception` failures to a `ValueError` prefixed with the export name. This is
+deliberately function-specific; `BaseException` is not caught and step
+implementations do not gain a new recovery policy. Return values continue
+through the existing expression and persistence boundaries without a separate
+type whitelist. The process-global filter table is read by concurrent fan-out
+work, so handlers must be stateless and thread-safe.
+
+## Shared package installation
+
+Step packages and function packages share their install, registry, URL and
+removal mechanics instead of each owning a copy:
+
+- `workflows/_package_install.py` owns path and symlink safety, tree walking and
+  copying (no-follow, excludes, entry/byte/depth budgets), the project lock,
+  publish-with-rollback, removal, and one `install_package` flow: reject
+  conflicts, validate the source before creating directories, stage, re-validate
+  the staged copy, then re-run every conflict check under the lock immediately
+  before publishing. It is parameterized by small value objects:
+  `InstallRequest` (what to install), `PackageSpec` (naming, error type and
+  limits; directory, lock and staging names derive from the noun),
+  `PackageRules` (the kind's locate/registry/validate/check/entry semantics) and
+  `StagingOps` (copy, publish, lock; `StagingOps.standard(spec)` by default).
+  Removal uses `RemoveRequest` and `RemoveHooks`.
+- `workflows/_package_registry.py` provides `PackageRegistry`, the symlink-safe,
+  atomically written JSON registry with rollback of in-memory state on a failed
+  write. `StepRegistry` and `FunctionRegistry` only name their file, directory,
+  record key and error type.
+- `workflows/_archive_install.py` is the CLI flow for `--from`: URL validation,
+  the untrusted-source prompt, GitHub asset resolution, download, archive
+  format agreement, safe extraction and cleanup. `_commands._print_package_error`
+  renders failures, including advisory notes as warnings.
+
+A new package kind supplies a `PackageSpec`, `PackageRules`, a registry subclass
+and its manifest parser; it should not copy any of the above. The step
+installer keeps its established names (`_copy_package_tree`, `_replace_install`,
+`_step_install_transaction`, ...) as thin bindings and builds its spec at call
+time from its module limits, so existing patch points and messages are stable.
+Function-specific additions stay in `function/`: manifest parsing, export-name
+checks against built-ins, the registry and the package directories on disk, and
+registry metadata.
+
 ## Adding a step type
 
 1. Implement the step in its own `step/<package>/` subpackage and register

docs/toc.yml

@@ -137,6 +137,8 @@
       href: workflows/api.md
     - name: Expressions
       href: workflows/expressions.md
+    - name: Workflow Functions
+      href: workflows/functions.md
     - name: Overlays
       href: workflows/overlays.md
     - name: Catalogs

docs/workflows/cli.md

@@ -188,6 +188,20 @@ archives follow the same validation and installation behavior.
 
 For exact-version catalog selection and its digest requirements, see [Catalog releases](catalogs.md#workflow-releases).
 
+## Workflow Functions
+
+```bash
+specify workflow function add <id> --dev <directory>
+specify workflow function add <id> --from <archive-url>
+specify workflow function list [--json]
+specify workflow function remove <id>
+```
+
+Exactly one source option is required for `add`. Function packages are Python
+code and are loaded before workflow validation; install only packages you have
+reviewed. `list --json` reports registry metadata and generic export usage
+without importing package code.
+
 ## Update Workflows
 
 ```bash

docs/workflows/expressions.md

@@ -13,6 +13,32 @@ Steps can reference inputs and previous step outputs using `{{ expression }}` sy
 
 Available filters: `default`, `join`, `contains`, `map`, `from_json`, `to_json`, `upper`, `lower`, `split`, `length`.
 
+## Custom filters (Workflow Functions)
+
+Projects can install Python callables as pipe filters. See
+[Workflow functions](functions.md) for package and CLI details.
+
+```yaml
+# After installing a package that exports trim:
+message: "{{ inputs.message | trim | upper }}"
+```
+
+A function receives the piped value followed by the explicit filter
+arguments. Manifest `min_args` and `max_args` bounds are checked during
+validation and evaluation. Bare form is available for zero-minimum functions;
+parenthesized calls contain one or more arguments. Function names share a flat
+namespace with built-in filters and other installed packages. Collisions are
+errors and built-ins are never shadowed.
+
+Function packages contain executable Python and are imported before workflow
+validation. Review package source before installing it. Handlers should be
+stateless and thread-safe because fan-out can evaluate filters concurrently.
+Function failures use the normal expression-error path, with the function name
+prefixed to the exception; return values use the existing expression consumer
+rules rather than a separate function-specific type contract. As with other
+workflow side effects, an interruption before a resume checkpoint can repeat a
+handler, so resume is at-least-once.
+
 ## Syntax
 
 | Form | Example |

docs/workflows/functions.md

@@ -0,0 +1,117 @@
+# Workflow functions
+
+Workflow functions are project-installed Python callables exposed as pipe
+filters in workflow expressions. They are installed under
+`.specify/workflows/functions/<id>/` and consist of a `function.yml` manifest
+and an `__init__.py` module.
+
+## Install and manage functions
+
+```bash
+specify workflow function add text --dev ./workflow-functions/text
+specify workflow function list
+specify workflow function list --json
+specify workflow function remove text
+```
+
+A package can also be installed from an archive URL:
+
+```bash
+specify workflow function add text --from https://example.test/text.zip
+```
+
+Exactly one of `--dev` and `--from` is required. `--force` replaces an existing
+package with the same package ID; it cannot replace a function export owned by
+another package.
+
+Installation follows the same rules as custom step packages: files are validated
+and copied through a private staging directory, symlinks and unsupported file
+types are rejected, `.git`, `__pycache__` and `.DS_Store` are not copied, and a
+package may contain at most 512 entries, 50 MiB and 32 directory levels. A
+`--from` install shows an untrusted-source prompt with the URL before any
+request, resolves GitHub release asset URLs, and requires the URL, final URL and
+`Content-Type` to agree on the archive format.
+
+`remove` deletes the package directory and its registry entry under the same
+project lock as installation. A directory with no registry entry is removed with
+a warning; if the directory cannot be deleted, the registry entry is restored.
+
+`list` reads the persisted registry only. It does not import package code and
+reports each export's generic usage form. A package that was copied into
+`.specify/workflows/functions/` by hand is loaded when a workflow runs but is not
+listed, because it has no registry entry.
+
+## Manifest
+
+```yaml
+id: text
+name: Text helpers
+version: 1.0.0
+description: Small text transformations.
+functions:
+  - name: trim
+    handler: trim
+    min_args: 0
+    max_args: 0
+    description: Remove surrounding whitespace.
+```
+
+Package IDs use the same safe path-component validation as custom workflow
+steps. Export names are lowercase identifiers (`a-z`, `0-9`, and `_`) and must
+be unique in a package. `min_args` and `max_args` are integers with
+`0 <= min_args <= max_args <= 16`; they count arguments after the value being
+piped. The defaults are zero. A
+bare filter is accepted only when `min_args` is zero. Parenthesized calls must
+contain at least one argument and must be within the declared bounds.
+
+The handler is called as `handler(value, *args)`. For example:
+
+```python
+
+def trim(value):
+    return value.strip()
+
+
+def surround(value, left, right):
+    return f"{left}{value}{right}"
+```
+
+```yaml
+- id: normalize
+  type: shell
+  strict: true
+  env:
+    VALUE: "{{ inputs.value | trim }}"
+  run: 'printf "%s\\n" "$VALUE"'
+```
+
+## Loading and trust
+
+Installing validates and copies package files but does not import them. Running,
+resuming, adding, or updating a workflow loads installed function modules before
+workflow validation. Loading executes Python code, so review packages and
+install only from sources you trust. Archive downloads use the same HTTPS and
+loopback-HTTP restrictions as other executable workflow packages.
+
+Function names share a flat namespace with built-in and other installed
+filters. Installation rejects a name that is a built-in filter or is already
+provided by another package, whether that package is in the registry or only
+present on disk (the error says "found on disk, not in the registry" in that
+case). Runtime loading also fails on a collision, naming every colliding
+function and both sources, if packages were copied manually or a new Spec Kit
+release introduces a conflicting built-in. The collision check happens before
+any package code is imported. Built-in filters are never replaced. If loading
+fails because a directory or file cannot be read, the command reports the error
+and exits with code 1 (a failed payload with `--json`).
+
+The filter registry is process-global while a workflow is loaded. A project
+reload replaces the previous project's custom filters. Function handlers must
+be stateless and thread-safe because fan-out steps can evaluate expressions
+concurrently. As with other workflow side effects, an interruption before a
+resume checkpoint can cause an operation to run again; resume provides
+at-least-once rather than exactly-once execution.
+
+Custom return values use the existing expression and step-output handling; no
+additional function-specific type or JSON whitelist is applied. Exceptions
+raised by a handler are normalized as `ValueError` messages prefixed with the
+function name, while `BaseException` subclasses are not intercepted.

src/specify_cli/workflows/__init__.py

@@ -87,18 +87,47 @@ def _register_builtin_steps() -> None:
 _CUSTOM_STEP_MODULES: set[str] = set()
 
 
-def _unload_custom_steps() -> None:
-    """Clear custom registrations and synthetic imports from a prior project."""
+def _import_package(module_name: str, package_dir: Path):
+    """Import an installed package under an isolated synthetic module name."""
+    import importlib.util
     import sys
 
-    for type_key in tuple(STEP_REGISTRY):
-        if type_key not in BUILTIN_STEP_TYPES:
-            del STEP_REGISTRY[type_key]
-    for module_name in _CUSTOM_STEP_MODULES:
+    init_py = Path(package_dir) / "__init__.py"
+    spec = importlib.util.spec_from_file_location(
+        module_name,
+        init_py,
+        submodule_search_locations=[str(package_dir)],
+    )
+    if spec is None or spec.loader is None:
+        raise ImportError(f"Could not create an import specification for '{package_dir}'")
+    module = importlib.util.module_from_spec(spec)
+    module.__package__ = module_name
+    sys.modules[module_name] = module
+    try:
+        spec.loader.exec_module(module)  # type: ignore[union-attr]
+    except BaseException:
+        _drop_modules({module_name})
+        raise
+    return module
+
+
+def _drop_modules(module_names: set[str] | list[str] | tuple[str, ...]) -> None:
+    """Remove synthetic modules and their relative-import children."""
+    import sys
+
+    for module_name in module_names:
         sys.modules.pop(module_name, None)
         prefix = module_name + "."
         for loaded_name in [name for name in sys.modules if name.startswith(prefix)]:
             sys.modules.pop(loaded_name, None)
+
+
+def _unload_custom_steps() -> None:
+    """Clear custom registrations and synthetic imports from a prior project."""
+    for type_key in tuple(STEP_REGISTRY):
+        if type_key not in BUILTIN_STEP_TYPES:
+            del STEP_REGISTRY[type_key]
+    _drop_modules(_CUSTOM_STEP_MODULES)
     _CUSTOM_STEP_MODULES.clear()
 
 
@@ -113,9 +142,7 @@ def load_custom_steps(project_root: Path) -> list[str]:
     Silently skips packages that fail to import or validate.
     """
     import hashlib as _hashlib
-    import importlib.util as _importlib_util
     import re as _re
-    import sys as _sys
 
     _unload_custom_steps()
     steps_dir = Path(project_root) / ".specify" / "workflows" / "steps"
@@ -174,21 +201,9 @@ def load_custom_steps(project_root: Path) -> list[str]:
 
             # Treat the step directory as a proper package so that relative
             # imports inside the step (e.g. ``from .helpers import …``) work.
-            spec = _importlib_util.spec_from_file_location(
-                module_name,
-                init_py,
-                submodule_search_locations=[str(step_dir)],
-            )
-            if spec is None or spec.loader is None:
-                continue
-            module = _importlib_util.module_from_spec(spec)
-            module.__package__ = module_name
-            # Register before exec so relative imports resolve correctly.
-            _sys.modules[module_name] = module
+            module = _import_package(module_name, step_dir)
             registered = False
             try:
-                spec.loader.exec_module(module)  # type: ignore[union-attr]
-
                 # Find the StepBase subclass in the module
                 from .base import StepBase as _StepBase
 
@@ -222,14 +237,16 @@ def load_custom_steps(project_root: Path) -> list[str]:
                 # a broken/skipped step package leaves no lingering import state
                 # behind.
                 if not registered:
-                    _sys.modules.pop(module_name, None)
-                    submodule_prefix = module_name + "."
-                    for _mod_key in [
-                        k for k in _sys.modules if k.startswith(submodule_prefix)
-                    ]:
-                        _sys.modules.pop(_mod_key, None)
+                    _drop_modules({module_name})
         except Exception:  # noqa: BLE001, S112
             # Silently skip broken step packages at load time
             continue
 
     return loaded
+
+
+def load_workflow_functions(project_root: Path) -> list[str]:
+    """Load project-installed workflow expression functions."""
+    from .function.loader import load_workflow_functions as loader
+
+    return loader(project_root)