[Feature]: Workflow Functions: custom filters for the workflow expression language #5285

Updated after discussion with @mnriem: the proposal is now independently packaged Workflow Functions, used as pipe filters first. The earlier options A–C are kept below under "Alternatives Considered".

Author's scope update: bounded multi-argument filters are now included in this proposal. This expands the earlier zero/one-argument scope and is pending maintainer confirmation.

Problem Statement

The workflow expression language ships a fixed set of pipe filters: default, join, map, contains, split, from_json, upper, lower, length and to_json. These are defined in FILTER_TABLE in src/specify_cli/workflows/expressions/filters.py. There is no way for a project or a community package to add one. A missing filter, such as trim, forces authors into a shell step plus from_json, or into a custom step, for a one-line transformation.

Proposed Solution

Add Workflow Functions: independently packaged Python callables, installed per project and usable as pipe filters in expressions. Other uses (for example direct calls such as trim(x)) are added only when justified. This is not a new value type in the expression language.

Package layout

.specify/workflows/functions/text/
├── function.yml
└── __init__.py

id: text
functions:
  - name: trim
    handler: trim
    min_args: 0
    max_args: 0

def trim(value: str) -> str:
    return value.strip()

Result: {{ steps.capture.output.stdout | trim }}.

Arguments: each function declares inclusive min_args / max_args bounds: non-negative integers, both defaulting to 0, with min_args <= max_args. Bounds count only the explicit arguments after the piped value; the handler is called as handler(value, *args). The bare form is allowed when min_args == 0. Parenthesized calls must contain at least one argument and satisfy the bounds; trim() is not an accepted zero-argument form. For example, a function declaring min_args: 2 and max_args: 2 can be used as {{ inputs.text | pad(12, '0') }}. Optional and bounded argument ranges are supported; unbounded variadic arguments remain out of scope.

Install: specify workflow function add <package>.

Why this fits the engine: the grammar already accepts any FILTER_NAME, so the parser needs no change. The dispatcher (_apply_filter) and the static validation (_check_filter / _evaluator_rejects) both read only FILTER_TABLE, so a registered function is validated and executed consistently.

Engine changes

1. Add a project-scoped registry with registration, collision checks against built-in filters and other packages, a BUILTIN_* snapshot, and a reset between projects (like _unload_custom_steps()).
2. Load the registry before validate_workflow and before execution. Otherwise validation reports "unknown filter".
3. Compute _EXPECTED_FILTERS dynamically. It is currently built once at import and would omit custom functions from error messages.
4. Wrap handler exceptions into ValueError("filter '<name>': ..."). Built-ins only raise ValueError, and the engine does not catch anything else.
5. Replace FilterSpec.argument / bare with the manifest's min_args / max_args bounds. Support zero, optional, single, and multiple explicit arguments through the same validation and evaluation path. Reuse the existing argument-list grammar; preserve built-in filter behavior.
6. Harden the loader as for steps (symlink guards, symlinked parent directories).

Alternatives Considered

• Filters inside step packages (earlier option B): reuses delivery, catalogs and the trust model, but a filter would have to live in a step package.
• A custom step per transformation (earlier option C): works today, but is verbose and not usable inline in conditions or interpolation.
• Declarative aliases in workflow YAML, such as filters: {csv: "map('x') | join(',')"}: no Python and less trust risk, but cannot express new logic like trim.
• Direct calls (trim(x)) and further hooks (input validators, gate verdicts): deferred. They need a grammar change and a policy for impure functions.
• Adding the missing built-ins one by one: cheap, but never ends.

Use Cases

1. Normalise captured shell output ({{ steps.x.output.stdout | trim }}). Today echo false yields "false\n", which already needed a workaround in evaluate_condition.
2. Team-specific formatting (branch slugs, ticket IDs) in command and prompt inputs without extra steps.
3. Reusable function packages shared between projects.

Acceptance Criteria

• A registered function works as a filter in {{ }} interpolation, in single-expression templates, and in conditions (if, while, switch).
• Manifest argument bounds are validated. Bare, optional, single-argument, and multi-argument forms work within those bounds; validation and evaluation consistently reject too few/many arguments and empty parenthesized calls.
• validate_workflow accepts a registered function and reports an unknown one, with the custom function listed in the "expected one of ..." message.
• A name collision with a built-in filter or another package is rejected, and the built-in is unchanged.
• A non-ValueError exception from a handler surfaces as ValueError naming the filter, not as a crash.
• Functions from project A are not visible in project B in the same process.
• specify workflow function add/remove/list manage packages.
• Positive and negative tests per CONTRIBUTING.md#testing-deterministic-behavior, and docs in docs/workflows/expressions.md and design/workflow-step.md.
• Docs state the trust model (loading executes Python code) and the resume behaviour: completed steps are replayed from recorded results, while an unfinished step's expressions are evaluated again when it is retried (at-least-once). Handlers should therefore be deterministic and free of side effects.

Additional Context

Rough estimate: several days including the loader, installer and CLI, the function.yml schema, tests and docs. A catalog is not part of the first iteration.

AI Disclosure

Original draft: Pi Coding Agent (github-copilot provider), model claude-sonnet-5.5, reasoning effort high, human-supervised. Used for reading the code, drafting this issue text and revising it after maintainer feedback; reviewed by the author before posting.

Multi-argument scope update: authored and applied on behalf of @markuswondrak by Pi Coding Agent (github-copilot provider), model gpt-6-astra, reasoning effort xhigh, user-directed interactive mode. AI assistance covered the argument contract, example manifest, engine-change description, and acceptance criterion in this issue-body update.