Surface durable execution badge in Airflow Registry

[amoghrajesh]

---

Was generative AI tooling used to co-author this PR?

• Yes (please specify the tool below)

---

closes: #67236

What

Through this PR, I am trying to surface which operators can survive a worker crash/retry (aka durable execution) without redoing expensive work on Airflow Registry. ResumableJobMixin already provides this capability (Spark, Databricks, so far, Snowflake and other operators in progress), but nothing in the registry's extraction pipeline or UI could distinguish an operator that genuinely implements it from one that merely inherits it, and the Registry itself (registry/ + dev/registry/, in this repo) had no capability concept at all, only type classification (operator/hook/sensor/etc).

Current behaviour

Today, browsing a provider page in the Registry shows every operator/hook/sensor with no indication of crash-recovery support. Inheriting ResumableJobMixin is not proof an operator is crash safe on its own. A subclass can leave the mixin's methods unimplemented, implement them but never call execute_resumable() from execute() (dead capability), or, as in SparkSubmitOperator, only reach it under specific hook configuration.

Proposed change

The change is a 3 step process

1. Detection (dev/registry/extract_parameters.py): is_durable_capable() checks three things: the class inherits ResumableJobMixin, none of its abstract methods are left unimplemented (via inspect.isabstract(), now that the mixin's 6 crash-recovery methods are real @abstractmethods as of Making ResumableJobMixin a abstract class and its methods abstractmethods #69607, already merged), and execute_resumable is reachable from execute()'s source. load_resumable_job_mixin() imports the mixin once and is threaded down through discover_classes_from_provider() and make_entry(), mirroring the existing base_classes injection pattern.
2. Schema (registry_contract_models.py): supports_durable_execution: bool = False added to ModuleContract.
3. Frontend (registry/src/provider-version.njk, main.css): a small teal "Durable" badge on module cards when supports_durable_execution is true, with a tooltip pointing at the caveats (durable=True default, config-dependent, no per-run guarantee).

User implications / backcompat

New field defaults to False; no schema break. Note ModuleContract validation uses exclude_unset semantics: the field is simply absent (not present-and-false) on payloads that never set it, e.g. legacy per-version data from older extraction runs.

Testing

Non UI surface testing

1. Tested if the extract-data works fine for databricks as well as spark:

Spark:

breeze registry extract-data --provider apache-spark

Result:

   {
      "id": "apache-spark-spark_submit-SparkSubmitOperator",
      "name": "SparkSubmitOperator",
      "type": "operator",
      "import_path": "airflow.providers.apache.spark.operators.spark_submit.SparkSubmitOperator",
      "module_path": "airflow.providers.apache.spark.operators.spark_submit",
      "short_description": "Wrap the spark-submit binary to kick off a spark-submit job; requires \"spark-submit\" binary in the PATH.",
      "docs_url": "https://airflow.apache.org/docs/apache-airflow-providers-apache-spark/stable/_api/airflow/providers/apache/spark/operators/spark_submit/index.html#airflow.providers.apache.spark.operators.spark_submit.SparkSubmitOperator",
      "source_url": "https://github.com/apache/airflow/blob/providers-apache-spark/6.2.0/providers/apache/spark/src/airflow/providers/apache/spark/operators/spark_submit.py#L69",
      "category": "apache-spark",
      "provider_id": "apache-spark",
      "provider_name": "Apache Spark",
      "supports_durable_execution": true
    },

Databricks:

breeze registry extract-data --provider databricks

Output:

Databricks

   {
      "id": "databricks-databricks-DatabricksRunNowOperator",
      "name": "DatabricksRunNowOperator",
      "type": "operator",
      "import_path": "airflow.providers.databricks.operators.databricks.DatabricksRunNowOperator",
      "module_path": "airflow.providers.databricks.operators.databricks",
      "short_description": "Runs an existing Spark job run to Databricks using the api/2.2/jobs/run-now API endpoint.",
      "docs_url": "https://airflow.apache.org/docs/apache-airflow-providers-databricks/stable/_api/airflow/providers/databricks/operators/databricks/index.html#airflow.providers.databricks.operators.databricks.DatabricksRunNowOperator",
      "source_url": "https://github.com/apache/airflow/blob/providers-databricks/7.17.0/providers/databricks/src/airflow/providers/databricks/operators/databricks.py#L983",
      "category": "databricks",
      "provider_id": "databricks",
      "provider_name": "Databricks",
      "supports_durable_execution": true
    },
    {
      "id": "databricks-databricks-DatabricksSubmitRunOperator",
      "name": "DatabricksSubmitRunOperator",
      "type": "operator",
      "import_path": "airflow.providers.databricks.operators.databricks.DatabricksSubmitRunOperator",
      "module_path": "airflow.providers.databricks.operators.databricks",
      "short_description": "Submits a Spark job run to Databricks using the api/2.2/jobs/runs/submit API endpoint.",
      "docs_url": "https://airflow.apache.org/docs/apache-airflow-providers-databricks/stable/_api/airflow/providers/databricks/operators/databricks/index.html#airflow.providers.databricks.operators.databricks.DatabricksSubmitRunOperator",
      "source_url": "https://github.com/apache/airflow/blob/providers-databricks/7.17.0/providers/databricks/src/airflow/providers/databricks/operators/databricks.py#L559",
      "category": "databricks",
      "provider_id": "databricks",
      "provider_name": "Databricks",
      "supports_durable_execution": true
    },

UI Testing

Build the UI with pnpm install, pnpm dev and visit the registry page locally buillt.

This is how it looks for spark:

What's next

• A registry wide (cross-provider) durable-execution filter was considered and dropped for now in favor of just the badge; revisit once more providers adopt the mixin.

---

• Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
• For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
• When adding dependency, check compliance with the ASF 3rd Party License Policy.
• For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

[github-actions]

Backport successfully created: v3-3-test

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
✅	v3-3-test