Cancel the Redshift statement when a user kills the deferred task

[steveahnahn]

Supersedes #69631 (closed during a verification rework; GitHub cannot reopen a PR after its branch is rebased).

Problem

RedshiftDataOperator.on_kill cancels the running statement, but once a deferrable run enters the triggerer that method can no longer run: the worker process exits at defer(). When a user then marks the deferred task failed, clears it, or marks it success, RedshiftDataTrigger had no on_kill hook (the BaseTrigger default is a no op), so the statement kept executing against the cluster or workgroup. The operator's documented kill semantics silently stop applying the moment the task defers. On Redshift Serverless the orphaned statement keeps billing RPU hours, and on a provisioned cluster it holds a WLM slot, until it finishes on its own.

Sibling deferrable triggers already cancel their remote work on user kill: EMR in this provider, and BigQuery, Dataproc and Dataflow in the Google provider.

Change

RedshiftDataTrigger gains an on_kill that cancels the running statement through the hook's async connection (cancel_statement). A cancel_on_kill flag (default True) is added to both RedshiftDataOperator and RedshiftDataTrigger so users can opt out, and the operator threads it into the trigger at the defer site. Cancellation is best effort: failures while cancelling are logged and swallowed so trigger teardown is not blocked.

Live verification

Verified in a running Airflow instance (airflow standalone: real scheduler, triggerer and API server) against a live redshift-data API endpoint served by moto, with no mocks anywhere in the execution path:

1. A deferrable RedshiftDataOperator submitted a statement (ExecuteStatement over HTTP, status STARTED) and the task reached deferred, with the trigger polling the statement from the triggerer's event loop through the real aiobotocore client.

2. Marking the deferred task failed from the UI drove the user action cancel path. Triggerer log:

   Trigger cancelled by user action, invoking on_kill
   Cancelling Redshift statement 93e26017-b365-4c9f-9c58-9d387af5e6ea.
   Redshift statement 93e26017-b365-4c9f-9c58-9d387af5e6ea cancelled.
   

   No exception escaped on_kill.

3. DescribeStatement afterwards returned Status: ABORTED, and without the change the same kill leaves the statement STARTED (the base on_kill is a no op, so nothing contacts the API).

This exercises the full deferred kill chain end to end: trigger serialization round trip, on_kill invocation by the triggerer, async connection creation inside the event loop, and the CancelStatement HTTP call. What it does not cover is AWS side cancellation semantics; the trigger issues the same cancel_statement call the operator's existing non deferred on_kill already ships and the unit tests cover.

Tests

Trigger tests cover cancel on kill, the disabled and missing statement id no ops, best effort error swallowing, and serialization of the new flag. Operator tests cover the cancel_on_kill=False guard and that the flag is threaded into the trigger on defer. Every new assertion fails without the source change.

---

Was generative AI tooling used to co-author this PR?

• Yes, Claude Code (Fable 5)

Generated-by: Claude Code (Fable 5) following the guidelines