cuda.core: retain graph node attachments as per-node CUDA user objects

[Andy-Jost]

Background

Part of #1330.

CUDA graph nodes can reference resources whose lifetimes are not managed by the node parameters themselves, including kernels, kernel-argument buffers, events, host callbacks, callback data, and memcpy/memset operands.

The current implementation stores every attachment in one mutable GraphSlotTable, then retains that table on the CUgraph as a single CUDA user object. Graph clones and executable graphs retain the same user object and therefore observe the same mutable table payload. Replacing or removing a table entry can release a resource that an existing CUgraphExec still references.

Design

Keep a node attachment table as metadata owned by GraphBox, but do not retain the table itself as a CUDA user object. Instead, each resource-bearing node has an immutable NodeAttachments bundle retained through its own CUuserObject.

Conceptually:

GraphBox
  CUgraph
  map<CUgraphNode, AttachmentRecord>

AttachmentRecord
  CUuserObject
  NodeAttachments*  # non-owning; lifetime transfers from CUuserObject to the cleanup queue

NodeAttachments
  collection of OpaqueHandle owners

Examples include:

• Kernel node: kernel handle and kernel-argument holder.
• Host node: callback owner and copied user-data owner.
• Memcpy node: source and destination owners.
• Memset node: destination owner.
• Event node: event owner.

CUDA automatically propagates these user objects when graphs are cloned, instantiated, or used for whole-graph updates. Releasing a definition's reference therefore does not invalidate clones, executable graphs, or in-flight launches that still require the bundle.

All wrappers for the same underlying CUgraph, including child and conditional graph views, must share the same attachment metadata rather than creating independent tables.

User-object destructor safety

CUDA may invoke a user-object destructor asynchronously on a shared internal thread. The destructor must not call CUDA APIs and should not block.

The destructor must not directly delete NodeAttachments. Its OpaqueHandle entries are type-erased owning references, and releasing the final reference may invoke CUDA-backed deleters such as cuMemFree, cuMemFreeAsync, cuEventDestroy, or cuLibraryUnload. Releasing a Python-backed owner may also execute arbitrary finalizer code.

Deferred cleanup mechanism

The CUDA callback transfers each intact NodeAttachments payload to a process-lifetime deferred-cleanup queue. Queue insertion must not allocate, release an OpaqueHandle, or invoke CUDA. The queue state must remain valid for as long as CUDA may invoke an outstanding user-object destructor.

One possible wakeup mechanism is Py_AddPendingCall. A proposed implementation is:

1. Intrusively push the payload onto a lock-free MPSC queue.
2. Use an atomic flag to ensure that at most one drain call is scheduled.
3. Call Py_AddPendingCall to schedule a C drain callback.
4. The drain callback atomically detaches the queued payloads and deletes them from a normal Python thread, where CUDA-backed and Python-backed deleters may run.
5. Clear the scheduled flag and reschedule if payloads arrived concurrently with the drain.
6. If scheduling fails, leave the payloads queued and retry from a later CUDA callback or safe cuda-core entry point. Never retry in a blocking loop or reclaim on CUDA's callback thread.

Py_AddPendingCall avoids introducing and shutting down a dedicated worker thread, but it has important limitations:

• Its main-thread queue is bounded (32 entries in CPython 3.14), so scheduling one pending call per payload is unsafe. cuda-core must coalesce cleanup behind one pending call.
• It returns -1 when the queue is full without setting an exception.
• CPython currently takes an internal mutex, so the call is not strictly non-blocking.
• Execution may be delayed while the main thread is outside the bytecode evaluator.
• It targets the main interpreter, requiring an explicit decision about subinterpreter support.
• Behavior must be tested on both GIL-enabled and free-threaded Python.
• Pending cleanup must stop during Python finalization; payloads that cannot be reclaimed safely are intentionally leaked.

A dedicated host worker remains an alternative if Py_AddPendingCall cannot satisfy callback latency, blocking, subinterpreter, or shutdown requirements. The implementation choice should be validated with stress tests covering more than 32 simultaneous user-object destructions, scheduling failure, concurrent enqueue/drain races, free-threaded Python, and interpreter finalization.

Attachment transactions

Provide internal operations for creating, installing, replacing, and removing immutable node attachment bundles.

Replacement follows a two-phase transaction:

1. Create the replacement bundle and retain its user object on the graph.
2. Perform the CUDA node-parameter update.
3. On success, publish the replacement in GraphBox and release the graph's old user-object reference.
4. On failure, release the replacement and leave the old record unchanged.

Node deletion releases its attachment only after cuGraphDestroyNode succeeds. Python handle invalidation and registry removal must also occur after driver success.

The public node-update APIs are out of scope for this issue, but this internal transaction support will be used by the follow-up definition-level update work.

Implementation scope

• Replace the user-object-owned GraphSlotTable in _cpp/resource_handles.*.
• Introduce immutable per-node attachment payloads and attachment records.
• Add dynamic loading and wrappers for cuGraphReleaseUserObject.
• Replace sequential graph_set_slot() calls with one attachment installation per resource-bearing node.
• Keep graph attachment metadata available across equivalent GraphHandle wrappers.
• Correct GraphNode.destroy() ordering and remove attachments after successful deletion.
• Add a process-lifetime deferred-cleanup queue for NodeAttachments payloads.
• Make the CUDA user-object destructor enqueue the intact payload without releasing any OpaqueHandle.
• Prototype and select a wakeup mechanism, using coalesced Py_AddPendingCall scheduling as the primary candidate and a dedicated host worker as the fallback.
• Define retry, initialization, and shutdown behavior, including outstanding callbacks and queued payloads during Python finalization.

Acceptance criteria

• Existing graph attachment lifetime tests continue to pass.
• An executable graph remains valid after its source node is deleted and source owners are dropped.
• A cloned graph retains attachments independently of source mutation or destruction.
• Reuse of a deleted CUgraphNode handle cannot invalidate an older executable graph.
• Failed node deletion leaves the Python node and its attachments valid.
• Nodes with multiple owners retain and release the complete attachment bundle together.
• Attachments are released after the last graph, executable graph, or in-flight launch reference disappears.
• The CUDA callback releases no OpaqueHandle and invokes no CUDA API. If Py_AddPendingCall is selected, it is the only permitted Python C API call from that callback.
• When a graph owns the final reference to a Buffer, its allocation remains valid through all dependent launches and is reclaimed from the deferred-cleanup worker.
• Final release of event, kernel/library, and Python-backed attachments occurs only from the selected deferred-cleanup context.
• Destroying an executable graph with an in-flight launch does not enqueue its attachments until CUDA determines that the launch no longer requires them.
• Stress coverage exceeds CPython's 32-entry pending-call capacity and exercises scheduling failure and concurrent enqueue/drain races.
• Deferred-cleanup shutdown cannot cause a callback to access destroyed queue state or invoke Python after finalization.

Non-goals

• Public GraphNode.update() methods.
• Individual executable-graph node updates.
• Importing metadata for graphs cloned or modified outside cuda.core.
• Changing the existing raw-handle lifetime contract.

[mdboom]

You may want to look at Py_AddPendingCall. You can provide a C callback that is guaranteed to run periodically when the GIL is held. Using that would mean you wouldn't have another thread (and manage its lifetime and that complexity). I'm not sure it's the right tool for the job, but something to experiment with against the alternatives.