Skip to content
Closed
1 change: 1 addition & 0 deletions docs/src/SUMMARY.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,3 +36,4 @@
* [Verification Model](./design/orchestrator/orchestrator-model.md)
* [State Machine](./design/orchestrator/orchestrator-machine.md)
* [Platform Architecture](./design/orchestrator/orchestrator-platform.md)
* [PLDM/Orchestrator IPC](./design/orchestrator/pldm-orchestrator-ipc.md)
193 changes: 193 additions & 0 deletions docs/src/design/orchestrator/pldm-orchestrator-ipc.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,193 @@
# PLDM/Orchestrator IPC

How the PLDM FirmwareDevice service and the orchestrator communicate during a

@rusty1968 rusty1968 Sep 9, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The firmware_device run_terminus_inner is currrently a tight loop interleaving the roles of initiator/responder - what you are assuming here is that we can fit the blocking requests to the orchestrator in the FdOps callbacks without disrupting the protocol.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed on the mechanism. handle_component, activate and
cancel_update_component all run inside the responder callback with the UA
waiting, so a transact there holds the UA's response for as long as the
orchestrator takes to answer.

FD_T1 and FD_T2 are ours to set, but the response timeout is the BMC's. Do you
have a number for what the UA tolerates? Your skeleton already carries a
bounded wait, so I assume you had one in mind.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed on the mechanism. handle_component, activate and cancel_update_component all run inside the responder callback with the UA waiting, so a transact there holds the UA's response for as long as the orchestrator takes to answer.

FD_T1 and FD_T2 are ours to set, but the response timeout is the BMC's. Do you have a number for what the UA tolerates? Your skeleton already carries a bounded wait, so I assume you had one in mind.

The timeouts were not selected by me, let's follow up with @CourtneyDrant .

firmware update. Two Pigweed kernel channels, both initiated by PLDM: a notify

@rusty1968 rusty1968 Sep 9, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Orchestrator is a supervisor of the update scenario. It is not a channel handler - it is the initiator. It will register for notifications from PLDM when it is ready to do so. Use the I2C/MCTP as a template (with the bounded wait of course which is already part of the skeleton I have laid out). PLDM should be the server/handler, and the orchestrator the client that gets notified.

Look at what we already do today - The i2c server (handler) nudges its client via object_set_peer_user_signal on an async event, and the client then issues a transact to pull the latched data.

@chrysh chrysh Sep 9, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the flipped model, how does the Accepted/Rejected verdict reach PLDM before the RequestUpdate response goes back to the UA, a second transact?

channel_transact is an initiator-only. In the flipped model you are suggesting, every exchange where PLDM needs an answer (veto, Offer, Complete, Activate, Abort) becomes: latch → nudge → orchestrator pulls event → orchestrator pushes verdict in a second transact. Two round trips per exchange, sync or async, while PLDM holds the UA's request pending. Current design: one transact.

It would make the status-push direction cleaner (Authenticating/Staging/Staged), but I am not even sure how important this information is for PLDM.

@CourtneyDrant What is your opinion?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rusty1968 In the case of I2C, the client only pulls latched bytes. In our case, it's PLDM that needs the answer, and should be the one initiating the interaction, because PLDM needs to answer the UA. PLDM would need to hold UA state across: latch, nudge, orchestrator pulls, orchestrator pushes verdict in a second transact.

Furthermore, it moves blocking into the orchestrator's loop. channel_transact blocks the caller. I wanted the orchestrator to be always responsive.

That being said, if we as a group think the PLDM being the server is the better software design, I am happy to rewrite it. But given that the request direction is always UA -> FD, it seems to me that pldm being the client asking the orchestrator feels like the more natural direction.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The flipped model works cleanly if the veto is allowed to land after the
RequestUpdate response: the orchestrator registers, gets nudged, pulls the
event, and gates whether the transfer starts. It does not work if the verdict
has to be in the response itself, because a handler cannot push to its
initiator, so PLDM would have to hold the UA's request across a nudge and a
second transact.

Which of the two do you have in mind? If it is the first, I will redraw with
the veto as a transfer gate and drop the in-response reject.

channel for pre-transfer veto, and an intake channel for control messages
(Offer, Complete, Poll, Activate, Abort) and the async effect chain. Firmware
bytes go direct to flash, never through IPC.

Design decisions:

- Activate is on the wire (ActivateFirmware from the UA), not implicit after
staging.
- Flash seam is async: poll_stage calls start_erase/start_program, returns,

@rusty1968 rusty1968 Sep 9, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The flash service API is blocking - what you are referring to here is the FlashDriver trait whose implementor resides on the flash service process.

See #365

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two questions then, because this decides whether the effect chain in this doc
works at all.

Does the flash service own the external SPI flash we stage into, or only
on-chip flash?

And is a split or deferred API on the service planned? FlashIpcClient makes
every erase, program and read a blocking transact, and FlashDriver's split
API (start_erase/is_busy/complete_op) stays inside the server process.
Without something equivalent across the boundary, the orchestrator's verifier
blocks for a full chunk read every loop pass and a staging erase blocks it for
10 to 100 ms. That is the "never wait in the handler" property the async effect
chain here is built on.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two questions then, because this decides whether the effect chain in this doc works at all.

Does the flash service own the external SPI flash we stage into, or only on-chip flash?

And is a split or deferred API on the service planned? FlashIpcClient makes every erase, program and read a blocking transact, and FlashDriver's split API (start_erase/is_busy/complete_op) stays inside the server process. Without something equivalent across the boundary, the orchestrator's verifier blocks for a full chunk read every loop pass and a staging erase blocks it for 10 to 100 ms. That is the "never wait in the handler" property the async effect chain here is built on.

It is one service per flash device, so the answer is yes count on being able to open a channel to the device server managing the spi where the image is stored. The orchestrator is not in the data path though.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you mean by not in the data path? The orchestrator will tell pldm where to put the firmware though, I assume?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you mean by not in the data path? The orchestrator will tell pldm where to put the firmware though, I assume?

The per-component staging area mapping is platform-specific - by not in the data path I meant to say the orchestrator doesperform any flash I/O.

checks is_busy on the next call and then complete_op, which is where the
operation's error surfaces. Uses FlashDriver's split API, not BlockingFlash.
- USER signal is level-triggered (verified from Pigweed kernel source): OR'd
into the peer's active_signals bitfield, persists until lowered. No lost
wakeups.
- MCTP server (separate process) buffers 4 messages while PLDM is in a
transact. Overflow drops silently, no backpressure to the bus. Recovery from
a dropped message is PLDM's, not this seam's: pldm-lib defaults FD_T1 (update
mode idle) to 120s and FD_T2 (RequestFirmwareData retry) to 5s.
- A Rejected veto becomes an error completion code in the RequestUpdate
response, ALREADY_IN_UPDATE_MODE when the reason is an update already
running; the UA retries.
- Activation reports, it does not roll back. The ActivateFirmware response
carries the spec's estimated_time and means accepted; the outcome of the SVN
bump reaches the UA as GetStatus AuxStateStatus (GenericError is the only
failure value the spec offers), and GetFirmwareParameters shows which version
is actually active. A failed activation leaves a bootable system, because
PLDM never writes the active image and staging is inert, so re-running the
update from RequestUpdate is always available. What the UA does with that is
the UA's.
- Receiving carries the staging base address, not just the total. The
orchestrator picks the region and programs the SMC write filter for it, so
the window PLDM writes through and the window the hardware allows come from
one place. PLDM holds no board layout.
- Write access is two layers: a typed StagingWindow inside PLDM (Rust, catches
offset bugs) backed by an SMC write filter PLDM cannot reprogram (catches a
compromised process). The orchestrator opens the filter on Offer and closes
it on Complete/Abort/timeout. How the filter registers are kept out of
PLDM's reach (separate MPU region, separate controller/CS, lock-until-reset)
depends on the AST10x0 register layout; see "Who owns the SPI flash
controller" in open questions.
- Transfer loop is zero-IPC: PLDM writes firmware bytes direct to flash and

@rusty1968 rusty1968 Sep 9, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is an IPC channel between the fw_update service and the flash driver binding that manages the staging partition. Perhaps the more appropriate label should be zero Orchestrator IPC?

tracks progress locally. Complete carries the byte count for the
orchestrator's coverage check (early-fail only, verify hashes the staged
image anyway). On a write error PLDM sends Abort to release staging and
reports the failure to the UA in TransferComplete's result code.

```mermaid
sequenceDiagram
participant UA as UA (BMC)<br/>remote, over MCTP
participant PLDM as PLDM FirmwareDevice<br/>single thread: run_terminus

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am thinking we should rename pldm as fw_update service. PLDM as a middleware crate is distinct from the service that uses it.

participant Orch as Orchestrator<br/>single thread: object_wait loop
participant Flash as Shared Storage<br/>ext. SPI flash

Note over UA, Orch: NOTIFY CHANNEL (pre-transfer veto)

UA->>PLDM: RequestUpdate (MCTP)
activate PLDM
PLDM->>Orch: channel_transact: Request::UpdateRequested
Note right of Orch: check state, policy
Orch-->>PLDM: Response::Accepted | Rejected
deactivate PLDM
PLDM-->>UA: RequestUpdate response (accept/reject)

Note over UA, Orch: if Accepted: INTAKE CHANNEL

activate PLDM
PLDM->>Orch: Offer { target: TargetId, total: u64 }
Note right of Orch: validate target + length,<br/>reserve staging,<br/>open the SMC write filter
Orch-->>PLDM: IntakeStatus::Receiving { base: FlashAddress, total }
deactivate PLDM

loop FD pulls chunks from UA via RequestFirmwareData
PLDM->>UA: RequestFirmwareData (MCTP)
UA-->>PLDM: firmware chunk response
PLDM-->>Flash: write firmware bytes (direct, no IPC)
Note right of PLDM: PLDM tracks its own write progress
end

activate PLDM
PLDM->>Orch: Complete { written: u64 }
Note right of Orch: check coverage,<br/>queue Pending::UpdateRequest
Orch-->>PLDM: IntakeStatus::Authenticating
deactivate PLDM

Note over UA, Flash: async: orchestrator event loop drains pending

PLDM->>UA: TransferComplete (MCTP)

Note over PLDM: PLDM FREE:<br/>services UA on MCTP<br/>MCTP responsive

Note over Orch, Flash: EFFECT CHAIN (non-blocking steps)<br/>1. poll_pending<br/>2. SM: Ready -> Updating<br/>3. poll_stage (one step)<br/>4. return to object_wait<br/>repeat 3-4 until phase done<br/>IPC responsive between steps

Orch-->>Flash: PayloadSource::read_at
Flash-->>Orch: payload bytes

Orch->>PLDM: object_set_peer_user_signal<br/>(dataless nudge, wakes WaitGroup)

loop wake on USER signal, poll status, send *Complete to UA
activate PLDM
PLDM->>Orch: Poll
Note right of Orch: read latched IntakeStatus
Orch-->>PLDM: Authenticating | Staging | Staged | Failed
deactivate PLDM
Note over PLDM, UA: when phase done:
PLDM->>UA: VerifyComplete (MCTP)
PLDM->>UA: ApplyComplete (MCTP)
Note right of PLDM: on failure: same commands<br/>with error completion code
end

UA->>PLDM: ActivateFirmware (MCTP, explicit)
activate PLDM
PLDM->>Orch: Activate
Orch-->>PLDM: IntakeStatus::Activating
deactivate PLDM
PLDM-->>UA: ActivateFirmware response (accepted, not done)
Note right of Orch: activation effect (async):<br/>bump SVN in OTP (irreversible),<br/>nudge + Poll reports Activated
UA->>PLDM: GetStatus (MCTP, until activation lands)
PLDM-->>UA: current state + AuxState

Note over UA, Orch: between Offer and Activate
UA->>PLDM: CancelUpdate (MCTP)
activate PLDM
PLDM->>Orch: Abort
Note right of Orch: in-flight flash step completes<br/>and is discarded
Orch-->>PLDM: IntakeStatus::Idle
deactivate PLDM
PLDM-->>UA: CancelUpdate response

Note over Orch: If PLDM dies mid-transfer (no Complete, no Abort),<br/>orchestrator-side timeout releases the staging reservation.<br/>The transfer loop is zero-IPC, so this bounds total transfer time:<br/>it must exceed worst-case transfer plus FD_T1 (120s),<br/>so a live PLDM always aborts first.

Note over UA, Flash: Blocking direction: always PLDM -> Orchestrator, never the reverse.<br/>Every IPC response is immediate. Effects run async via poll_stage (one step, return, repeat).<br/>PLDM stays free to service UA on MCTP. USER signal nudge replaces blind polling.<br/>FD initiates TransferComplete, VerifyComplete, ApplyComplete. UA initiates ActivateFirmware, GetStatus and CancelUpdate.
```

## Write-access containment

PLDM writes firmware bytes direct to flash (zero-IPC, see above), so write
access must be confined to the inactive slot and only for the duration of the
transfer. Two layers, each catching a different class of failure:

The first layer is a typed StagingWindow inside the PLDM process. When PLDM
receives a Receiving response it constructs the window from the base and total
it carries: a bounded handle over the staging region, capped to its length. All
writes go through the window; it translates offsets and rejects anything
outside the region. The window is dropped on Complete, Abort, or timeout, so
PLDM holds no flash handle outside an active transfer. This catches offset bugs
and use-after-transfer bugs but not a compromised process, because PLDM still
has the underlying flash mapped.

The second layer is a hardware write filter that PLDM cannot reprogram. The SMC
raises SmcInterrupt::WriteProtected on writes outside an allowed region. The
orchestrator (or a dedicated flash-service process) opens the filter for the
staging region on Offer and closes it on Complete/Abort/timeout. PLDM needs the
SMC control registers that drive erase/program commands, but must not be able
to touch the filter/write-protect registers. Whether those register sets are
separable (distinct MPU pages, separate controller/CS, or lock-until-reset
bits) depends on the AST10x0 register layout and is folded into the "who owns
the SPI flash controller" open question below.

The net effect: bugs hit the Rust window check, a compromised process hits the
hardware filter, and both "inactive slot only" and "only during an update" are
enforced. Even a fully rogue PLDM can at worst corrupt the staging area and
fail verify; the active image is never written by PLDM at any point in the
flow, and activation is orchestrator-side metadata plus the SVN bump in OTP.

## Open questions

Who owns the SPI flash controller. The diagram has PLDM writing the staging
region and the orchestrator reading it, but FlashDriver takes `&mut self` and
says nothing about multiple clients. Either each process drives its own
controller over disjoint regions, or one process owns the driver and the other
reaches it over IPC. Sequencing keeps the two off the same bytes at the same
time (the orchestrator reads only after Complete), so this is about the driver
and the controller, not about the protocol. A related constraint from the
write-access containment section: PLDM needs the erase/program control
registers but must not reach the write-protect/filter registers. Whether those
register sets fall on separate MPU pages on the AST10x0 (datasheet needed)
determines whether pw_kernel can enforce the split, or whether a dedicated
flash-service process must own the entire SMC and proxy writes.

How the orchestrator learns that PLDM died, short of the timeout. Abort is a
message a live PLDM sends; the timeout covers the case where it can send
nothing. pw_kernel has no peer-closed signal: the set is READABLE, WRITEABLE,
ERROR, JOINABLE, USER and the interrupt bits. The one existing path is
ChannelInitiatorObject::reset, which raises ERROR on the handler, and only if a
transaction was in flight and only once someone joins the dead process. During
the zero-IPC transfer loop no transaction is in flight, so the orchestrator
sees nothing. Replacing the timeout means the orchestrator waits on JOINABLE on
PLDM's process object, or the supervisor that joins PLDM tells it. That is a
supervisor question, not a channel one.