-
Notifications
You must be signed in to change notification settings - Fork 28
docs: PLDM/orchestrator IPC design #458
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
f77bad7
e1a0d23
185b384
a3a2113
69eb02e
3ce2768
5682e71
63dcb7b
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,193 @@ | ||
| # PLDM/Orchestrator IPC | ||
|
|
||
| How the PLDM FirmwareDevice service and the orchestrator communicate during a | ||
| firmware update. Two Pigweed kernel channels, both initiated by PLDM: a notify | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The Orchestrator is a supervisor of the update scenario. It is not a channel handler - it is the initiator. It will register for notifications from PLDM when it is ready to do so. Use the I2C/MCTP as a template (with the bounded wait of course which is already part of the skeleton I have laid out). PLDM should be the server/handler, and the orchestrator the client that gets notified. Look at what we already do today - The i2c server (handler) nudges its client via object_set_peer_user_signal on an async event, and the client then issues a transact to pull the latched data.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. In the flipped model, how does the Accepted/Rejected verdict reach PLDM before the RequestUpdate response goes back to the UA, a second transact? channel_transact is an initiator-only. In the flipped model you are suggesting, every exchange where PLDM needs an answer (veto, Offer, Complete, Activate, Abort) becomes: latch → nudge → orchestrator pulls event → orchestrator pushes verdict in a second transact. Two round trips per exchange, sync or async, while PLDM holds the UA's request pending. Current design: one transact. It would make the status-push direction cleaner (Authenticating/Staging/Staged), but I am not even sure how important this information is for PLDM. @CourtneyDrant What is your opinion?
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @rusty1968 In the case of I2C, the client only pulls latched bytes. In our case, it's PLDM that needs the answer, and should be the one initiating the interaction, because PLDM needs to answer the UA. PLDM would need to hold UA state across: latch, nudge, orchestrator pulls, orchestrator pushes verdict in a second transact. Furthermore, it moves blocking into the orchestrator's loop. channel_transact blocks the caller. I wanted the orchestrator to be always responsive. That being said, if we as a group think the PLDM being the server is the better software design, I am happy to rewrite it. But given that the request direction is always UA -> FD, it seems to me that pldm being the client asking the orchestrator feels like the more natural direction.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The flipped model works cleanly if the veto is allowed to land after the Which of the two do you have in mind? If it is the first, I will redraw with |
||
| channel for pre-transfer veto, and an intake channel for control messages | ||
| (Offer, Complete, Poll, Activate, Abort) and the async effect chain. Firmware | ||
| bytes go direct to flash, never through IPC. | ||
|
|
||
| Design decisions: | ||
|
|
||
| - Activate is on the wire (ActivateFirmware from the UA), not implicit after | ||
| staging. | ||
| - Flash seam is async: poll_stage calls start_erase/start_program, returns, | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The flash service API is blocking - what you are referring to here is the FlashDriver trait whose implementor resides on the flash service process. See #365
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Two questions then, because this decides whether the effect chain in this doc Does the flash service own the external SPI flash we stage into, or only And is a split or deferred API on the service planned?
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
It is one service per flash device, so the answer is yes count on being able to open a channel to the device server managing the spi where the image is stored. The orchestrator is not in the data path though.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. What do you mean by not in the data path? The orchestrator will tell pldm where to put the firmware though, I assume?
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
The per-component staging area mapping is platform-specific - by not in the data path I meant to say the orchestrator doesperform any flash I/O. |
||
| checks is_busy on the next call and then complete_op, which is where the | ||
| operation's error surfaces. Uses FlashDriver's split API, not BlockingFlash. | ||
| - USER signal is level-triggered (verified from Pigweed kernel source): OR'd | ||
| into the peer's active_signals bitfield, persists until lowered. No lost | ||
| wakeups. | ||
| - MCTP server (separate process) buffers 4 messages while PLDM is in a | ||
| transact. Overflow drops silently, no backpressure to the bus. Recovery from | ||
| a dropped message is PLDM's, not this seam's: pldm-lib defaults FD_T1 (update | ||
| mode idle) to 120s and FD_T2 (RequestFirmwareData retry) to 5s. | ||
| - A Rejected veto becomes an error completion code in the RequestUpdate | ||
| response, ALREADY_IN_UPDATE_MODE when the reason is an update already | ||
| running; the UA retries. | ||
| - Activation reports, it does not roll back. The ActivateFirmware response | ||
| carries the spec's estimated_time and means accepted; the outcome of the SVN | ||
| bump reaches the UA as GetStatus AuxStateStatus (GenericError is the only | ||
| failure value the spec offers), and GetFirmwareParameters shows which version | ||
| is actually active. A failed activation leaves a bootable system, because | ||
| PLDM never writes the active image and staging is inert, so re-running the | ||
| update from RequestUpdate is always available. What the UA does with that is | ||
| the UA's. | ||
| - Receiving carries the staging base address, not just the total. The | ||
| orchestrator picks the region and programs the SMC write filter for it, so | ||
| the window PLDM writes through and the window the hardware allows come from | ||
| one place. PLDM holds no board layout. | ||
| - Write access is two layers: a typed StagingWindow inside PLDM (Rust, catches | ||
| offset bugs) backed by an SMC write filter PLDM cannot reprogram (catches a | ||
| compromised process). The orchestrator opens the filter on Offer and closes | ||
| it on Complete/Abort/timeout. How the filter registers are kept out of | ||
| PLDM's reach (separate MPU region, separate controller/CS, lock-until-reset) | ||
| depends on the AST10x0 register layout; see "Who owns the SPI flash | ||
| controller" in open questions. | ||
| - Transfer loop is zero-IPC: PLDM writes firmware bytes direct to flash and | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. There is an IPC channel between the fw_update service and the flash driver binding that manages the staging partition. Perhaps the more appropriate label should be zero Orchestrator IPC? |
||
| tracks progress locally. Complete carries the byte count for the | ||
| orchestrator's coverage check (early-fail only, verify hashes the staged | ||
| image anyway). On a write error PLDM sends Abort to release staging and | ||
| reports the failure to the UA in TransferComplete's result code. | ||
|
|
||
| ```mermaid | ||
| sequenceDiagram | ||
| participant UA as UA (BMC)<br/>remote, over MCTP | ||
| participant PLDM as PLDM FirmwareDevice<br/>single thread: run_terminus | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I am thinking we should rename pldm as fw_update service. PLDM as a middleware crate is distinct from the service that uses it. |
||
| participant Orch as Orchestrator<br/>single thread: object_wait loop | ||
| participant Flash as Shared Storage<br/>ext. SPI flash | ||
|
|
||
| Note over UA, Orch: NOTIFY CHANNEL (pre-transfer veto) | ||
|
|
||
| UA->>PLDM: RequestUpdate (MCTP) | ||
| activate PLDM | ||
| PLDM->>Orch: channel_transact: Request::UpdateRequested | ||
| Note right of Orch: check state, policy | ||
| Orch-->>PLDM: Response::Accepted | Rejected | ||
| deactivate PLDM | ||
| PLDM-->>UA: RequestUpdate response (accept/reject) | ||
|
|
||
| Note over UA, Orch: if Accepted: INTAKE CHANNEL | ||
|
|
||
| activate PLDM | ||
| PLDM->>Orch: Offer { target: TargetId, total: u64 } | ||
| Note right of Orch: validate target + length,<br/>reserve staging,<br/>open the SMC write filter | ||
| Orch-->>PLDM: IntakeStatus::Receiving { base: FlashAddress, total } | ||
| deactivate PLDM | ||
|
|
||
| loop FD pulls chunks from UA via RequestFirmwareData | ||
| PLDM->>UA: RequestFirmwareData (MCTP) | ||
| UA-->>PLDM: firmware chunk response | ||
| PLDM-->>Flash: write firmware bytes (direct, no IPC) | ||
| Note right of PLDM: PLDM tracks its own write progress | ||
| end | ||
|
|
||
| activate PLDM | ||
| PLDM->>Orch: Complete { written: u64 } | ||
| Note right of Orch: check coverage,<br/>queue Pending::UpdateRequest | ||
| Orch-->>PLDM: IntakeStatus::Authenticating | ||
| deactivate PLDM | ||
|
|
||
| Note over UA, Flash: async: orchestrator event loop drains pending | ||
|
|
||
| PLDM->>UA: TransferComplete (MCTP) | ||
|
|
||
| Note over PLDM: PLDM FREE:<br/>services UA on MCTP<br/>MCTP responsive | ||
|
|
||
| Note over Orch, Flash: EFFECT CHAIN (non-blocking steps)<br/>1. poll_pending<br/>2. SM: Ready -> Updating<br/>3. poll_stage (one step)<br/>4. return to object_wait<br/>repeat 3-4 until phase done<br/>IPC responsive between steps | ||
|
|
||
| Orch-->>Flash: PayloadSource::read_at | ||
| Flash-->>Orch: payload bytes | ||
|
|
||
| Orch->>PLDM: object_set_peer_user_signal<br/>(dataless nudge, wakes WaitGroup) | ||
|
|
||
| loop wake on USER signal, poll status, send *Complete to UA | ||
| activate PLDM | ||
| PLDM->>Orch: Poll | ||
| Note right of Orch: read latched IntakeStatus | ||
| Orch-->>PLDM: Authenticating | Staging | Staged | Failed | ||
| deactivate PLDM | ||
| Note over PLDM, UA: when phase done: | ||
| PLDM->>UA: VerifyComplete (MCTP) | ||
| PLDM->>UA: ApplyComplete (MCTP) | ||
| Note right of PLDM: on failure: same commands<br/>with error completion code | ||
| end | ||
|
|
||
| UA->>PLDM: ActivateFirmware (MCTP, explicit) | ||
| activate PLDM | ||
| PLDM->>Orch: Activate | ||
| Orch-->>PLDM: IntakeStatus::Activating | ||
| deactivate PLDM | ||
| PLDM-->>UA: ActivateFirmware response (accepted, not done) | ||
| Note right of Orch: activation effect (async):<br/>bump SVN in OTP (irreversible),<br/>nudge + Poll reports Activated | ||
| UA->>PLDM: GetStatus (MCTP, until activation lands) | ||
| PLDM-->>UA: current state + AuxState | ||
|
|
||
| Note over UA, Orch: between Offer and Activate | ||
| UA->>PLDM: CancelUpdate (MCTP) | ||
| activate PLDM | ||
| PLDM->>Orch: Abort | ||
| Note right of Orch: in-flight flash step completes<br/>and is discarded | ||
| Orch-->>PLDM: IntakeStatus::Idle | ||
| deactivate PLDM | ||
| PLDM-->>UA: CancelUpdate response | ||
|
|
||
| Note over Orch: If PLDM dies mid-transfer (no Complete, no Abort),<br/>orchestrator-side timeout releases the staging reservation.<br/>The transfer loop is zero-IPC, so this bounds total transfer time:<br/>it must exceed worst-case transfer plus FD_T1 (120s),<br/>so a live PLDM always aborts first. | ||
|
|
||
| Note over UA, Flash: Blocking direction: always PLDM -> Orchestrator, never the reverse.<br/>Every IPC response is immediate. Effects run async via poll_stage (one step, return, repeat).<br/>PLDM stays free to service UA on MCTP. USER signal nudge replaces blind polling.<br/>FD initiates TransferComplete, VerifyComplete, ApplyComplete. UA initiates ActivateFirmware, GetStatus and CancelUpdate. | ||
| ``` | ||
|
|
||
| ## Write-access containment | ||
|
|
||
| PLDM writes firmware bytes direct to flash (zero-IPC, see above), so write | ||
| access must be confined to the inactive slot and only for the duration of the | ||
| transfer. Two layers, each catching a different class of failure: | ||
|
|
||
| The first layer is a typed StagingWindow inside the PLDM process. When PLDM | ||
| receives a Receiving response it constructs the window from the base and total | ||
| it carries: a bounded handle over the staging region, capped to its length. All | ||
| writes go through the window; it translates offsets and rejects anything | ||
| outside the region. The window is dropped on Complete, Abort, or timeout, so | ||
| PLDM holds no flash handle outside an active transfer. This catches offset bugs | ||
| and use-after-transfer bugs but not a compromised process, because PLDM still | ||
| has the underlying flash mapped. | ||
|
|
||
| The second layer is a hardware write filter that PLDM cannot reprogram. The SMC | ||
| raises SmcInterrupt::WriteProtected on writes outside an allowed region. The | ||
| orchestrator (or a dedicated flash-service process) opens the filter for the | ||
| staging region on Offer and closes it on Complete/Abort/timeout. PLDM needs the | ||
| SMC control registers that drive erase/program commands, but must not be able | ||
| to touch the filter/write-protect registers. Whether those register sets are | ||
| separable (distinct MPU pages, separate controller/CS, or lock-until-reset | ||
| bits) depends on the AST10x0 register layout and is folded into the "who owns | ||
| the SPI flash controller" open question below. | ||
|
|
||
| The net effect: bugs hit the Rust window check, a compromised process hits the | ||
| hardware filter, and both "inactive slot only" and "only during an update" are | ||
| enforced. Even a fully rogue PLDM can at worst corrupt the staging area and | ||
| fail verify; the active image is never written by PLDM at any point in the | ||
| flow, and activation is orchestrator-side metadata plus the SVN bump in OTP. | ||
|
|
||
| ## Open questions | ||
|
|
||
| Who owns the SPI flash controller. The diagram has PLDM writing the staging | ||
| region and the orchestrator reading it, but FlashDriver takes `&mut self` and | ||
| says nothing about multiple clients. Either each process drives its own | ||
| controller over disjoint regions, or one process owns the driver and the other | ||
| reaches it over IPC. Sequencing keeps the two off the same bytes at the same | ||
| time (the orchestrator reads only after Complete), so this is about the driver | ||
| and the controller, not about the protocol. A related constraint from the | ||
| write-access containment section: PLDM needs the erase/program control | ||
| registers but must not reach the write-protect/filter registers. Whether those | ||
| register sets fall on separate MPU pages on the AST10x0 (datasheet needed) | ||
| determines whether pw_kernel can enforce the split, or whether a dedicated | ||
| flash-service process must own the entire SMC and proxy writes. | ||
|
|
||
| How the orchestrator learns that PLDM died, short of the timeout. Abort is a | ||
| message a live PLDM sends; the timeout covers the case where it can send | ||
| nothing. pw_kernel has no peer-closed signal: the set is READABLE, WRITEABLE, | ||
| ERROR, JOINABLE, USER and the interrupt bits. The one existing path is | ||
| ChannelInitiatorObject::reset, which raises ERROR on the handler, and only if a | ||
| transaction was in flight and only once someone joins the dead process. During | ||
| the zero-IPC transfer loop no transaction is in flight, so the orchestrator | ||
| sees nothing. Replacing the timeout means the orchestrator waits on JOINABLE on | ||
| PLDM's process object, or the supervisor that joins PLDM tells it. That is a | ||
| supervisor question, not a channel one. | ||
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The firmware_device run_terminus_inner is currrently a tight loop interleaving the roles of initiator/responder - what you are assuming here is that we can fit the blocking requests to the orchestrator in the FdOps callbacks without disrupting the protocol.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Agreed on the mechanism.
handle_component,activateandcancel_update_componentall run inside the responder callback with the UAwaiting, so a transact there holds the UA's response for as long as the
orchestrator takes to answer.
FD_T1 and FD_T2 are ours to set, but the response timeout is the BMC's. Do you
have a number for what the UA tolerates? Your skeleton already carries a
bounded wait, so I assume you had one in mind.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The timeouts were not selected by me, let's follow up with @CourtneyDrant .