fix(agent): stop reconnects and restarts erasing device truth from the status topic

[impuls42]

Closes #12.

What was happening

Two causes behind one symptom, and the one in the issue comment is the one that actually bites.

_on_connect published a hardcoded bare PrinterStatus, retained. That fires on every reconnect, not just the first connect, and paho reconnects silently — the broker's Istio route caps at a 24h timeout and flaps besides — so it ran several times a day. It was not merely uninformative, it was lossy: it overwrote a retained message that had been correct a moment earlier, and the printer sleeps between jobs so nothing could put it back. Hence an agent 25h up, NRestarts=0, every device field null.

The rarer cause is the one the issue was filed about: a fresh process knew nothing at all, because the device fields lived in six attributes on PrintWorker that were only ever filled inside a send.

Both are the same missing idea — nowhere to keep what the printer said.

What this does

DeviceSnapshot (agent/device_state.py) is that place, persisted to the spool DB after every capture and read back at startup. The MQTT source remembers the last retained status it published, seeded from the spool, and republishes that on connect. The will and the shutdown notice carry it too: losing the link is news about reachability, not grounds for forgetting the serial.

device_seen_at is new on PrinterStatus. Remembered truth published as though it were live is its own kind of lie — the same failure the tri-state media_ok exists to avoid, one layer along. An absolute UTC instant rather than an age, because the message is retained: an age is computed once and then sits on the broker getting wronger. It only advances when the printer actually reported something, so a connection that answered nothing cannot launder three-day-old media state into looking current.

{"v":1,"printer_id":"d30-workshop","state":"idle","model":"phomemo-d30",
 "serial":"Q223P4C31420105","firmware":"2.1.2","battery_pct":100,"voltage_v":4.17,
 "media_ok":true,"tape_width_mm":15.0,"pending_labels":0,"error":null,
 "device_seen_at":"2026-08-02T09:14:03Z"}

contracts/job-v1.schema.json is generated from PrintJob only, so the schema gate is untouched, and the InvenTree driver reads the payload with .get(), so the new field is backwards compatible.

device.probe_on_start (on by default) surveys the printer once at startup, so an agent coming up next to an awake printer publishes live truth immediately rather than waiting for the next job. This cannot wake a sleeping unit — AUTO_POWER_TIME really does power the radio down and only the button brings it back — so a miss costs one connect timeout and the stored snapshot is published unchanged.

It runs on the print loop and only at startup, deliberately not from on_connect. That callback is paho's network thread, so probing there would touch the printer concurrently with a print — the one thing the single-threaded loop exists to prevent — and it would fire on every broker reconnect, several times a day, over a link that says nothing about the printer.

One behaviour change worth flagging: with the probe on, an agent configured without device.mac now fails at startup rather than at first print. That config could never have printed anyway, so failing where systemd will show it seems the better of the two.

Verified

Full suite green (264 passed), ruff clean, schema in sync. End to end without hardware or a broker — print one job, drop the process, bring a new Spool + MqttSource up on the same DB, fire _on_connect:

what the retained topic holds after the restart:
 "serial": "Q223P4C31420105", "firmware": "2.1.2", "battery_pct": 100,
 "voltage_v": 4.17, "media_ok": true, "device_seen_at": "..."

after a reconnect, serial is still: Q223P4C31420105

InvenTree renders: ('CONNECTED', 'fw 2.1.2 · 100% (4.17V) · media ok · Q223P4C31420105')

against CONNECTED — media unreported today.

tests/test_source_mqtt.py is new. That module had no coverage at all, which is how a hardcoded blank status survived in the one callback that runs most often.

Also

The README still claimed "There is no read channel… low battery [is] undetectable". That was falsified by #9 and this change makes it actively misleading, so it is corrected here. HARDWARE-NOTES.md was already right.

Consumer-side follow-up in sengine-cloud/inventree-label-dispatch renders the age on the Machines page, so remembered truth is not shown as live there either.

What the will can and cannot promise

Raised in review, and worth having written down because the code reads as if the will is always current and it is not.

The will rides in the CONNECT packet, so the broker holds one fixed payload for the life of a connection. Nothing the agent does while connected can change what fires if that connection dies — a mid-session drop publishes whatever was armed when the session opened.

That is only half the story, though. paho rebuilds CONNECT from the same _will_* fields on every automatic reconnect — will_set has no connected-state guard and _send_connect reads _will_payload at packet-build time, both checked against 2.1.0 — so _on_connect re-arms it. That bounds the staleness by the reconnect interval, a few hours here, rather than by the process lifetime, which for a long-lived agent would have meant the will still describing boot weeks later.

device_seen_at rides along regardless, so even the frozen-within-a-session case is labelled rather than misleading.

Two related deliberate choices, also noted in review:

• _disconnected() keeps error and media_ok rather than clearing them. A consumer switching on state alone would miss a latched fault; that is the correct trade here, because the alternative is a disconnect notice that forgets the printer had no tape. The InvenTree driver renders from the whole payload.
• probe_device stamps seen_at after the settle sleep, so it can read up to a second late. The replies arrive during that window, so the error is bounded by the settle and immaterial at the resolution this field is published in (whole seconds).

The window the will leaves open, and what closes it

Raised on the consumer PR, and it is a real residual rather than a hypothetical.

A fault learned during a session cannot reach that session's will — the broker holds it from CONNECT. So: agent connects while the tape is fine, a print an hour later finds the tape gone and correctly publishes error/media_ok: false, the agent is then killed, and the broker publishes the older healthier reading over the newer one. The topic goes backwards, which is the same shape as the bug this PR fixes.

Prevention is not available. Re-arming on reconnect (above) bounds it by the reconnect interval instead of the process lifetime, but cannot touch a live session. on_disconnect cannot help either — it fires with no connection to publish on — and a SIGKILL runs no callback at all.

What bounds it is recovery. The unit is Restart=always / RestartSec=5, and a restarted agent seeds _last_status from the spool — which does hold the fault, because _capture_feedback persists before the crash — and republishes it on connect. The stale-healthy row is therefore seconds wide, not indefinite, and it is marked DISCONNECTED throughout.

That property was carrying the whole argument while being untested, so test_a_stale_will_is_corrected_by_the_restart_that_follows_it now asserts it end to end: arm a healthy will, learn a fault mid-session, restart on the same spool, and check the republished status carries the fault and a device_seen_at strictly newer than the will's.

Documented on the consumer side too, since that is where it is visible.

[impuls42]

Consumer half is up: sengine-cloud/inventree-label-dispatch#7 renders device_seen_at as · seen 6h ago on the Machines page, so remembered truth is not shown as live there either.

[impuls42]

Thanks — acted on the LWT point, and it turned out to be partly fixable rather than only documentable.

Correct that the will is frozen for the life of a connection: it rides in the CONNECT packet and the broker holds it, so nothing changes what fires if that session drops. But paho rebuilds CONNECT from the same _will_* fields on every automatic reconnect — will_set has no connected-state guard, and _send_connect reads _will_payload at packet-build time, both checked against 2.1.0. So _on_connect now re-arms it (09c6487). That moves the staleness bound from the process lifetime — which for a long-lived agent meant the will still describing boot weeks later — down to the reconnect interval, a few hours here. Armed from paho's network thread, the same thread that builds the packet, so topic and payload can't come from different passes.

The residual constraint is in the PR description now, along with the two cosmetics: _disconnected() deliberately keeps error/media_ok (a disconnect notice that forgets the printer had no tape is worse than one a state-only consumer under-reads), and probe_device's ≤1s late seen_at is bounded by the settle window and below the whole-second resolution the field publishes at.

265 tests green, ruff clean.