[WIP] Ryot V11 - #1832
Draft
IgnisDa wants to merge 3770 commits into
Draft
[WIP] Ryot V11#1832IgnisDa wants to merge 3770 commits into
IgnisDa wants to merge 3770 commits into
Conversation
Narrow the leaving stack entry explicitly and type DOM query results as HTMLElement so the full Turbo check completes without linter warnings.
Reconcile each host location synchronously into one navigation-store snapshot so edge state, retained screens, and pop transitions cannot diverge. Replace imperative header updates with declarative route metadata, correlate publications to history identity and plugin ownership, and restore retained titles on pop without compatibility paths or version changes. Normalize Motion edge velocity to pixels per second, add focused component and store coverage, migrate the fixture plugin, and update the client plugin design contract.
Update every workspace package identity, dependency, import, compiler allowlist, sandbox runtime key, Turbo filter, Docker command, test fixture, and documentation reference from @ryot/* to @ryot-app/*. Regenerate bun.lock and keep plugin archives and compiler workers on the new scope without compatibility aliases.
Align all legacy relationship inserts with the current five-column relationship identity, including plugin provenance. This restores PostgreSQL conflict inference for global episodic and user-authored relationship migration rows.
Register the documented /e/:entityId and /v/:viewSlug authenticated routes in the TanStack route tree, with TODO-only page content ready for later renderers.
Use explicit global and user entity scopes so authenticated updates, provider population, generic imports, saved views, signals, and sandbox entity creation resolve the caller’s effective definition snapshot instead of silently consulting the system registry. Capture definition snapshots through durable activities, preserve plugin ownership for entities and relationships, remove optional scope fallbacks, and update focused backend and provider e2e coverage plus plugin author documentation. chore: more compatibility fixes
Retry uninstall only while durable workflows retain transient plugin references. Surface entity, integration, saved-view, and other persistent conflicts immediately instead of hiding them until the polling timeout. Update plugin lifecycle and E2E fixture documentation to describe the explicit cleanup requirement. Keep best-effort plugin teardown non-failing Treat persistent plugin reference conflicts as completed best-effort teardown while continuing to poll transient workflow references. Preserve strict uninstall behavior for tests that explicitly assert plugin removal and document the distinction for fixture authors.
Load saved-view records and execute their persisted grid RyotQL document through a dedicated Effect service. Render first-page field slots with stable formatting, entity links, managed image resolution, and nonfatal image fallbacks. Add route-level pending, empty, not-found, malformed-definition, and retryable failure states, plus service, asset, formatting, and authenticated route coverage.
Add the admin API and in-memory God Mode session services, user administration and migration report screens, and standalone route wiring outside the authenticated shell. Preserve safe onboarding redirects into God Mode and include the related route, service, component, and navigation coverage. chore: add capacitor share
Replace inline administration actions with an accessible three-dot popover that preserves keyboard navigation, focus restoration, hardware Back handling, disabled states, and destructive confirmations. Compact the generated reset-link card and apply balanced vertical spacing around the result row. Expand focused coverage for menu interaction, reset availability, confirmations, and link transfer.
Add the public kernel-client reset page and submit reset tokens through Better Auth without entering the OAuth guard. Preserve native server onboarding, stable invalid-token errors, and regression coverage for validation, success, and routing.
Port grid, list, and table saved-view layouts to the kernel client with persisted layout selection, debounced search, cursor pagination, entity deduplication, managed assets, lower-bound counts, and distinct Count all queries. Guard page and count workflows with query identity and request generations so stale layout and search results cannot replace the active view. Add the TanStack-backed client UI SDK DataTable, migrate God Mode users to it, expose the table SDK entry to client plugin compilation, and cover controller, storage, recipe, service, route, and component behavior.
Port the saved-view affordances the renderer migration dropped: the desktop Filters button with its active-filter pill, the accent Add button and its A shortcut, the compact-viewport add FAB, the empty-state online-search action, and the no-matches online-search suggestion. Gate every add affordance on the record's entitySchemaSlug and leave the handlers logging a TODO until the provider add flow exists. Split the header side by side at lg so four controls and the title stop competing for one row, and position the FAB against a route wrapper because the shell content carries the drawer's motion transform. Lift the chrome those controls need into the client UI SDK as useShortcut, SearchField, SegmentedControl, and Badge, so kernel and plugin call sites share one implementation, and delete the route-local search and layout selector. Cover the SDK primitives, the entity-schema gating, and the empty and no-matches states, and drive the shortcuts and viewport swap through a browser end-to-end test. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The shell resolved the sidebar workspace from the first path segment and fell back to the remembered workspace only under /settings, so kernel-owned routes whose segment is a reserved slug rendered "No workspace". /v/:viewSlug and /e/:entityId both hit this. Make the remembered last workspace the sole authority for workspace context on every authenticated route, including plugin routes: reaching /fitness/workouts/1 through a Media collection must not move the sidebar off Media. Only a deliberate choice writes it now, so drop the effect that persisted whichever plugin route the user landed on and leave the workspace switcher and the / bootstrap redirect as the only writers. Split the two questions that homeActive was answering. The Home row highlight follows the remembered workspace, while the edge resolver's atRoot stays route-derived through a new isWorkspaceRoot helper, so a plugin workspace root still resolves to the drawer rather than a back gesture when it is not the remembered workspace. Amend section 16 and the section 17 shell chrome to match, dropping the rule that let a saved view's plugin association determine surrounding context. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The kernel hand-rolled overlay behaviour three separate times. The confirmation dialog, the user actions menu, and the mobile drawer each carried their own focus-trap loop, their own `focusable` selector constant, their own scroll lock, and their own Escape handling, and the workspace switcher carried a fourth outside-dismiss listener. Three copies of the same Tab containment block is three places for it to drift. Move that behaviour here as `Modal`, `Menu`, and the hooks underneath them: `useFocusTrap`, `useScrollLock`, `useDismissOnOutside`, and `useRestoreFocus`. `useScrollLock` returns an idempotent `unlock` rather than only releasing on cleanup, because the drawer restores overflow and navigates inside one `flushSync` and an unmount-only release would land the next route on a frozen page. Keep the kernel's policy out of here. Back interception arrives as an injected `onInterceptBack` that returns true when the caller consumed the close request, so this package never learns about the router, native back, or Capacitor. `Modal` takes its panel classes from the caller too, since the kernel's `ui-*` utilities live in the kernel stylesheet that a plugin document never loads. Add the presentational controls the schema form will need and this package lacked: `Switch`, `TextField`, `Chip`, and `MultiSelect`. `SearchField` hard-wired its shortcut to `enabled: shortcut !== undefined`, so a caller could not silence it. An overlay above the page needs to, or `/` steals focus back to the page behind it. Add `shortcutEnabled`. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Replace the three hand-rolled copies with the shared primitives. The confirmation dialog and the user actions menu become `Modal` and `Menu` children, losing their local portals, positioning block, focus-trap loops, scroll lock, and duplicated `focusable` constant. The workspace switcher takes `useDismissOnOutside` but stays inline rather than portalled, because it is anchored inside the drawer's transformed, scrolling panel. The drawer keeps its motion, progress value, and edge gesture: it is deliberately not a `<dialog>`, and `showModal()` cannot be dragged progressively open. It now consumes `useFocusTrap` and `useScrollLock` so there is one implementation of each, and releases the lock through `unlock` before `closeThen` navigates. Behaviour that had to survive and now has coverage: the dialog refuses to dismiss while an operation is pending, and the menu restores focus on Escape but not on an outside click. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Uploading a file meant creating an intent, PUTting the bytes to the returned URL, then completing the intent. Every caller that wanted a token had to know that sequence. Add `uploads.uploadTemporary`, which takes a file and returns a token. The three steps are invisible to callers. It follows the shape the adapter already uses for `navigate`, `theme`, and `invokeOperation`: optional on `RyotClientAdapter`, with a missing implementation raising `unsupported-capability`. The kernel backs it through `AuthenticatedApi`, honouring the method and headers the intent returns rather than assuming PUT, and passes the `Blob` straight to `fetch` so the file is never read into memory. The plugin adapter deliberately does not provide it. Every bridge message is validated against a `JsonValue` payload schema, so a `Blob` cannot cross the port; a plugin calling it gets `unsupported-capability` until the bridge can carry binary. There is a TODO at the adapter construction saying so. Encoding the bytes to squeeze them through the existing envelope would trade a clear boundary for a hidden one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Port the schema-driven form from the old client so provider search options, and anything else described by an `AppSchema`, can render here. It covers every supported property kind: boolean, string, number, integer, date, datetime, enum, enum-array, scalar arrays, and upload fields, along with the conditional visibility rules and validation the old form carried. Put it behind `@ryot-app/client-ui-sdk/schema-form` rather than the root barrel. This package is inlined into every plugin artifact, and the form pulls `@ryot-app/contract`, `effect`, and `@tanstack/react-form`. On the root export that weight would land in every plugin, including ones that only import `Button`. A new subpath has to be registered in the compiler's trusted-module list, so add it there and to the type-entry map. The upload transport stays injected as a `SchemaFileUpload` prop, as it was before, so this package still knows nothing about servers, auth, or scopes. Drop the pieces that do not apply: the expo document picker and the React Native drop-zone stub, since the browser picker covers web and the Capacitor webview both, and the review summary, which nothing here uses. Correct the plugin design doc, which advertised a `Dialog` and charts and gestures subpaths. Separate what ships from what is still aspirational. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The recipe joined `sandbox_provider.pluginSlug` to `plugin.slug`, but that column has never existed: the table and the RyotQL catalog expose `pluginId` only. Every call failed validation before reaching the database, so provider discovery was broken for every caller. That join is fixed now, but the test that should have caught it asserted `expect(query.joins).toHaveLength(2)` and would have passed either way. Assert both join conditions by field so a wrong column fails here rather than at runtime. Verified by mutation: restoring the old `pluginSlug`/`slug` join fails this test. Nothing yet checks recipe column references against the catalog in general, which is why this survived. Five other recipes declare joins and remain unguarded. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Port the state machines from the old client unchanged: the search reducer with its request tokens and pagination, the import controller with its enqueue-then- poll schedule and user-facing failure messages, the provider options state, the preferred-provider selection, and the result display mapping. Only import paths changed, so these arrive with their existing tests intact. Replace the atom layer, which does not exist here. `ProviderAddService` follows `SavedViewsService`, and splits by transport rather than by convenience: the two RyotQL recipes go through the client SDK's `data.query`, while provider search, search options, import, and import polling are contract endpoints reached through `AuthenticatedApi`. Failures carry the stage they came from. `importProviderEntity` stays a pure function over injected `start` and `poll` effects, so the polling policy remains testable without a server. Persist the remembered provider through `ClientStorage`, keyed by scope and entity schema, alongside the existing saved-view layout key. Update the route test fixtures for the new service and the two storage operations. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The saved-view screen already rendered every add affordance, all five wired to a placeholder that logged a TODO. Replace it with the real flow: provider chips selected from the remembered provider, a debounced search, the advanced options form driven by the provider's materialized `AppSchema`, in-library detection, and an import that polls to completion. The panel and result row are DOM rewrites of the old React Native components with their class strings and copy carried across, so the screen matches the old client rather than being redesigned. Keep the open state in the route's search params, so browser and Android back dismiss the flow and the no-matches action can seed the query. Mount it from `SavedViewPage` rather than inside `SavedViewContent`, which is keyed on the record, so a refetch cannot unmount the flow mid-import. Refresh once when the flow closes having imported something, never once per import. That required keying the content on the identity of the loader data: an import changes no saved-view record field, so keying on `updatedAt` alone meant `router.invalidate()` refetched but the fresh page could never seed the controller and the imported item never appeared. Suppress the page's `A` and `/` shortcuts while the flow is open, or they reach the screen behind it. The filters control stays deliberately inert. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Exercising the provider add flow required IGDB credentials, since the media plugin owned the only provider declaring search options. Add a credential-free one backed by PokeAPI, so `bun dev` can drive the whole loop. The plugin declares its own `pokemon` entity schema and roots the provider at it, which user-installed plugins can now do. Search options carry a static enum-array of the eighteen types plus an include-alternate-forms boolean, which is what makes the advanced options form reachable at all. Query matching is honest about the API: PokeAPI's list endpoint has no text search, so an unfiltered query fetches the index and filters names, while a type filter intersects the per-type endpoints. Details map only fields PokeAPI actually supplies. Add a saved view over the schema at /v/all-pokemon so there is somewhere to run the flow. Ordering it by pokedex number needs an explicit numeric cast, because a raw JSON path is not a scalar and the kernel rejects it as an ordering expression. The image is an asset array rather than a bare URL string: the card and table image slots decode their value as an `AssetLocator`, so a JSON object is required, and RyotQL has no object constructor to build one from a string. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The browser suite asserted the console line the placeholder logged, so it broke the moment the real flow landed. Rewrite it to drive the flow: each affordance opening the dialog, `add` appearing and clearing in the URL, the no-matches action seeding the query, an import reaching in-library and then showing up in the view, browser back dismissing it, and the desktop and compact affordances swapping. It installs a deterministic provider rooted at a suite-unique entity schema, so results and the imported entity are fixed values rather than whatever other suites left behind. The two shortcut guards needed care to test. Hotkeys are ignored while a text input holds focus, so asserting from the autofocused search box passes whether or not the guards exist. Focus a provider chip first; verified by mutation that both assertions fail without the guards. Add a live PokeAPI smoke test behind `RUN_LIVE_PROVIDER_TESTS`, alongside the media one, covering discovery through `providerSearchRecipe`, materialized search options, a type-filtered search, and a real import. `uninstallTestPlugin` warned on every persistent-reference conflict, so a suite that imported anything logged a cleanup failure that was really the expected outcome. Match the semantics `releasePrivatePlugin` now has: only a workflow-referenced conflict is transient and worth reporting. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The edge gesture commits on distance or velocity: `dx > width / 3 || vx > 0.5`. The two router drag tests built their pointer events without a timestamp, so jsdom stamped each at construction and `vx` came out of whatever the clock did between three synchronous dispatches. A drag of 28px across a 300px root is well under the distance arm, so the below-threshold test passed only when both events happened to land on the same tick; when they straddled one, `vx` reached hundreds of times the threshold and the gesture committed. It failed reliably under a parallel test run and never in isolation. Give the events explicit timestamps 400ms apart. The past-threshold drag now commits on distance alone at vx 0.395, and the released drag sits at vx 0.07, so both assert the wiring rather than the clock. The base has to be non-zero. React resolves a synthetic event's timestamp as `nativeEvent.timeStamp || Date.now()`, so stamping the pointerdown 0 silently falls back to the real clock and leaves the elapsed time negative — which made `vx` zero for a reason that had nothing to do with the intended drag. Both arms of `shouldCommit` are already covered as units in gesture.test.ts, so these stay wiring tests. Verified by mutation: shortening the span to 20ms makes the released drag commit and fails the test with the same error the flake produced. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The plugin owned one provider, so the add flow only ever exercised a boolean
and an enum-array option against a card with an image. Root a second provider
at a new `move` entity schema to cover the single-select `enum` control and the
imageless card path, which nothing reached before.
Search options resolve entirely from list endpoints, so filtering never has to
fetch a candidate to discard it: `/type/{n}`, `/move-damage-class/{n}`, and
`/generation/{n}` each expose a flat `moves` array. A `minPower` option was
considered and dropped because power lives only on the detail payload, and
filtering on it would make `totalItems` and `nextPage` lie.
`typeNames` unions rather than intersects. A move has exactly one type, so the
Pokemon provider's intersect semantics would return nothing for any two
selections; the test pins this by searching electric+fire for "punch" and
expecting both Fire Punch and Thunder Punch.
Split the shared helpers into `pokeapi-shared.ts` and rename the Pokemon module
and its sandbox entries, since `pokeapi.ts` was Pokemon-specific under a generic
name. Slugs are unchanged, so stored providers and operations are unaffected.
The eighteen type choices stay duplicated across both search manifests instead
of moving into the shared module. The sandbox manifest extractor reads
`defineManifest` by static analysis and rejects an imported identifier with
"Manifest values must be JSON-safe literals" — a build emits a correct
manifest.json regardless, so this only surfaces when the archive is installed.
Display expressions stay uncast. Casting a JSON `null` to numeric errors in
Postgres, which every status move would have hit through `power` and `accuracy`;
the numeric cast belongs only in the ordering expression the kernel demands.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NFgYbJQP5yF6VWmvsCeYfC
The page body is `grid min-h-full gap-5`, and a grid container's default `align-content: normal` resolves to `stretch`, which grows every auto-sized row until the tracks fill the container. Whenever the results were shorter than the viewport the leftover height was split between the header, the results, and the pagination rows: a gap opened under the result count and "End of ..." floated far below the last card. Native never showed this because a scroll view sizes to its content, and mobile web hid it because the content overflowed. `content-start` keeps the rows at their intrinsic heights; `min-h-full` still paints the background down the viewport, and the empty, no-match, and searching states are unaffected since each carries its own `min-h-96 place-content-center`. The list row is `[image][text][callout]`, and the old renderer gave the text block `flex-1` so it absorbed the row width. The port dropped it, so the block sized to its content and the callout landed right after the title at a different offset on every row. Restore it, and pin `shrink-0` on the callout: React Native defaults to `flexShrink: 0` and the web to `1`, so without it a narrow row would compress the callout where the old layout kept it whole. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0159rN3HA7wsQFN7CkHaYZBq
`ryot plugin build` accepted plugins the server rejects at install. A sandbox manifest that read `values: TYPE_CHOICES` from a sibling module passed `tsc`, passed oxlint, and produced a correct manifest.json, because the build resolves the import. The install-time extractor reads `defineManifest` by static analysis and never executes the module, so it failed with "Manifest values must be JSON-safe literals" — after the archive had already shipped. The install gate is two checks, not one: the sandbox compile, and a parity comparison between declared and compiled script metadata. Having the CLI call only the compiler would have left the second half unenforced, and reimplementing it would have created a second definition of what a valid plugin is. Move both into `compilePluginManifestScripts` and have the pipeline and the CLI share it. The shared function fails with a `PluginScriptCompileMismatch` tagged error rather than the backend's `PluginValidationError`, which lives in kernel-backend and drives the HTTP `validation-failed` reason code. The backend catches the tag and maps it back through the shared message builder, so API error codes and issue strings are unchanged; the CLI renders the same error for a terminal. The `build-plugin` test fixture was `export const main = "initial"`, which is not a compilable sandbox script and would have failed the new check. It is now a real `defineScript` that still contains "initial" so existing assertions hold. No first-party plugin needed changing: media, fitness, and fixture all build clean under the new check. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NFgYbJQP5yF6VWmvsCeYfC
The CLI globs `backend/**/*.ts` into the archive, so `backend/` was the shipped sandbox tree. It was also where host-side code lived. `backend/schemas/*` and `backend/shared/view-helpers.ts` are imported only by `manifest.ts`, `saved-views.ts`, and `query-recipes.ts`, and were shipped in every archive as sandbox source no script can reach. Nothing in the tree said which side a file belonged to. Make the boundary the top-level split. `host/` is the TypeScript the server and web client import; `backend/` and `client/` are the two archived trees and keep their names, so the archive layout and the `backend/`/`client/` prefixes in plugin-archive and the ingestion pipeline are untouched. `backend/` and `client/` never import `host/`; production `host/` code reaches the sandbox tree only through `backend/contracts/`, which holds the schemas, recipes, and helpers both sides need. Package exports follow: `./operations/schemas` is now `./contracts/operations`. Drop the `scripts/` path segment. It carried no information — every backend path was `backend/scripts/<area>/` and `.sandbox.ts` already marks entrypoints — but removing it also merges the two vocabularies the tree had grown. `backend/imports/` held parsers and `backend/scripts/imports/` held entrypoints, so `goodreads.sandbox.ts` reached its parser through `../../imports/goodreads`; there were two directories named `operations` two levels apart. Parser, entrypoint, and test are now siblings. Address providers by path: `backend/providers/<rootEntitySchemaSlug>/<vendor>/<operation>.sandbox.ts`. Provider slugs are exactly `<entity>.<vendor>`, and the 51 directories now match the 51 slugs. `media-group/` had mixed six different `-group` schemas into one bucket under vendor-only filenames, so `media-group/hardcover-details.sandbox.ts` gave no hint it was `book-group.hardcover`; the `media/` segment above `movie/` and `show/` was noise that also forced `company/` and `person/` to a different depth. The delimiter is now `/`, which ends the dot-versus-dash split that had `person/tmdb.details.sandbox.ts` next to `company/tmdb-details.sandbox.ts`. Vendor helpers move from thirteen flat `providers/*-shared.ts` files to `backend/lib/vendors/`, alongside the cross-area helpers from `backend/shared/`. Tests colocate with their subjects. `plugins/media/shared/` had held only tests whose implementations were in `backend/shared/`, while `plugins/fitness/shared/` held both source and test for the same `view-helpers.ts` concept. Two tests carried path assumptions rather than behavior: a glob rooted at `backend/scripts/**` and a package root derived from `import.meta.url`, which now resolves inside `host/`. `script-catalog.ts` moves to `host/` unchanged. It is 1029 hand-maintained lines whose imports are aliased `manifest0` through `manifest182`, and it is the only reason `host/` imports all 182 sandbox modules. Deriving it needs the compiler to discover a script's kind instead of verifying a declared one, which is a separate change to the manifest contract. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`script-catalog.ts` was 1064 lines across media and fitness whose only content was a second statement of what each sandbox module already declares. Its imports were aliased `manifest0` through `manifest182` in no order, and every entry restated its own import path as a string literal. A test existed solely to assert the catalog equalled a glob, and the compiler then compared the catalog against each module a third time. The compiler already extracts everything the catalog held. `inspectSandboxSource` reads `kind` from `defineManifest` and `operation` from the `defineProvider` call, and `compileSandboxPackageEntries` took a declaration map only to compare against those. Nothing had to change in how a script is understood — the declarations map stays for callers that hold a built manifest, and the entry compiler now also returns the provider operation it found. `entry` is the path, and `providerSlug` is the directory, so both come from the glob under the layout the previous commit established. The path under `backend/providers/` is the provider slug with `/` for `.`, at any depth, so a single-segment slug uses a single directory and `movie.tmdb` uses two. Each derived script is decoded through `PluginScript` rather than asserted into it, so a provider entry outside `backend/providers/` fails the build instead of producing an invalid manifest. `PluginManifest` splits in two. `definePlugin` now takes the authored form, which has no `scripts` and validates what does not depend on them; the built form adds `scripts` and keeps the existing cross-reference checks in full. The CLI assembles the built manifest and validates it before writing, so a cron, binding, or provider operation naming a script that does not exist still fails the build. Installation keeps its gate. `compilePluginManifestScripts` derives the same list from the archive's sources and compares it to the shipped `manifest.json`, so a hand-edited archive cannot widen what a script declares. The pipeline is unchanged. Kernel plugin fixtures moved from `scripts/*.sandbox.ts` to the archive layout they were standing in for. Entry discovery matches any `.sandbox.ts` in the files it is given rather than re-asserting the `backend/` prefix, which `plugin-archive` already enforces and which the pipeline has already applied by that point. Assertions that read metadata off `manifest.scripts` now read it off the sandbox module that owns it. The media catalog-versus-glob test is replaced by one that checks the convention the derivation now rests on: every provider directory maps to a declared provider, and its operation files match that provider's declared operations. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The client compiler injects TypeScript declarations for supported CSS and binary asset imports. Remove the fixture declaration file that redeclared those wildcard modules and caused TypeScript 7 duplicate-identifier diagnostics during fixture compilation.
Configure Jaeger to use non-ephemeral Badger storage and mount it on a named Docker volume so development traces survive container restarts. Keep the Jaeger configuration limited to the OTLP and query paths used by the dev stack.
… in activities Provider population and the durable sandbox host strategies reached LifecycleExecution.after and executePolicy from inside workflow activities. Each hook run started a non-discard child AutomationRunWorkflow parented to the enclosing workflow, so the activity was stored Suspended and every child completion replayed the parent and re-ran the whole activity body (one related entity transaction ran four times, ~5 min to populate 20 movies). - add src/lib/infrastructure/workflow-scope.ts: makeActivity marks activity bodies, implementWorkflow resets the marker for workflow bodies, and LifecycleExecution.after/executePolicy die when they run under the marker - enforce the boundary in scripts/check-architecture.ts (Activity.make( and .toLayer( only in workflow-scope.ts, tests included) with a unit test, and convert every activity and workflow registration in the kernel backend - add LifecycleExecution.dispatch(plans) plus schema-derived dispatch plans and drop the duplicated executeCommittedPlans loops in entities, relationships, events, signals, user-state and provider-entities - add src/lib/infrastructure/lifecycle-workflow-step.ts and split entity, relationship, collection, generic-import, event-create and provider-population writes into prepare (activity) -> policies (body) -> recorded policy outcome -> commit (activity) -> dispatch, with a resumable cursor for upsertGlobalEntities - dispatch upsertGlobalEntities, changeUserRelationships and upsertGlobalRelationships through a new lifecycle-workflow host strategy that keeps today's host failure messages and success payloads - cover the guard, the recorded policy outcome across replays, population dispatch ordering and activity memoisation, and the host capability steps Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Populating the all-movies saved view planned ~940 automation runs for 20
items. Most could not do any work (library hooks on global people exit
immediately on a null execution user) or re-read the same entities once
per credit relationship. Each run claims a sandbox slot and starts a Deno
process.
Hooks can now declare when they are worth running, so the planner never
creates those runs.
Contract
- After hooks accept optional executionScope ("user" | "global") and
frequency ("item" | "batch"). Batch frequency is rejected unless every
target is an entity, event or relationship.
- New AutomationBatchChangePayload (category change, operation batch,
single-resource non-empty items) is part of AutomationChangePayload.
AutomationTriggerKind gains the batch variant. Item change payloads keep
their own population; the batch carries none of its own.
- Named item change payload types (entity/event/relationship) replace
Extract<> lookups that would otherwise have matched batches.
Planner
- executionScope is compared with the run executionUserId, so
per-recipient signal runs survive.
- Item hooks match only item triggers and batch hooks only batch
triggers, so each hook sees every change exactly once. A batch matches
a target when any of its items does.
- LifecyclePlanner.planBatch groups a write item plans by scope user and
chunks them in plan order by AUTOMATIONS_BATCH_MAX_ITEMS (default 200)
and by half the sandbox context limit, so every batch run input fits.
Trigger ids derive from the write identity, scope and chunk index, so
replays reuse them.
Write paths (batch planned in the write transaction, dispatched after
item plans)
- Entities: create/update/delete, deleteByIds, ensureUserEntities,
global upsert, and a new batch-scoped persistPlannedProviderUpserts.
Provider population child and related-entity loops use it.
- Relationships: mutation pipeline, one batch per reconciliation group,
single-write replay.
- Events: one batch per create workflow call, planned in its own
activity and dispatched in the workflow body; single update/delete via
EventsService.mutate.
- Prepared persists return item plans only; user-state clear and merge
plan one batch per resource for the whole write.
Database
- automation_trigger_kind_check allows change/batch for entity, event
and relationship. The single squashed baseline was regenerated.
History
- Run-history redaction recurses into batch items, so secret-marked
properties stay hidden for batch hook runs.
Media plugin
- ensure-library-membership and record-library-membership-event are
user-scoped; their null-user early returns are gone.
- media.association runs per batch: one entity read for every credited
entity, same skip rules, signal slugs and per-role discriminators,
limited to person/company credits on media or group entities.
- media.relationship-sync runs per batch and acts on the population
batch leader item, so a split batch emits once.
Docs: plugin-kit lifecycle hooks, automations README, media README,
kernel backend AGENTS.md.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every catalog lookup used to drag along data it never used: `activePluginFields` selected each plugin's whole manifest through correlated subqueries, and `toStoredPlugin` loaded every `sandbox_script` row of the active revision, `compiled_code` included, just to learn a provider id or that a details script exists. Populating one saved view made 1,363 `PluginRepository.list` calls and moved ~26GB of compiled code. The rule this establishes: catalog reads never load script bodies, never decrypt configuration, and never re-decode an immutable revision. They read small mutable pointer rows inside the caller's transaction and take all immutable revision content from an in-process cache. Only the execution path loads compiled code, by script id. - Split `PluginScriptDescriptor` (manifest entry + content hash) from `NormalizedPluginScript`, so code-bearing scripts survive only in the compile -> persist pipeline. `pipeline.toPluginScriptDescriptor` is the single projection, applied before building a registry entry. - Add a revision cache to `PluginRepository` keyed by `plugin_revision.id`, with a batched `readRevisions` and a `readRevision`. The active-revision pointer stays a database read on every lookup, so reads still reflect what is committed under the shared plugin-ingestion advisory lock. `persist` invalidates the one key it writes, because a garbage-collected revision's scripts are reinserted there. - Replace `activePluginFields` with `pluginPointerFields`, drop script bodies from `storedScriptFields`, and turn the three `sourceHash` correlated subqueries into `plugin_revision` joins. - Make installation reads state-only by default; `listHydratedForUser` serves the three callers that genuinely read `config`. This removes two queries and an AES decrypt per installation row from the whole runtime path. - Narrow the automation planner and attempt-artifact projections to the compared columns, batch the planner's per-plugin config-revision reads, and restrict its catalog query to the scopes the loop already admitted. - Resolve each provider slug once per related-entity group instead of once per entity, and memoize built definition snapshots per loader snapshot. Semantics are unchanged: no authorization, activeness, or scope check is weakened, and `lockCatalog` locking and ordering are untouched. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`LifecyclePlanner.plan` pins plugin/configuration revisions and must keep those pins valid, so it locks every active plugin's `plugin-config:` key. That key was taken exclusively, which made two planners that only read the catalog exclude each other and turned planning into a global mutex over every write that plans automations. Planners now take `pg_advisory_xact_lock_shared`; configuration writers keep the exclusive form and stay the only thing that excludes planning. Because those locks pin the catalog for the rest of the transaction, the resolver also memoizes the lock set and each user's catalog on the active `Database` value, so a transaction that plans many triggers resolves both once. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Require pinned script projections, derive bounded sandbox inputs from immutable triggers, replace full policy transforms with validated canonical patches, and add precise preparation diagnostics.
Problem as observed ------------------- Populating a page of shows was slow, logged "provider entity population completed with automation warnings", and left items stuck "populating". `media.auto-complete-episodic-parent` was a `delivery: "required"` after hook that targeted, besides episode completions, `update` on the `show` and `podcast` entity schemas. Provider population updates GLOBAL show and podcast entities, so a first population of a show that has already ended (`productionStatus` going from null to "Ended") planned a global run of this hook with no execution user. Because the hook is required, the population workflow then waited on that run before it could finish. Why that run could never work ----------------------------- - With no execution user the script is not scoped to anyone's data, so its lifecycle replay read every user's episode events for the parent. For "Manifest" that was 62 episodes and 37,645 progress events, paged 100 at a time with one sandbox replay per page, running for 10+ minutes while the population waited. - When it did get far enough to complete a parent it failed anyway: system executions are denied `listEventSchemas`, observed as the business failure "Sandbox durable host denied: listEventSchemas is not available for system executions" on "Good Omens". - Even if both were solved, completing a parent is a per-user fact; a global run has no user to complete it for. Design and rationale -------------------- Auto-completing a parent when production ends now fans out per user through the existing signal system instead of an entity-update hook. - Reuse `media.status.changed` rather than adding a new signal or changing any kernel or contract surface. A production-status change is a domain fact; the notification hook is only one of its consumers. - Its `related_users` audience (media-monitoring owners, subject side "source") makes the planner create one run per recipient, executed as that user. A hook that targets the signal but is not the signal's `notificationHookSlug` runs for every recipient without notification preference gating, so muting the notification does not disable auto-completion. - `entitySchemaSlug` is added to the signal's properties so the script can pick the show or podcast episodic config without an extra query. Why two hooks share one script ------------------------------ - `media.auto-complete-episodic-parent` stays `required` but now targets only `show-episode:complete` and `podcast-episode:complete`. A user completing the last episode sees the parent complete within the same request. - `media.auto-complete-on-status-change` is a new `async` hook on the `media.status.changed` emit. The signal is itself emitted from the async `media.entity-updated` hook that nobody waits on, so there is no request to hold open and nothing benefits from `required` delivery. - Both declare `executionScope: "user"`, so no global run is ever planned for either of them. Consequences and trade-offs --------------------------- - Provider population no longer plans this hook at all: no automation warnings, no waiting on it, no global runs, no cross-user event reads. - `media.status.changed` fires only when the old status is known (and only for roots that were previously populated), so the first population of an already-ended show never auto-completes it. Completing the final episode afterwards still does, through the event path. - Users who have a show in their library but do not monitor it are not auto-completed when production ends; they are auto-completed on their next final episode completion instead. What changed ------------ - plugins/media/host/schemas/signal.ts: `media.status.changed` gains the required string property `entitySchemaSlug`. - plugins/media/backend/automations/media-entity-updated.sandbox.ts: the status emission includes `entitySchemaSlug: after.entitySchemaSlug`. - plugins/media/host/plugin.ts: `media.auto-complete-episodic-parent` targets only the two episode-complete events and declares `executionScope: "user"`; new `media.auto-complete-on-status-change` (after, async, user scope) targets the `media.status.changed` emit with the same `automation.media-auto-complete-episodic-parent` script. - plugins/media/backend/automations/auto-complete-episodic-parent.sandbox.ts: the entity-update trigger branch is replaced by a signal branch. For `media.status.changed` it selects the config from `entitySchemaSlug` (show or podcast only), requires `subjectEntityId`, and triggers only when the new status is terminal and the old one was not. The input projection drops `entity` and selects `signal` properties `entitySchemaSlug`, `oldStatus`, `newStatus`. - plugins/media/README.md: the Lifecycle section describes the two hooks. Tests ----- - auto-complete-episodic-parent.test.ts: entity-update cases become signal cases; terminal transitions complete shows and podcasts, already-terminal or non-terminal transitions, non show/podcast schemas, and a missing `subjectEntityId` do not query at all; coverage, on-hold, and resume cases now start from the signal; the manifest projection is asserted. - media-entity-updated.test.ts: the status emission carries `entitySchemaSlug`. - host/plugin.test.ts: both hook declarations (required event-only hook and async signal hook, both user-scoped) are asserted exactly. - input-projections.test.ts: the script's pinned projection is updated. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Replace paginated episode and event replay with one parent-rooted RyotQL snapshot that derives coverage closure and shared consumed-on values in the query. Remove the event-schema lookup capability, create parent completion events by slug, and update focused tests and lifecycle documentation.
Update every active Effect package, the root platform override, and the Deno sandbox import map from rc.115 to rc.116. Regenerate bun.lock with the new package metadata and integrity hashes.
Add a deterministic direct sandbox scenario at the lower payload boundary required by the Effect durable-deferred regression validation. Keep the existing 1 MiB and near-4 MiB scenarios unchanged so the targeted remote run can verify terminal completion across representative accepted result sizes.
Update e2e expectations for add-to-library events, active client-page frames, asynchronous results tables, and batched media automation execution. Narrow projected event change schemas, read workflow journals from one Redis snapshot, and preserve focused regression coverage for browser, backup, automation, and event lifecycle behavior.
Validate the Effect 4.0.0-rc.116 fix for the lost DurableDeferred wake reported in Effect issue 8238 against the deployed PR image on the canonical 2-vCPU, 4-GB, no-swap benchmark host. Run five discarded direct sandbox executions at each required result boundary: 64 KiB, 1 MiB, and 3,900,000 bytes just below the 4 MiB limit. All 15 accepted executions reached the completed terminal state. The observed p50/p95 latencies were 1,398/1,455 ms, 2,342/2,406 ms, and 3,770/3,981 ms respectively. No request failed, timed out, reported a failure stage, or increased the cgroup OOM-kill counter, and the safety watchdog did not trigger. Record the deployed image, host, runtime, configuration, application samples, per-repetition scenario artifacts, summary, and evidence-based report. Mark host sampling unhealthy because sequential Docker statistics produced 8-9 second intervals and the local stream ended early; retain the valid sub-second application evidence while explicitly excluding host-resource conclusions from this targeted correctness run. Remove the workflow TODO now that rc.116 is installed and every required payload boundary has completed repeatedly without a lost wake.
IgnisDa
marked this pull request as draft
September 13, 2026 08:43
|
|
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
There was a problem hiding this comment.
CodeQL found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.
IgnisDa
force-pushed
the
ultra-rewrite
branch
5 times, most recently
from
September 19, 2026 07:11
4dbb4d4 to
0e5c10a
Compare
The workflow journal bridge now reads the journal from one HGETALL snapshot, so the test Redis fakes must provide hgetall instead of hget/hmget. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ing and add profiling Sampling ran on a sleep-after-request loop and called `docker stats` once per container, so the first baseline's host evidence was invalid. Both samplers now use deadline scheduling with missed-slot accounting, the host series reads procfs and cgroup v2 directly, and benchmark containers resolve by exact compose service instead of a name substring. Application snapshots are collected on the benchmark host, because the public route costs more than the 200 ms interval in latency. Provider import metrics counted workflow body attempts as imports. They now name attempts as attempts, trace each phase attempt per execution so replays are not double counted, and leave logical import state to benchmark-owned records. Deno and Bun profiling is admin-gated and fails closed unless SANDBOX_BENCHMARK_PROFILE_DIR is set: one armed correlation token profiles a logical execution and every replay attempt, with checkpoint memory, smaps and heap snapshots written to restricted files. Repetition aggregation, run provenance, retention classification and profile sanitizing round out the tools for the follow-up run. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
@coderabbitai ignore