Repository navigation
fix(pipeline): preserve concurrent manage_adr write across reindex swap - #2315
0xb007ab1e wants to merge 12 commits into
Conversation
|
Thanks for opening this — it has been seen, and it is queued. This note is automated, but it is not a brush-off: it exists so you know where your PR stands instead of having to guess from silence. Current review status: working through a backlog. What that means for this PR, concretely:
Things that will genuinely speed it up whenever review does happen:
If this fixes a bug, a reproduction we can run is worth more than a description of the symptom. Thanks for contributing, and sorry in advance for the wait. |
DeusData
left a comment
There was a problem hiding this comment.
Thank you for #2314 and #2315. Silently losing an ADR is exactly the kind of bug nobody notices until the decision record is gone, and your root-cause write-up (a shared project_summaries row, early capture and late swap, in-process guards only) is precise. We appreciate the honest risk section too.
We'd like to go with the direction you named as Alternative 1: a separate per-project ADR store that a reindex never replaces. Here is why we prefer it over the lock:
- It removes the race instead of narrowing it. When the ADR doesn't live in the file a reindex regenerates, there is no capture, re-read or re-apply to keep in step, and no lock to coordinate across processes.
- The lock as written conflicts with two of our standing rules. We don't put sleeps or polling waits into the code, and we don't let a timeout decide an outcome. The 10 ms retry loop, with its 30 s fail-open, does both: a slow peer silently brings the data loss back.
- The re-apply happens after the swap. For a moment the published DB shows the stale ADR, and if the re-apply write fails, the rescued ADR is lost with only a log line. An ADR deleted during the reindex would also come back from the early capture.
What we'd look for in the reworked PR:
- ADRs stored in their own file beside the project DB (your
<project>.adr.dbnaming is fine), read and written bymanage_adr, and never touched by index publication. - A one-time migration: an existing ADR in
project_summariesmoves to the new store the first time it is read or written, and the old row is no longer used as the source of truth. Existing indexes must keep their ADRs without a reindex. delete_projectremoves the ADR store with the project, or keeps it deliberately and says so. Please pick one and document it.- Tests: an ADR written between the start of a reindex and its publish survives (your hook-based test adapts directly); an ADR created before upgrading is migrated and still readable after a reindex; delete behaves as documented.
- The early-capture and re-apply code in the pipeline can then go away, which makes publication simpler.
If you'd rather not take on the larger change, tell us and we will pick it up ourselves, crediting your analysis and test. Either way, thank you. This was a careful find.
9527694 to
c27a535
Compare
|
Reworked to Alternative 1 as you asked — thanks for the precise review; agreed on removing the race rather than narrowing it, and on the no-sleep/no-timeout rules.
Tests (file-based): an ADR seeded in the legacy Built clean under |
DeusData
left a comment
There was a problem hiding this comment.
Thank you, @0xb007ab1e, for such a fast and thorough rework. You took the structural fix from your own "Alternatives" section, and the result is much simpler: publication no longer writes ADRs at all, the capture and re-apply machinery and its fail-capture hook are gone, and BEGIN IMMEDIATE now runs on the connection it is meant to protect. That is exactly the direction we hoped for.
A few things before we can merge, each with the reason:
- Migrate before a rebuild can destroy the legacy row. Migration currently runs lazily, on the first ADR read or write. But the format-change route (
pipeline.caround 1749-1758) and the forced-full route (around 1770-1775) unlink the old generation, and a full rebuild publishes a freshproject_summaries. On an upgrade whose first action is a full reindex (a format bump, a forced full, or the watcher's startup reindex), the ADR is therefore lost before anything copies it. Could you migrate the legacy row into the sidecar at the start of routing, whenever an existing generation is present and before any unlink or publish, and preserve and abort if that fails? A done-marker in the sidecar (for example anadr_metarow) would make the migration truly one-time. Today it re-runs on every new connection and could bring back a deleted ADR from a delta-cloned legacy row. - Lock behaviour of the sidecar. Please open it the way the graph store is opened, with
PRAGMA busy_timeout = 10000andjournal_mode = WAL(the baresqlite3_openinstore.caround 9694). Without them, a second process's write fails immediately withSQLITE_BUSY, and a read during a commit reports "no ADR". That is precisely the cross-process case this PR exists for. - Reads must not write. Query-only stores also carry
db_path, somanage_adr get,index_repository'sadr_presentandGET /api/adrcurrently create<db>.adr.db. On a read-only cache, a legacy ADR that used to be readable now errors out. For reads, please open the sidecarSQLITE_OPEN_READONLYif it exists, and otherwise read the legacy row from the graph DB without migrating. - Path helper. Please build the sidecar path through
cbm_path_for_file_apiwith a 4096-byte buffer, likestore_open_internal. On Windows that adds the long-path\\?\prefix; without it, a long cache path opens the graph but not the ADR. - Memory-core gate.
store.cgoes from 358 to 359 raw allocator sites against a baseline of 358, somake -f Makefile.cbm lint-memory-corewill fail.adr_db_pathis written but never read, so dropping the field and itsfreefixes it. Lowering the baselines forpipeline.c(to 129) andpipeline_incremental.c(to 134) in the same change is welcome. - The ADR should travel with the exported artifact. The team-shared
.codebase-memory/graph.db.zstused to carry the ADR inproject_summaries, so a teammate's first index started with the team's decisions. With the sidecar that silently stops. Please carry it across: write the sidecar's ADR into the exported DB, and restore it into the sidecar on import when no local ADR exists. - Tests. Please add:
- (a) an ADR written from a pipeline test hook between the start of the reindex and publication survives, on both the full and the delta route;
- (b) the upgrade order: seed a legacy row, run a forced-full or format-change reindex with no ADR read first, then read the ADR;
- (c)
delete_projectthrough the MCP handler removes<db>.adr.dband its-wal/-shm/-journal, and a re-indexed project of the same name starts with no ADR; - (d) the artifact round trip keeps the ADR.
- Docs and description. Please mention in the
delete_projecttool description and in the README that deleting a project also deletes its ADR, and update the PR description to the sidecar design. It still describes the lock andcompat_fs.c.
A smaller note, not a blocker: the sidecar name ends in .db, so the CLI's cbm_list_indexes / count_db_indexes count it as an index. A one-line exclusion there would help.
Thanks again. The hard part is done, and this is close.
c27a535 to
2d2c0fd
Compare
|
Round 2 pushed — thank you for the precise, itemized review. All eight plus the minor:
Verified locally: |
DeusData
left a comment
There was a problem hiding this comment.
Thank you for round 2, @0xb007ab1e. This is a careful, thorough rework:
- Moving ADRs into the per-project sidecar removes the race rather than narrowing it, and deleting the capture and re-apply machinery made publication noticeably simpler.
- Reusing
stage_root_lengthfor the staging-to-final derivation was the right call. It recognises only the exact minted.stage.XXXXXXbasename, so a real path containing.stage.is safe. - You brought the memory-core baselines down in
pipeline.candpipeline_incremental.cwhile keepingstore.cat 358. Appreciated.
Items 2, 3, 4, 5, 7c, 7d and 8 are done. A few small things before we merge:
- The migrate-before-delete step fails open (
pipeline.caround 1728-1731).cbm_store_open_path_queryreturns NULL for open failures as well as for a missing file, and on this route the final generation is known to exist. Soif (!mig) return true;can let a rebuild publish over a legacy ADR it never moved. Suggested:if (!mig) return !cbm_file_exists(final_path);. - A deleted ADR can come back through the artifact. After migration, the legacy
project_summariesrow stays in the graph DB, and incremental and closure-repair clones keep carrying it.VACUUM INTOcopies it into the snapshot. When the sidecar has no ADR, nothing is injected, the stale row ships, and a teammate's import restores it. Suggested: insnapshot_inject_adr, alwaysDELETE FROM project_summariesfirst, then insert only when an ADR exists. - Make the one-time migration atomic (
store.caround 9673-9768). Please wrap the marker check, the legacy copy and the marker insert in oneBEGIN IMMEDIATE … COMMIT. Otherwise two processes migrating at once can re-copy a row the other just deleted. It would also help ifcbm_store_adr_getfell back to the legacy graph row when the sidecar exists but has nomigratedmarker, so a half-created sidecar never hides an ADR. - Tests.
- Please add the delta / closure-repair variant of the ADR-written-mid-reindex test.
- Pin the route in both new pipeline tests with
cbm_pipeline_incremental_test_last_route()(FORCED_FULL, and CLOSURE_REPAIR for the delta one), so they stay bound to their routes if routing changes. - The preserve-and-abort path on migration failure lost its test when the old hook went away. Putting a directory at
<db>.adr.dbshould make it fail deterministically with no hook: assertCBM_PIPELINE_ABORT_PRESERVE_DBand that the old generation is intact.
Nits, take them or leave them:
- Give the read-only sidecar handle the same 10 s busy timeout.
adr_sidecar_pathshould compare(size_t)n >= raw_szrather than against an(int)cast.- In
delete_project, remove the sidecar only once the graph DB unlink has succeeded, and sizeadr_pathlikepathplus the suffix. is_project_db_file(mcp.c) and the cross-repo cache scan could skip*.adr.dbthe way the CLI listing now does.- In the PR body, "migrated on first access" now means "on first write or before the next rebuild".
Thanks again. This is close, and the design is exactly where we wanted it.
2d2c0fd to
a1ac807
Compare
|
Round 3 pushed — thank you again for the depth here. Correctness
Tests
Nits — all applied: the read-only handle gets the 10 s busy timeout; Verified locally: If any other maintainers or contributors want to weigh in on the sidecar approach, the migration ordering, or edges I might have missed, I'd welcome the input — happy to adjust. |
DeusData
left a comment
There was a problem hiding this comment.
Thank you, @0xb007ab1e. Round 4 closes every item we raised, and in the way we'd have written it ourselves: the fail-closed open, the DELETE-first snapshot, one BEGIN IMMEDIATE covering marker, copy and marker insert, and a fallback so a half-created sidecar never hides an ADR. The directory at <db>.adr.db is a neat, hook-free way to make migration fail deterministically, and pinning both survival tests to their routes means they can't silently turn into full rebuilds.
One edge case is left, in import. The restore goes through cbm_store_adr_migrate_once, which does nothing once the local sidecar is marked, and any earlier rebuild or ADR write sets that marker. So a teammate who has already indexed the project and has no local ADR gets nothing from the artifact's ADR (your artifact_import_does_not_clobber_local_adr comment notes this too). On main, import replaced the whole DB, so the shared ADR always came through.
We'd like import to copy the artifact's ADR into the sidecar whenever the local sidecar has no row for the project, while still never overwriting a local one. A test would pin it: index the destination first (marked, no ADR), import an artifact carrying ADR "A", and expect "A". Everything else is ready. Thanks again for sticking with this through four rounds; the sidecar design came out clean.
|
Done — thank you, that was the right call. Import no longer relies on the marker-gated migrate-once. It now uses a new Test as you specified — Verified locally: |
a1ac807 to
3327ee5
Compare
|
Thank you, @0xb007ab1e. Round 5 is exactly the policy we asked for, and it verified cleanly on macOS: 1,375 tests pass merged with current On Our reading: both tests import while the destination graph DB still exists (and may still be open). Windows refuses to rename over an existing or open file, so The smallest fix is probably in the tests: close any store handle and remove the destination graph DB before calling Once Windows is green, we'll approve and merge. The three small nits from our verification (a silent failed restore in |
|
Fixed — thanks for the precise diagnosis, that was exactly it. Both new import tests now remove the destination graph DB (and its No production change — I agree import over an existing DB isn't the right shape, so I kept it test-only. Verified locally: Thanks for handling the three nits in a follow-up. |
3327ee5 to
232730c
Compare
|
Gentle check-in, no rush. The Windows fix (232730c) is green now — the Is there anything else you'd like me to address before approval? Happy to adjust. And thanks again for the thorough reviews — the sidecar design is much better for them. |
232730c to
7b3c0f5
Compare
|
Rebased onto Only one real conflict, in Local validation on the rebased head:
Should be mergeable once the review's cleared whenever it's convenient. Thanks again for the reviews. |
|
Thanks for addressing the Windows import-test setup and for the subsequent rebase. I have read the green-Windows update and your follow-up. Rechecking the revised head and making the landing decision are now maintainer-side steps; no additional contributor work is requested here. |
Reworks the ADR-loss-across-reindex fix to Alternative 1 (per the PR review):
remove the race rather than narrow it. ADRs live in a per-project sidecar SQLite
DB ("<graph_db>.adr.db"), separate from the graph DB a reindex rebuilds or
deletes, so there is no capture, re-read, re-apply, or lock — and no sleeps,
polling, or timeout deciding an outcome.
store.c:
- ADR writes (store/delete/update_sections) use a cached read-write handle opened
SQLITE_OPEN_READWRITE|CREATE and configured like the graph store (PRAGMA
busy_timeout=10000 + journal_mode=WAL) so cross-process access waits rather
than failing SQLITE_BUSY. A :memory: store keeps ADRs in the graph DB.
- Reads never write: cbm_store_adr_get uses the cached RW handle if open, else
opens the sidecar SQLITE_OPEN_READONLY (with the same busy timeout) when the
file exists, else reads the legacy graph row without creating anything. When
the sidecar exists but is NOT marked migrated (half-created), it falls back to
the legacy graph row so a partial sidecar never hides an ADR.
- One-time migration is atomic: adr_copy_legacy_once wraps the marker check,
legacy copy, and marker insert in a single BEGIN IMMEDIATE...COMMIT
(re-checking the marker under the lock; ROLLBACK->ERR on failure). New public
cbm_store_adr_migrate_once. Sidecar path via cbm_path_for_file_api (Windows
long-path). Dropped the write-only adr_db_path field.
pipeline.c: migrate a legacy ADR into the sidecar BEFORE the format-change and
forced-full routes unlink the old generation; preserve-and-abort on failure, and
do NOT fail open — a query-open returning NULL preserves unless the final DB
genuinely does not exist. Migration targets the FINAL path (staging suffix
stripped). The early-capture/re-apply machinery and its test hook are gone.
artifact.c: the exported artifact carries the ADR — snapshot_inject_adr always
DELETEs project_summaries first, then injects only when an ADR is present, so a
deleted ADR never resurrects through a lingering legacy row. Import restores the
ADR into the destination sidecar via cbm_store_adr_restore_from_graph_if_absent:
it copies the artifact's ADR (INSERT OR IGNORE — never clobbers a local one)
regardless of the migrated marker, so a teammate who already indexed the project
(marked sidecar, no ADR) still receives the shared decision record. store.c
factors the row copy into adr_copy_rows, shared by the one-time legacy migration
and the ungated restore.
mcp.c: delete_project removes "<db>.adr.db" (+ -wal/-shm/-journal) only after the
graph unlink succeeds; is_project_db_file skips *.adr.db. cli.c: exclude
*.adr.db from index listing/counting. README + memory-core baselines updated.
Tests (file-based): ADR written mid-reindex survives on BOTH the full and delta
routes (route-pinned FORCED_FULL / CLOSURE_REPAIR); a legacy row migrates before
a forced-full/format reindex with no prior read; migration failure (a directory
at <db>.adr.db) preserves-and-aborts with the old generation intact;
delete_project via the MCP handler removes the sidecar + journals; the artifact
export->import round trip keeps the ADR; a deleted ADR does not resurrect on
import; an import does not clobber a local ADR; an import into an already-indexed
destination (marked sidecar, no ADR) restores the artifact's ADR; a half-created
sidecar falls back to the legacy row. The import tests remove the destination
graph DB (keeping the sidecar + marker) before importing, matching production
(try_artifact_bootstrap imports only when the graph DB is missing) so the rename
succeeds on Windows, not only on POSIX.
Closes DeusData#516
Signed-off-by: 0xb007ab1e <134006168+0xb007ab1e@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
7b3c0f5 to
001fbe2
Compare
|
Rebased onto current The round-4 edge case (import must restore the artifact's ADR into a marked-but-empty local sidecar) was already addressed in this commit: import calls Conflict resolution of note: Focused ADR suites pass locally under ASan/UBSan (store_arch/artifact: round-4 + sidecar; pipeline: full+delta reindex-survival + migration; mcp: delete-sidecar). The network-gated |
|
Thank you for the October 22 refresh and for spelling out the marked-but-empty import case, centralized sidecar exclusion, and preservation tests. The revised head is now waiting on our review; no repeat rebase is requested. Sorry for the delay—I’ll follow up soon. |
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
Thanks again, @0xb007ab1e. Main moved under this branch (four of the files you touch changed upstream), so we merged current main into your branch with a plain signed-off merge commit (8cfefa6). There were no conflicts and your commits are unchanged; this just lets CI re-validate the combination. If you push again, please pull first so the merge commit is kept. We will merge once the required checks are green. |
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
Thanks again, @0xb007ab1e. The checks on your branch came back green, but another change touching tests/test_pipeline.c landed on main after that run started, so to keep the green honest we merged current main into your branch once more (one signed-off merge commit, d07f6d6; your change is untouched). We will merge as soon as the required checks are green on it. Please pull before pushing again. |
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
Thanks again, @0xb007ab1e. Several other pull requests landed on main this evening, including changes in files this one also touches, so we merged current main into your branch once more (one signed-off merge commit, 04cb937; your changes are untouched). That keeps the required checks honest against today's tree; we will merge once they are green. Please pull before pushing again. |
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
Thank you for your patience, @0xb007ab1e. CI went green on this branch, but tonight's merges on main touched src/mcp/mcp.c, src/pipeline/pipeline_incremental.c and their tests, which this pull request also changes, so that green no longer covered today's main. We merged current main into the branch once more (one merge commit, 791e378; your changes are untouched) and will merge once the fresh checks are green. Please pull before pushing again. |
tests/test_pipeline.c: main and this branch added tests at the same spot; both are kept unchanged. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
Thank you for your patience, @0xb007ab1e. This evening #2305 landed on main and added new tests in tests/test_pipeline.c at the same spot as yours, so the files conflicted. We merged current main into your branch (one merge commit, 8fae39e) and kept both sets of tests unchanged; the rest of your change is untouched. We will merge once the fresh checks are green; please pull before pushing again. |
Main and this branch added tests at the same spot of tests/test_pipeline.c. Joining the two conflict sides lost the closing lines git had moved out of the conflict, so a test function was left open and the file did not compile. Rebuild the file as main plus this branch own changes to it. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
Sorry, @0xb007ab1e, our merge of main earlier this evening broke the build of tests/test_pipeline.c on this branch. Both sides had added tests at the same spot, and joining them lost a few closing lines, so one test function was left open. dda8f8a rebuilds the file as main plus exactly your changes to it (the diff against main is identical to your own change), and it compiles cleanly with the full test flags. Nothing else changed. Please pull before pushing again. |
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
Thank you for your patience, @0xb007ab1e. A pull request that merged into main a few minutes ago (#2290) changes src/cli/cli.c, which this pull request also changes, so the CI run in progress here could no longer vouch for today's main. We merged current main into your branch once more (one merge commit, b66ac84; it merged cleanly and your own change is untouched) and started a fresh run. We will merge once the required checks are green. Please pull before pushing again. |
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
Thank you for your patience, @0xb007ab1e. A pull request that merged into main a few minutes ago (#2769) changes src/pipeline/pipeline.c, which this pull request also changes, so the CI run in progress here could no longer vouch for today's main. We merged current main into your branch once more (one merge commit, 0947903; it merged cleanly and your own change is untouched) and started a fresh run. We will merge once the required checks are green. Please pull before pushing again. |
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
Thank you for your patience, @0xb007ab1e. A pull request that merged into main a few minutes ago (#2600) changes src/mcp/mcp.c, which this pull request also changes, so the CI run in progress here could no longer vouch for today's main. We merged current main into your branch once more (one merge commit, 9f70764; it merged cleanly and your own change is untouched) and started a fresh run. We will merge once the required checks are green. Please pull before pushing again. |
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
Thank you for your patience, @0xb007ab1e. A pull request that merged into main tonight (#2601) changes src/pipeline/pipeline_incremental.c, which this pull request also changes, so CI on this branch could no longer vouch for today's main. We merged current main into your branch once more (one merge commit, 218123b; it merged cleanly and your own change is untouched) and started a fresh run. We will merge once the required checks are green. Please pull before pushing again. |
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
Thank you for your patience, @0xb007ab1e. Main has moved since the last CI run on this branch, and some of the new commits change src/cli/cli.c, src/mcp/mcp.c and src/pipeline/pipeline_incremental.c, which this pull request also changes, so that run can no longer vouch for today's main. Separately, the Windows leg of that run hit our 15-minute per-suite limit on a slow runner while every test was passing; that is a runner-speed issue on our side, not your change. We merged current main into your branch (one merge commit, 359bd2f; it merged cleanly and your own change is untouched) and started a fresh run. We will merge once the required checks are green. Please pull before pushing again. |
ADRs in a per-project sidecar DB, untouched by reindex
Fixes the silent loss of an ADR across a reindex. On
mainan ADR lives in the graph DB'sproject_summaries, which a reindex (format-change / forced-full / watcher startup) rebuilds or unlinks — so a decision written before or during publication is lost. The earlier lock-based approach narrowed the race but conflicted with the no-sleeps/no-timeout rules; this is the structural fix the reviewers steered toward (Alternative 1).Design
"<db>.adr.db"(its own SQLite file beside the graph DB), openedREADWRITE|CREATEwithbusy_timeout=10000+WAL, read/written bymanage_adr. Index publication never writes ADRs — there is nothing to capture or re-apply, and no lock.adr_copy_legacy_once): an existingproject_summariesADR moves into the sidecar atomically under a singleBEGIN IMMEDIATE(marker check + copy + marker insert), before any unlink/publish in the format-change and forced-full routes — preserve-and-abort on failure. A half-created sidecar falls back to the legacy row.READONLYwhen it exists, else read the legacy row without creating anything.delete_projectremoves"<db>.adr.db"(+-wal/-shm/-journal); a fullcbm_remove_indexeswipe removes sidecars too.cbm_store_adr_restore_from_graph_if_absent, ungated by the migrate marker,INSERT OR IGNOREso a local ADR is never clobbered (closes the round-4 edge case: a teammate who already indexed, marked sidecar, no ADR, still receives the artifact's ADR)..adr.dbcentrally incbm_is_project_index_db(all MCP loops + the CLI'scbm_list_indexes/count_db_indexes).Tests
Full + delta route: an ADR written between the start of a reindex and publish survives; legacy row migrates before a forced-full/format-change with no prior read; half-created-sidecar fallback;
delete_projectremoves the sidecar; artifact round-trip keeps the ADR; import restores into a marked-but-empty local sidecar without clobbering a local one.Rebase note
Rebased onto current
main(the branch had gone 619 commits stale). Conflicts resolved instore.c,pipeline.c,pipeline_incremental.c,mcp.c,README.md, and the memory-core baseline; the.adr.dbindex-scan exclusion was folded into the now-centralcbm_is_project_index_db, and the delta publish reconciled withmain's new doc-candidatedump_and_persistsignature. memory-core baselines ratcheted to measured (pipeline.c129,pipeline_incremental.c130). Focused ADR suites (store_arch / artifact / pipeline / mcp) pass locally under ASan/UBSan; the network-gated incremental fixtures run in CI.🤖 Generated with Claude Code