Establish a golden evaluation for memory usefulness

[miniLV]

This was generated by AI during triage.

What to build

Establish a small deterministic golden evaluation that proves the memory system helps answer future engineering questions, rather than only proving that each pipeline step runs.

The evaluation should exercise a complete path from representative local session fixtures through the bounded evidence and Wiki surfaces to the read-only query contract. It should make regressions visible without requiring live personal session logs or a model-as-judge.

Acceptance criteria

• The corpus contains 10–20 representative questions covering recent state, exact tickets and paths, historical decisions, root causes, superseded conclusions, reusable guidance, and no-match cases.
• At least one long multi-workstream session fixture verifies whether important earlier outcomes survive the bounded Evidence Snapshot.
• Expected facts, expected evidence pages or cards, and expected NO_MATCH behavior are explicit and reviewable.
• Evaluation output reports fact hits, citation correctness, false reuse, and no-match correctness per case and in summary.
• The first version is deterministic and local-substitutable; it does not require an external model judge.
• Fixtures contain no private user data and are safe to keep in the repository.
• Existing tests and strict Wiki lint remain green.

Blocked by

None — can start immediately.

[miniLV]

This was generated by AI during triage.

Agent Brief

Category: enhancement
Summary: Add deterministic end-to-end goldens that measure whether engineering memory produces correct, cited, non-fabricated answers.

Current behavior:
The repository has strong tests for capture, setup, workflow, and lint mechanics. It does not have a stable acceptance surface for future-question usefulness, so Snapshot recall and query compatibility risks cannot be measured before refactoring.

Desired behavior:
A small repository-safe corpus runs locally and reports whether expected facts and citations survive the memory path, whether stale or unreviewed guidance is incorrectly reused, and whether absent information returns NO_MATCH.

Key interfaces:

• Golden case — fixture sessions, user question, expected facts, expected evidence, and expected match state.
• Evaluation result — per-case fact hits, citation correctness, false reuse, and no-match correctness plus a summary.
• Memory path under evaluation — bounded Snapshot, Daily or Concept evidence, and read-only query behavior.

Acceptance criteria:

• The corpus includes 10–20 safe, representative cases across the supported query intents.
• A multi-workstream long-session case detects loss of an important early outcome.
• A superseded-conclusion case requires the current decision rather than the stale one.
• A no-evidence case requires NO_MATCH and rejects fabricated support.
• Results are deterministic and require no live user logs or external model judge.
• Output is useful both in tests and as a human-readable baseline report.
• Existing tests and strict lint remain green.

Out of scope:

• Changing Snapshot selection policy in this issue.
• Replacing the query Skill or adding a vector database.
• Introducing an LLM judge framework.

[hegu-1]

A golden eval for usefulness is exactly what these memory systems need. Pipeline tests prove that files move; they do not prove that future agents make better decisions.

I would include questions that force the system to distinguish:

• current vs superseded conclusions
• exact source path/ticket recall vs broad semantic recall
• user-authored decisions vs agent-inferred summaries
• root cause vs symptom
• answerable from memory vs requires fresh inspection

The last category matters: good memory should also know when not to answer from memory.

I have been using a personal memory vault with the same concern: raw evidence, wiki nodes, playbooks, and generated outputs are separate so evaluation can ask which layer supported the answer: https://github.com/hegu-1/personal-memory-vault-starter

[miniLV]

@hegu-1 Your earlier distinction between memory-only support and cases that require fresh inspection helped shape the evaluation boundary here.

I have opened a small, privacy-preserving first-Daily pilot in #6 to measure the earlier activation boundary: preflight → setup → first Daily. If you already use Codex App on macOS and are interested, your perspective from a layered memory vault would be especially relevant. No star or public session data is requested, and no pressure if the environment is not a fit.