fix(detect): honor .graphifyignore for the memory tree (fixes #2267)

[Munawarx]

Problem

.graphifyignore patterns are silently bypassed for graphify-out/memory/, so users cannot exclude their own saved Q&A output from the corpus. The tree gets re-ingested on every run and node counts grow for reasons unrelated to the source tree.

Root cause

In graphify/detect.py the ignore check is guarded:

if not in_memory and _is_ignored(p, root, ignore_patterns, _cache=ignore_cache):
    ignored.append(str(p)); continue

The not in_memory condition means user ignore patterns never apply to the memory tree. This is issue #2267, Option 1 (the maintainer-friendly fix the reporter requested).

Solution

Remove the not in_memory guard so _is_ignored applies to the memory tree like every other path. Default behavior is unchanged: memory is still ingested unless explicitly ignored.

As a side benefit, the hidden/noise filter (which also keyed off in_memory) now correctly applies to the memory tree, so stray dotfiles dropped there are no longer unconditionally picked up.

Testing

• Added test_graphifyignore_excludes_memory_tree in tests/test_detect.py (matches existing style, uses tmp_path).
• Verified the test FAILS on the unfixed code and PASSES with the fix.
• Regression: with no .graphifyignore, memory is still ingested (default loop intact).
• Edge cases checked: parent pattern graphify-out/, nested proj/graphify-out/memory/, and negation patterns behave correctly.

Expected impact

Users can finally scope the graph to the source tree only. Removes silent config-ignored behavior.

Fixes #2267

[safishamsi]

Thanks @Munawarx — the gap is real (user ignore patterns never reach graphify-out/memory/). But the one-line change over-reaches: ignore_patterns includes .gitignore and $GIT_DIR/info/exclude, not just .graphifyignore. Since graphify-out/ is generated output and is commonly gitignored, dropping the guard silently disables the memory round-trip by default for those repos — I verified _is_ignored(<root>/graphify-out/memory/query_1.md, ...) returns True with graphify-out/ in .gitignore, which contradicts the 'default unchanged' claim. Please scope this to .graphifyignore-sourced patterns only (track pattern provenance and apply just those to the memory tree), and add a regression test that a gitignored graphify-out/ still keeps memory included. Happy to re-review.

[Munawarx]

Thanks @safishamsi for the detailed review. I've reworked the fix to track pattern provenance exactly as you described.

What changed:

Each ignore pattern now carries a source tag (3rd tuple element): 'gitignore', 'graphifyignore', 'info_exclude', or 'cli_exclude'.

• _load_dir_own_ignore: tags each pattern by its origin file (.gitignore vs .graphifyignore)
• _load_graphifyignore: tags /info/exclude entries and propagates per-directory tags
• _is_ignored: new optional sources parameter filters by provenance
• detect() main loop: memory-tree files call _is_ignored with sources={'graphifyignore'} — only .graphifyignore-sourced patterns are evaluated for memory

This means a gitignored graphify-out/ no longer drops memory files, while .graphifyignore patterns still apply to the memory tree as before.

Regression tests added:

1. test_gitignored_graphify_out_keeps_memory_included — graphify-out/ in .gitignore → memory files stay included (the exact scenario you verified was broken)
2. test_info_exclude_does_not_drop_memory — same protection for /info/exclude
3. test_graphifyignore_excludes_memory_tree — user's .graphifyignore still excludes memory tree

All 3 new tests pass; no regressions in the existing suite (225 passed). Happy to address anything else.

[graphify-labs]

Looks safe to merge — no coupling regressions and no blocking issues, checked against the code graph (not a self-assessment).

Graphify reviewed this change.

Looks safe to merge — no coupling regressions and no blocking issues, checked against the code graph (not a self-assessment).

---

Graphify review — findings

This PR adds provenance tracking to the ignore-pattern loading in graphify/detect.py, so each pattern is tagged with its source (.gitignore, .graphifyignore, $GIT_DIR/info/exclude, or CLI exclude) by changing the pattern tuples from (anchor, pattern) to (anchor, pattern, source). The _is_ignored function gains an optional sources filter, and detect() uses it so that files under the graphify-out/memory/ tree are only filtered by .graphifyignore-sourced patterns rather than all ignore sources. It also adds tests covering the memory-tree exclusion behavior for .graphifyignore, gitignored graphify-out/, and $GIT_DIR/info/exclude. Surface area is _load_dir_own_ignore, _load_graphifyignore, _is_ignored, the memory-handling branch in detect, plus new/updated tests in tests/test_detect.py.

Worth a look

• _is_ignored cache reuses results across different source filters — graphify/detect.py:1109 · Escalate · medium
  • agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review

Analysis details — impact, health, verification

Impact & health

Graphify review

Impact — 1151 functions depend on the 425 functions this change touches.

Health — this change adds coupling hotspots:

• worse: detect() — 83 callers, 11 callees

Verification — 1151 functions in the blast radius were not formally verified this run (proofs are advisory here).

Gate & verification

graphify gate

PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.

Advisory (not blocking):

• verification_scope: 618 function(s) in the blast radius were not formally verified this run

· 1 more finding(s) on lines outside this diff (see the check run).

[Munawarx]

Also addressed the cache collision flagged by the Graphify review bot.

Issue: _is_ignored cached results keyed by path alone. Since the same _cache dict is shared across calls in one scan, a non-memory file under graphify-out/ (evaluated with all sources including .gitignore) would poison the cache — when a memory file's ancestor walk hit that cached entry, it returned True and wrongly dropped the memory file.

Fix: Cache key is now (path, frozenset(sources)) when a source filter is active. Evaluations under different source filters never collide.

Test added: test_memory_cache_not_poisoned_by_non_memory_eval — places a non-memory output.json + memory query_1.md under the same gitignored graphify-out/ and verifies the memory file survives.

All tests pass: 226 passed, 0 new failures.

[graphify-labs]

Looks safe to merge — no coupling regressions and no blocking issues, checked against the code graph (not a self-assessment).

Graphify reviewed this change.

Looks safe to merge — no coupling regressions and no blocking issues, checked against the code graph (not a self-assessment).

---

Graphify review — findings

This PR adds provenance tracking to graphify's ignore-pattern system, changing the internal (anchor, pattern) tuples into (anchor, pattern, source) triples tagged with constants like _IGNORE_SOURCE_GITIGNORE and _IGNORE_SOURCE_GRAPHIFYIGNORE. The _is_ignored function gains an optional sources filter (and correspondingly keyed cache) so callers can evaluate only patterns from selected origins. In detect(), files under graphify-out/memory/ are now filtered using only .graphifyignore-sourced patterns rather than being fully exempt from ignore checks. The surface area spans _load_dir_own_ignore, _load_graphifyignore, _is_ignored, and the walk loop in detect(), plus two new tests covering memory-tree exclusion via .graphifyignore and non-exclusion via .gitignore. Most of the listed changed test symbols appear to be rationale/fixture entries rather than substantive logic changes.

Worth a look

• sources={} filter uses truthy check causing empty set to behave as no filter for cache key but as filter for evaluation — graphify/detect.py:1122 · Escalate · medium
  • agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
• Shared ignore_cache collides between filtered and unfiltered lookups when sources is empty — graphify/detect.py:1123 · Escalate · medium
  • agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review

Analysis details — impact, health, verification

Impact & health

Graphify review

Impact — 1153 functions depend on the 427 functions this change touches.

Health — this change adds coupling hotspots:

• worse: detect() — 84 callers, 11 callees

Verification — 1153 functions in the blast radius were not formally verified this run (proofs are advisory here).

Gate & verification

graphify gate

PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.

Advisory (not blocking):

• verification_scope: 620 function(s) in the blast radius were not formally verified this run

· 1 more finding(s) on lines outside this diff (see the check run).