Summary
_origin: "ast" is the marker several code paths use to decide whether a node came from AST or
semantic extraction. But it is only stamped by the current extract path (graphify/extract.py:5433),
so graphs carrying nodes written by older versions have a large population of AST nodes with no
_origin. Provenance-aware logic then misclassifies them as semantic.
Evidence
Real repo, graph built up across 0.4.x → 0.7.10 → 0.9.27:
document nodes total: 12,243
- with
_origin == "ast": 783
- with no
_origin at all: 11,460
All 12,243 are structurally identical — file_type=document, source_location="L<line>" — i.e. all
of them are markdown-extractor output (extractors/markdown.py emits file + heading nodes with
L<line>). Semantic extraction emits source_location: null per the extraction spec. So the true
AST population is 12,243 and only 6% of it is marked.
Why it matters
Consumers treat missing _origin as "semantic":
graphify/watch.py:568-583 — preserved_nodes keeps anything without _origin=="ast"
graphify/watch.py:601 — same for edges
graphify/build.py:401, :724 — AST-wins-on-collision resolution
In this repo the misclassification happens to fail safe (legacy AST nodes are preserved rather than
replaced), but it is unintentional, and it makes the marker unusable as the basis for the tier-scoped
merge fix proposed in #2333. A migration that infers _origin where it is missing, or a fallback that
treats source_location =~ ^L\d as AST, would make provenance reliable.
I used the source_location fallback locally and it partitioned the graph correctly where _origin
did not (scenario C vs B in #2333: +2,639 nodes vs -8,290).
Suggested fix
Either (a) backfill _origin on load for nodes matching the AST shape, or (b) make the provenance
predicate a helper that falls back to source_location when _origin is absent, and use that helper
everywhere instead of the raw == "ast" comparison.
Environment
graphifyy 0.9.27, Python 3.11, Windows 11.
Summary
_origin: "ast"is the marker several code paths use to decide whether a node came from AST orsemantic extraction. But it is only stamped by the current extract path (
graphify/extract.py:5433),so graphs carrying nodes written by older versions have a large population of AST nodes with no
_origin. Provenance-aware logic then misclassifies them as semantic.Evidence
Real repo, graph built up across 0.4.x → 0.7.10 → 0.9.27:
documentnodes total: 12,243_origin == "ast": 783_originat all: 11,460All 12,243 are structurally identical —
file_type=document,source_location="L<line>"— i.e. allof them are markdown-extractor output (
extractors/markdown.pyemits file + heading nodes withL<line>). Semantic extraction emitssource_location: nullper the extraction spec. So the trueAST population is 12,243 and only 6% of it is marked.
Why it matters
Consumers treat missing
_originas "semantic":graphify/watch.py:568-583—preserved_nodeskeeps anything without_origin=="ast"graphify/watch.py:601— same for edgesgraphify/build.py:401,:724— AST-wins-on-collision resolutionIn this repo the misclassification happens to fail safe (legacy AST nodes are preserved rather than
replaced), but it is unintentional, and it makes the marker unusable as the basis for the tier-scoped
merge fix proposed in #2333. A migration that infers
_originwhere it is missing, or a fallback thattreats
source_location =~ ^L\das AST, would make provenance reliable.I used the
source_locationfallback locally and it partitioned the graph correctly where_origindid not (scenario C vs B in #2333: +2,639 nodes vs -8,290).
Suggested fix
Either (a) backfill
_originon load for nodes matching the AST shape, or (b) make the provenancepredicate a helper that falls back to
source_locationwhen_originis absent, and use that helpereverywhere instead of the raw
== "ast"comparison.Environment
graphifyy 0.9.27, Python 3.11, Windows 11.