Skip to content

_origin provenance marker is only stamped by current extract path — legacy AST nodes are misread as semantic #2334

Description

@SungKyeol

Summary

_origin: "ast" is the marker several code paths use to decide whether a node came from AST or
semantic extraction. But it is only stamped by the current extract path (graphify/extract.py:5433),
so graphs carrying nodes written by older versions have a large population of AST nodes with no
_origin
. Provenance-aware logic then misclassifies them as semantic.

Evidence

Real repo, graph built up across 0.4.x → 0.7.10 → 0.9.27:

  • document nodes total: 12,243
  • with _origin == "ast": 783
  • with no _origin at all: 11,460

All 12,243 are structurally identical — file_type=document, source_location="L<line>" — i.e. all
of them are markdown-extractor output (extractors/markdown.py emits file + heading nodes with
L<line>). Semantic extraction emits source_location: null per the extraction spec. So the true
AST population is 12,243 and only 6% of it is marked.

Why it matters

Consumers treat missing _origin as "semantic":

  • graphify/watch.py:568-583preserved_nodes keeps anything without _origin=="ast"
  • graphify/watch.py:601 — same for edges
  • graphify/build.py:401, :724 — AST-wins-on-collision resolution

In this repo the misclassification happens to fail safe (legacy AST nodes are preserved rather than
replaced), but it is unintentional, and it makes the marker unusable as the basis for the tier-scoped
merge fix proposed in #2333. A migration that infers _origin where it is missing, or a fallback that
treats source_location =~ ^L\d as AST, would make provenance reliable.

I used the source_location fallback locally and it partitioned the graph correctly where _origin
did not (scenario C vs B in #2333: +2,639 nodes vs -8,290).

Suggested fix

Either (a) backfill _origin on load for nodes matching the AST shape, or (b) make the provenance
predicate a helper that falls back to source_location when _origin is absent, and use that helper
everywhere instead of the raw == "ast" comparison.

Environment

graphifyy 0.9.27, Python 3.11, Windows 11.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions