-
Notifications
You must be signed in to change notification settings - Fork 52
feat(devkit): add dependency topology scanner script #361
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
jensenojs
wants to merge
2
commits into
sudo-tee:main
Choose a base branch
from
jensenojs:feat/dependency-topology-tool
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
2 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,71 @@ | ||
| # Dependency Topology Scanner | ||
|
|
||
| Static analysis tool for Lua codebase dependency architecture. | ||
|
|
||
| ## File Structure | ||
|
|
||
| ``` | ||
| scripts/dependency-topology/ | ||
| ├── scan_topology.py # CLI entry: scan / diff subcommands | ||
| ├── scan_analysis.py # Core analysis: groups, edge rules, payload builders | ||
| ├── graph_utils.py # Pure graph algorithms (Tarjan SCC, back edges, degree) | ||
| ├── html_renderer.py # Interactive dagre-d3 + d3v5 HTML visualization | ||
| └── topology.jsonc # Group definitions + review comments (strategy file) | ||
| ``` | ||
|
|
||
| ## Quick Start | ||
|
|
||
| ```bash | ||
| # Scan current HEAD → generate interactive HTML | ||
| python3 scripts/dependency-topology/scan_topology.py scan | ||
|
|
||
| # Output to specific path | ||
| python3 scripts/dependency-topology/scan_topology.py scan -o /tmp/deps.html | ||
|
|
||
| # JSON output (for scripts/agents) | ||
| python3 scripts/dependency-topology/scan_topology.py scan --json | ||
|
|
||
| # Diff — smart default: | ||
| # worktree has uncommitted Lua changes → HEAD vs worktree | ||
| # worktree is clean → HEAD~1 vs HEAD (last commit) | ||
| python3 scripts/dependency-topology/scan_topology.py diff | ||
|
|
||
| # Compare specific refs (branch names, commit SHAs, remote refs) | ||
| python3 scripts/dependency-topology/scan_topology.py diff --from upstream/main --to clean-code-remove-core | ||
| python3 scripts/dependency-topology/scan_topology.py diff --from HEAD~5 --to HEAD | ||
| ``` | ||
|
|
||
| ## Snapshot References | ||
|
|
||
| - `worktree` — current working tree (uncommitted changes) | ||
| - `HEAD` — latest commit | ||
| - Any git ref — branch name (e.g. `upstream/main`), tag, short or full commit SHA | ||
| - Relative refs — `HEAD~1`, `HEAD^` | ||
|
|
||
| **diff defaults (no args):** | ||
| - Worktree has uncommitted Lua changes → `HEAD` vs `worktree` | ||
| - Worktree is clean → `HEAD~1` vs `HEAD` | ||
|
|
||
| Note: ambiguous short names (e.g. `upstream` when both a local branch and remote exist) | ||
| produce a git warning. Prefer fully-qualified refs: `upstream/main`, `refs/heads/mybranch`. | ||
|
|
||
| ## Output | ||
|
|
||
| **scan:** One-line summary + HTML file path | ||
| ``` | ||
| 4 cycles, 20 violations, violations=20 → /path/to/dependency-graph.html | ||
| ``` | ||
|
|
||
| **diff:** Change direction summary | ||
| ``` | ||
| HEAD → worktree: +2/-1 edges, improved=1, regressed=0 | ||
| ``` | ||
|
|
||
| ## JSON Output Signals | ||
|
|
||
| When using `--json`: | ||
|
|
||
| - `health` — one-glance status for cycles / violations / ungrouped coverage | ||
| - `cycles` — SCC details with severity, members_by_layer, example_cycle, back_edges_in_scc | ||
| - `violations` — policy violations grouped by rule with full edge lists | ||
| - `group_coverage` — module counts per layer (including ungrouped) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,225 @@ | ||
| #!/usr/bin/env python3 | ||
| """Repository-local static Lua dependency graph helpers. | ||
|
|
||
| Mechanism only: | ||
| - Parse `require('opencode.*')` edges from `lua/opencode/**/*.lua` | ||
| - Build snapshot graph from worktree or git ref | ||
| - Provide SCC / back-edge utilities | ||
| """ | ||
|
|
||
| from __future__ import annotations | ||
|
|
||
| from collections import Counter, defaultdict | ||
| from dataclasses import dataclass | ||
| from pathlib import Path | ||
| import re | ||
| import subprocess | ||
| from typing import Dict, Iterable, List, Optional, Sequence, Set, Tuple | ||
|
|
||
|
|
||
| REQUIRE_PATTERNS = [ | ||
| re.compile(r"require\s*\(\s*['\"](opencode(?:\.[^'\"]+)?)['\"]\s*\)"), | ||
| re.compile(r"require\s+['\"](opencode(?:\.[^'\"]+)?)['\"]"), | ||
| ] | ||
|
|
||
|
|
||
| @dataclass | ||
| class SnapshotGraph: | ||
| snapshot: str | ||
| files: int | ||
| nodes: Dict[str, str] # module -> relative file path | ||
| edges: Set[Tuple[str, str]] | ||
|
|
||
|
|
||
| def module_from_relpath(relpath: str) -> Optional[str]: | ||
| if not relpath.startswith("lua/opencode/") or not relpath.endswith(".lua"): | ||
| return None | ||
| mod = relpath[len("lua/") : -len(".lua")] | ||
| if mod.endswith("/init"): | ||
| mod = mod[: -len("/init")] | ||
| return mod.replace("/", ".") | ||
|
|
||
|
|
||
| def _worktree_files(repo: Path) -> List[Tuple[str, str]]: | ||
| out: List[Tuple[str, str]] = [] | ||
| base = repo / "lua" / "opencode" | ||
| for fp in base.rglob("*.lua"): | ||
| rel = fp.relative_to(repo).as_posix() | ||
| text = fp.read_text(encoding="utf-8", errors="ignore") | ||
| out.append((rel, text)) | ||
| return out | ||
|
|
||
|
|
||
| def _git_files(repo: Path, ref: str) -> List[Tuple[str, str]]: | ||
| cmd = ["git", "ls-tree", "-r", "--name-only", ref, "lua/opencode"] | ||
| try: | ||
| ls = subprocess.check_output(cmd, cwd=repo, text=True, stderr=subprocess.PIPE) | ||
| except subprocess.CalledProcessError as e: | ||
| stderr = e.stderr.strip() if e.stderr else "" | ||
| raise ValueError( | ||
| f"Invalid snapshot ref '{ref}'. Valid values: HEAD, worktree, branch name, commit SHA.\n" | ||
| f"git error: {stderr}" | ||
| ) from None | ||
|
|
||
| out: List[Tuple[str, str]] = [] | ||
| for rel in ls.splitlines(): | ||
| if not rel.endswith(".lua"): | ||
| continue | ||
| show_cmd = ["git", "show", f"{ref}:{rel}"] | ||
| try: | ||
| text = subprocess.check_output(show_cmd, cwd=repo, text=True, stderr=subprocess.DEVNULL) | ||
| except subprocess.CalledProcessError: | ||
| continue | ||
| out.append((rel, text)) | ||
| return out | ||
|
|
||
|
|
||
| def load_snapshot_graph(repo: Path, snapshot: str) -> SnapshotGraph: | ||
| files = _worktree_files(repo) if snapshot == "worktree" else _git_files(repo, snapshot) | ||
|
|
||
| nodes: Dict[str, str] = {} | ||
| for rel, _ in files: | ||
| module = module_from_relpath(rel) | ||
| if module: | ||
| nodes[module] = rel | ||
|
|
||
| edges: Set[Tuple[str, str]] = set() | ||
| for rel, content in files: | ||
| src = module_from_relpath(rel) | ||
| if not src: | ||
| continue | ||
|
|
||
| deps: Set[str] = set() | ||
| for pat in REQUIRE_PATTERNS: | ||
| deps.update(m.group(1) for m in pat.finditer(content)) | ||
|
|
||
| for dep in deps: | ||
| if dep in nodes: | ||
| edges.add((src, dep)) | ||
|
|
||
| return SnapshotGraph(snapshot=snapshot, files=len(files), nodes=nodes, edges=edges) | ||
|
|
||
|
|
||
| def tarjan_scc(nodes: Iterable[str], edges: Iterable[Tuple[str, str]]) -> List[List[str]]: | ||
| graph: Dict[str, List[str]] = defaultdict(list) | ||
| for a, b in edges: | ||
| graph[a].append(b) | ||
|
|
||
| index = 0 | ||
| stack: List[str] = [] | ||
| on_stack: Set[str] = set() | ||
| indices: Dict[str, int] = {} | ||
| lowlink: Dict[str, int] = {} | ||
| result: List[List[str]] = [] | ||
|
|
||
| def strongconnect(v: str) -> None: | ||
| nonlocal index | ||
| indices[v] = index | ||
| lowlink[v] = index | ||
| index += 1 | ||
| stack.append(v) | ||
| on_stack.add(v) | ||
|
|
||
| for w in graph[v]: | ||
| if w not in indices: | ||
| strongconnect(w) | ||
| lowlink[v] = min(lowlink[v], lowlink[w]) | ||
| elif w in on_stack: | ||
| lowlink[v] = min(lowlink[v], indices[w]) | ||
|
|
||
| if lowlink[v] == indices[v]: | ||
| comp: List[str] = [] | ||
| while True: | ||
| w = stack.pop() | ||
| on_stack.remove(w) | ||
| comp.append(w) | ||
| if w == v: | ||
| break | ||
| result.append(comp) | ||
|
|
||
| for n in sorted(set(nodes)): | ||
| if n not in indices: | ||
| strongconnect(n) | ||
|
|
||
| return result | ||
|
|
||
|
|
||
| def back_edges(nodes: Iterable[str], edges: Iterable[Tuple[str, str]]) -> Set[Tuple[str, str]]: | ||
| graph: Dict[str, List[str]] = defaultdict(list) | ||
| for a, b in edges: | ||
| graph[a].append(b) | ||
| for n in graph: | ||
| graph[n] = sorted(set(graph[n])) | ||
|
|
||
| white, gray, black = 0, 1, 2 | ||
| color: Dict[str, int] = {n: white for n in set(nodes)} | ||
| backs: Set[Tuple[str, str]] = set() | ||
|
|
||
| def dfs(v: str) -> None: | ||
| color[v] = gray | ||
| for w in graph[v]: | ||
| c = color.get(w, white) | ||
| if c == white: | ||
| dfs(w) | ||
| elif c == gray: | ||
| backs.add((v, w)) | ||
| color[v] = black | ||
|
|
||
| for n in sorted(color.keys()): | ||
| if color[n] == white: | ||
| dfs(n) | ||
|
|
||
| return backs | ||
|
|
||
|
|
||
| def degree(edges: Iterable[Tuple[str, str]]) -> Tuple[Counter, Counter]: | ||
| indeg: Counter = Counter() | ||
| outdeg: Counter = Counter() | ||
| for src, dst in edges: | ||
| outdeg[src] += 1 | ||
| indeg[dst] += 1 | ||
| return indeg, outdeg | ||
|
|
||
|
|
||
| def find_cycle_in_scc(members: List[str], edges: Iterable[Tuple[str, str]]) -> List[str]: | ||
| """Return one concrete cycle path within an SCC, e.g. [a, b, c, a]. | ||
|
|
||
| Uses DFS from the first member; backtracks until a back-edge is found. | ||
| Returns [] if no cycle is found (shouldn't happen for a real SCC > 1). | ||
| """ | ||
| member_set = set(members) | ||
| graph: Dict[str, List[str]] = defaultdict(list) | ||
| for a, b in edges: | ||
| if a in member_set and b in member_set: | ||
| graph[a].append(b) | ||
| for n in graph: | ||
| graph[n] = sorted(set(graph[n])) | ||
|
|
||
| path: List[str] = [] | ||
| on_path: Dict[str, int] = {} # node -> index in path | ||
| visited: Set[str] = set() | ||
|
|
||
| def dfs(v: str) -> List[str]: | ||
| path.append(v) | ||
| on_path[v] = len(path) - 1 | ||
| for w in graph[v]: | ||
| if w in on_path: | ||
| # Found cycle: extract from w's position to end, close it | ||
| return path[on_path[w]:] + [w] | ||
| if w not in visited: | ||
| visited.add(w) | ||
| result = dfs(w) | ||
| if result: | ||
| return result | ||
| path.pop() | ||
| del on_path[v] | ||
| return [] | ||
|
|
||
| start = sorted(members)[0] | ||
| visited.add(start) | ||
| return dfs(start) | ||
|
|
||
|
|
||
| def largest_scc_size(comps: Sequence[Sequence[str]]) -> int: | ||
| nontrivial = [c for c in comps if len(c) > 1] | ||
| return max((len(c) for c in nontrivial), default=0) | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.