Benchmarks for finch-tensor: subgraph-matching (counting) queries on the G-CARE and Sun & Luo graphs, and microbenchmarks (SpGEMM, SDDMM, chain products, ...) against Galley.jl, Scorch and SparseAuto.
You need git and pixi (curl -fsSL https://pixi.sh/install.sh | sh).
Clone with the submodules; pixi resolves every environment, and the global
one installs Scorch from external/scorch, so a clone without them can't
install anything:
git clone --recurse-submodules https://github.com/finch-tensor/FinchLiteBenchmarks.git
cd FinchLiteBenchmarks
pixi installpixi.lock pins every package, including the finch-tensor commit, so all
machines get the same environment. Julia is installed into the environment by
juliacall the first time finch runs, with its packages in .pixi/julia-depot;
nothing needs to be on your PATH.
Run one dblp query to check the setup:
pixi run python -m finchtensorbench.benchmarks.subgraph.subgraph_matching \
--dataset dblp --configs galley --queries dense_4/query_dense_4_2 --runs 1The first run takes several minutes: it downloads Julia, precompiles Finch.jl, and downloads the dataset. It should end with a line like
ok dense_4/query_dense_4_2 exec 0.01027s plan 2.738s first 10.22s count 37913 (gt 37913)
Every count is checked against the ground truth (gt).
- Datasets download on first use into
data/(setGCARE_DATA_DIRto put them elsewhere), from pinned commits with checksums.yeast,dblpandyoutubeare the graphs from Galley.jl's experiments, with its "lite" query sample.dblp_extandyoutube_extadd the largerdense_8andsparse_16query families. The G-CARE graphs (human,aids, ...) come from the G-CARE archives on Google Drive. - Configs are presets from
src/finchtensorbench/benchmarks/autoschedule.py(default,galley,galley-cost,galley-lp,galley-sampling), orlogic/loop/formatter/stats. - Results: a CSV and each query's plan go in
graph/subgraph_matching/(--output-dirto change it). - Parallel runs:
--jobs Nruns N workers, each pinned to its own core (--cpusto choose them) and capped at--memory-limit-gb. Queries that exceed--timeoutare recorded as timeouts. Pinning usesos.sched_setaffinity, so the runner currently works on Linux only.
run_subgraph.sh runs several datasets and configs and plots them:
src/finchtensorbench/benchmarks/subgraph/run_subgraph.sh -d "dblp youtube" -c galley -j 4galley_jl_subgraph runs Galley.jl on the same queries, so its CSV lines up
with finch's. It needs Julia 1.11 (its pinned Manifest was resolved with
1.11.7), which pixi does not provide. Install it, for example with
juliaup (juliaup add 1.11), then:
pixi run python -m finchtensorbench.benchmarks.subgraph.galley_jl_subgraph \
--dataset dblp --julia "julia +1.11"
pixi run python -m finchtensorbench.benchmarks.subgraph.compare_subgraph \
graph/subgraph_matching/*.csv --baseline galley.jlThe first run installs and precompiles Galley.jl. run_subgraph.sh -g runs
it after finch and compares the two; pass the Julia command through JULIA,
e.g. JULIA="julia +1.11".
Each backend has its own pixi environment and task, which writes a CSV to
graph/scripts/:
| Backend | Command | Notes |
|---|---|---|
| finch | pixi run run-finch |
|
| Galley.jl | pixi run -e galley run-galley |
through galley-jl-python |
| Scorch | pixi run -e scorch run-scorch |
builds Scorch from external/scorch, with the Command Line Tools' clang on macOS and conda-forge's GCC on Linux |
| SparseAuto | pixi run -e sparseauto build-sparseauto, then pixi run -e sparseauto run-sparseauto |
builds TACO from external/sparseLNR |
pixi run run-all [backends...] runs them all (or the ones named) and plots
the results; THREADS=N sets the thread count (default 1).
- Tests:
pixi run -e test test. - To benchmark another finch-tensor revision, change its
branchin[tool.pixi.pypi-dependencies]inpyproject.toml, then runpixi update finch-tensorand commitpixi.lock. To run a local checkout instead of the installed copy, setPYTHONPATHinside the environment, since pixi's activation resets it tosrc:pixi run env PYTHONPATH=src:/path/to/finch-tensor/src python -m .... Each subgraph CSV row records the finch-tensor revision it ran. - The
globalenvironment combines every backend, for an IDE.