Skip to content

test: add a corpus test that formats real files repeatedly - #834

Merged
dsherret merged 2 commits into
mainfrom
corpus-test
Oct 31, 2026
Merged

dsherret merged 2 commits into
mainfrom
corpus-test

Conversation

@dsherret

@dsherret dsherret commented Oct 31, 2026 •

Copy link
Copy Markdown
Member

The specs are small, so they don't catch problems that only show up on larger real code. This adds a corpus test that formats every JS/TS file in a set of directories and reports:

  • the files that are still changing after being formatted 10 times, with the first line that changed;
  • the files whose formatted text doesn't parse;
  • the files that panic;
  • the slowest files.

Files that change when formatted again but stop changing within 10 formats are only counted, since they settle.

It's skipped unless DPRINT_TS_CORPUS is set to the corpus directories, so it doesn't change cargo test. In a git repository it only checks the files git tracks (including in submodules), so local files don't change the results. Files that don't parse are skipped.

DPRINT_TS_CORPUS=../deno cargo test --release --test corpus -- --baseline tests/corpus_baselines/deno.txt
  • --baseline <file> only fails for the files that aren't in the baseline, and lists the ones that no longer have problems.
  • --write-baseline <file> writes the baseline.
  • --write-hashes <file> writes a hash of each file's formatted text, for comparing two versions to find the files whose formatting changed.
  • --reduce <dir> reduces each file that keeps changing to a small snippet that still does, by removing statements and list items and replacing expressions with an identifier (largest first, up to two minutes per file), and writes the distinct snippets along with a summary.md of how each formats.

Results for deno

This adds a baseline for deno at 87c5ddbdc92ed65385dc0800dc66645620331080 with its submodules (91,295 files, about 45 seconds in a release build on my machine):

  • No files are still changing after 10 formats. 337 change when formatted again but settle, almost all vendored or bundled code (the most formats one took was 9).
  • 163 files' formatted text doesn't parse. They're all test262 tests that are meant to be syntax errors (ex. an escaped await), which swc parses anyway.
  • 3 files panic in swc's lexer, which are also test262 tests.

Running it in CI would need a checkout of deno and its submodules, so that's left for later.

The specs are small, so they don't catch formatting that changes when formatted
again on larger real code. The corpus test formats every JS/TS file in the
directories in `DPRINT_TS_CORPUS` twice and reports the files that change, whose
formatted text doesn't parse, or that panic. It only fails for files that aren't
in the baseline passed with `--baseline`, and adds a baseline for the deno repo.
…a reducer

Formatting that changes when formatted again, but stops changing after a few
formats, is now only counted, and the files with problems are the ones still
changing after 10 formats. `--reduce <dir>` reduces those files to small
snippets that still keep changing, by removing statements and list items and
replacing expressions, and writes them along with how each formats.
@dsherret dsherret changed the title test: add a corpus test that formats real files twice test: add a corpus test that formats real files repeatedly Oct 31, 2026
@dsherret
dsherret merged commit 14b4767 into main Oct 31, 2026
1 check passed
@dsherret
dsherret deleted the corpus-test branch October 31, 2026 23:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant