Repository navigation
How do you check that a regression test detects the intended bug? #15390
Replies: 2 comments
|
A recent public example is our Neo.mjs PR. It uses Playwright rather than pytest, but the verification problem was similar. An early test fixture hadn’t initialized the table’s view data, so its failure wasn’t evidence of the intended bug. We corrected the setup before recording the comparison. With the corrected tests unchanged between the before-and-after runs, the initial 14 cases produced eight failures and six passing controls on the old implementation. All 14 passed with the fix. The manual work was checking why each test failed and keeping the environments comparable. I’d find a report useful if it preserved the exact revisions, test patch and failing assertions, while distinguishing setup errors from failures that actually demonstrate the regression. |
|
A related example is Dask #12731, which was developed with AI assistance. The issue involved set iteration order affecting the order of MultiIndex levels in Parquet metadata. The regression test covers both index-name orders and one or two partitions. The PR records 2 failures and 2 passes before the fix with |
Uh oh!
There was an error while loading. Please reload this page.
I'm comparing existing ways to check pytest regression tests before and after a fix. In a local comparison using a public SQLGlot fix and three synthetic controls, Proof distinguished a real regression, an already-passing test, a setup failure and a test that still fails.
On your last bug-fix PR, how did you establish that the unchanged new test failed for the intended reason on the old code? What, if anything, needed awkward manual work after using your existing tools?
Public or sanitized examples would be helpful.
All reactions