Fix vacuous CI matrix meta-test and add missing entries#3129
Fix vacuous CI matrix meta-test and add missing entries#3129adamtheturtle wants to merge 4 commits intomainfrom
Conversation
The in-process pytest.main() in ci/test_custom_linters.py installs its own capture, so capsys saw an empty string and tests passed vacuously. Switch to a collection-hook plugin that reads node IDs directly. This surfaces three test files missing from the ci-tests matrix (test_respx_mock_usage.py, test_target_validators.py, docs/source/httpx-example.rst) plus the meta-tests themselves; add them. Closes #3128.
adamtheturtle
temporarily deployed
to
development
GitHub Actions
Inactive
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
Previously ran pytest.main() per pattern (~120 invocations, 89s). Now do a single in-process collection and check each pattern against the cached node IDs in pure Python via a small boundary-aware prefix matcher. Runtime drops from 89s to under 1s.
Previously the return code of pytest.main() was discarded, so a collection error (import failure, syntax error) would silently leave the fixture with whatever items were captured before the crash. Assert the exit code is OK so collection errors fail the meta-tests loudly.
The prefix-matching shortcut was a hand-rolled approximation of pytest's selector. Go back to invoking pytest.main() per pattern so selection matches CI exactly, even though it's slower (~89s). Keep the exit-code assertion so collection errors fail loudly.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit d907796. Configure here.
| tests = _tests_from_pattern(ci_pattern=ci_pattern) | ||
| message = f'"{ci_pattern}" does not match any tests.' | ||
| assert collect_only_result == 0, message | ||
| assert tests, message |
There was a problem hiding this comment.
Custom error message in test_ci_patterns_valid is unreachable
Low Severity
When a CI pattern matches no tests, pytest.main(--collect-only) returns ExitCode.NO_TESTS_COLLECTED (5), not ExitCode.OK. The hard assert exit_code == pytest.ExitCode.OK inside _tests_from_pattern fires first, so the function can never return an empty set. This makes the assert tests, message check in test_ci_patterns_valid dead code — the helpful "…does not match any tests" message is unreachable, replaced by a less informative "Collection…failed with exit code" error.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit d907796. Configure here.
1 participant
Summary
ci/test_custom_linters.py::test_tests_collected_oncewas passing vacuously because the in-processpytest.main()installed its own capture, socapsysreturned an empty string and every pattern collected 0 tests.pytest_collection_modifyitemsand reads node IDs directly — still in-process (no subprocess overhead), and the meta-test now does real work (~89s vs 1.6s).tests/mock_vws/test_respx_mock_usage.py,tests/mock_vws/test_target_validators.py,docs/source/httpx-example.rst, andci/(so the meta-tests themselves run in CI).Test plan
uv run --extra=dev python -m pytest -v ci/test_custom_linters.py— both tests pass and runtime reflects real collection work.Note
Medium Risk
Touches CI workflow matrix and the meta-test that validates it; risk is mainly false positives/negatives or increased CI runtime due to real collection across patterns.
Overview
Fixes the
ci/test_custom_linters.pymeta-tests so they no longer pass vacuously: replaces stdout parsing/capsyswith an in-process pytest collection plugin (pytest_collection_modifyitems) that records collected node IDs and asserts collection succeeds.Updates the GitHub Actions
ci_patternmatrix in.github/workflows/test.ymlto include previously unlisted test/doc/ci/paths that the now-working meta-test detects (so all patterns are validated and the meta-tests themselves run in CI).Reviewed by Cursor Bugbot for commit d907796. Bugbot is set up for automated code reviews on this repo. Configure here.