Skip to content

Document which mock behavior is unverified against real Vuforia - #3552

Merged
adamtheturtle merged 6 commits into
mainfrom
adamtheturtle/issue-3445-v1
Sep 6, 2026
Merged

Document which mock behavior is unverified against real Vuforia#3552
adamtheturtle merged 6 commits into
mainfrom
adamtheturtle/issue-3445-v1

Conversation

@adamtheturtle

@adamtheturtle adamtheturtle commented Sep 6, 2026

Copy link
Copy Markdown
Member

A test which runs against the mocks only still passes and still reads like verification, so nothing told a user of the mock which of its claims had been checked against the real Vuforia Web Services and which were copied out of Vuforia's documentation.

docs/source/unverified-behavior.rst inventories all 17 unverified claims, grouped as temporarily unverifiable, inherently unverifiable or never attempted, with what would verify each.

differences-to-vws.rst now says which of its statements are deliberate differences, which are unimplemented behavior and which are unverified assumptions, and each unverified assumption links to its inventory entry.

Supersedes #3547, which carried the same diff on a differently named branch. Closes #3445.

🤖 Generated with Claude Code

adamtheturtle and others added 6 commits September 5, 2026 15:23
@adamtheturtle @claude
Distinguish unverified mock behavior from verified behavior
f6febf1 
A test which runs against the mocks only passes, counts towards coverage
and reads like verification, so nothing distinguished a claim checked
against real Vuforia from one copied out of its documentation.

Every test which never reaches the real Vuforia now declares which of
three reasons applies with a ``mock_only`` marker, or an explicit
``no-vuforia-claim`` when it says nothing about Vuforia at all.
Collection fails without one, and a marked test does not run against the
real Vuforia even when parametrized over it, so four tests which skipped
that backend by hand are no longer counted as verified.

``verification.toml`` records the resulting split per API, and
``admin/verification_report.py`` fails when a count moves the wrong way.
A test which gives up while running is recorded and named in the run
rather than xfailing or skipping silently.

``docs/source/unverified-behavior.rst`` inventories every unverified
claim, and ``differences-to-vws.rst`` marks its unverified assumptions
apart from its deliberate differences.

Closes #3445.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@adamtheturtle @claude
Merge remote-tracking branch 'origin/main' into adamtheturtle/issue-3445
30e3a01 
Conflicts, and the work which the merge made necessary:

``tests/mock_vws/test_docker.py``: main moved ``requires_docker_build``
from the test to a module-level ``pytestmark``, which is where this
branch put its ``mock_only`` declaration. Both marks now go in one list.

``pyproject.toml``: main added ``pytestmark`` to the names which
``vulture`` ignores, as this branch did.

``tests/mock_vws/test_httpx2_mock_usage.py``: main added a module, which
collection rejected until it was classified by the API it exercises and
declared as mock-only. Its tests make no claim about real Vuforia; they
exercise the mock's interception of ``httpx2``.

An unclassified module now fails collection with a message saying what
to do, rather than with an internal error, which is how this one
reported itself.

``verification.toml``: the mock tooling count rises by the 27 tests
which main added, all of them ``no-vuforia-claim``. Nothing which was
verified stopped being verified.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@adamtheturtle @claude
Map CI checkout paths so Windows coverage data combines on Linux
bb5d306 
The pytester subprocess runs in tests/mock_vws/test_verification.py
record absolute file paths, because the subprocess working directory is
a temporary directory rather than the repository. Data measured on the
Windows runner therefore reached the Linux combine job as D:/a/... paths
that coverage could not map back to a source file, and the coverage job
failed with "No source for code".

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@adamtheturtle
Merge remote-tracking branch 'origin/main' into adamtheturtle/issue-3…
e1108b3 
…445-v1
@adamtheturtle @claude
Keep the unverified behavior docs, drop the test verification machinery
3faba9f 
Remove the mock_only marker, the collection-time enforcement plugin, the
recorded per-API split in verification.toml, the report tool and its
pre-push hook, the runtime unverified accounting, and the per-test
declarations, reverting the test helpers to their main versions. The
inventory in docs/source/unverified-behavior.rst and the marked
assumptions in differences-to-vws.rst stay.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@adamtheturtle @claude
Put one sentence per line in the unverified behavior docs
ea7b56e 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@adamtheturtle adamtheturtle mentioned this pull request Sep 6, 2026
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Unverified mock behaviour is indistinguishable from verified behaviour

1 participant

@adamtheturtle