Document which mock behavior is unverified against real Vuforia

docs/source/differences-to-vws.rst

@@ -4,6 +4,15 @@ Differences between the mock and the real Vuforia Web Services
 The mock attempts to be realistic, but it was built without access to the source code of the original API.
 Please report any issues `here <https://github.com/VWS-Python/vws-python-mock/issues>`__.
 
+This document mixes three kinds of statement, and it says which is which:
+
+* A deliberate difference, where the mock does something else on purpose.
+  The image matchers are one.
+* Behavior which the mock does not implement.
+* An unverified assumption, where the mock follows Vuforia's documentation and nobody has checked that the documentation is accurate.
+  Each of these carries a note pointing at its entry in :doc:`unverified-behavior`, which says what would verify it.
+  These are the ones which can bite: the mock passes its tests, your tests pass, and the divergence appears in production.
+
 Image matching
 --------------
 
@@ -126,7 +135,11 @@ NGINX Error cases
 
 Vuforia uses NGINX.
 This has error handling which is not duplicated in the mock.
-For example, Vuforia returns a 400 (``BAD REQUEST``) response if a header or cookie is given which is larger than 8 KiB.
+For example, Vuforia is documented as returning a 400 (``BAD REQUEST``) response if a header or cookie is given which is larger than 8 KiB.
+
+.. admonition:: Unverified assumption
+
+   :ref:`unverified-nginx-oversized-header-or-cookie`
 
 Result codes
 ------------
@@ -142,11 +155,12 @@ These are:
 Request quota exhaustion
 ------------------------
 
-The mock returns ``RequestQuotaReached`` when a
-:class:`mock_vws.database.CloudDatabase` is created with
-``request_quota=0``. This behavior follows the public Vuforia documentation,
-but the response has not been verified against a real database with an
-exhausted quota.
+The mock returns ``RequestQuotaReached`` when a :class:`mock_vws.database.CloudDatabase` is created with ``request_quota=0``.
+This behavior follows the public Vuforia documentation.
+
+.. admonition:: Unverified assumption
+
+   :ref:`unverified-request-quota-exhaustion`
 
 Request rate limits
 -------------------
@@ -156,11 +170,14 @@ endpoints in general, with 45 requests per second for
 ``GET /targets/{target_id}``, 10 requests per second for
 ``GET /duplicates/{target_id}``, and 1 request per minute for ``GET /targets``.
 
-The mock models these limits separately for each group of endpoints, but it
-applies no limit by default. The documented numbers have not been verified
-against a real database, and applying a limit of 1 request per minute to
-``GET /targets`` by default would break the tests of anything which uses the
-mock. Set ``request_rate_limits`` to
+The mock models these limits separately for each group of endpoints, but it applies no limit by default.
+Applying a limit of 1 request per minute to ``GET /targets`` by default would break the tests of anything which uses the mock.
+
+.. admonition:: Unverified assumption
+
+   :ref:`unverified-request-rate-limits`
+
+Set ``request_rate_limits`` to
 :data:`mock_vws.request_rate_limits.DOCUMENTED_REQUEST_RATE_LIMITS` to apply
 the documented limits::
 
@@ -182,9 +199,11 @@ the documented limits::
 VWS endpoints together, and it is tracked separately from the per-endpoint
 limits.
 
-Vuforia also documents that ``GET /targets`` fails for databases with more than
-1 million images, but the mock does not implement this, as the behavior is not
-reproducible against a test account.
+Vuforia also documents that ``GET /targets`` fails for databases with more than 1 million images, which the mock does not implement.
+
+.. admonition:: Unverified assumption
+
+   :ref:`unverified-targets-over-one-million-images`
 
 Configurable Cloud Query failures
 ---------------------------------
@@ -244,8 +263,12 @@ supported by the Flask/Docker backend.
 Other configurable result codes
 -------------------------------
 
-The mock also supports four other result codes which have not been verified
-against real databases in the corresponding states:
+The mock also supports four other result codes which come from Vuforia's result codes table rather than from a response which a real database gave:
+
+.. admonition:: Unverified assumption
+
+   :ref:`unverified-additional-result-codes`
+
 
 * ``TargetQuotaReached`` is returned when adding a target to a
   :class:`mock_vws.database.CloudDatabase` which already contains
@@ -363,14 +386,15 @@ Real Vuforia uses ``userId:<numeric-user-id>`` where the numeric portion is per-
 Standard and advanced routes share datasets by UUID. Access to each route
 family is separated by its corresponding OAuth scope.
 
-Some Model Target Web API paths remain mock-only in
-``tests/mock_vws/test_model_target_web_api.py::TestAdditionalBehaviors``.
+Some Model Target Web API paths remain mock-only in ``tests/mock_vws/test_model_target_web_api.py::TestAdditionalBehaviors``.
 Downloads of still-processing datasets are mock-only because exercising the path against real Vuforia would require creating a dataset on every test run; the mock drives the processing window deterministically.
-A download request for a dataset which is not ready reports the dataset's
-training status. The mock reports ``not-started`` for the whole processing
-window, as real Vuforia does for a dataset which was just created, and
-``failed`` for a dataset whose generation failed. The name which real Vuforia
-reports for a failed dataset has not been observed.
+A download request for a dataset which is not ready reports the dataset's training status.
+The mock reports ``not-started`` for the whole processing window, as real Vuforia does for a dataset which was just created, and ``failed`` for a dataset whose generation failed.
+
+.. admonition:: Unverified assumption
+
+   :ref:`unverified-model-target-failed-dataset-name`
+
 Some malformed State-Based Model Target configuration documents remain
 mock-only because real Vuforia returns an internal server error for them.
 
@@ -389,8 +413,11 @@ Each row ends with a carriage return and a line feed, as the header row does.
 The mock takes the counts when the report is requested, so counts which are
 set after that are not in that report.
 The mock orders the rows by target ID.
-Real Vuforia's order is not known, because no report with rows has been
-observed.
+
+.. admonition:: Unverified assumption
+
+   :ref:`unverified-reco-counts-report-row-order`
+
 Setting recognition counts is mock-only, because real Vuforia's counts are
 delayed for longer than a test runs, so the tests for reports with rows in
 ``tests/mock_vws/test_reco_counts_report.py`` run against the mocks only.
@@ -432,8 +459,10 @@ The mock's URL returns a 404 response until the report is ready, and requires
 no authorization.
 The lack of authorization matches real Vuforia, whose URL carries its own
 signature.
-The 404 has not been verified, because no request for a real report has caught
-one before it was generated.
+
+.. admonition:: Unverified assumption
+
+   :ref:`unverified-reco-counts-report-not-ready`
 
 Paths which the mock does not serve
 -----------------------------------

docs/source/index.rst

@@ -36,6 +36,7 @@ Reference
    docker
    mock-api-reference
    differences-to-vws
+   unverified-behavior
    versioning-and-api-stability
    contributing
 

docs/source/unverified-behavior.rst

@@ -0,0 +1,239 @@
+Unverified behavior
+===================
+
+The mock is kept honest by running one test suite against the real Vuforia Web Services and against each mock, and asserting the same things about each.
+Not every claim which the mock makes can be checked that way, and a test which runs against the mocks only still passes, still counts towards coverage and still reads like verification.
+
+This document lists every claim which the mock makes and which nobody has observed real Vuforia make, so that a user of the mock can tell which behaviors are copied from the service and which are copied from its documentation.
+
+Categories
+----------
+
+Each claim below is one of these:
+
+Temporarily unverifiable
+   The behavior is observable in principle, but not right now.
+   The credentials, the account scope or the account allowance needed to provoke it are missing.
+   These are the ones which are expected to become verified.
+
+Inherently unverifiable
+   The behavior cannot be provoked against real Vuforia at all.
+   These are mock-only forever.
+
+Never attempted
+   The mock implements something from Vuforia's public documentation and nobody has checked that the documentation is accurate.
+   This is the category which bites: the mock looks verified, and a divergence appears in production.
+
+.. _unverified-request-quota-exhaustion:
+
+Request quota exhaustion
+------------------------
+
+:Category: never-attempted
+:API: VWS Target API
+
+The mock returns a ``RequestQuotaReached`` response for a database created with ``request_quota=0``.
+The status code and the body shape come from Vuforia's documentation and from the shape which other VWS errors have.
+
+A real database with an exhausted request quota would verify this.
+No such response has been seen.
+
+.. _unverified-request-rate-limits:
+
+Request rate limits
+-------------------
+
+:Category: never-attempted
+:API: VWS Target API
+
+Vuforia documents a limit of 15 requests per second for VWS endpoints in general, 45 per second for ``GET /targets/{target_id}``, 10 per second for ``GET /duplicates/{target_id}`` and one per minute for ``GET /targets``.
+The mock models the limits separately for each group of endpoints, and applies them only when it is asked to.
+
+Sending more than the documented number of requests to a real database, and seeing what it returns, would verify this.
+
+.. _unverified-additional-result-codes:
+
+Additional result codes
+-----------------------
+
+:Category: never-attempted
+:API: VWS Target API
+
+``ProjectSuspended``, ``ProjectHasNoApiAccess``, ``TargetQuotaReached`` and ``TooManyRequests`` come from Vuforia's result codes table.
+No response from a real database in any of those states has been seen, which is why the mock's ``ProjectHasNoApiAccess`` casing is the table's casing rather than an observed one.
+
+A database put into each state by the Target Manager portal would verify these.
+
+.. _unverified-targets-over-one-million-images:
+
+``GET /targets`` for very large databases
+-----------------------------------------
+
+:Category: never-attempted
+:API: VWS Target API
+
+Vuforia documents that ``GET /targets`` fails for a database with more than one million images.
+The mock does not implement this, so a user of the mock sees a successful response where real Vuforia may not.
+
+A database with more than a million images would verify this, which a test account cannot hold.
+
+.. _unverified-nginx-oversized-header-or-cookie:
+
+Large headers and cookies
+-------------------------
+
+:Category: never-attempted
+:API: Cross-cutting request handling
+
+Vuforia runs behind NGINX, which is documented as returning a 400 (``BAD REQUEST``) response for a header or a cookie larger than 8 KiB.
+The mock does not implement this, and no test sends such a request to either.
+
+Sending a request with a header larger than 8 KiB to a real database would verify this.
+
+.. _unverified-reco-counts-report-not-ready:
+
+A reco counts report which is not ready
+---------------------------------------
+
+:Category: never-attempted
+:API: Reco Counts Report API
+
+The URL which the mock returns for a reco counts report gives a 404 response until the report is ready.
+
+A request for a real report which caught it before it was generated would verify this.
+Every real report requested so far has been ready by the time it was asked for.
+
+.. _unverified-reco-counts-report-row-order:
+
+The order of reco counts report rows
+------------------------------------
+
+:Category: inherently-unverifiable
+:API: Reco Counts Report API
+
+The mock orders report rows by target ID.
+
+Real Vuforia's recognition counts lag behind its queries by far longer than a test runs, so no real report with rows in it has been seen, and no order has been observed.
+
+.. _unverified-seeded-recognition-counts:
+
+Recognition counts in the target summary report
+-----------------------------------------------
+
+:Category: inherently-unverifiable
+:API: VWS Target API
+
+The mock does not count recognitions.
+Counts are set on a target instead, and the target summary report and the reco counts report show them.
+
+Real counts lag behind real queries by longer than a test runs, so no test can make a real summary show a recognition.
+
+.. _unverified-database-summary-processing-images:
+
+Processing images in the database summary
+-----------------------------------------
+
+:Category: inherently-unverifiable
+:API: VWS Target API
+
+The mock's database summary is accurate immediately, so it counts an image which is still processing.
+
+The real summary lags behind the targets in the database, and sometimes skips the processing state altogether, so it cannot be relied on to show one.
+
+.. _unverified-vumark-processing-target:
+
+VuMark instance generation for a processing target
+--------------------------------------------------
+
+:Category: inherently-unverifiable
+:API: VuMark Instance Generation API
+
+The mock returns ``TargetStatusNotSuccess`` for an instance generation request for a VuMark target which is still processing.
+
+VuMark targets are created through the Target Manager portal rather than through the API, so no test can hold a real VuMark target in the processing state.
+
+.. _unverified-model-target-generation-failure:
+
+Model Target generation failures
+--------------------------------
+
+:Category: inherently-unverifiable
+:API: Model Target Web API
+
+The mock can finish dataset generation with a ``failed`` status and an ``error`` object.
+
+Real Vuforia cannot be made to fail generation on demand.
+
+.. _unverified-model-target-generation-warning:
+
+Model Target generation warnings
+--------------------------------
+
+:Category: never-attempted
+:API: Model Target Web API
+
+The mock can add a ``warning`` object to a dataset which finished successfully.
+
+The shape of that object comes from Vuforia's documentation rather than from a warning which a real dataset carried.
+
+.. _unverified-model-target-failed-dataset-name:
+
+The name of a failed Model Target dataset
+-----------------------------------------
+
+:Category: inherently-unverifiable
+:API: Model Target Web API
+
+A download request for a dataset which is not ready reports the dataset's training status.
+The mock reports ``not-started`` for the whole processing window, as real Vuforia does for a dataset which was just created, and ``failed`` for a dataset whose generation failed.
+
+The name which real Vuforia reports for a failed dataset has not been seen, because generation cannot be made to fail.
+
+.. _unverified-model-target-invalid-state-based-configuration:
+
+Malformed State-Based configuration documents
+---------------------------------------------
+
+:Category: inherently-unverifiable
+:API: Model Target Web API
+
+The mock reports validation errors for malformed State-Based Model Target configuration documents.
+
+Real Vuforia returns an internal server error for them, so there is no behavior to match.
+
+.. _unverified-model-target-training-allowance:
+
+The exhausted training allowance response
+-----------------------------------------
+
+:Category: inherently-unverifiable
+:API: Model Target Web API
+
+The mock can return Vuforia's ``TRAINING_ALLOWANCE_EXCEEDED`` response.
+
+The response shape comes from a real rejection seen in CI rather than from documentation, but the allowance cannot be exhausted on demand, and cannot be restored afterwards, so no test can provoke it.
+
+.. _unverified-model-target-signed-dataset-creation:
+
+Signed Model Target dataset creation
+------------------------------------
+
+:Category: temporarily-unverifiable
+:API: Model Target Web API
+
+Creating a signed dataset, such as an advanced dataset with a state-based configuration, consumes the Vuforia account's Model Target training allowance.
+The allowance is small, is shared across all CI jobs, and cannot be raised or reset.
+
+These cases reach real Vuforia only when ``--verify-model-target-signing`` is given, so by default the mock's behavior for them is unverified.
+
+.. _unverified-model-target-oauth2-client-credentials:
+
+OAuth2 client-credential management
+-----------------------------------
+
+:Category: temporarily-unverifiable
+:API: Model Target Web API
+
+The mock serves the OAuth2 client-credential management routes, including creating, listing, updating and deleting credentials, and Vuforia's limit of 100 created credentials per account.
+
+The Model Target test account does not have the scope for those routes, so they cannot be reached with the credentials which CI has.

newsfragments/3445.change

@@ -0,0 +1 @@
+Say which claims about the real Vuforia Web Services the mock has not had checked against them. :doc:`unverified-behavior` lists each such claim and what would verify it, and ``differences-to-vws`` marks the unverified assumptions among its deliberate differences.