docs: list non-conformant and unregistered metadata in a checked ledger

[d-v-b]

🤖 AI text below 🤖

A new Metadata compatibility page lists every kind of stored metadata that zarr reads or writes although it does not conform to a Zarr specification, or uses a name the zarr-extensions registry does not define. For each kind, the page says which software wrote it, how zarr reads it now, and whether zarrs, tensorstore and zarr 2 read it. The page is generated from zarr.core.metadata.ledger, which a test checks against the code that reads and writes such metadata.

The idea follows zarrs' doc/correctness_issues.md (zarrs/zarrs#488).

What changes

• zarr.core.metadata.ledger.LEDGER has 17 entries. Each entry records:
  • what the stored metadata looks like;
  • non-conformant (a specification forbids it) or unregistered (a name zarr-extensions does not define);
  • which software and zarr releases wrote it;
  • whether zarr still writes it;
  • how zarr reads it now, and whether it warns;
  • whether zarrs, tensorstore and zarr 2 read it;
  • the repairs in zarr.core.metadata.repair that read it.
• Every site in src/zarr that reads or writes such metadata carries a # NON-CONFORMANT: <key> or # UNREGISTERED: <key> comment, as zarrs does.
• tests/test_ledger.py fails if any of these hold:
  • a function in ARRAY_REPAIRS is in no entry;
  • a marker names no entry, or uses the wrong tag;
  • an entry has no marker;
  • a v3_unstable_dtype_warning call is unmarked;
  • the generated page drops an entry.
• docs/metadata-compatibility.md is filled in by mkdocs_hooks.py at build time and listed in the nav after the release notes. The contributing guide gains a paragraph on the ledger, under data format compatibility.
• The repair module docstring no longer says it is the only place invalid metadata is read leniently. Other lenient readings, such as data type aliases and fill-value spellings, live with the code that reads them, and the ledger now lists them.

Warnings describe the metadata only. Release versions appear only in the ledger and on the docs page, and no warning in src/zarr names a release.

How the entries were checked

• Version ranges: the agent installed zarr 2.18.7 and the 3.0.0 to 3.4.1 releases, and wrote sample metadata with each. For example, attributes holding NaN were written as bare NaN tokens by 3.0.0 to 3.0.6, as "NaN" strings by 3.0.7 to 3.0.10, refused by 3.1.0, and bare again from 3.1.1.
• "Current behavior" claims: each was checked against this branch, both what the metadata is read as and whether a warning is given.
• "Other implementations" column: the agent built a sample store for each entry. The structured, endian-less bytes and zarr 2 string-fill samples were written by zarr 3.1.6 and zarr 2.18.7 themselves. Each sample was read with zarrs 0.23.14 (a small Rust program: Array::open and read every element, else Group::open), tensorstore 0.1.85 (the zarr and zarr3 drivers), and zarr 2.18.7 for the Zarr format 2 samples. Which numcodecs.* codecs zarrs reads depends on the crate features the program enabled.

Left out on spec grounds

• v2 <i1/>b1/<V4 spellings: the v2 spec allows </> on any typestr, so reading them is not lenient.
• v2 extra .zarray keys: the spec says readers SHOULD ignore them.

Open questions for review

• The ledger is private (zarr.core.metadata.ledger). The docs page is public.
• The "other implementations" results are a snapshot. The scripts that produced them are not in this PR. A follow-up could commit a sample generator, so the column can be rerun when zarrs or tensorstore release.

Tests: tests/test_ledger.py plus the metadata, dtype, codec, group, chunk-grid, v2, JSON, CLI and docs suites pass (4269 passed). mkdocs build --strict passes. The agent also broke the code three ways to check that the ledger tests catch drift: an unlisted repair, a deleted marker, and a wrong tag.

Author attestation

• I am a human, these are my changes, and I have reviewed and understood every change and can explain why each is correct.

Stack and related work

• Independent of draft feat(docs): index every deprecation from a single declaration #4454, the deprecations index. Both add an on_page_markdown hook to mkdocs_hooks.py, so whichever merges second takes a small conflict there.

• While building the ledger, the agent found five bugs in reading stored metadata:

  • the registered bytes data type with an int-array fill value, which every zarr 3.0 bytes array has, is rejected;
  • a v2 .zgroup of {} opens as Zarr format 3;
  • the registered "endian" codec alias cannot be read;
  • a bool fill value of "false" is read as True;
  • dimension_names: "x" is split into characters.

  Fixes are being prepared separately. The variable-length-bytes entry describes the current behavior (the [] fill value is rejected) and should be updated when that fix lands.

• Rebased on bd0dc840f.

🤖 Generated with Claude Code