[alejoe91]

Fixes #4014 - UPDATED 16/10/2026

Implement bulk changes from zarr v2 to zarr v3

• create_dataset -> create_array
• Serialization of non JSON serializable objects (recording/sorting/extensions) not supported anymore
• Compressors use zarr.codecs, which wrap numcodecs compressor (e.g., Blosc -> BloscCodec)
• Codec Pipeline: improved generation of codec pipeline to automatically chain filters (ArrayArrayCodec), serializers (standard unless an ArrayBytesCodec is used - e.g. Wavpack), and compressors (BytesBytesCodec)
• Sharding: new sharding option in the zarr save functions (save(..., format="zarr")), with tests on chunks and shards
• Backward compatibility: loading objects written with zarr v2 (recordings, sortings, templates, SortingAnalyzer with extensions), including zarr_class_info, channel_ids dtype handling, and storage_options
• NWB / zarr:
  • zarr links are read from _LINKS for pynwb
  • zarr children iteration is wrapped to skip arrays whose metadata zarr v3 can't parse
  • streaming zarr (stream_mode="zarr") now works with zarr>=3 and its test is re-enabled
  • NWB-zarr tests only run the hdf5 backend when hdmf-zarr is not installed
• Python: requires-python >= 3.11. The hdmf-zarr>=0.14.0 extras are restricted to python_version>='3.12', since hdmf-zarr 0.14.0 requires Python >=3.12. NWB-zarr (pynwb) is therefore unavailable on 3.11, while plain zarr reading/writing and zarr streaming work everywhere.
• Removed the requires_zarr_write test marker (zarr writing is always available) and cleaned up tests
• CI:
  • cross-version serialization workflow checks v2 -> v3 compatibility (including dev) on all supported versions
  • numpy pinned <2.5 in the cross-serialization tests
  • deepinterpolation action disabled
• Docs: updated the compression section

Depends on:

• ProbeInterface: Support zarr v3 probeinterface#387
• NWB (hdmf-zarr) 0.14.0

[alejoe91]

Thank you so much @h-mayorquin

I agree with the proposed split up. Merged #4873 #4875 and #4876 (and removed the _has_zarr_python_3() function), dropped core-tests.yml and removed the unused default_zarr_version config.

[chrishalcrow]

Suggested change

	-shards: tuple | None, default: None
	- shards: tuple | None, default: None

Also shard size must be a multiple of chunk_size??

[h-mayorquin]

Thanks @alejoe91. #4881 and #4882 are the last two peel-offs, the format change (4 in my review). After that, we will only have the zarr backwards compatability and the core changes on this PR.

Now for the review here, I will skip the backward compatibility, as I think you are in a better position to judge it and you will be the main person affected. There is one thing in it that breaks the new format, though: get_sorting_provenance() returns None for every new zarr analyzer. The new provenance is an attribute of the sorting group, but it is now read from the root group, and the legacy fallback condition is inverted (if sort_dict is not None, which should be is None). No test catches it because the only caller, the code behind save_as, select_units, merge_units and split_units, silently falls back to the analyzer's copy of the sorting, so the results stay correct but a derived analyzer (after a merge during curation, for example) loses the link to the original sorting. I think the fix is:

sort_dict = zarr_root["sorting"].attrs.get("provenance")
if sort_dict is None:
    sort_dict = get_zarr_attr_or_legacy_object(zarr_root, "sorting_provenance")

and a test that creates an analyzer from a saved sorting in both formats, reloads it, and checks that get_sorting_provenance() is not None.

I will do another review once the format change in #4881 and #4882 is settled as I need to run now.

[h-mayorquin]

Last round of review

1. On main, the spike fields of a zarr sorting were chunked by default and that was lost. We should add it back with chunks="auto" and then, after this PR, think about what a good chunking is for time-ordered spikes.

2. There is a wrong import in the compression example of doc/modules/core.rst. It still imports LZMA and Delta from numcodecs, the zarr 2 classes, so the example throws an error. It should be from zarr.codecs.numcodecs import LZMA, Delta.

3. I would remove the commented-out structured writer and its reader branch. I like ZarrSpikeVector (as I mentioned in the meeting) as it decouples how the spike vector is accessed from how it is stored, and with it we don't need a structured array on disk. Just to re-state this here, the advantages of decoupling the access API from storage are:

• The code that uses the spike vector keeps the structured array interface (spikes["sample_index"], slices, np.asarray(spikes)) and the storage can change without touching it.
• Nothing is read at construction and a slice reads only the chunks it covers. That keeps lazy loading cheap, especially for a remote analyzer.
• segment_index is derived from segment_slices instead of stored per spike.
• Each field gets its own filter and chunks, and reading one field doesn't decompress the others. This is also important because different fields might work better with different filter parameters.

3 is optional of course, 2 I think we should do. I am unsure if 1 one was deliberate and in such case I would like to hear why.

After that, this should be good to go for me and maybe we should plan on a release of probeinterface?

[alejoe91]

thanks @h-mayorquin

Implemented all! For auto-chunking, instead of using "auto" I set the target_chunk_size_bytes default to 10MB.

We don't need a release in probeinterface, but can we approve this? SpikeInterface/probeinterface#387
So we can point to main before merging this one!

[chrishalcrow]

I'm a bit out of my depth here, but did look through the code in full and seems reasonable.

I also made some analyzers with zarr v3 and messed with them: attaching labels, doing iterative curation in the GUI etc. Honestly the new json thing fixes my main issue with zarr. Absolutely love having some human readable files in the zarr folder. Sharding also seems great!

So, from a non-zarr expert, but complex-user standpoint, I approve!

Very impressive work @alejoe91 and @h-mayorquin !!

[chrishalcrow]

These failing tests are passing for me locally on all python versions.

[chrishalcrow]

Suggested change

	return BloscCodec(cname="zstd", clevel=clevel, shuffle=BloscShuffle.bitshuffle)
	return BloscCodec(cname="zstd", clevel=clevel, shuffle="bitshuffle")

I'm getting a deprecation warning for BloscShuffle.bitshuffle
https://zarr.readthedocs.io/en/stable/release-notes/#improved-documentation_1

[h-mayorquin]

LGTM.
Thanks for carrying this through.

[h-mayorquin]

I think this is called again on line 868, unsure why, something to quickly check.

[h-mayorquin]

I think the errors are unrelated and noisy. Another of those flaky tests.