Zarr 2.14.0 includes experimental support for sharding

Technical
,
1

The latest release of zarr (2.14.0) includes experimental support for sharding, along with a number of other features from the v3 spec.

Previous zarr specs stored one chunk per storage object. This can be problematic for zarr stores with a large number of chunks due to design constraints of the underlying storage (e.g. inode limits). Sharding allows storing multiple chunks in one storage object.

Full zarr v3 spec here
Details on zarr sharding here

1 Like
2

Thanks for sharing @dougiesquire

In practice are there recommended minimum sizes for storage objects in S3 buckets? Or put another way, maximum number of objects per bucket?

I can see the utility of reducing the inode count on lustre file systems, but I thought the current work-around for this was to compress the whole zarr directory tree? I suppose that gets problematic for extremely large datasets? Are there are any other reasons to favour sharding over compressing the whole zarr directory structure?

It is great that this functionality now exists, but I can now see a similar issue arising for zarr that weā€™ve had for a long time with netCDF: aligning read/write chunk size to object size on disk. Not a deal breaker, but a subtlety that often needs to be taken into account.

3

My understanding is that there is no limit to the number of objects you can store in a bucket, but latency overheads still can be problematic for large numbers of small objects.

Yes, Iā€™ve zipped some pretty large zarr stores (order 10s TB), but it doesnā€™t feel like an ideal work-around:

  • zarr.ZipStores are not safe to write in multiple processes and thereā€™s no way to update an existing zip file without unzipping and rezipping. Zipping/unzipping can take a really long time for large stores.
  • Iā€™ve hit hard limits on file size on some systems.
  • Dividing a dataset across multiple ZipStores helps with the above, but this can be a pain to manage and Iā€™ve experienced unexplained performance issues doing this (newer versions of dask may be better).

Yes, though from my skim through the specs Iā€™m not sure how this issue has changed for zarr with the introduction of sharding. My (ignorant) understanding is that read/write will still occur at the chunk level.

4

Hi,

A bit late this this thread but I have some huge zarr stores that Iā€™ve had good success converting to sharded Zarr V3 stores to significantly reduce inode usage. However since the hh5/xp65 analysis envs use V2 these V3 stores are not usable for the wider community.

  1. Iā€™ve gotten sharding to work really well with Zarr V3 but I have to use my own conda env to have this version of Zarr as the hh5/xp65 conda envs only have Zarr V2.18.3. Is there any reason why the Zarr package hasnā€™t been updated to Zarr V3 as it has some nice upgrades from V2?

  2. @dougiesquire did you ever manage to get this experimental support for sharding in V2 to work? Iā€™m trying to use:

import os
os.environ["ZARR_V3_SHARDING"] = "1"

from zarr._storage.v3_storage_transformers import ShardingStorageTransformer

but I canā€™t get it to recognise that Iā€™ve enabled "ZARR_V3_SHARDING".

Thanks,
Sam

1 Like
5

Ping @rbeucher, @CharlesTurner

Iā€™ve never tried sorry @sam.green. Sounds like updating to V3 is the way to go if thereā€™s nothing stopping that.

2 Likes
6

Hi @sam.green

I was looking into this. Intake-ESM is blocking the upgrade to Zarr v3.0.
@CharlesTurner is the maintainer of the package. Charles, could you look into this and see if we can safely move to Zarr v3.0?

2 Likes
7

We pinned to zarr v2 when we released intake-esm v2025.2.3, see here.

I agree upgrading to zarr v3 if we can would be ideal, but from memory there were enough breaking changes that it might not be straightforward.

Iā€™ll start looking into it.

3 Likes
8

Great, thanks @CharlesTurner @rbeucher @dougiesquire.

It would be great to upgrade to Zarr v3 but I understand if it takes a while to make intake-esm v2025.2.3 compatible. Let me know if thereā€™s anything I can do.

1 Like
9

Hi Sam,

Sorry itā€™s taken me so long to get back to you on this.

I started trying to relax the zarr pin, but itā€™s proving to be surprisingly difficult. It looks like the zarr and fsspec ecosystems are moving in different directions - see eg. Zarr Python 3 tracking issue Ā· Issue #514 Ā· fsspec/kerchunk Ā· GitHub.

I suspect it can be done - looks like some major progress was made here, but itā€™s just a case of me getting a few days to sit down and really wrap my head around the changes and how they affect intake-esm.

Tentatively, Iā€™m going to say that unless I get some unexpected time to work on this or it turns out to be easier than I thought, Iā€™ll aim to get it cracked in spring.

10

It appears that some updated have been made to some relevant dependencies in the meantime - Iā€™m guessing the zarr v3.1 release (I havenā€™t checked yet) & I think weā€™ll be able to unpin zarr v2 in intake-esm sooner rather than later.

11

Hi @CharlesTurner ,

Sorry I didnā€™t get a notification that you replied to this thread, just seen your 2 messages now. No problem on the length of time, I understand that dependancies can be problematic. Thanks for keeping this in your to-do list!

1 Like
12

No worries - the fix has been merged into main on intake-esm now. Just waiting on one more PR to go in and Iā€™ll trigger a new release, at which point weā€™ll be able to upgrade zarr.

Hopefully this should be within the next week or two. :crossed_fingers:

1 Like
13

Oh awesome, canā€™t wait!