# Rechunking large data at constant memory in Dask

**URL:** <https://forum.access-hive.org.au/t/rechunking-large-data-at-constant-memory-in-dask/573>\
**Category:** Technical\
**Tags:** python, dask\
**Created:** [20 March 2023 22:39 UTC](https://forum.access-hive.org.au/t/rechunking-large-data-at-constant-memory-in-dask/573 "2023-03-20T22:39:52Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![dougiesquire](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/dougiesquire/32/16_2.png) [@dougiesquire](https://forum.access-hive.org.au/u/dougiesquire)\
**Post date:** [20 March 2023 22:39 UTC](https://forum.access-hive.org.au/t/rechunking-large-data-at-constant-memory-in-dask/573/1 "2023-03-20T22:39:52Z")

</div>

(Re-posting from the [Pangeo discourse](https://discourse.pangeo.io/))

With release `2023.3.1`, `dask` offers a new method for rechunking arrays called `p2p` that rechunks large arrays at constant memory. This enables workloads that were previously un-runnable and required you to rechunk datasets ahead of time using tools like `rechunker`.

See [Rechunking large data at constant memory in Dask [experimental] - HPC - Pangeo](https://discourse.pangeo.io/t/rechunking-large-data-at-constant-memory-in-dask-experimental/3266)

---

<div class="post-metadata">

**Author:** ![Aidan](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/aidan/32/42_2.png) [@Aidan](https://forum.access-hive.org.au/u/Aidan)\
**Post date:** [21 March 2023 22:07 UTC](https://forum.access-hive.org.au/t/rechunking-large-data-at-constant-memory-in-dask/573/2 "2023-03-21T22:07:06Z")

</div>

Very interesting! Seems quite sensitive to the number of tasks, hence the chunk size.

Llike almost everything in `xarray`/`dask`-land it’s all about tha chunks, 'bout tha chunks, hey? (apologies to Meghan Trainor)