Efficient daily resampling using xarray

[Aidan (Aidan Heerdegen, ACCESS-NRI Release Team Lead)]

Before giving up on xarray altogether make sure you’re using the latest version of dask as there has been a major update that fixes a lot of these memory issues

Latest version of dask (2022.11.0) could fix many workflow issues

The latest version of dask (2022.11.0) uses a new mode of scheduling by default that can significantly reduce the memory usage in a lot of typical geoscience workflows. The new default tries to address the problem of “root task overproduction” - if you’ve ever inexplicably run out of memory while computing a climatology, for example, you were probably experiencing root task overproduction. Details here: Reducing memory usage in Dask workloads by 80%. It would be great to explore/benchmark any i…

It is something @Scott looked at in the past, and developed climtas for this purpose

https://climtas.readthedocs.io/en/latest/

but definitely check the latest dask doesn’t solve the problem before trying climtas