# Efficient daily resampling using xarray

**URL:** <https://forum.access-hive.org.au/t/efficient-daily-resampling-using-xarray/233>\
**Category:** Technical\
**Tags:** python\
**Created:** [5 December 2022 05:30 UTC](https://forum.access-hive.org.au/t/efficient-daily-resampling-using-xarray/233 "2022-12-05T05:30:07Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![Michael\_Barnes](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/michael_barnes/32/247_2.png) [@Michael\_Barnes](https://forum.access-hive.org.au/u/Michael_Barnes)\
**Post date:** [5 December 2022 05:30 UTC](https://forum.access-hive.org.au/t/efficient-daily-resampling-using-xarray/233/1 "2022-12-05T05:30:07Z")

</div>

Good day.  
I often resample hourly netcdf data (eg. ERA5) into daily means with xarray. However, when doing this for large 4D dataset (eg. ERA5 file) it is very slow. Does anyone have a more efficient method that is effective and returns an xarray object? An example of code below:

import xarray as xr  
tfile = ‘/g/data/rt52/era5/pressure-levels/reanalysis/t/2022/t\_era5\_oper\_pl\_20220301-20220331.nc’  
ft=xr.open\_dataset(tfile).resample(time=‘1D’).mean(keep\_attrs=True)

---

<div class="post-metadata">

**Author:** ![dougiesquire](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/dougiesquire/32/16_2.png) [@dougiesquire](https://forum.access-hive.org.au/u/dougiesquire)\
**Post date:** [5 December 2022 05:49 UTC](https://forum.access-hive.org.au/t/efficient-daily-resampling-using-xarray/233/2 "2022-12-05T05:49:19Z")

</div>

Hi @Michael_Barnes. I think you will have better experiences with `coarsen`. E.g.

```python
ft=xr.open_dataset(tfile).coarsen(time=24).mean(keep_attrs=True)

```

will average blocks of 24 hours. Obviously, you’ll need to be a little careful that your “days” correspond to the period you want. E.g. if the first time in your dataset is a 6am, then your resulting days will be the average over 6am-5am, whereas if the first time is 12pm, then your days will be 12pm-11am.

---

<div class="post-metadata">

**Author:** ![Michael\_Barnes](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/michael_barnes/32/247_2.png) [@Michael\_Barnes](https://forum.access-hive.org.au/u/Michael_Barnes)\
**Post date:** [5 December 2022 06:37 UTC](https://forum.access-hive.org.au/t/efficient-daily-resampling-using-xarray/233/3 "2022-12-05T06:37:29Z")

</div>

Thanks @dougiesquire for the suggestion.  
When I use coarsen, it doesn’t seem to be more efficient (time-wise) and seems to also eat up a lot of RAM which the resample option does not.

---

<div class="post-metadata">

**Author:** ![coreyrobinson](https://avatars.discourse-cdn.com/v4/letter/c/51bf81/32.png) [@coreyrobinson](https://forum.access-hive.org.au/u/coreyrobinson)\
**Post date:** [5 December 2022 23:47 UTC](https://forum.access-hive.org.au/t/efficient-daily-resampling-using-xarray/233/4 "2022-12-05T23:47:04Z")

</div>

Chipping in here since it’s something I’ve also struggled with, even for “medium”-sized data…

I’ve had moderate success using CDO rather than Python/xarray for daily means. An example I’ve used is something like this:

```nohighlight
FILES=/g/data/rt52/era5/pressure-levels/reanalysis/v/
OUTDIR=/g/data/k10/cr7888/era5_daily_means/v250/

for year in {1959..2021}; 
do 
    for month in {01,02,03,04,05,06,07,08,09,10,11,12}; 
    do 
        # select 250 hPa, regrid to 1x1 deg, take daily mean (in that order)
        cdo -L -daymean -remapcon,r360x180 -sellevel,250 $FILES/$year/v_era5_oper_pl_$year$month\01*.nc /scratch/k10/cr7888/v250/$year$month.nc
    done
    # concat monthly files into yearly
    cdo mergetime /scratch/k10/cr7888/v250/$year*.nc $OUTDIR/era5_v250_daily_mean_$year.nc
done

```

Not sure if/how CDO does parallelisation but it’s possible the loops slow things down (if there’s any CDO experts here feel free to make suggestions since I’m a novice myself!)

Only other thing I can think of with Python/xarray is to mess around with chunk sizes, allocated compute/memory resources, etc. Though in my experience that seems to only make small differences.

---

<div class="post-metadata">

**Author:** ![Michael\_Barnes](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/michael_barnes/32/247_2.png) [@Michael\_Barnes](https://forum.access-hive.org.au/u/Michael_Barnes)\
**Post date:** [6 December 2022 00:20 UTC](https://forum.access-hive.org.au/t/efficient-daily-resampling-using-xarray/233/5 "2022-12-06T00:20:43Z")

</div>

Hi @coreyrobinson. A good suggestion.  
CDO does seem to be more efficient. I have tested the python tools that use cdo (nctoolkit) and it does seem to be better than xarray and compatible to the unix CDO speed. It also produces an xarray object which is convenient for my application. Below the code which I think solves the issue and is equivalent to the original.

import nctoolkit as nc  
import xarray as XR  
tfile = ‘/g/data/rt52/era5/pressure-levels/reanalysis/t/2022/t\_era5\_oper\_pl\_20220301-20220331.nc’  
cdo\_sub\_command = “-daymean”  
ds = nc.open\_data(tfile)  
ds.cdo\_command(cdo\_sub\_command)  
ft = ds.to\_xarray()

---

<div class="post-metadata">

**Author:** ![dougiesquire](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/dougiesquire/32/16_2.png) [@dougiesquire](https://forum.access-hive.org.au/u/dougiesquire)\
**Post date:** [6 December 2022 02:47 UTC](https://forum.access-hive.org.au/t/efficient-daily-resampling-using-xarray/233/6 "2022-12-06T02:47:59Z")

</div>

@Michael_Barnes, it looks like you’re happy with the cdo solution, which is great. But, just to add to my previous answer, using dask and appropriate chunking will help here (sorry, I didn’t originally realize the size of your input data).

For example, your initial code took me approximately 12 mins to run using a “Large” ARE instance (7 cpus, 32GB mem). The following takes 1.5 mins:

```python
import xarray as xr
from distributed import Client
Client()

tfile = "/g/data/rt52/era5/pressure-levels/reanalysis/t/2022/t_era5_oper_pl_20220301-20220331.nc"
hourly = xr.open_dataset(tfile, chunks={"time": 24, "level": 2})
daily = hourly.coarsen(time=24).mean(keep_attrs=True).compute()

```

Is this competitive with using cdo?

Note, using `resample` instead of `coarsen` above takes approximately 3 mins.

---

<div class="post-metadata">

**Author:** ![Aidan](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/aidan/32/42_2.png) [@Aidan](https://forum.access-hive.org.au/u/Aidan)\
**Post date:** [6 December 2022 02:48 UTC](https://forum.access-hive.org.au/t/efficient-daily-resampling-using-xarray/233/7 "2022-12-06T02:48:20Z")

</div>

Before giving up on `xarray` altogether make sure you’re using the latest version of `dask` as there has been a major update that fixes a lot of these memory issues

> [@Latest version of dask (2022.11.0) could fix many workflow issues](https://forum.access-hive.org.au/t/latest-version-of-dask-2022-11-0-could-fix-many-workflow-issues/149):
>
> The latest version of dask (2022.11.0) uses a new mode of scheduling by default that can significantly reduce the memory usage in a lot of typical geoscience workflows. The new default tries to address the problem of “root task overproduction” - if you’ve ever inexplicably run out of memory while computing a climatology, for example, you were probably experiencing root task overproduction. Details here: [Reducing memory usage in Dask workloads by 80%](https://www.coiled.io/blog/reducing-dask-memory-usage). It would be great to explore/benchmark any i…

It is something @Scott looked at in the past, and developed `climtas` for this purpose

[https://climtas.readthedocs.io/en/latest/](https://climtas.readthedocs.io/en/latest/)

but definitely check the latest `dask` doesn’t solve the problem before trying `climtas`

---

<div class="post-metadata">

**Author:** ![Aidan](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/aidan/32/42_2.png) [@Aidan](https://forum.access-hive.org.au/u/Aidan)\
**Post date:** [6 December 2022 02:49 UTC](https://forum.access-hive.org.au/t/efficient-daily-resampling-using-xarray/233/8 "2022-12-06T02:49:42Z")

</div>

> [@dougiesquire](#):
>
> The following takes 1.5 mins

Heh, we crossed quotes @dougiesquire

Is this with the latest `dask` version?

---

<div class="post-metadata">

**Author:** ![dougiesquire](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/dougiesquire/32/16_2.png) [@dougiesquire](https://forum.access-hive.org.au/u/dougiesquire)\
**Post date:** [6 December 2022 02:52 UTC](https://forum.access-hive.org.au/t/efficient-daily-resampling-using-xarray/233/9 "2022-12-06T02:52:24Z")

</div>

> [@Aidan](#):
>
> Is this with the latest `dask` version?

No, this is actually using an old version of dask. @Michael_Barnes’ original code doesn’t actually use dask at all. Just for fun, I’ll try update dask and see if this makes any difference. I suspect not as memory is actually kept under control in this workflow, but who knows??

---

<div class="post-metadata">

**Author:** ![dougiesquire](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/dougiesquire/32/16_2.png) [@dougiesquire](https://forum.access-hive.org.au/u/dougiesquire)\
**Post date:** [6 December 2022 03:05 UTC](https://forum.access-hive.org.au/t/efficient-daily-resampling-using-xarray/233/10 "2022-12-06T03:05:32Z")

</div>

> [@dougiesquire](#):
>
> The following takes 1.5 mins

Using dask version 2022.12.0 (instead of 2022.9.2) seems to slow things down marginally. Closer to 2 mins.

---

<div class="post-metadata">

**Author:** ![Aidan](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/aidan/32/42_2.png) [@Aidan](https://forum.access-hive.org.au/u/Aidan)\
**Post date:** [6 December 2022 03:07 UTC](https://forum.access-hive.org.au/t/efficient-daily-resampling-using-xarray/233/11 "2022-12-06T03:07:31Z")

</div>

> [@dougiesquire](#):
>
> @Michael_Barnes’ original code doesn’t actually use dask at all

Oh right, well isn’t that the answer then? Without using `dask` and chunking `xarray` will try and load alll the data at once, and blow out memory, possibly leading to spilling to on disk swap, which is horribly slow.

---

<div class="post-metadata">

**Author:** ![Aidan](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/aidan/32/42_2.png) [@Aidan](https://forum.access-hive.org.au/u/Aidan)\
**Post date:** [6 December 2022 03:11 UTC](https://forum.access-hive.org.au/t/efficient-daily-resampling-using-xarray/233/12 "2022-12-06T03:11:36Z")

</div>

> [@dougiesquire](#):
>
> Using dask version 2022.12.0 (instead of 2022.9.2) seems to slow things down marginally. Closer to 2 mins.

Might be interesting to know how many years you have to process before it starts to struggle with the memory over-production problem, i.e. where is the cutover in speed between old and new `dask`.

I would expect that the new `dask` default queuing would runtime would remain approximately linear in speed with increasing size of the time dimension, but that the previous implementation would start to display rapidly diverging run time before failing to run at all.