# Dask remove time chunks for Fourier transforms **URL:** \ **Category:** Technical\ **Tags:** dask, xarray, chunking\ **Created:** [29 August 2023 23:15 UTC](https://forum.access-hive.org.au/t/dask-remove-time-chunks-for-fourier-transforms/1163 "2023-08-29T23:15:38Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![ashjbarnes](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/ashjbarnes/32/205_2.png) [@ashjbarnes](https://forum.access-hive.org.au/u/ashjbarnes)\ **Post date:** [29 August 2023 23:15 UTC](https://forum.access-hive.org.au/t/dask-remove-time-chunks-for-fourier-transforms/1163/1 "2023-08-29T23:15:38Z")
I’ve got a bit of a dask riddle: I have several 25Gb files on disk that are chunked in time **as separate files on disk**. I want to read in a very skinny slice of this data amounting to ~50Mb and perform a Fourier transform in time. However, even when I explicitly call `.rechunk` or set `chunks = {"time":None}`, the Fourier transform fails because as far as Dask is concerned my data is still chunked in time, and FFTs require time to be in a single chunk. As a workaround I’m doing the following: - open slice of data with `xr.open_mfdataset( ... ,chunks = {"zi":1,"xq":500}).isel( yh = 50 )` - Perform simple interpolation along one spatial dimension `.interp(xq = xh)` - Load my slice into memory with `.load()` TAKES 12 MINS PER EXPERIMENT ON WHOLE NODE - Perform remaining analysis (takes 0.3s) doing a simple spatial interpolation just before the FFT step. However this takes about 12 mins to do which seems stupidly slow for such a simple operation, followed by about 0.3 seconds to perform the FFT on loaded data. This is annoying because I need to perform this operation more than 100 times and I know the current method is super inefficient Is there a better way to approach this? ---
**Author:** ![aekiss](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/aekiss/32/46_2.png) [@aekiss](https://forum.access-hive.org.au/u/aekiss)\ **Post date:** [29 August 2023 23:34 UTC](https://forum.access-hive.org.au/t/dask-remove-time-chunks-for-fourier-transforms/1163/2 "2023-08-29T23:34:40Z")
Might be worth asking Paige Martin how she handled this in her model. ---
**Author:** ![dougiesquire](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/dougiesquire/32/16_2.png) [@dougiesquire](https://forum.access-hive.org.au/u/dougiesquire)\ **Post date:** [29 August 2023 23:45 UTC](https://forum.access-hive.org.au/t/dask-remove-time-chunks-for-fourier-transforms/1163/3 "2023-08-29T23:45:20Z")
> [@ashjbarnes](#): > > However, even when I explicitly call `.rechunk` or set `chunks = {"time":None}`, the Fourier transform fails because as far as Dask is concerned my data is still chunked in time Does calling `data.chunk(chunks={"time": -1})` produce an array with more than one chunk in the time dimension? Also, what are you using to do the fft? `xrft`? Depending on how much data you’re dealing with, you may have more luck first saving an intermediate dataset with the chunking you need and then computing the ffts on that. ---
**Author:** ![ashjbarnes](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/ashjbarnes/32/205_2.png) [@ashjbarnes](https://forum.access-hive.org.au/u/ashjbarnes)\ **Post date:** [30 August 2023 00:12 UTC](https://forum.access-hive.org.au/t/dask-remove-time-chunks-for-fourier-transforms/1163/4 "2023-08-30T00:12:30Z")
Hmm doing the -1 chunking still leaves the spatial chunks there as circled below ![Screenshot from 2023-08-30 10-08-06](https://us1.discourse-cdn.com/flex020/uploads/access1/original/1X/e59bb32fedfe3e654b0daf8f3e087e5f8762c07b.png) The only way I’ve been able to make the FFT work is using the ‘load’ command - nothing else seems to work. Re-saving would also work but I’d expect that re-saving to disk would take at least as long as loading the data to memory right? ---
**Author:** ![ashjbarnes](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/ashjbarnes/32/205_2.png) [@ashjbarnes](https://forum.access-hive.org.au/u/ashjbarnes)\ **Post date:** [30 August 2023 00:31 UTC](https://forum.access-hive.org.au/t/dask-remove-time-chunks-for-fourier-transforms/1163/5 "2023-08-30T00:31:05Z")
Ok so for those following along at home: Using the .rechunk() method does work. BUT this can only be applied directly to dask arrays, not xarrays. It seems that the `chunks` option which xarray uses to wrap the dask chunking simply doesn’t do what you expect. However, the below shenanigans _do_ work. Basically I’m overwriting the xarray data with a manually rechunked dask array ![image](https://us1.discourse-cdn.com/flex020/uploads/access1/original/1X/5347069a19f398c98f4e2c9af5e88a48c09fd922.png) Unless I’m missing something here maybe I’ll raise an issue on xarray to point this out? ---
**Author:** ![dougiesquire](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/dougiesquire/32/16_2.png) [@dougiesquire](https://forum.access-hive.org.au/u/dougiesquire)\ **Post date:** [30 August 2023 00:36 UTC](https://forum.access-hive.org.au/t/dask-remove-time-chunks-for-fourier-transforms/1163/6 "2023-08-30T00:36:33Z")
The time chunking will need to be done _after_ the `open_mfdataset` step because you have separate files in time. Does calling ```python eslice.chunk(chunks={"time": -1}) ``` still not give you a single chunk in time? ---
**Author:** ![ashjbarnes](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/ashjbarnes/32/205_2.png) [@ashjbarnes](https://forum.access-hive.org.au/u/ashjbarnes)\ **Post date:** [30 August 2023 01:18 UTC](https://forum.access-hive.org.au/t/dask-remove-time-chunks-for-fourier-transforms/1163/7 "2023-08-30T01:18:37Z")
That works! Thanks so much - I had no idea about the difference. So the `chunks` argument will chunk each _individual_ file as its loaded, but the `.chunk()` method is the equivalent of `.rechunk()` for dask arrays? In the past I guess it’s confused me that `.rechunk()` doesn’t work on xarrays but I guess I forget about the `.chunk` method as being the xarray equivalent / wrapper ---
**Author:** ![dougiesquire](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/dougiesquire/32/16_2.png) [@dougiesquire](https://forum.access-hive.org.au/u/dougiesquire)\ **Post date:** [30 August 2023 01:59 UTC](https://forum.access-hive.org.au/t/dask-remove-time-chunks-for-fourier-transforms/1163/8 "2023-08-30T01:59:11Z")
Yup, its most efficient to use the `chunks` argument when opening the data, where possible (see [here](https://docs.xarray.dev/en/stable/user-guide/dask.html#reading-and-writing-data)). But this chunking is applied before the combine step in `open_mfdataset`. In your case, this means chunk sizes in time will be truncated at the length of the time dimension in the files you’re combining. I think it’s probably a good idea to do all the chunking you can using the `chunks` argument in `open_mfdataset`, and only chunk time using the `.chunk()` method, e.g., ```python eslice = xr.open_mfdataset( paths, chunks={"time": -1, "zi": 1, "xh": 500, etc}, etc ).chunk(chunks={"time": -1}) ``` ---
**Author:** ![ashjbarnes](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/ashjbarnes/32/205_2.png) [@ashjbarnes](https://forum.access-hive.org.au/u/ashjbarnes)\ **Post date:** [30 August 2023 02:03 UTC](https://forum.access-hive.org.au/t/dask-remove-time-chunks-for-fourier-transforms/1163/9 "2023-08-30T02:03:22Z")
Thanks Dougie! This makes a lot of sense. 12-15 mins of processing time down to 30s is a good effort ---
**Author:** ![Aidan](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/aidan/32/42_2.png) [@Aidan](https://forum.access-hive.org.au/u/Aidan)\ **Post date:** [30 August 2023 06:02 UTC](https://forum.access-hive.org.au/t/dask-remove-time-chunks-for-fourier-transforms/1163/10 "2023-08-30T06:02:38Z")
> [@ashjbarnes](#): > > 12-15 mins of processing time down to 30s is a good effort This seems so useful it would be good to “surface” this somehow. Probably the best way would be a COSIMA Recipe @ashjbarnes, assuming this is COSIMA data? I took the liberty of sprinkling some tag-magic to help with folks finding this.