I have been trying to use intake to do some analysis with @KZCurtin using the 1/20th degree PanAntarctic model. I don’t personally use intake - I find that using xr.open_mfdataset with a preprocessing function works smoothly 99.9% of the time - but intake seems a cool tool with potential. Anyway, w…

Sorry for taking so long to get to this @JuliaN - I was sick last week. I think this difference is due to the files for dates prior to 2000, which you are including when using Intake-ESM, but not when you open directly with xr.open_mfdataset. Are these files formatted differently that the later dat…

@JuliaN Thanks for sending the notebook to illustrate the issue. @dougiesquire will look into it and will let you know what he finds so others can find the information if they have the same issue.

Hi Dougie, thanks for testing this out! I am not aware of any format differences prior or post 2000s. I am only interested in plotting the last 10 years of experiment, hence why I restrict with mfdataset. I was not aware of the possibility of using start_date with intake! That’s super useful, thank …

[image] JuliaN: Minute-to-minute update: I have been running intake for +30min and its still not finished (mfdataset took 11min) Are you using my code above , specifically: xarray_open_kwargs=dict( chunks="auto" ) as you were doing with open_mfdataset?

[image] JuliaN: I am not aware of any format differences prior or post 2000s. I am only interested in plotting the last 10 years of experiment, hence why I restrict with mfdataset. I was not aware of the possibility of using start_date with intake! That’s super useful, thank you. Is it possible…

Ok I was not using your kwargs, looks like switching what I was using from the cosima recipes to ‘auto’ chunks works! It takes the same time as mfdataset yeay! I’ll try to remember to pass this as kwargs - @KZCurtin this might be useful :slight_smile: I can’t find differences between the files in t…

[image] JuliaN: Ok I was not using your kwargs, looks like switching what I was using from the cosima recipes to ‘auto’ chunks works! @dougiesquire should we update the COSIMA Tutorial to use chunks="auto" as the default suggestion if this is crucial? Or are there times when this wouldn’t be …

In this case, without preprocessing the kernel crashes. I think it would be super useful to add it to COSIMA, but potentially it can wait until start_time update is complete and incorporate that too?

Cool, @JuliaN given you know more than me about all this, maybe do you want to open a COSIMA Recipe issue, linking to this post, to remind us to make these changes to the Tutorial at some stage?

Just to clarify, chunks='auto' uses the chunking from the netcdfs and the default chunks=None uses whole files as individual chunks? Is it possible to change the intake default in xp65? Happy to do so if given a few pointers on where to make the modification

Intake vs mfdataset

Technical

clairecarouge (Claire Carouge, ACCESS-NRI Land Modelling Team Lead) 30 April 2025 04:10 4

@dougiesquire See @CharlesTurner 's reply here, he says intake ESM has a problem in analysis3-25.04, I have no idea if this is linked to what is happening here, but it might be worth trying with a previous version first.

Topic		Replies	Views
ACCESS-NRI Intake catalogue for CM2-025 simulation Earth System help , cosima , model-evaluation	18	210	24 February 2025
Dask remove time chunks for Fourier transforms Technical dask , xarray , chunking	9	448	30 August 2023
Making Intake Datastore for panantarctic COSIMA help , mom6 , intake	16	255	7 February 2025
Xarray warnings while loading data using cosima cookbook Technical python , help , cosima	15	605	5 November 2024
Intake Virtual Icechunk v0.2.0 Release Announcement: intake-esm style catalogues backed by Icechunk ACCESS-NRI Releases python , data , intake	0	35	27 May 2026

Intake vs mfdataset

Related topics