# Intake-ESM and 3hr data **URL:** \ **Category:** Technical\ **Tags:** help\ **Created:** [29 July 2024 00:07 UTC](https://forum.access-hive.org.au/t/intake-esm-and-3hr-data/2259 "2024-07-29T00:07:16Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![rb4844](https://avatars.discourse-cdn.com/v4/letter/r/e47c2d/32.png) [@rb4844](https://forum.access-hive.org.au/u/rb4844)\ **Post date:** [29 July 2024 00:07 UTC](https://forum.access-hive.org.au/t/intake-esm-and-3hr-data/2259/1 "2024-07-29T00:07:16Z")
I am trying to identify which CMIP6 models at NCI have 3hr data, without manually trawling through them all. Something like this: ``` import intake cmip6 = intake.open_esm_datastore("/g/data/dk92/catalog/v2/esm/cmip6-oi10/catalog.json") values_dict = cmip6.unique() ``` models\_list = values\_dict.source\_id # but take a subset for testing small\_list = [‘EC-Earth3’, ‘CESM2’, ‘GFDL-ESM2M’, ‘GFDL-ESM4’, ‘MIROC6’, ‘NorESM2-MM’, ‘CMCC-ESM2’] # variables of interest three\_hr\_data = [‘huss’, ‘tas’, ‘uas’, ‘vas’, ‘ps’, ‘pr’] # testing with one model cat\_subset = cmip6.search( source\_id=[‘EC-Earth3’], experiment\_id=[“ssp370”], table\_id=“3hr”, variable\_id=“huss”, grid\_label=[“gn”, ‘gr’], ) cat\_subset ``` cmip6-oi10 catalog with 5 dataset(s) from 430 asset(s): etc ``` dset\_dict = cat\_subset.to\_dataset\_dict( xarray\_open\_kwargs={“consolidated”: True, “decode\_times”: True, “use\_cftime”: True} ) Which then fails with this error: ESMDataSourceError: Failed to load dataset with key=‘f.ScenarioMIP.EC-Earth-Consortium.EC-Earth3.ssp370.r6i1p1f1.3hrPt.atmos.3hr.huss.gr.v20200201’ You can use `cat['f.ScenarioMIP.EC-Earth-Consortium.EC-Earth3.ssp370.r6i1p1f1.3hrPt.atmos.3hr.huss.gr.v20200201'].df` to inspect the assets/files for this key. Why does it say ‘3hrPt.atmos.3hr’ ? Not what I was expecting. Actual path is /g/data/oi10/replicas/CMIP6/ScenarioMIP/EC-Earth-Consortium/EC-Earth3/ssp370/r1i1p1f1/3hr/huss/gr/v20200310 Can you suggest how to get this to work in a useful way? At the end of the day, I want a list of paths similar to the above, for all the models with 3hr data in the three\_hr\_data list. Many thanks. ---
**Author:** ![Thomas-Moore](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/thomas-moore/32/329_2.png) [@Thomas-Moore](https://forum.access-hive.org.au/u/Thomas-Moore)\ **Post date:** [29 July 2024 00:20 UTC](https://forum.access-hive.org.au/t/intake-esm-and-3hr-data/2259/2 "2024-07-29T00:20:17Z")
FYI - we’re working on this over here, for those interested: [Using Intake-ESM · Issue #5 · shared-climate-data-problems/CMIP-data-problems · GitHub](https://github.com/shared-climate-data-problems/CMIP-data-problems/issues/5) ---
**Author:** ![dougiesquire](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/dougiesquire/32/16_2.png) [@dougiesquire](https://forum.access-hive.org.au/u/dougiesquire)\ **Post date:** [29 July 2024 04:34 UTC](https://forum.access-hive.org.au/t/intake-esm-and-3hr-data/2259/3 "2024-07-29T04:34:57Z")
@rb4844 did you include `gdata/oi10` in the list of storage paths when you started your job (presumably on the ARE)? My guess is that’s why you can’t open the data. Regarding the `3hrPt` in the key name (which I assume is what you were surprised to see), this is the entry in the “frequency” column for these assets. If you think this might be wrong, I’d suggest reaching out to the [NCI helpdesk](https://nci.org.au/users/nci-helpdesk) since NCI created and manage these datastores. ---
**Author:** ![rb4844](https://avatars.discourse-cdn.com/v4/letter/r/e47c2d/32.png) [@rb4844](https://forum.access-hive.org.au/u/rb4844)\ **Post date:** [29 July 2024 05:07 UTC](https://forum.access-hive.org.au/t/intake-esm-and-3hr-data/2259/4 "2024-07-29T05:07:09Z")
Hi Dougie, Yes, I have gdata/oi10. Environment/module on ARE with /g/data/hh5/public/modules but I assume this is ok as import intake works. Storage is gdata/dk7+gdata/dk92+gdata/p73+gdata/rn45+gdata/hh5+gdata/xv83+gdata/oi10+gdata/r87+gdata/fs38+gdata/rr3+gdata/cj37+gdata/rt52+gdata/lp01 ---
**Author:** ![dougiesquire](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/dougiesquire/32/16_2.png) [@dougiesquire](https://forum.access-hive.org.au/u/dougiesquire)\ **Post date:** [29 July 2024 05:21 UTC](https://forum.access-hive.org.au/t/intake-esm-and-3hr-data/2259/5 "2024-07-29T05:21:33Z")
`consolidated` is not a valid kwarg for `xarray.open_dataset` with netcdf files. Removing that, I can open the data using your code: ```python import intake from distributed import Client client = Client(threads_per_worker=1) cmip6 = intake.open_esm_datastore( "/g/data/dk92/catalog/v2/esm/cmip6-oi10/catalog.json" ) cat_subset = cmip6.search( source_id=['EC-Earth3'], experiment_id=["ssp370"], table_id="3hr", variable_id="huss", grid_label=["gn", 'gr'], ) dset_dict = cat_subset.to_dataset_dict( xarray_open_kwargs={"decode_times": True, "use_cftime": True} ) ``` ---
**Author:** ![rb4844](https://avatars.discourse-cdn.com/v4/letter/r/e47c2d/32.png) [@rb4844](https://forum.access-hive.org.au/u/rb4844)\ **Post date:** [29 July 2024 05:51 UTC](https://forum.access-hive.org.au/t/intake-esm-and-3hr-data/2259/6 "2024-07-29T05:51:41Z")
Awesome, that worked now. Now to figure out what the result means. … Many thanks ---
**Author:** ![Thomas-Moore](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/thomas-moore/32/329_2.png) [@Thomas-Moore](https://forum.access-hive.org.au/u/Thomas-Moore)\ **Post date:** [29 July 2024 06:04 UTC](https://forum.access-hive.org.au/t/intake-esm-and-3hr-data/2259/7 "2024-07-29T06:04:03Z")
Any ideas @rb4844 why this wasn’t a solution? [Using Intake-ESM · Issue #5 · shared-climate-data-problems/CMIP-data-problems · GitHub](https://github.com/shared-climate-data-problems/CMIP-data-problems/issues/5#issuecomment-2254762290) ---
**Author:** ![rb4844](https://avatars.discourse-cdn.com/v4/letter/r/e47c2d/32.png) [@rb4844](https://forum.access-hive.org.au/u/rb4844)\ **Post date:** [29 July 2024 06:20 UTC](https://forum.access-hive.org.au/t/intake-esm-and-3hr-data/2259/8 "2024-07-29T06:20:26Z")
Not really. There must be some error in my original, longer version of the code that I can’t see (not the first time that’s happened, lol). Using the simpler version does work, so next step is to develop that to loop over more models. Thanks for looking at this though, appreciated. ---
**Author:** ![Thomas-Moore](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/thomas-moore/32/329_2.png) [@Thomas-Moore](https://forum.access-hive.org.au/u/Thomas-Moore)\ **Post date:** [29 July 2024 06:49 UTC](https://forum.access-hive.org.au/t/intake-esm-and-3hr-data/2259/9 "2024-07-29T06:49:48Z")
no worries - always good to get a solution! ---
**Author:** ![atteggiani](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/atteggiani/32/212_2.png) [@atteggiani](https://forum.access-hive.org.au/u/atteggiani)\ **Post date:** [13 August 2024 01:17 UTC](https://forum.access-hive.org.au/t/intake-esm-and-3hr-data/2259/10 "2024-08-13T01:17:46Z")