# Using intake catalog to pull CMIP models that match multiple criteria

**URL:** <https://forum.access-hive.org.au/t/using-intake-catalog-to-pull-cmip-models-that-match-multiple-criteria/196>\
**Category:** Technical\
**Tags:** python, model-evaluation, med\
**Created:** [25 November 2022 01:43 UTC](https://forum.access-hive.org.au/t/using-intake-catalog-to-pull-cmip-models-that-match-multiple-criteria/196 "2022-11-25T01:43:36Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![taimoorsohail](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/taimoorsohail/32/201_2.png) [@taimoorsohail](https://forum.access-hive.org.au/u/taimoorsohail)\
**Post date:** [25 November 2022 01:43 UTC](https://forum.access-hive.org.au/t/using-intake-catalog-to-pull-cmip-models-that-match-multiple-criteria/196/1 "2022-11-25T01:43:36Z")

</div>

Hello,

I am trying to use @Paola-CMS [intake catalog](https://github.com/coecms/nci-intake-catalogue/blob/main/docs/intake_cmip6_demo.ipynb) to pull multiple CMIP6 variables that match multiple criteria. My code is:

```auto
import intake
cat = intake.cat.nci
cmip6 = cat['esgf'].cmip6
# Let's select a subset passing the search() method some constraints
subset1 = cmip6.search(activity_id='DAMIP', experiment_id='hist-nat', table_id='Omon', variable_id='thetao')
subset2 = cmip6.search(activity_id='DAMIP', experiment_id='hist-aer', table_id='Omon', variable_id='thetao')
subset3 = cmip6.search(activity_id='DAMIP', experiment_id='hist-GHG', table_id='Omon', variable_id='thetao')
subset4 = cmip6.search(activity_id='CMIP', experiment_id='historical', table_id='Omon', variable_id='thetao')

```

Basically, I only want the models/ensemble members that match _all_ of the above criteria (i.e. subset1 & subset2 & subset3 & subset4). In the historical activity, for example, I don’t want all the instances with `thetao`, but rather the ones that have the same ensemble member and model as the matched hist-nat, hist-aer and hist-GHG runs. Is there any hint how this can be done? Thank you!

---

<div class="post-metadata">

**Author:** ![Paola-CMS](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/paola-cms/32/57_2.png) [@Paola-CMS](https://forum.access-hive.org.au/u/Paola-CMS)\
**Post date:** [25 November 2022 02:29 UTC](https://forum.access-hive.org.au/t/using-intake-catalog-to-pull-cmip-models-that-match-multiple-criteria/196/2 "2022-11-25T02:29:09Z")

</div>

@taimoorsohail this is already [possible in intake:

First simplify your query by using lists where you need to include more than one value for a query facet, as for experiment\_id and activity\_id (probably redundant)

then use the require\_all\_on=[‘source\_id’] as in this case you want to select the models that have all the selected experiments.

subset = cmip6.search(require\_all\_on=[‘source\_id’], experiment\_id=['hist-nat, ‘historical’,…], …)

> **[Enforce search query criteria via require\_all\_on argument](https://intake-esm.readthedocs.io/en/stable/how-to/enforce-search-query-criteria-via-require-all-on.html)**
>
> By default, intake-esm’s search() method returns entries that fulfill any of the criteria specified in the query. Intake-esm can return entries that fulfill all query criteria when the user supplie...

---

<div class="post-metadata">

**Author:** ![taimoorsohail](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/taimoorsohail/32/201_2.png) [@taimoorsohail](https://forum.access-hive.org.au/u/taimoorsohail)\
**Post date:** [25 November 2022 03:34 UTC](https://forum.access-hive.org.au/t/using-intake-catalog-to-pull-cmip-models-that-match-multiple-criteria/196/3 "2022-11-25T03:34:35Z")

</div>

Thanks @Paola-CMS ! This is exactly what I wanted, and it _almost_ works - but I am having a strange bug. If I follow your instructions and pull all models (and their associated ensemble members) which have both `thetao` and `tas` it works and gives me a lot of datasets:

```auto
import intake

cat = intake.cat.nci
cmip6 = cat['esgf'].cmip6
subset = cmip6.search(require_all_on=['source_id'], \
experiment_id=['hist-nat','hist-aer','hist-GHG','historical'], \
variable_id=['thetao','tas'])

```

But this includes daily and hourly fields. I just want monthly. So when I add `table_id` to this:

```auto
import intake

cat = intake.cat.nci
cmip6 = cat['esgf'].cmip6
subset = cmip6.search(require_all_on=['source_id'], \
experiment_id=['hist-nat','hist-aer','hist-GHG','historical'], \
table_id = ['Omon', 'Amon'], variable_id=['thetao','tas'])
subset

```

I get the error:

```auto
ValueError: Length of values (0) does not match length of index (12)

```

I know for a fact that there are models and ensemble members which match my search criteria (I have manually found them in `fs38` and `oi10`) so not sure what this error is!

---

<div class="post-metadata">

**Author:** ![Paola-CMS](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/paola-cms/32/57_2.png) [@Paola-CMS](https://forum.access-hive.org.au/u/Paola-CMS)\
**Post date:** [25 November 2022 04:07 UTC](https://forum.access-hive.org.au/t/using-intake-catalog-to-pull-cmip-models-that-match-multiple-criteria/196/4 "2022-11-25T04:07:49Z")

</div>

It is weird, for the moment you could bypass it by setting the search without table\_id and adding it later to further filter the results:

subset = …  
subset2 = subset.search(table\_id=[‘Omon’, ‘Amon’])

---

<div class="post-metadata">

**Author:** ![Paola-CMS](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/paola-cms/32/57_2.png) [@Paola-CMS](https://forum.access-hive.org.au/u/Paola-CMS)\
**Post date:** [25 November 2022 05:15 UTC](https://forum.access-hive.org.au/t/using-intake-catalog-to-pull-cmip-models-that-match-multiple-criteria/196/5 "2022-11-25T05:15:53Z")

</div>

I now also know why the other approach doesn’t work.  
When you add the Omon, Amon constraints in the first query you are creating a “condition” which is impossible to satisfy, as it will try to return all possible combination of your constraints, it will return only models that have the following files:

(‘hist-nat’, ‘tas’, ‘Amon’), (‘hist-aer’, ‘thetao’, ‘Amon’), (‘hist-nat’, ‘thetao’, ‘Amon’), (‘hist-GHG’, ‘thetao’, ‘Amon’), (‘hist-nat’, ‘tas’, ‘Omon’), (‘hist-aer’, ‘thetao’, ‘Omon’), (‘historical’, ‘thetao’, ‘Amon’), (‘hist-nat’, ‘thetao’, ‘Omon’), (‘hist-aer’, ‘tas’, ‘Omon’), (‘hist-aer’, ‘tas’, ‘Amon’), (‘historical’, ‘tas’, ‘Amon’), (‘historical’, ‘thetao’, ‘Omon’), (‘hist-GHG’, ‘tas’, ‘Amon’), (‘historical’, ‘tas’, ‘Omon’), (‘hist-GHG’, ‘tas’, ‘Omon’), (‘hist-GHG’, ‘thetao’, ‘Omon’)}

As it cannot find `tas` in `Omon` of `thetao` in `Amon` it always returns zero results. Using the workaround I proposed before you avoid this issue.

---

<div class="post-metadata">

**Author:** ![taimoorsohail](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/taimoorsohail/32/201_2.png) [@taimoorsohail](https://forum.access-hive.org.au/u/taimoorsohail)\
**Post date:** [9 January 2023 01:07 UTC](https://forum.access-hive.org.au/t/using-intake-catalog-to-pull-cmip-models-that-match-multiple-criteria/196/7 "2023-01-09T01:07:23Z")

</div>

Hi @Paola-CMS I’m having an error when trying to use the intake module:

```auto
import intake

cat = intake.cat.nci
# CMIP6 is included in `esgf` which is itself a catalogue
# so we are using list() again to see the sub-catalogues
cmip6 = cat['esgf'].cmip6

```

Throws up the error:

```auto
TypeError: __init__ () missing 1 required positional argument: 'esmcol_obj'

```

This is new - previously the module was working fine…

---

<div class="post-metadata">

**Author:** ![clairecarouge](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/clairecarouge/32/380_2.png) [@clairecarouge](https://forum.access-hive.org.au/u/clairecarouge)\
**Post date:** [17 January 2023 04:39 UTC](https://forum.access-hive.org.au/t/using-intake-catalog-to-pull-cmip-models-that-match-multiple-criteria/196/8 "2023-01-17T04:39:52Z")

</div>

Hi @taimoorsohail , Paola replied to the same issue on Slack for someone else and the answer is to use `conda/analysis3-22.10` module. There was an update to `intake` and now the catalogue is only compatible with the latest `intake` version.

---

<div class="post-metadata">

**Author:** ![taimoorsohail](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/taimoorsohail/32/201_2.png) [@taimoorsohail](https://forum.access-hive.org.au/u/taimoorsohail)\
**Post date:** [20 January 2023 01:12 UTC](https://forum.access-hive.org.au/t/using-intake-catalog-to-pull-cmip-models-that-match-multiple-criteria/196/9 "2023-01-20T01:12:27Z")

</div>

Thanks @clairecarouge ! Unfortunately even with `conda/analysis3-22.10` I get the error `TypeError: __init__ () missing 1 required positional argument: 'obj'`

---

<div class="post-metadata">

**Author:** ![Aidan](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/aidan/32/42_2.png) [@Aidan](https://forum.access-hive.org.au/u/Aidan)\
**Post date:** [23 January 2023 05:58 UTC](https://forum.access-hive.org.au/t/using-intake-catalog-to-pull-cmip-models-that-match-multiple-criteria/196/10 "2023-01-23T05:58:47Z")

</div>

I reproduced your error @taimoorsohail

Weirdly it works with `conda/analysis3-22.07`, for me anyway

 ![Screenshot 2023-01-23 at 4.57.36 pm](https://us1.discourse-cdn.com/flex020/uploads/access1/original/1X/ebaccf7cc427332a674abef4d09f692354e5184a.png)

Any ideas @Paola-CMS?

---

<div class="post-metadata">

**Author:** ![Paola-CMS](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/paola-cms/32/57_2.png) [@Paola-CMS](https://forum.access-hive.org.au/u/Paola-CMS)\
**Post date:** [23 January 2023 21:56 UTC](https://forum.access-hive.org.au/t/using-intake-catalog-to-pull-cmip-models-that-match-multiple-criteria/196/11 "2023-01-23T21:56:53Z")

</div>

It’s unfortunately a change in the new intake-esm version that makes new catalogue incompatible with older versions so  
so if you use the unstable you can only load the catalogue opening directly the new catalogue file:  
cat = intake.open\_catalog(‘/g/data/hh5/public/apps/nci-intake-catalogue/catalogue\_new.yaml’)

using older conda env or an intake-esm \<2022.09 version:  
cat = intake.open\_catalog(‘/g/data/hh5/public/apps/nci-intake-catalogue/catalogue.yaml’)  
or  
cat = intake.cat.nci

NB that the last might change soon and be the reversed so cat =intake.cat.nci will work only with new intake-esm versions at our next update.  
For issues like this it might be better to put something in the helpdesk, or on [github](https://github.com/coecms/nci-intake-catalogue), as currently it’s impossible for me to check the forum with the same frequency.