# Combining intake datastores and missing files from custom experiment datastore
**URL:** https://forum.access-hive.org.au/t/combining-intake-datastores-and-missing-files-from-custom-experiment-datastore/4741
**Category:** Technical
**Tags:** help, intake
**Created:** [29 July 2025 01:54 UTC](https://forum.access-hive.org.au/t/combining-intake-datastores-and-missing-files-from-custom-experiment-datastore/4741 "2025-07-29T01:54:34Z")
**Posts on this page:** 6
**Page:** 1
### Author:  [@claireyung](https://forum.access-hive.org.au/u/claireyung)
#### Post date: [29 July 2025 01:54 UTC](https://forum.access-hive.org.au/t/combining-intake-datastores-and-missing-files-from-custom-experiment-datastore/4741/1 "2025-07-29T01:54:34Z")
Hi Intake Team,
I have a couple of questions about intake datastores. I have run a regional OM3 model using the ACCESS-OM3 infrastructure, which automatically made `intake_esm_ds.json` files at the end of the experiment. I have two issues which I was hoping you may be able to help with:
1. I ran the model in three different folders (slight parameter changes in each case, but same diagnostics and file structures). This means I have three datastores. Is it possible with the current infrastructure to merge three datastores in the notebook where you are analysing them? Alternatively, should I be remaking a datastore by puttting them all in one folder and using the script at the end of [Building Intake-ESM datastores of ACCESS model output — COSIMA Cookbook documentation](https://cosima-recipes.readthedocs.io/en/latest/Cooking-Lessons-101-Tutorials/Make_Your_Own_Intake_Datastore.html)? Can this script pick up output in symbolic links?
2. Some of the diagnostics didn’t get picked up by the esm datastore making script. I think this is because they were not named with the `access-om3.mom6.XX.nc` convention, instead they are called something like `ocean_daily_z.nc`. Do you have any suggestions that might allow these files to be added to the datastore?
Thank you!
Claire
---
### Author:  [@CharlesTurner](https://forum.access-hive.org.au/u/CharlesTurner)
#### Post date: [29 July 2025 06:12 UTC](https://forum.access-hive.org.au/t/combining-intake-datastores-and-missing-files-from-custom-experiment-datastore/4741/2 "2025-07-29T06:12:36Z")
Hi Claire,
1. Currently, it’s not straightforward to merge two datastores, at least not to my knowledge, although I don’t think there’s a good reason for this - I’m pretty sure the functionality exists, it’s just that it’s not that well exposed. I’ll do some digging and you get a snippet of code that you should be able to combine the three, because I’m pretty sure it should be easy. It strikes me that this is probably something we’d like to expose utility tools for too - could you open an issue on the [catalog repo](https://github.com/ACCESS-NRI/access-nri-intake-catalog) and I’ll get to it as soon as I get the chance?
2. This is an irritating and somewhat fundamental issue with the way our builders currently scan directories to build esm-datastores. @marc.white has done some fantastic work to fix this that we’re just putting the final touches on, but for now if you can point me to the directories where these files live I can update the filename conventions in the builder so that it’ll pick up these files.
Cheers, Charles
---
### Author:  [@CharlesTurner](https://forum.access-hive.org.au/u/CharlesTurner)
#### Post date: [29 July 2025 06:33 UTC](https://forum.access-hive.org.au/t/combining-intake-datastores-and-missing-files-from-custom-experiment-datastore/4741/3 "2025-07-29T06:33:24Z")
Okay, this code should work to combine your datastores:
````python
from intake_esm.core import esm_datastore
import pandas as pd
def combine_datastores(*esm_datastores) -> esm_datastore:
"""
Takes a bunch of esm_datastores and returns a combined esm_datastore.
Still very much a first draft but should work for consistently shaped
datastores.
Usage: combine_datastores(esm_datastore_1, esm_datastore_2, ... , esm_datastore_n)
To get your datastores:
```python
import intake
esm_ds1 = intake.open_esm_datastore(
"/path/to/datastore1.json",
columns_with_iterables = ['variable'], # Probably
)
"""
esmcat_dict = esm_datastores[0].esmcat.dict()
df_list = [esm_datastore.df for esm_datastore in esm_datastores]
df = pd.concat(df_list)
return esm_datastore({'esmcat': esmcat_dict, 'df': df})
````
Once you have a combined datastore, you can save it with
```python
esm_datastore.serialize(
name = "experiment_datastore", # saves to `experiment_datastore.json`
directory = "/path/to/a/dir", # Puts it in directory `/path/to/a/dir`
catalog_type='file'
)
```
Let me know if you have any issues & I’ll do my best to help!
Cheers, Charles
---
### Author:  [@CharlesTurner](https://forum.access-hive.org.au/u/CharlesTurner)
#### Post date: [29 July 2025 06:50 UTC](https://forum.access-hive.org.au/t/combining-intake-datastores-and-missing-files-from-custom-experiment-datastore/4741/4 "2025-07-29T06:50:45Z")
---
### Author:  [@claireyung](https://forum.access-hive.org.au/u/claireyung)
#### Post date: [29 July 2025 07:02 UTC](https://forum.access-hive.org.au/t/combining-intake-datastores-and-missing-files-from-custom-experiment-datastore/4741/5 "2025-07-29T07:02:02Z")
Awesome, thanks so much @CharlesTurner, that script combined the datastores!
On the files that don’t make it in - I’ve moved all the output files to `/g/data/ol01/cy8964/access-om3/archive/8km_jra_ryf_obc_Charrassin/`, the ones that don’t make it are `ocean_month_z.nc` and `ocean_month.nc`.
Thanks!
---
### Author:  [@clairecarouge](https://forum.access-hive.org.au/u/clairecarouge)
#### Post date: [27 July 2026 05:08 UTC](https://forum.access-hive.org.au/t/combining-intake-datastores-and-missing-files-from-custom-experiment-datastore/4741/6 "2026-07-27T05:08:01Z")