Datastore making - unidentified realm

COSIMA
1

Hello

I am getting an error in making a datastore for processed ACESS-OM2 files. These are density-binned files procesed offline from the 01deg_jra55v140_iaf_cycle3_antarctic_tracers run. The error appears related to the fact that the files are not identified to have any standard intake catalog “realm”, presumably due to the offline density binning. Code and error message detalis are below. Anyone has an idea of how to deal with this?

Thanks,

Aviv

p.s.1 the 01deg_jra55v140_iaf_cycle3_antarctic_tracers run is in the process of integrated into the intake catalog.

p.s.2 the datastore making script

from access_nri_intake.source import builders
from access_nri_intake.experiment import use_datastore

FoldSigBinned = ‘/g/data/g40/access-om2/archive/01deg_jra55v140_iaf_cycle3_antarctic_tracers_isopyc/’

FoldDBs = ‘/scratch/v45/as2408/access-om2/01deg_jra55v140_iaf_cycle3_antarctic_tracers/DBs/’

BUILDER = builders.AccessOm2Builder
CATALOG_DIR = FoldDBs
DATASTORE_NAME = “Exp9tracersSigAvgs”

esm_ds = use_datastore(
builder=BUILDER,
experiment_dir=FoldSigBinned,
catalog_dir=CATALOG_DIR,
datastore_name=DATASTORE_NAME
)

esm_ds

p.s.3 The error message:

---------------------------------------------------------------------------
ParserError                               Traceback (most recent call last)
Cell In[112], line 5
      2 CATALOG_DIR = FoldDBs# Path("~/catalog_dir").expanduser() # We'll save our datastore in a directory called catalog_dir in our home dir
      3 DATASTORE_NAME = "Exp9tracersSigAvgs"
----> 5 esm_ds = use_datastore(
      6     builder=BUILDER,
      7     experiment_dir=FoldSigBinned,
      8     catalog_dir=CATALOG_DIR,
      9     datastore_name=DATASTORE_NAME
     10 )
     12 esm_ds

File /g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/access_nri_intake/experiment/main.py:122, in use_datastore(experiment_dir, builder, catalog_dir, builder_kwargs, open_ds, datastore_name, description)
    120 builder_instance: Builder = builder(path=str(experiment_dir), **builder_kwargs)
    121 print(f"{f_info}Building esm-datastore...{f_reset}")
--> 122 builder_instance.get_assets().build()
    123 print(f"{f_success}Sucessfully built esm-datastore!{f_reset}")
    124 print(
    125     f"{f_info}Saving esm-datastore to {f_path}{str(catalog_dir.absolute())}{f_reset}"
    126 )

File /g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/access_nri_intake/source/builders.py:237, in BaseBuilder.build(self)
    232 def build(self):
    233     """
    234     Builds a datastore from a list of netCDF files or zarr stores.
    235     """
--> 237     self.get_assets().validate_parser().parse().clean_dataframe()
    239     return self

File /g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/access_nri_intake/source/builders.py:227, in BaseBuilder.validate_parser(self)
    224         validate_against_schema(info, ESM_JSONSCHEMA)
    225         return self
--> 227 raise ParserError(f"""Parser returns no valid assets.
    228     Try parsing a single file with Builder.parser(file)
    229     Last failed asset: {asset}
    230     Asset parser return: {info}""")

ParserError: Parser returns no valid assets.
            Try parsing a single file with Builder.parser(file)
            Last failed asset: /g/data/g40/access-om2/archive/01deg_jra55v140_iaf_cycle3_antarctic_tracers_isopyc/passive_wilkes_tr_DensityBinned_monthly-mean-ym_1999_12.nc
            Asset parser return: {'INVALID_ASSET': '/g/data/g40/access-om2/archive/01deg_jra55v140_iaf_cycle3_antarctic_tracers_isopyc/passive_wilkes_tr_DensityBinned_monthly-mean-ym_1999_12.nc', 'TRACEBACK': 'Traceback (most recent call last):\n  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/access_nri_intake/source/builders.py", line 207, in _parser_catch_invalid\n    return cls.parser(file)\n           ^^^^^^^^^^^^^^^^\n  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-26.01/lib/python3.11/site-packages/access_nri_intake/source/builders.py", line 470, in parser\n    raise ParserError(f"Cannot determine realm for file {file}")\naccess_nri_intake.source.builders.ParserError: Cannot determine realm for file /g/data/g40/access-om2/archive/01deg_jra55v140_iaf_cycle3_antarctic_tracers_isopyc/passive_wilkes_tr_DensityBinned_monthly-mean-ym_1999_12.nc\n'}