I am struggling to access the future perturbation runs by @MATTENGLAND on NCI using the COSIMA cookbook (experiment = ‘01deg_jra55v13_ryf9091_qian_wthmp’)
FileNotFoundError: [Errno 2] No such file or directory: b'/g/data/ik11/outputs/access-om2-01/01deg_jra55v13_ryf9091_qian_wthmp/output878/ocean/ocean_month.nc'
2023-10-04 13:43:25,853 - distributed.worker - WARNING - Compute Failed
Key: open_dataset-15e0c6d3-a0d8-45f2-b7da-69099463b995
Function: execute_task
args: ((<function apply at 0x153ebc720f70>, <function open_dataset at 0x153e9f82ef70>, ['/g/data/ik11/outputs/access-om2-01/01deg_jra55v13_ryf9091_qian_wthmp/output934/ocean/ocean_month.nc'], (<class 'dict'>, [['engine', None], ['chunks', (<class 'dict'>, [['xu_ocean', 3600]])]])))
kwargs: {}
Exception: "FileNotFoundError(2, 'No such file or directory')"
Note that this data was recently moved from ik11 to cj50, so not sure why the database is still pointing to ik11.
1 Like
Aidan
(Aidan Heerdegen, ACCESS-NRI Release Team Lead)
2
Who moved the data? @aekiss? I ask because the metadata.yaml file didn’t make it across. Both are still listed in the database, but only the ik11 location has metadata associated with it:
189|01deg_jra55v13_ryf9091_qian_wthmp|/g/data/cj50/access-om2/raw-output/access-om2-01/01deg_jra55v13_ryf9091_qian_wthmp||||||
135|01deg_jra55v13_ryf9091_qian_wthmp|/g/data/ik11/outputs/access-om2-01/01deg_jra55v13_ryf9091_qian_wthmp|Qian Li|qian.li5@unsw.edu.au|2021-04-20||This simulation is designed for current and future climate projections using the forcing fields of wind, thermal and meltwater based on JRA55-do, CMIP6 scenario (SSP5-8.5) and Golledge et al. (2019). For the details in forcing fields, please contact Qian Li (qian.li5@unsw.edu.au). The projections are available for the period 2001-2040 on ik11 and also extended to 2070. If you would like to look into the model outputs, please also contact the project PI Matt England (m.england@unsw.edu.au) and the simulation contributor Andy Hogg (andy.hogg@anu.edu.au). |If you require any further information, feel free to contact Qian.
see
$ ls /g/data/cj50/access-om2/raw-output/access-om2-01/01deg_jra55v13_ryf9091_qian_wthmp/metadata.yaml
ls: cannot access '/g/data/cj50/access-om2/raw-output/access-om2-01/01deg_jra55v13_ryf9091_qian_wthmp/metadata.yaml': No such file or directory
Moving data was always a bit fraught. The best approach is to move all the data out of the directory but leave the directory there until the database has been updated, so that all entries for the old location are deleted correctly.
So I’d suggest reinstating /g/data/ik11/outputs/access-om2-01/01deg_jra55v13_ryf9091_qian_wthmp and running the update.
In the mean time it should be possible to mark present as FALSE for all the ncfiles with experiment_id = 135. It’s no longer my remit to muck about with that database, but @micael or @angus-g might be able to do it.
But that metadata.yaml file information should be reinstated in the new location.
Aidan
(Aidan Heerdegen, ACCESS-NRI Release Team Lead)
3
I don’t have the correct permissions, but if either @micael or @aekiss are able to, I think
Done - hopefully that fixes the problem. I don’t know who moved the data, but it’s all owned by ql1694 who I guess is Qian?
Aidan
(Aidan Heerdegen, ACCESS-NRI Release Team Lead)
5
I’ve reconstituted the metadata.yaml as much as I could from the existing database entry if that is useful
contact: Qian Li
email: qian.li5@unsw.edu.au
created: 2021-04-20
name: 01deg_jra55v13_ryf9091_qian_wthmp
model:
- ACCESS-OM2
nominal_resolution:
- 0.1 degrees
version:
description: >-
0.1 degree ACCESS-OM2 global model configuration with JRA55-do v1.3 RYF9091
repeat year forcing (May 1990 to Apr 1991)
This simulation is designed for current and future climate projections using the forcing fields of wind,
thermal and meltwater based on JRA55-do, CMIP6 scenario (SSP5-8.5) and Golledge et al. (2019). For the
details in forcing fields, please contact Qian Li (qian.li5@unsw.edu.au). The projections are available
for the period 2001-2040 on ik11 and also extended to 2070. If you would like to look into the model outputs,
please also contact the project PI Matt England (m.england@unsw.edu.au) and the simulation contributor Andy Hogg (andy.hogg@anu.edu.au).
notes: >-
If you require any further information, feel free to contact Qian.
keywords:
- tenth
- cosima
- access-om2-01
- jra55-do-v1.3
- ryf
I haven’t included an experiment_uuid field, as this isn’t used by the Cookbook, and AFAICT this experiment was not included in the ACCESS-NRI Intake catalog so there is no way to know if a unique ID had already been defined.
I very much doubt the metadata.yaml had an experiment_uuid field. I’ve been adding this field as data are added to the Intake catalog, and you’re correct that this experiment is not yet included. Probably best to leave it out for now and add one if/when this experiment gets added to the catalog.
Aidan
(Aidan Heerdegen, ACCESS-NRI Release Team Lead)
7
Yes that seems to have had the desired effect of removing all the moved files from the database:
sqlite> select * from experiments where experiment = '01deg_jra55v13_ryf9091_qian_wthmp';
189|01deg_jra55v13_ryf9091_qian_wthmp|/g/data/cj50/access-om2/raw-output/access-om2-01/01deg_jra55v13_ryf9091_qian_wthmp||||||
135|01deg_jra55v13_ryf9091_qian_wthmp|/g/data/ik11/outputs/access-om2-01/01deg_jra55v13_ryf9091_qian_wthmp|Qian Li|qian.li5@unsw.edu.au|2021-04-20||This simulation is designed for current and future climate projections using the forcing fields of wind, thermal and meltwater based on JRA55-do, CMIP6 scenario (SSP5-8.5) and Golledge et al. (2019). For the details in forcing fields, please contact Qian Li (qian.li5@unsw.edu.au). The projections are available for the period 2001-2040 on ik11 and also extended to 2070. If you would like to look into the model outputs, please also contact the project PI Matt England (m.england@unsw.edu.au) and the simulation contributor Andy Hogg (andy.hogg@anu.edu.au). |If you require any further information, feel free to contact Qian.
sqlite> select count(*) from ncfiles where experiment_id = 135;
0
sqlite>
Now you should probably remove the ik11 directory and consider making a metadata.yaml directory in the cj50 location.
Try loading that data now @taimoorsohail and see if the problem is fixed.
Thanks @aekiss and @Aidan. That had the desired effect for that experiment, but I am having the same issue with 01deg_jra55v13_ryf9091_qian_wthp. Do I have permissions to make the directory in ik11? Otherwise you may have to do this again @aekiss.