Hi,
I am not able to access outputs from the run 01deg_jra55v13_ryf9091 -
ds = process_dataset(cc.querying.getvar('01deg_jra55v13_ryf9091', 'dzt' , session, frequency='1 monthly'), 'xt_ocean')
PermissionError: [Errno 13] Permission denied: '/g/data/ik11/outputs/access-om2-01/01deg_jra55v13_ryf9091/output1111/ocean/ocean.nc'
Seems like the read permission has been disabled for this directory -
@AndyHoggANU do you have any idea ?
Yes the RYF data has now moved to cj50, so it’s publicly available. I think we should update the cookbook database, because some of the Recipes still don’t work with Intake.
Anyone know how we go about updating the cookbook database?
This used to be updated automatically but is now triggered manually, using these scripts IIRC: GitHub - COSIMA/master_index: Scripts to generate COSIMA master index
@micael do I just do update_db_start.sh, check logs, then update_db_finish.sh?
Does it matter that the old location /g/data/ik11/outputs/access-om2-01/01deg_jra55v13_ryf9091 still exists, but is inaccessible?
See here for context: ACCESS-OM2-01 `01deg_jra55v13_ryf9091` moved from ik11 to cj50
@adele-morrison are you suggesting that the 01deg_jra55v13_ryf9091 is not accessible by the intake catalog or was your comment a generic one?
The RYF is accessible by Intake (we updated the catalog already). But it’s no longer accessible from the main cookbook database. I think until all of the COSIMA recipes no longer use Intake, we should still maintain the cookbook database. Some new students are struggling with using intake with some of the recipes that still use the cookbook.
OK, but if @sb4233 wants to try converting tot he Intake catalogue for this analysis that should work, and might be the quickest route.
@aekiss Yes, those two commands should work. Just make sure you launch them the same day.
@sb4233 Thanks for your post - I am just marking this as out of scope for ACCESS-NRI triage admin purposes but looks like there is lots of help available so please continue the discussion.
OK thanks @micael, I’ve launched it. Do recall roughly how long it will take?
Also, will it remove /g/data/ik11/outputs/access-om2-01/01deg_jra55v13_ryf9091 from the database now that it’s unreadable?
It depends a lot on the number of new files to index. Anywhere between a couple of hours to more than 10 hours.
No. That was one of the issues of the database, as it would require manual pruning.
As an aside, we recently had a discussion at ANU that we really don’t like posts being tagged as ‘outofscope’. It could discourage others in the community from replying if ACCESS-NRI is tagging that the post is not important. And it could make new users feel terrible for finding the courage to ask questions. We already have enough trouble trying to convince students to post for help. Is it possible that for COSIMA at least we could change this ‘outofscope’ tag to something like ‘community-question’ instead?
Haha, we are just discussing that exact question at our staff retreat! We were suggesting #community-help?
Thanks, and yes I have made a custom intake data store for the cj50 files following this : Building Intake-ESM datastores of ACCESS model output — COSIMA Cookbook documentation
So, this seems to work for me now : )
Great that Intake is working for you @sb4233, that’s a better option than the old database.
I’ve updated the cookbook master database to /g/data/ik11/databases/daily/cosima_master_2025-05-27.db in case this trips anyone else up - can you check whether your old method works now? (I have my doubts, as the DB may need /g/data/ik11/outputs/access-om2-01/01deg_jra55v13_ryf9091 to be manually pruned to avoid interfering with the new location.)
If you wanted to do this, you could use the sqlite repl on the database:
select * from experiments where id = 136;
> "136|01deg_jra55v13_ryf9091|/g/data/ik11/outputs/access-om2-01/01deg_jra55v13_ryf9091|
Andy Hogg|andy.hogg@anu.edu.au|2020-06-11|
https://github.com/COSIMA/01deg_jra55_ryf/tree/01deg_jra55v13_ryf9091|
0.1 degree ACCESS-OM2 global model configuration with JRA55-do v1.3 RYF9091
repeat year forcing (May 1990 to Apr 1991)|Additional daily outputs saved from
1 Jan 1950 to 31 Dec 1969 and 1 Jan 2086 to 31 Dec 2100"
select count(*) from ncfiles where experiment_id = 136;
> 0
delete from experiments where id = 136;
delete from keyword_assoc where expt_id = 136;
Thanks @angus-g, how do I invoke the sqlite repl? is there a module I need to load?
Just sqlite3 /g/data/ik11/databases/daily/cosima_master_2025-05-27.db should do the trick, no modules needed!
Btw, the cookbook is working now!