"database disk image is malformed" // COSIMA Cookbook failure?

COSIMA
, ,
1

Hi all. Thanks to those credited with all the different cool tools & documentation for data discovery across the millions of netcdf files of model output we have before us.

I’m trying to 100% confirm that while the 01deg_jra55v140_iaf_cycle4 OM2 10th of a degree run does have daily 3D BGC output, it does not have 3D daily output for physical variables like temp or salt.

So far I’ve confirmed this using (A) Mk1 eyeball squinting my way through the ik11 forest of netcdf files and (B) making use of the new ACCESS-NRI intake catalog.

Because I’ve got trust issues and also always assume I’ve done something wrong I’m trying to triple confirm this fact using the COSIMA COOKBOOK.

However I can’t get the CC tutorial working?

DatabaseError: (sqlite3.DatabaseError) database disk image is malformed is the error I get when I run cc.querying.get_experiments(session).

I’m following the tutorial and working on ARE using the conda/analysis3-unstable module on NCI, as directed.

What am I doing wrong that is silly? Thanks in advance for the support.

2

01deg_jra55v140_iaf_cycle4 doesn’t have any daily 3d output (the 3d BGC is 5-daily samples of 1-day means). See here: https://cosima.org.au/index.php/2020/07/29/data-available-0-1-1958-2018-access-om2-iaf-run/

1 Like
3

Thanks for that confirmation @aekiss, much appreciated.

4

RE: CC database

While some tutorial code appears to fail, other commands work and provide answers on the frequencies available for 3D temp across the 01deg_jra55v140_iaf_cycle4 experiment. Example:

cc.querying.get_variables(frequency='1 monthly',session=session,experiment='01deg_jra55v140_iaf_cycle4',search='temp')

vs

cc.querying.get_variables(frequency='1 daily',session=session,experiment='01deg_jra55v140_iaf_cycle4',search='temp')

It appears possibly that any CC database query that hits all experiments results in the database disk image is malformed error?

5

Hey @Thomas-Moore. Sometimes the DB gets corrupted, and you don’t necessarily see it if you’re not hitting the bad bits. So yeah, if you’re doing something that accesses more of it, you’re more likely to get the error.

There are DB backups, so you could try and older one and see if that works. If so maybe let @micael know, and he can make sure the bad ones are removed.

Edit: Forgot to mention where to find the DB backups

/g/data/ik11/databases/daily/
1 Like
6

Thanks for your help @Aidan - I’m basically a noob to the CC who couldn’t get the tutorial to run so thought I’d flag the error. I can get the answer I need via CC and also now have the authoritive confirmation from @aekiss.

I guess it’s just an issue for anyone new trying the CC tutorial? database disk image is malformed is a scary looking error - doesn’t build confidence if you’re new to something. :smile:

Actually this little exploration has shown how robust and useful the ACCESS-NRI intake catalog is. I never should have doubted it. :wink:

1 Like
7

Pull quote for the poster @dougiesquire

2 Likes
8

Oh no, looks like the database has been corrupted :frowning:

I’ve removed the corrupted versions. Let’s see if the next update works correctly.

It seems clear that intake is the future for the COSIMA recipes.

1 Like