# Cosima cookbook updating needs **URL:** \ **Category:** Workshops\ **Tags:** cosima-workshop-2022\ **Created:** [10 November 2022 00:36 UTC](https://forum.access-hive.org.au/t/cosima-cookbook-updating-needs/130 "2022-11-10T00:36:18Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![JuliaN](https://avatars.discourse-cdn.com/v4/letter/j/74df32/32.png) [@JuliaN](https://forum.access-hive.org.au/u/JuliaN)\ **Post date:** [10 November 2022 00:36 UTC](https://forum.access-hive.org.au/t/cosima-cookbook-updating-needs/130/1 "2022-11-10T00:36:18Z")
We discussed briefly during the breakout sessions possible improvements to the cookbook: 1. It is currently not scaling well with higher resolution models (\>1/10 of a degree). Could this be improved? 2. Many experiments in the database are not well documented and lack metadata. We should enforce mandatory metadata for the experiments indexed in the database. Please add to the list any other points for improvement 🙂 ---
**Author:** ![Aidan](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/aidan/32/42_2.png) [@Aidan](https://forum.access-hive.org.au/u/Aidan)\ **Post date:** [10 November 2022 01:22 UTC](https://forum.access-hive.org.au/t/cosima-cookbook-updating-needs/130/2 "2022-11-10T01:22:37Z")
Good points @JuliaN. We’ve long aspired to have some test codes to use as a basis for optimisation, but it hasn’t gone very far. Still a worthy goal! > > > Many people report problems with running calculations on large datasets, and wou…ld like some general advice on the best approaches for tackling large problems. > > There are lots of parameters that determine the success/efficiency of a calculation: > > 1. Order of operations > 2. Calculating intermediate results > 3. Dask chunking > 4. netCDF chunking on disk > 5. Number of dask workers (or not using a scheduler/dask at all) > 6. Number of threads and amount of memory per worker > > It becomes very complex very quickly. > > One approach is to have some representative test calculations that can then be used as a target for optimisation. These test calculations can be run whenever there are infrastructure or algorithm changes to check there has been no degradation in performance, or if they might be further improved. > > If that sounds like a useful idea then we need people to propose calculations that they know to be strenuous as possibilities for optimisation/best-practicification\*. Ideally these would be fairly compact, reproducible chunks of code. > > ping @AndyHoggANU @aekiss @adele157 @navidcy @angus-g > > > > \* not a real word The idea of only indexing data with sufficient metadata [has been floated before](https://github.com/COSIMA/cosima-cookbook/issues/299#issue-1288136695), and I agree, it is long overdue. ---
**Author:** ![rbeucher](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/rbeucher/32/11_2.png) [@rbeucher](https://forum.access-hive.org.au/u/rbeucher)\ **Post date:** [11 November 2022 01:39 UTC](https://forum.access-hive.org.au/t/cosima-cookbook-updating-needs/130/3 "2022-11-11T01:39:19Z")
Hi @JuliaN , Do you have a specific recipe in mind? I am looking at setting up an environment on Gadi to do that sort of scaling exercise. I do think that is something that we need to run routinely. A use case to get us started would be great! Thanks! Romain ---
**Author:** ![navidcy](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/navidcy/32/101_2.png) [@navidcy](https://forum.access-hive.org.au/u/navidcy)\ **Post date:** [14 November 2022 00:17 UTC](https://forum.access-hive.org.au/t/cosima-cookbook-updating-needs/130/4 "2022-11-14T00:17:39Z")
From the top of my head I remember that the overturning stream function example can be quite demanding when using the 0.1 degree output. [https://cosima-recipes.readthedocs.io/en/latest/documented\_examples/Zonally\_Averaged\_Global\_Meridional\_Overturning\_Circulation.html#gallery-documented-examples-zonally-averaged-global-meridional-overturning-circulation-ipynb](https://cosima-recipes.readthedocs.io/en/latest/documented_examples/Zonally_Averaged_Global_Meridional_Overturning_Circulation.html#gallery-documented-examples-zonally-averaged-global-meridional-overturning-circulation-ipynb) @AndyHoggANU might have more in mind? ---
**Author:** ![navidcy](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/navidcy/32/101_2.png) [@navidcy](https://forum.access-hive.org.au/u/navidcy)\ **Post date:** [14 November 2022 00:27 UTC](https://forum.access-hive.org.au/t/cosima-cookbook-updating-needs/130/5 "2022-11-14T00:27:43Z")
We also discussed that we want some sort of automation to ensure that all notebooks run using the latest conda-analysis environment and that no output has moved or deleted, etc. This is related to [Automating jupyterbook running with CI tools - #3 by navidcy](https://forum.access-hive.org.au/t/automating-jupyterbook-running-with-ci-tools/83/3) ---
**Author:** ![adele-morrison](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/adele-morrison/32/90_2.png) [@adele-morrison](https://forum.access-hive.org.au/u/adele-morrison)\ **Post date:** [14 November 2022 02:28 UTC](https://forum.access-hive.org.au/t/cosima-cookbook-updating-needs/130/6 "2022-11-14T02:28:42Z")
Yes I second using overturning as a test example. Calculating time series of overturning for a couple of hundred years of the RYF takes a long time and is a metric that is used a lot. ---
**Author:** ![Aidan](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/aidan/32/42_2.png) [@Aidan](https://forum.access-hive.org.au/u/Aidan)\ **Post date:** [14 November 2022 04:05 UTC](https://forum.access-hive.org.au/t/cosima-cookbook-updating-needs/130/7 "2022-11-14T04:05:27Z")
> [@navidcy](#): > > We also discussed that we want some sort of automation to ensure that all notebooks run using the latest conda-analysis environment and that no output has moved or deleted, etc. This can be done relatively easily with an automated job that updates the kernel version in the file, without even having to run it. It is definitely something worthwhile to automate so that recipes are always correct when run at NCI. ---
**Author:** ![rbeucher](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/rbeucher/32/11_2.png) [@rbeucher](https://forum.access-hive.org.au/u/rbeucher)\ **Post date:** [14 November 2022 22:38 UTC](https://forum.access-hive.org.au/t/cosima-cookbook-updating-needs/130/8 "2022-11-14T22:38:31Z")
OK. I’ll add that one to my To-do list then