# Latest version of dask (2022.11.0) could fix many workflow issues **URL:** \ **Category:** Technical\ **Tags:** python, dask\ **Created:** [16 November 2022 20:51 UTC](https://forum.access-hive.org.au/t/latest-version-of-dask-2022-11-0-could-fix-many-workflow-issues/149 "2022-11-16T20:51:23Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![dougiesquire](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/dougiesquire/32/16_2.png) [@dougiesquire](https://forum.access-hive.org.au/u/dougiesquire)\ **Post date:** [16 November 2022 20:51 UTC](https://forum.access-hive.org.au/t/latest-version-of-dask-2022-11-0-could-fix-many-workflow-issues/149/1 "2022-11-16T20:51:23Z")
The latest version of dask (2022.11.0) uses a new mode of scheduling by default that can significantly reduce the memory usage in a lot of typical geoscience workflows. The new default tries to address the problem of “root task overproduction” - if you’ve ever inexplicably run out of memory while computing a climatology, for example, you were probably experiencing root task overproduction. Details here: [Reducing memory usage in Dask workloads by 80%](https://www.coiled.io/blog/reducing-dask-memory-usage). It would be great to explore/benchmark any improvements from this to COSIMA workflows as part of the [COSIMA hackathon](https://forum.access-hive.org.au/t/cosima-hackathon-january/142). ---
**Author:** ![angus-g](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/angus-g/32/89_2.png) [@angus-g](https://forum.access-hive.org.au/u/angus-g)\ **Post date:** [16 November 2022 21:12 UTC](https://forum.access-hive.org.au/t/latest-version-of-dask-2022-11-0-could-fix-many-workflow-issues/149/2 "2022-11-16T21:12:32Z")
Nice! I’d be really interested to see how this pans out. It sounds like there’s the potential for some slight slowdown as a tradeoff for the reduction in memory usage, but also that it can speed things up by not spilling to disk. ---
**Author:** ![rbeucher](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/rbeucher/32/11_2.png) [@rbeucher](https://forum.access-hive.org.au/u/rbeucher)\ **Post date:** [16 November 2022 22:48 UTC](https://forum.access-hive.org.au/t/latest-version-of-dask-2022-11-0-could-fix-many-workflow-issues/149/3 "2022-11-16T22:48:09Z")
Good idea. I like the idea of bench-marking the versions to see how that plays out. ---
**Author:** ![Aidan](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/aidan/32/42_2.png) [@Aidan](https://forum.access-hive.org.au/u/Aidan)\ **Post date:** [17 November 2022 03:24 UTC](https://forum.access-hive.org.au/t/latest-version-of-dask-2022-11-0-could-fix-many-workflow-issues/149/4 "2022-11-17T03:24:41Z")
I have started an issue to get this installed in the hh5 conda environment > > > It would be great to update to the latest version of dask as it has a new defaul…t scheduler design that can drastically reduce memory usage, which might potentially mean many climate and ocean analyses can run with significantly reduced memory use > > https://forum.access-hive.org.au/t/latest-version-of-dask-2022-11-0-could-fix-many-workflow-issues/149 > > I would have just asked to re-run the Jenkins job, but it appears it has been failing for a while > > https://accessdev.nci.org.au/jenkins/blue/organizations/jenkins/conda%2Fanalysis3-unstable/activity/ > > Do you need any assistance debugging the problems? ---
**Author:** ![Aidan](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/aidan/32/42_2.png) [@Aidan](https://forum.access-hive.org.au/u/Aidan)\ **Post date:** [21 November 2022 01:52 UTC](https://forum.access-hive.org.au/t/latest-version-of-dask-2022-11-0-could-fix-many-workflow-issues/149/5 "2022-11-21T01:52:20Z")
Latest [dask](https://www.dask.org) version is now available in `conda/analysis3-unstable`: ```auto $ conda list dask # packages in environment at /g/data3/hh5/public/apps/miniconda3/envs/analysis3-22.07: # # Name Version Build Channel dask 2022.11.1 pyhd8ed1ab_0 conda-forge ``` Ok big data analysts (@navidcy @adele-morrison @AndyHoggANU). Give it your best shot ---
**Author:** ![navidcy](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/navidcy/32/101_2.png) [@navidcy](https://forum.access-hive.org.au/u/navidcy)\ **Post date:** [21 November 2022 02:04 UTC](https://forum.access-hive.org.au/t/latest-version-of-dask-2022-11-0-could-fix-many-workflow-issues/149/6 "2022-11-21T02:04:05Z")
cc: @claireyung and @polinash ---
**Author:** ![dougiesquire](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/dougiesquire/32/16_2.png) [@dougiesquire](https://forum.access-hive.org.au/u/dougiesquire)\ **Post date:** [5 January 2023 19:12 UTC](https://forum.access-hive.org.au/t/latest-version-of-dask-2022-11-0-could-fix-many-workflow-issues/149/7 "2023-01-05T19:12:10Z")
Here’s a blog post on how this development came about and what we can learn from it: [Dask.distributed and Pangeo: Better performance for everyone thanks to science / software collaboration | by Tom Nicholas | pangeo | Jan, 2023 | Medium](https://medium.com/pangeo/dask-distributed-and-pangeo-better-performance-for-everyone-thanks-to-science-software-63f85310a36b)