# How to efficiently chunk data for faster processing and plotting? **URL:** \ **Category:** Technical\ **Tags:** python, cosima, access-om2\ **Created:** [27 September 2024 06:28 UTC](https://forum.access-hive.org.au/t/how-to-efficiently-chunk-data-for-faster-processing-and-plotting/3735 "2024-09-27T06:28:30Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![sb4233](https://avatars.discourse-cdn.com/v4/letter/s/85f322/32.png) [@sb4233](https://forum.access-hive.org.au/u/sb4233)\ **Post date:** [27 September 2024 06:28 UTC](https://forum.access-hive.org.au/t/how-to-efficiently-chunk-data-for-faster-processing-and-plotting/3735/1 "2024-09-27T06:28:30Z")
Hi all, I have been using the wonderful `COSIMA COOKBOOK` to access the OM2 datasets for a while. It’s great for searching and loading datasets, but I am facing a bit of issue with the chunking as I am new to this type of coding. Due to being chunked it is highly efficient to do my processing but when I try to plot these it takes forever. For example, below is my processed data array for coherence between two variables at every grid point. Now if I want to plot this data (for example using `plt.contourf` etc.), I think it tries to load the whole data into memory and that’s where it takes forever. I wanted to know if there is a nicer and faster way to do this. Any help would be much appreciated. ![image](https://us1.discourse-cdn.com/flex020/uploads/access1/original/2X/3/30e78e21cd57ae9a7cd019d775b38a59fc50814e.png) ---
**Author:** ![atteggiani](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/atteggiani/32/212_2.png) [@atteggiani](https://forum.access-hive.org.au/u/atteggiani)\ **Post date:** [27 September 2024 09:50 UTC](https://forum.access-hive.org.au/t/how-to-efficiently-chunk-data-for-faster-processing-and-plotting/3735/2 "2024-09-27T09:50:43Z")
Hi Sougata, When you say that your “processing” is very efficient due to chunking, it is not exactly what is happening under the hood. `xarray` uses [Dask](https://docs.dask.org/en/stable/) under the hood to store DataArrays, which provides features like [chunking](https://docs.dask.org/en/stable/array-chunks.html) and [lazy evaluation](https://docs.dask.org/en/stable/user-interfaces.html?highlight=lazy%20evaluation#laziness-and-computing). Lazy evaluation means most of the processing you do on the data is not computed immediately. Instead, Dask stores all the operations that need to be carried out in a [task graph](https://docs.dask.org/en/stable/graphviz.html), that can be viewed with the DataArray method `.visualize()` (for the low-level task graph) or `.dask.visualize()` (for the high-level layer graph). All these tasks (and so the actual processing) will be actually executed only when the data is computed. In your case, this is when you plot the data. That is why you are experiencing very long processing times only at the time of plotting. To better understand why it is taking so long to process your data, you can take a look at the total number of tasks your Dask graph has and what that is influenced by. In general, the more tasks your Dask graph has, the more time it will take to do the processing. Although, this is also influenced by the specific algorithm (operation) you are performing. In addition, chunking greatly influences this performance as well, and will influence the total number of tasks in the Dask graph. In your case, the `Dask graph` field in your DataArray says “552 chunks in 1246 graph layers”, which means there are 1246 total “operations” (the nodes in the high-level layer graph) that need to be carried out. Each of this operations has 552 tasks (the number of your chunks), for a total of approximately 552\*1246 = 687792 total tasks!! ( you can also check the total number of tasks with `len(yourDataArray.dask)`). I don’t use `xarray` as much anymore, so I might be wrong here, but that sounds like a lot of tasks!! The main reasons might be your chunks being too small, so you might be able to have less chunks but bigger in size. Also note that every chunk is currently 6MB, which can be safely increased without any memory-related issues. The main chunking direction (coordinates along which you are chunking) also highly influences performance, and depending on your operations there might be better/worse way to chunk your data. As mentioned above, I don’t use `xarray` and `dask` often anymore, so other people might be able to give you more practical indications on best practices for chunking and performance. As a starting point, you might want to check out [Array Best Practices](https://docs.dask.org/en/stable/array-best-practices.html) and [Dask Best Practices](https://docs.dask.org/en/stable/best-practices.html). Hope this helps. Cheers Davide ---
**Author:** ![sb4233](https://avatars.discourse-cdn.com/v4/letter/s/85f322/32.png) [@sb4233](https://forum.access-hive.org.au/u/sb4233)\ **Post date:** [27 September 2024 10:51 UTC](https://forum.access-hive.org.au/t/how-to-efficiently-chunk-data-for-faster-processing-and-plotting/3735/3 "2024-09-27T10:51:56Z")
Thanks very much @atteggiani for such a detailed and insightful answer. It cleared much of doubts! I will try out your suggestions : ) ---
**Author:** ![navidcy](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/navidcy/32/101_2.png) [@navidcy](https://forum.access-hive.org.au/u/navidcy)\ **Post date:** [28 September 2024 02:07 UTC](https://forum.access-hive.org.au/t/how-to-efficiently-chunk-data-for-faster-processing-and-plotting/3735/4 "2024-09-28T02:07:05Z")
(if @atteggiani answers clarifies your questions @sb4233 then click the `Select if this reply solves the problem` at the bottom of the post?) ---
**Author:** ![Thomas-Moore](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/thomas-moore/32/329_2.png) [@Thomas-Moore](https://forum.access-hive.org.au/u/Thomas-Moore)\ **Post date:** [29 September 2024 03:43 UTC](https://forum.access-hive.org.au/t/how-to-efficiently-chunk-data-for-faster-processing-and-plotting/3735/5 "2024-09-29T03:43:21Z")
Hey @sb4233 - if you are working with OM2 regularly and want to explore and help document how to improve xarray / dask workflows then consider jumping over here and documenting more specifics about your use case. > [@Join in! Investigating analysis-ready data (ARD) strategies to increase impact of ocean and climate model archives at NCI](https://forum.access-hive.org.au/t/join-in-investigating-analysis-ready-data-ard-strategies-to-increase-impact-of-ocean-and-climate-model-archives-at-nci/3723): > > rocket vn19 rocketInvestigating analysis-ready data (ARD) strategies to increase impact of ocean and climate model archives at NCI The purpose of this NCI project is to provide resources to develop and test Analysis-Ready Data (ARD) workflows for climate and ocean modelling on High-Performance Computing (HPC) systems at NCI. This project, supported by the CSIRO, aims to bring together members of the COSIMA community, the Australian Climate Service\*, and other interested parties to explore … We have spare compute and TB’s of storage to enable analysis-ready data (ARD) workflows in the new `vn19` project. ---
**Author:** ![sb4233](https://avatars.discourse-cdn.com/v4/letter/s/85f322/32.png) [@sb4233](https://forum.access-hive.org.au/u/sb4233)\ **Post date:** [29 September 2024 05:29 UTC](https://forum.access-hive.org.au/t/how-to-efficiently-chunk-data-for-faster-processing-and-plotting/3735/6 "2024-09-29T05:29:20Z")
Thanks @Thomas-Moore for introducing vn19! It seems like a great initiative for addressing questions related to analyzing large datasets. Will try to be an active participant 🙂