# Experiment Proposal: Processing Global km-scale Hackathon Data

**URL:** <https://forum.access-hive.org.au/t/experiment-proposal-processing-global-km-scale-hackathon-data/5347>\
**Category:** Experiments\
**Tags:** experiment, machine-learning, working-group\
**Created:** [2 October 2025 06:16 UTC](https://forum.access-hive.org.au/t/experiment-proposal-processing-global-km-scale-hackathon-data/5347 "2025-10-02T06:16:11Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![sam.green](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/sam.green/32/345_2.png) [@sam.green](https://forum.access-hive.org.au/u/sam.green)\
**Post date:** [2 October 2025 06:16 UTC](https://forum.access-hive.org.au/t/experiment-proposal-processing-global-km-scale-hackathon-data/5347/1 "2025-10-02T06:16:12Z")

</div>

**Experiment title 🔔:** Global km-scale Hackathon Data

**Summary 🔔:**

In May 2025 21st Century Weather, together with NCI and ACCCESS-NRI took part in the Global km-scale Hackathon [km-Scale Hackathon | HK25 homepage](https://digital-earths-global-hackathon.github.io/hk25/). Several simulations were run by different global teams covering 2020-03-01 to 2021-03-01. We transferred 3 models to Gadi: ICON Global, 2.5 km, UM Global, 5 km & 10 km, SCREAM Global, 3.25 km / 128 levels totalling ~100Tb in Zarr format.  
Project `qx55` is where this data is located and where it will be stored long-term. However before it is ready for long-term storage significant processing needs to be carried out to reduce inode usage (better chunking/sharding) and storage usage (higher compression). SUs are needed for this and some storage is required as a staging ground.

**Scientific motivation:**

There were many different science goals for this data: [km-Scale Hackathon | HK25 homepage](https://digital-earths-global-hackathon.github.io/hk25/scienceteams/)  
For this ML group the motivation is that the data was very hard to get, it took a month of continuous transfers. It is of very high resolution and global, and could be used to train an AI emulator for downscaling. There have also been several interesting data-driven models that trained on dataset with this type of grid and show good performance.

**Experiment Name 🔔:**  
**People 🔔:** Sam Green @sam.green  
**Model:** UM, ICON, SCREAM  
**Configuration:**  
**Initial conditions:**  
**Run plan:**  
**Simulation details:** [km-Scale Hackathon | HK25 homepage](https://digital-earths-global-hackathon.github.io/hk25/simulations/)  
**Total KSUs required 🔔:** 50kSU  
**Total storage required 🔔:** 50TB  
**Storage lifetime 🔔:** High chance just this quarter; Low chance the next quarter too.  
**Long term data plan 🔔:** qx55 with further application to NCI or ACCESS-NRI to host qx55  
**Outputs:** UM/ICON/SCREAM Zarr datasets with optimised storage/inode usage.  
**Restarts:**  
**Related articles:**

**Analysis:**

[GitHub - digital-earths-global-hackathon/tools: Pre-processing and preparation of data/ simulations.](https://github.com/digital-earths-global-hackathon/tools) has notebooks to show how to analyse the data.  
[Digital Earths Global Hackathon Data Catalog](https://digital-earths-global-hackathon.github.io/catalog/) is a catalogue containing all datatsets that were available during the Hackathon.

**Conclusion:**

The Hackathon was very successful and has produced large datasets that provide benefits to the Australian community for simulation analysis and training ML models. If there are other models/datasets in [Digital Earths Global Hackathon Data Catalog](https://digital-earths-global-hackathon.github.io/catalog/) that may also be useful to this ML community then we can transfer and store them too.

---

<div class="post-metadata">

**Author:** ![Sanaa](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/sanaa/32/2354_2.png) [@Sanaa](https://forum.access-hive.org.au/u/Sanaa)\
**Post date:** [3 October 2025 02:38 UTC](https://forum.access-hive.org.au/t/experiment-proposal-processing-global-km-scale-hackathon-data/5347/2 "2025-10-03T02:38:53Z")

</div>

Hi Sam, the dataset is definitely valuable. The co-chairs have approved your request and mentioned that, if possible, you could make it more generally available, for example on the Thredds server.