Earthmover.io are open sourcing Icechunk, the back-end software for their ArrayLake offering:
This is a seriously impressive bit of software that adds transactions and versioning to zarr datastores.
1 Like
I’m hoping Pawsey HPC engineers are having a serious look for their warm tier object storage cluster, Acacia? And given the hope that NCI will increasingly include on-premises object store capacity that NCI HPC engineers are getting across this to? @Aidan, what is your view on best ways to socialise this across the community?
Find some like-minded folks around here and look into uses cases? I know @anton and @MartinDix are very keen on improving versioning of important data, like model inputs.
Proof of concept demos?
Get someone who is familiar with it to give a presentation?
1 Like
We might get a list together of key people and then ask EarthMover folks to think about tailoring a virtual talk about how to use Icechunk for on-prem object store?
1 Like
2 Likes
I had a chat with Ryan today, we’re setting up a trial account which I’ll use to explore Pawsey object storage. I’ve struggled with the Rust dependency on the docker/Singularity images I use on Pawsey, I know it’s not that hard but it’s just another thing to add to my Python challenges. I think earthmover would jump at working with anyone with GADI experience and object storage.
As another thing, has anyone looked at Arkouda? Pangeo Showcase: "Arkouda as an XArray backend for HPC!" - Pangeo Showcase - Pangeo
Just watched that and they’re interested in testers on HPC. I’m pretty comfy on Pawsey now, but only have limited experience on GADI with Python tooling (so if anyone wants to explore and hand-hold with me that’d be awesome).
2 Likes
This is awesome.
I’m currently in an email chat with Ryan about setting up a virtual showcase for NCI / Pawsey / Australian folks in late January or early February.
4 Likes
That sounds great!
Hey @mdsumner et al
Ryan would love to have a chat with core folks before he gives a wider showcase.
He’d like to speak to data users ( and those who care about them ) about what the current pain points and problems are. He’d like to ask us some questions to shape his presentation.
Can we target this preliminary chat for late January? Who should be on it? I’m happy to help coordinate and organise.
I’m keen. Anton Steketee, Lenneke Jong, Ben Raymond come to mind.
1 Like