Hi,
I am trying to set up ACCESS-OM2-025 on Nesi (NZ supercomputer).
I followed the build steps for NeSI provided by Harshula (many thanks). Now I’m trying to replicate the 0.25 deg release, version 3.0.
I notice that CICE has to be custom-built for each resolution. The 0.25 deg version (as per the 3.0 release) uses an exe called cice_auscom_1440x1080_48x40_480p.exe. I have built an exe with the same dimensions and name as that one. (As per Harshula’s advice.)
That is evidently set up as a 48x40 layout, (mirroring the ocean layout), but the naming implies it would use 480 CPUs, and yet in the config.yaml and cice_in.nml it is only using 361 CPUs.
There is some mention in the Makefile about “ghost cells”… can anyone please provide a quick primer as to how it ends up being 480p and ultimately 361 CPUs.
I’m asking becuase the job size limits on NeSI are smaller than Gadi, and I’m going to have to cut down the number of CPUs. help
I wrote a CMake build for cice5 which helped a bit to make this clearer and is used in the latest releases. @manodeep is well on the way to getting rid of the static memory allocations in cice5, which means that the layout can be changed without re-compiling.
The key detail is that cice has multiple computational blocks per processor (task) and that there are several ways of choosing how to distribute the blocks amongst processors. The goal is to balance the computational load between the two poles (as there’s more ice in the Arctic when there’s less in the Antarctic, and vice-versa). The blocks are sized such that block_size * mxblcks covers the entire grid, where mxblcks is the maximum number of blocks per processor. For the Makefile build, mxblcks was calculated automatically (and not always correctly, depending on the block decomposition used).
Segmenting the grid into 48 in x and 40 in y, gives blocks sizes of 30x27 cells. To cover the whole grid, there is 1920 blocks and dividing that by 480 tasks, gives a mxblcks of 4. This is what is used at build time.
However, looking in the runtime logs, shows something subtly different:
Domain Information
Horizontal domain: nx = 1440
ny = 1080
...
Processors: total = 361
Processor shape: square-ice
Distribution type: roundrobin
Distribution weight: latitude
...
Number of ghost cells: 1
ice: total number of blocks is 1442
Block size: nx_block = 32
ny_block = 29
Internally it’s calculated it only needs 1442 blocks. (The others are presumably wholly land). 1442 divided by 4 (mxblocks) rounds up to 361. It also looks like the ghost cells just get added on inside the code.
It’s hard to give concrete advice on how to get a good new layout as there are too many variables. I suggest just setting ntask to the correct number of tasks which make sense on Nesi, then finding a block size which is roughly square and gives 5-9 blocks per task. And just running with that. The impact of mxblcks being slightly larger than required is probably moot (it will allocate some extra memory that isn’t needed). The ice_diag log will report the number of blocks actually needed.