FRE-NCTools - "make_topog": MPI error when generating topography

Technical
1

Hi all - I’m attempting to generate a topography using a subset of GEBCO2022, and a 1/30th degree hgrid i’ve generated using FRE-NCTools make_hgrid and make_solo_mosaic. I’m getting an error when executing the “make_topog” command after about 10 minutes of processing, stating:
FATAL Error: nxgrid is greater than MAXXGRID/nthreads, increase MAXXGRID, decrease nthreads, or increase number of MPI ranks
The error message seems quite straightforward, however I don’t yet know enough about MPI to make these suggested changes.

The code which produced this error is:
FRE-NC-toolpath/make_topog --mosaic ocean_mosaic.nc --topog_type realistic --topog_file eac2.nc --topog_field elevation

I’ve tried this within a gadi_jupyter-lab session and ARE jupyterlab session (using subprocess).

The toolpath I’m using is in /g/data/ul08/FRE_tools/bin/bin/make_topog
The runpath can be found in /scratch/v45/jr5971/mom6/regional_configs/hgrid_hi-res2/

Any pointers?

2

Given that it’s taking a while, you could probably run the tool with MPI. Make sure you’re building with mpif90 and mpicc, then you can mpiexec -np <num_ranks> make_topog. I don’t really have any advice for how many ranks to use, but you could trial and error your way to something that runs acceptably fast.

Otherwise, if you want to run single-threaded you can bump up the MAXXGRID file defined in create_xgrid.h and rebuild the tools.

3

Awesome, thanks Angus, I’ll give those suggestions a go and update when I’ve got a result.

4

Hi again Angus - I’ve got a few embarrasing questions to ask based on your suggestions.

  1. Is the best/only way to run mpiexec to submit a PBS job? If so, would you happen to have an example PBS script for when you’ve done something similar?
  2. When you say the following…
  • Are you talking about recompiling the make_topog tool? Or do I need to setup a whole new config file to feed into mpiexec that somehow incorporates mpif90 & mpicc?
  1. Have you got any examples of this approach for other tasks you’ve done? Just at odds on where to start.

Apologies for all the questions! This is entirely new territory for me.

5
  1. Is the best/only way to run mpiexec to submit a PBS job? If so, would you happen to have an example PBS script for when you’ve done something similar?

You could do it inside a gadi_jupyterlab session that uses multiple cores (maybe also ARE, but I don’t have any experience running MPI programs there). Otherwise, if you want to use a dedicated job, you only really need to load the OpenMPI module and run your program, prefixing it with mpiexec. There’s an example on NCI Help.

  1. … Are you talking about recompiling the make_topog tool? Or do I need to setup a whole new config file to feed into mpiexec that somehow incorporates mpif90 & mpicc?

Yes, you need to recompile make_topog using the MPI-enabled compilers, so that you can run it under MPI with mpiexec. Usually it’s just a matter of changing the compiler commands. I don’t know how you’ve built the tool though, so I can’t provide more specific information.

6

Thanks again Angus. Managed to recompile the tools with the “./configure --with-mpi=yes” argument - made another mistake which was to not have openmpi loaded before compiling but now it’s all working very smoothly using make_topog_parallel in a PBS job.

2 Likes
7

PS: I did have to increase the MAXXGRID variable in the create_xgrid.h file as well to make it work, so thanks for that pointer as well!

2 Likes