Payu generated symlinks don't work with ParallelIO library

anton · 21 November 2023 03:32

I am posting here because I don’t know where it fits better!

I am trying to configure CICE in OM3 to use Parallel IO (which supports Netcdf 4, rather than the current Netcdf Classic - Parallel IO in CICE · Issue #81 · COSIMA/access-om3 · GitHub). This is configurable in Nuopc - by changing nuopc.runconfig here.

When an attempt to parallel read a restart file is made, we get this error:

get_stripe failed: 61 (No data available)
Abort with message NetCDF: Error initializing for parallel access in file /jobfs/98914803.gadi-pbs/mo1833/spack-stage/spack-stage-parallelio-2.5.10-hyj75i7d5yy5zbqc7jm6whlkduofib2k/spack-src/src/clib/pioc_support.c at line 2832

This get_stripe failed is because payu provides a symlink to the restart file rather than the file. The payu ‘work’ directiory has an ‘input’ folder with symlinks ( e.g. input/iced.1900-01-01-10800.nc -> /g/data/ik11/inputs/access-om3/0.x.0/1deg/cice/iced.1900-01-01-10800.nc)

Trying this bash shows what is going on:

$ lfs getstripe input/iced.1900-01-01-10800.nc 
input/iced.1900-01-01-10800.nc has no stripe info

when the expected result is

$ lfs getstripe /g/data/ik11/inputs/access-om3/0.x.0/1deg/cice/iced.1900-01-01-10800.nc
/g/data/ik11/inputs/access-om3/0.x.0/1deg/cice/iced.1900-01-01-10800.nc
lmm_stripe_count:  1
lmm_stripe_size:   1048576
lmm_pattern:       raid0
lmm_layout_gen:    0
lmm_stripe_offset: 6
	obdidx		 objid		 objid		 group
	     6	     138480481	    0x8410b61	   0x3c0000400

I have always thought of symlinks as pretty robust!

Options here are possibly:

Update payu to point directly to the file instead of symlinking (would this need a copy of the restart files in the work directory?)
Raise the issue with the developers of the ParrallelIO library
Something else ?

I haven’t yet thought through how updating payu would work. We wouldn’t want to have to make a copy of every restart file, every time a model component run by payu is initialised.

I have focussed on testing this with CICE & NUOPC, but every model component run by Payu will have the same issue. To update to Netcdf4 and parallel reads for any other component (I tested with the data-atmosphere but all would be impacted) some change will need to be made.

Aidan · 21 November 2023 06:06

I agree this is a decent place to talk about it, as it involves a lot more than just payu.

So why does PIO in CICE5 work with ACCESS-OM2 with symlinks? Is nuopc doing some other step that involves interrogating the striping?

This would constitute a pretty large change in the logic of payu and would be an option of last resort.

I’d plump for figuring out if you can turn off this ifs interrogation step. My recollection from Nic Hannah’s testing was that he didn’t get much IO improvement when he changed the default PIO configuration.

@rui.yang is the one who knows about optimising Lustre striping though.

anton · 22 November 2023 03:45

Yeah this is curious. Ill keep investigating.

The big change was probably from switching to Parallel IO, the configuration of it is probably less important as long as it is reasonable (i.e. 12 vs 24 threads for IO might not change much in the final result due to the limitations in disk access).

Similarly, I expect the default configuration for striping will be fine at our file sizes. It just needs to work.

Aidan · 22 November 2023 03:56

To address something you mentioned in your original post, IIRC there was no discernible difference in performance for 1 deg and even 0.25 deg wasn’t much. It was the tenth that it really made a difference, but @aekiss might recall, or have some actual numbers.

There is some useful stuff in the TWG minutes, search for PIO and start sifting for nuggets of wisdom:

https://cosima.org.au/index.php/category/minutes/

(Feeling vindication for the time it took me to write those damn things)

Aidan · 22 November 2023 10:48

I am so full of it. cice5 already does this because the driver wanted to modify restarts (and inputs I guess) in-place

github.com

payu-org/payu/blob/master/payu/models/cice5.py#L42


      
                  'cice_in.nml',
                  'input_ice.nml',
                  'input_ice_gfdl.nml',
                  'input_ice_monin.nml'
              ]
          
              self.ice_nml_fname = 'cice_in.nml'
          
              self.set_timestep = self.set_local_timestep
          
              self.copy_restarts = True
              self.copy_inputs = True
          
          def set_local_timestep(self, t_step):
              dt = self.ice_in['setup_nml']['dt']
              npt = self.ice_in['setup_nml']['npt']
          
              self.ice_in['setup_nml']['dt'] = t_step
              self.ice_in['setup_nml']['npt'] = (int(dt) * int(npt)) // int(t_step)
          
              ice_in_path = os.path.join(self.work_path, self.ice_nml_fname)

So if you needed a work-around it might be your ticket. Would be good not to need it though.

I should really have remembered this, I wrote it. I sort of did, but thought it didn’t apply, so apologies if this has held you up.

(Just want to say, how nice are those one-box code snippet previews. Nice one discourse!)

anton · 26 November 2023 22:45

Ok, great! I was starting to get mighty confused about where the difference from OM2 to OM3 was.

Do I need to build my own payu to to test that? (Are there instructions for building payu?)

anton · 28 November 2023 05:45

This step is buried deep in in the open-mpi library:

github.com

open-mpi/ompi/blob/8ddc99764eadc23c58c2c8a8bdfde4cb6930f91a/ompi/mca/fs/lustre/fs_lustre_file_open.c#L157


      
              if ( 0 > fh->fd) {
                  return mca_fs_base_get_mpi_err(errno);
              }
          }
          
          lump = alloc_lum();
          if (NULL == lump) {
              fprintf(stderr,"Cannot allocate memory for extracting stripe size\n");
              return OMPI_ERROR;
          }
          rc = llapi_file_get_stripe(filename, lump);
          if (rc != 0) {
              opal_output(1, "get_stripe failed: %d (%s)\n", errno, strerror(errno));
              free(lump);
              return OMPI_ERROR;
          }
          fh->f_stripe_size   = lump->lmm_stripe_size;
          fh->f_stripe_count  = lump->lmm_stripe_count;
          fh->f_fs_block_size = lump->lmm_stripe_size;
          fh->f_flags |= OMPIO_LOCK_NEVER;
          free(lump);

Which makes it kind of challenging to do anything about. Fortran also doesn’t have a good way to just read the path it’s trying to point the symlink to.

We could:

Update the rpointer files in Payu to use full paths for the restart and input files (instead of using the symlinks).
Copy the restart files (as above) in Payu
Patch CICE to read using serial input and only write using parallel output. (IO is controlled in nuopc.run config, so it’s messy). This would mean all model components would have to use serial input and have a similar patch applied if we wanted parallel output (I don’t know if that would be slow, the datm files are a lot bigger than the CICE ones but still less than 2GB each).
Possibly raise an issue with open-mpi, and make the case the behaviour with Lustre is wrong and they should be checking for symlinks when opening files.

Aidan · 28 November 2023 09:11

I don’t think there are. I usually load a vanilla conda environment (one without payu):

module use /g/data/hh5/public/modules
module load conda/python3

and then

pip install -e . --user

in the payu source directory.

That installs into ~/.local/bin/payu.

Wow. Good find.

That would break the manifests as they’re currently implemented.

That’s a decent work-around for the time being. Does require a patched payu however.

I think this is worth doing regardless.