Payu generated symlinks don't work with ParallelIO library

anton · 21 November 2023 03:32

I am posting here because I don’t know where it fits better!

I am trying to configure CICE in OM3 to use Parallel IO (which supports Netcdf 4, rather than the current Netcdf Classic - Parallel IO in CICE · Issue #81 · COSIMA/access-om3 · GitHub). This is configurable in Nuopc - by changing nuopc.runconfig here.

When an attempt to parallel read a restart file is made, we get this error:

get_stripe failed: 61 (No data available)
Abort with message NetCDF: Error initializing for parallel access in file /jobfs/98914803.gadi-pbs/mo1833/spack-stage/spack-stage-parallelio-2.5.10-hyj75i7d5yy5zbqc7jm6whlkduofib2k/spack-src/src/clib/pioc_support.c at line 2832

This get_stripe failed is because payu provides a symlink to the restart file rather than the file. The payu ‘work’ directiory has an ‘input’ folder with symlinks ( e.g. input/iced.1900-01-01-10800.nc -> /g/data/ik11/inputs/access-om3/0.x.0/1deg/cice/iced.1900-01-01-10800.nc)

Trying this bash shows what is going on:

$ lfs getstripe input/iced.1900-01-01-10800.nc 
input/iced.1900-01-01-10800.nc has no stripe info

when the expected result is

$ lfs getstripe /g/data/ik11/inputs/access-om3/0.x.0/1deg/cice/iced.1900-01-01-10800.nc
/g/data/ik11/inputs/access-om3/0.x.0/1deg/cice/iced.1900-01-01-10800.nc
lmm_stripe_count:  1
lmm_stripe_size:   1048576
lmm_pattern:       raid0
lmm_layout_gen:    0
lmm_stripe_offset: 6
	obdidx		 objid		 objid		 group
	     6	     138480481	    0x8410b61	   0x3c0000400

I have always thought of symlinks as pretty robust!

Options here are possibly:

Update payu to point directly to the file instead of symlinking (would this need a copy of the restart files in the work directory?)
Raise the issue with the developers of the ParrallelIO library
Something else ?

I haven’t yet thought through how updating payu would work. We wouldn’t want to have to make a copy of every restart file, every time a model component run by payu is initialised.

I have focussed on testing this with CICE & NUOPC, but every model component run by Payu will have the same issue. To update to Netcdf4 and parallel reads for any other component (I tested with the data-atmosphere but all would be impacted) some change will need to be made.

Aidan · 21 November 2023 06:06

I agree this is a decent place to talk about it, as it involves a lot more than just payu.

So why does PIO in CICE5 work with ACCESS-OM2 with symlinks? Is nuopc doing some other step that involves interrogating the striping?

This would constitute a pretty large change in the logic of payu and would be an option of last resort.

I’d plump for figuring out if you can turn off this ifs interrogation step. My recollection from Nic Hannah’s testing was that he didn’t get much IO improvement when he changed the default PIO configuration.

@rui.yang is the one who knows about optimising Lustre striping though.

anton · 22 November 2023 03:45

Yeah this is curious. Ill keep investigating.

The big change was probably from switching to Parallel IO, the configuration of it is probably less important as long as it is reasonable (i.e. 12 vs 24 threads for IO might not change much in the final result due to the limitations in disk access).

Similarly, I expect the default configuration for striping will be fine at our file sizes. It just needs to work.

Aidan · 22 November 2023 03:56

To address something you mentioned in your original post, IIRC there was no discernible difference in performance for 1 deg and even 0.25 deg wasn’t much. It was the tenth that it really made a difference, but @aekiss might recall, or have some actual numbers.

There is some useful stuff in the TWG minutes, search for PIO and start sifting for nuggets of wisdom:

https://cosima.org.au/index.php/category/minutes/

(Feeling vindication for the time it took me to write those damn things)

Aidan · 22 November 2023 10:48

I am so full of it. cice5 already does this because the driver wanted to modify restarts (and inputs I guess) in-place

github.com

payu-org/payu/blob/master/payu/models/cice5.py#L42


      
                  'cice_in.nml',
                  'input_ice.nml',
                  'input_ice_gfdl.nml',
                  'input_ice_monin.nml'
              ]
          
              self.ice_nml_fname = 'cice_in.nml'
          
              self.set_timestep = self.set_local_timestep
          
              self.copy_restarts = True
              self.copy_inputs = True
          
          def set_local_timestep(self, t_step):
              dt = self.ice_in['setup_nml']['dt']
              npt = self.ice_in['setup_nml']['npt']
          
              self.ice_in['setup_nml']['dt'] = t_step
              self.ice_in['setup_nml']['npt'] = (int(dt) * int(npt)) // int(t_step)
          
              ice_in_path = os.path.join(self.work_path, self.ice_nml_fname)

So if you needed a work-around it might be your ticket. Would be good not to need it though.

I should really have remembered this, I wrote it. I sort of did, but thought it didn’t apply, so apologies if this has held you up.

(Just want to say, how nice are those one-box code snippet previews. Nice one discourse!)

anton · 26 November 2023 22:45

Ok, great! I was starting to get mighty confused about where the difference from OM2 to OM3 was.

Do I need to build my own payu to to test that? (Are there instructions for building payu?)

anton · 28 November 2023 05:45

This step is buried deep in in the open-mpi library:

github.com

open-mpi/ompi/blob/8ddc99764eadc23c58c2c8a8bdfde4cb6930f91a/ompi/mca/fs/lustre/fs_lustre_file_open.c#L157


      
              if ( 0 > fh->fd) {
                  return mca_fs_base_get_mpi_err(errno);
              }
          }
          
          lump = alloc_lum();
          if (NULL == lump) {
              fprintf(stderr,"Cannot allocate memory for extracting stripe size\n");
              return OMPI_ERROR;
          }
          rc = llapi_file_get_stripe(filename, lump);
          if (rc != 0) {
              opal_output(1, "get_stripe failed: %d (%s)\n", errno, strerror(errno));
              free(lump);
              return OMPI_ERROR;
          }
          fh->f_stripe_size   = lump->lmm_stripe_size;
          fh->f_stripe_count  = lump->lmm_stripe_count;
          fh->f_fs_block_size = lump->lmm_stripe_size;
          fh->f_flags |= OMPIO_LOCK_NEVER;
          free(lump);

Which makes it kind of challenging to do anything about. Fortran also doesn’t have a good way to just read the path it’s trying to point the symlink to.

We could:

Update the rpointer files in Payu to use full paths for the restart and input files (instead of using the symlinks).
Copy the restart files (as above) in Payu
Patch CICE to read using serial input and only write using parallel output. (IO is controlled in nuopc.run config, so it’s messy). This would mean all model components would have to use serial input and have a similar patch applied if we wanted parallel output (I don’t know if that would be slow, the datm files are a lot bigger than the CICE ones but still less than 2GB each).
Possibly raise an issue with open-mpi, and make the case the behaviour with Lustre is wrong and they should be checking for symlinks when opening files.

Aidan · 28 November 2023 09:11

I don’t think there are. I usually load a vanilla conda environment (one without payu):

module use /g/data/hh5/public/modules
module load conda/python3

and then

pip install -e . --user

in the payu source directory.

That installs into ~/.local/bin/payu.

Wow. Good find.

That would break the manifests as they’re currently implemented.

That’s a decent work-around for the time being. Does require a patched payu however.

I think this is worth doing regardless.

anton · 1 December 2023 01:08

I raised it here:

github.com/open-mpi/ompi

mpi_file_open fails on symlink with Lustre filesystem

opened 01:07AM - 01 Dec 23 UTC

anton-seaice

Thank you for taking the time to submit an issue! ## Background information … ### What version of Open MPI are you using? (e.g., v3.0.5, v4.0.2, git branch name and hash, etc.) v4.1.4 ### Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.) This is provided on this platform. I assume its through a spack distribution ### Please describe the system on which you are running * Operating system/version: Rocky Linux 8.8 (Green Obsidian) * Computer hardware: Intel Xeon Scalable ‘Cascade Lake’ * Network type: ? (see https://nci.org.au/our-systems/hpc-systems) ----------------------------- ## Details of the problem When I attempt to open a file through its symlink, it fails with the error : ``` get_stripe failed: 61 (No data available) MPI_ERR_OTHER: known error not in list ``` The expected result is that the symlink to the file opens the same way the file does. This is my minimal test example: ``` program mpi_file_open_example implicit none include 'mpif.h' integer, parameter :: nprocs = 4 integer :: rank, ierr, file_handle, status, len character(len=MPI_MAX_ERROR_STRING) :: error_string character(len=80) :: filename = "parallel_file.txt", link = "link_to_parallel_file.txt" ! Confirm the file can be opened using open open(newunit=file_handle, file=link, status="old", action="read", & iostat=ierr, iomsg=error_string) if (ierr /= 0) then print *, trim(error_string) call abort() end if close(file_handle) ! Initialize MPI call MPI_Init(ierr) call MPI_Comm_rank(MPI_COMM_WORLD, rank, ierr) ! Open the file in parallel write(*, *) "Open the file in parallel: " call MPI_File_open(MPI_COMM_WORLD, filename, & MPI_MODE_RDONLY, MPI_INFO_NULL, file_handle, ierr) if (ierr /= MPI_SUCCESS) then call MPI_Error_string(ierr, error_string, len, ierr) write(*, *) "Error opening ", trim(filename), " : ", trim(error_string) else write(*, *) "Success opening ", trim(filename) end if ! Close the file call MPI_File_close(file_handle, ierr) write(*, *) "===================" ! Open the symlink in parallel write(*, *) "Open the symlink in parallel: " call MPI_File_open(MPI_COMM_WORLD, link, & MPI_MODE_RDONLY, MPI_INFO_NULL, file_handle, ierr) if (ierr /= MPI_SUCCESS) then call MPI_Error_string(ierr, error_string, len, ierr) write(*, *) "Error opening ", trim(link), " : ", trim(error_string) else write(*, *) "Success opening ", trim(link) end if ! Close the file call MPI_File_close(file_handle, ierr) ! Finalize MPI call MPI_Finalize(ierr) end program mpi_file_open_example ``` to reproduce: ``` module purge module load intel-compiler/2021.6.0 openmpi/4.1.4 mpifort mpi_file_open.f90 -o mpi_file_open.exe touch parallel_file.txt ln -s parallel_file.txt link_to_parallel_file.txt ./mpi_file_open.exe ```

Aidan · 1 December 2023 04:26

Just for general information, the OpenMPI libraries used, even in spack builds, are the ones provided by NCI.

You can get more (A LOT MORE) information about the mpi installation with ompi_info in case that is required:

$ ompi_info                        
                 Package: Open MPI apps@gadi-cpu-clx-2915.gadi.nci.org.au    
                          Distribution                                  
                Open MPI: 4.1.4                                          
  Open MPI repo revision: v4.1.4                                          
   Open MPI release date: May 26, 2022                                   
                Open RTE: 4.1.4                                         
  Open RTE repo revision: v4.1.4                                           
   Open RTE release date: May 26, 2022                                      
                    OPAL: 4.1.4                                              
      OPAL repo revision: v4.1.4                                          
       OPAL release date: May 26, 2022                                     
                 MPI API: 3.1.0                                              
            Ident string: 4.1.4                                            
                  Prefix: /apps/openmpi-mofed5.6-pbs2021.1/4.1.4          
 Configured architecture: x86_64-pc-linux-gnu                             
          Configure host: gadi-cpu-clx-2915.gadi.nci.org.au               
           Configured by: apps                                             
           Configured on: Mon Aug  1 04:47:24 UTC 2022                       
          Configure host: gadi-cpu-clx-2915.gadi.nci.org.au                 
  Configure command line: '--prefix=/apps/openmpi-mofed5.6-pbs2021.1/4.1.4' 
                          '--disable-dependency-tracking'                 
                          '--disable-heterogeneous' '--disable-ipv6'     
                          '--enable-orterun-prefix-by-default'            
                          '--enable-sparse-groups' '--enable-mpi-fortran'
                          '--enable-mpi-cxx' '--enable-mpi1-compatibility'
                          '--enable-shared' '--disable-static'          
                          '--disable-wrapper-rpath'                        
                          '--disable-wrapper-runpath' '--disable-mpi-java' 
                          '--enable-mca-static' '--enable-hwloc-pci'    
                          '--enable-visibility' '--with-zlib'             
                          '--with-cuda=/apps/cuda/11.7.0' '--without-pmi'    
                          '--with-ucx=/apps/ucx/1.13.0' '--without-verbs'
                          '--without-verbs-usnic' '--without-portals4'  
                          '--without-ugni' '--without-usnic' '--without-ofi' 
                          '--without-cray-xpmem' '--with-xpmem'            
                          '--with-knem=/opt/knem-1.1.4.90mlnx1' '--with-cma'
                          '--without-x' '--without-memkind'              
                          '--without-cray-pmi' '--without-alps'         
                          '--without-flux-pmi' '--without-udreg'        
                          '--without-lsf' '--without-slurm'                
                          '--with-tm=/opt/pbs/default' '--without-sge'   
                          '--without-moab' '--without-singularity'        
                          '--without-fca' '--with-hcoll=/apps/hcoll/4.7.3208'
                          '--with-ucc=/apps/ucc/1.0.0' '--without-ime'      
                          '--without-pvfs2' '--with-lustre'                
                          '--with-io-romio-flags=--with-file-system=lustre+ufs'
                          '--without-psm' '--without-psm2' '--without-mxm'   
                          '--disable-mem-debug' '--disable-mem-profile' 
                          '--disable-picky' '--disable-debug'          
                          '--disable-timing' '--disable-event-debug'  
                          '--disable-memchecker' '--disable-pmix-timing'
                          '--with-mpi-param-check=runtime'             
                          '--with-oshmem-param-check=never'             
                          '--without-valgrind'                                                                                                                                        [144/625]
                Built by: apps
                Built on: Mon Aug  1 05:04:18 UTC 2022
              Built host: gadi-cpu-clx-2915.gadi.nci.org.au
              C bindings: yes
            C++ bindings: yes
             Fort mpif.h: yes (all)
            Fort use mpi: yes (full: ignore TKR)
       Fort use mpi size: deprecated-ompi-info-value
        Fort use mpi_f08: yes
 Fort mpi_f08 compliance: The mpi_f08 module is available, but due to
                          limitations in the gfortran -march=broadwell
                          compiler and/or Open MPI, does not support the
                          following: array subsections, direct passthru
                          (where possible) to underlying Open MPI's C
                          functionality
  Fort mpi_f08 subarrays: no
           Java bindings: no
  Wrapper compiler rpath: disabled
              C compiler: gcc -march=broadwell
     C compiler absolute: /opt/nci/bin/gcc
  C compiler family name: GNU
      C compiler version: 8.5.0
            C++ compiler: g++ -march=broadwell
   C++ compiler absolute: /opt/nci/bin/g++
           Fort compiler: gfortran -march=broadwell
       Fort compiler abs: /opt/nci/bin/gfortran
         Fort ignore TKR: yes (!GCC$ ATTRIBUTES NO_ARG_CHECK ::)
   Fort 08 assumed shape: yes
      Fort optional args: yes
          Fort INTERFACE: yes
    Fort ISO_FORTRAN_ENV: yes
       Fort STORAGE_SIZE: yes
      Fort BIND(C) (all): yes
      Fort ISO_C_BINDING: yes
 Fort SUBROUTINE BIND(C): yes
       Fort TYPE,BIND(C): yes
 Fort T,BIND(C,name="a"): yes
            Fort PRIVATE: yes
          Fort PROTECTED: yes
           Fort ABSTRACT: yes
       Fort ASYNCHRONOUS: yes
          Fort PROCEDURE: yes
         Fort USE...ONLY: yes
           Fort C_FUNLOC: yes
 Fort f08 using wrappers: yes
         Fort MPI_SIZEOF: yes
             C profiling: yes
           C++ profiling: yes
   Fort mpif.h profiling: yes
  Fort use mpi profiling: yes
   Fort use mpi_f08 prof: yes
          C++ exceptions: no
          Thread support: posix (MPI_THREAD_MULTIPLE: yes, OPAL support: yes,
                          OMPI progress: no, ORTE progress: yes, Event lib:
                          yes)
           Sparse Groups: yes
  Internal debug support: no
  MPI interface warnings: yes
     MPI parameter check: runtime
Memory profiling support: no
Memory debugging support: no
              dl support: yes
   Heterogeneous support: no
 mpirun default --prefix: yes
       MPI_WTIME support: native
     Symbol vis. support: yes
   Host topology support: yes
            IPv6 support: no
      MPI1 compatibility: yes
          MPI extensions: affinity, cuda, pcollreq
   FT Checkpoint support: no (checkpoint thread: no)
   C/R Enabled Debugging: no
  MPI_MAX_PROCESSOR_NAME: 256
    MPI_MAX_ERROR_STRING: 256
     MPI_MAX_OBJECT_NAME: 64                                                                                                                                                                           
        MPI_MAX_INFO_KEY: 36
        MPI_MAX_INFO_VAL: 256                                                                                                                                                                          
       MPI_MAX_PORT_NAME: 1024
  MPI_MAX_DATAREP_STRING: 128
           MCA allocator: basic (MCA v2.1.0, API v2.0.0, Component v4.1.4)
           MCA allocator: bucket (MCA v2.1.0, API v2.0.0, Component v4.1.4)
           MCA backtrace: execinfo (MCA v2.1.0, API v2.0.0, Component v4.1.4)
                 MCA btl: self (MCA v2.1.0, API v3.1.0, Component v4.1.4)
                 MCA btl: smcuda (MCA v2.1.0, API v3.1.0, Component v4.1.4)
                 MCA btl: tcp (MCA v2.1.0, API v3.1.0, Component v4.1.4)
                 MCA btl: vader (MCA v2.1.0, API v3.1.0, Component v4.1.4)
            MCA compress: bzip (MCA v2.1.0, API v2.0.0, Component v4.1.4)
            MCA compress: gzip (MCA v2.1.0, API v2.0.0, Component v4.1.4)
                 MCA crs: none (MCA v2.1.0, API v2.0.0, Component v4.1.4)
                  MCA dl: dlopen (MCA v2.1.0, API v1.0.0, Component v4.1.4)
               MCA event: libevent2022 (MCA v2.1.0, API v2.0.0, Component
                          v4.1.4)
               MCA hwloc: external (MCA v2.1.0, API v2.0.0, Component v4.1.4)
                  MCA if: linux_ipv6 (MCA v2.1.0, API v2.0.0, Component
                          v4.1.4)
                  MCA if: posix_ipv4 (MCA v2.1.0, API v2.0.0, Component
                          v4.1.4)
         MCA installdirs: env (MCA v2.1.0, API v2.0.0, Component v4.1.4)
         MCA installdirs: config (MCA v2.1.0, API v2.0.0, Component v4.1.4)
              MCA memory: patcher (MCA v2.1.0, API v2.0.0, Component v4.1.4)
               MCA mpool: hugepage (MCA v2.1.0, API v3.0.0, Component v4.1.4)
             MCA patcher: overwrite (MCA v2.1.0, API v1.0.0, Component
                          v4.1.4)
                MCA pmix: isolated (MCA v2.1.0, API v2.0.0, Component v4.1.4)
                MCA pmix: pmix3x (MCA v2.1.0, API v2.0.0, Component v4.1.4)
               MCA pstat: linux (MCA v2.1.0, API v2.0.0, Component v4.1.4)
              MCA rcache: grdma (MCA v2.1.0, API v3.3.0, Component v4.1.4)
              MCA rcache: gpusm (MCA v2.1.0, API v3.3.0, Component v4.1.4)
              MCA rcache: rgpusm (MCA v2.1.0, API v3.3.0, Component v4.1.4)
           MCA reachable: weighted (MCA v2.1.0, API v2.0.0, Component v4.1.4)
           MCA reachable: netlink (MCA v2.1.0, API v2.0.0, Component v4.1.4)
               MCA shmem: mmap (MCA v2.1.0, API v2.0.0, Component v4.1.4)
               MCA shmem: posix (MCA v2.1.0, API v2.0.0, Component v4.1.4)
               MCA shmem: sysv (MCA v2.1.0, API v2.0.0, Component v4.1.4)
               MCA timer: linux (MCA v2.1.0, API v2.0.0, Component v4.1.4)
              MCA errmgr: default_app (MCA v2.1.0, API v3.0.0, Component
                          v4.1.4)
              MCA errmgr: default_hnp (MCA v2.1.0, API v3.0.0, Component
                          v4.1.4)
              MCA errmgr: default_orted (MCA v2.1.0, API v3.0.0, Component
                          v4.1.4)
              MCA errmgr: default_tool (MCA v2.1.0, API v3.0.0, Component
                          v4.1.4)
                 MCA ess: env (MCA v2.1.0, API v3.0.0, Component v4.1.4)
                 MCA ess: hnp (MCA v2.1.0, API v3.0.0, Component v4.1.4)
                 MCA ess: pmi (MCA v2.1.0, API v3.0.0, Component v4.1.4)
                 MCA ess: singleton (MCA v2.1.0, API v3.0.0, Component
                          v4.1.4)
                 MCA ess: tool (MCA v2.1.0, API v3.0.0, Component v4.1.4)
                 MCA ess: tm (MCA v2.1.0, API v3.0.0, Component v4.1.4)
               MCA filem: raw (MCA v2.1.0, API v2.0.0, Component v4.1.4)
             MCA grpcomm: direct (MCA v2.1.0, API v3.0.0, Component v4.1.4)
                 MCA iof: hnp (MCA v2.1.0, API v2.0.0, Component v4.1.4)
                 MCA iof: orted (MCA v2.1.0, API v2.0.0, Component v4.1.4)
                 MCA iof: tool (MCA v2.1.0, API v2.0.0, Component v4.1.4)
                MCA odls: default (MCA v2.1.0, API v2.0.0, Component v4.1.4)
                MCA odls: pspawn (MCA v2.1.0, API v2.0.0, Component v4.1.4)
                 MCA oob: tcp (MCA v2.1.0, API v2.0.0, Component v4.1.4)
                 MCA plm: isolated (MCA v2.1.0, API v2.0.0, Component v4.1.4)
                 MCA plm: rsh (MCA v2.1.0, API v2.0.0, Component v4.1.4)
                 MCA plm: tm (MCA v2.1.0, API v2.0.0, Component v4.1.4)
                 MCA ras: simulator (MCA v2.1.0, API v2.0.0, Component
                          v4.1.4)
                 MCA ras: tm (MCA v2.1.0, API v2.0.0, Component v4.1.4)
                MCA regx: fwd (MCA v2.1.0, API v1.0.0, Component v4.1.4)
                MCA regx: naive (MCA v2.1.0, API v1.0.0, Component v4.1.4)
                MCA regx: reverse (MCA v2.1.0, API v1.0.0, Component v4.1.4)
               MCA rmaps: mindist (MCA v2.1.0, API v2.0.0, Component v4.1.4)
               MCA rmaps: ppr (MCA v2.1.0, API v2.0.0, Component v4.1.4)
               MCA rmaps: rank_file (MCA v2.1.0, API v2.0.0, Component
                          v4.1.4)
               MCA rmaps: resilient (MCA v2.1.0, API v2.0.0, Component
                          v4.1.4)
               MCA rmaps: round_robin (MCA v2.1.0, API v2.0.0, Component
                          v4.1.4)
               MCA rmaps: seq (MCA v2.1.0, API v2.0.0, Component v4.1.4)
                 MCA rml: oob (MCA v2.1.0, API v3.0.0, Component v4.1.4)
              MCA routed: binomial (MCA v2.1.0, API v3.0.0, Component v4.1.4)
              MCA routed: direct (MCA v2.1.0, API v3.0.0, Component v4.1.4)
              MCA routed: radix (MCA v2.1.0, API v3.0.0, Component v4.1.4)
                 MCA rtc: hwloc (MCA v2.1.0, API v1.0.0, Component v4.1.4)
              MCA schizo: flux (MCA v2.1.0, API v1.0.0, Component v4.1.4)
              MCA schizo: ompi (MCA v2.1.0, API v1.0.0, Component v4.1.4)
              MCA schizo: orte (MCA v2.1.0, API v1.0.0, Component v4.1.4)
              MCA schizo: jsm (MCA v2.1.0, API v1.0.0, Component v4.1.4)
               MCA state: app (MCA v2.1.0, API v1.0.0, Component v4.1.4)
               MCA state: hnp (MCA v2.1.0, API v1.0.0, Component v4.1.4)
               MCA state: novm (MCA v2.1.0, API v1.0.0, Component v4.1.4)
               MCA state: orted (MCA v2.1.0, API v1.0.0, Component v4.1.4)
               MCA state: tool (MCA v2.1.0, API v1.0.0, Component v4.1.4)
                 MCA bml: r2 (MCA v2.1.0, API v2.0.0, Component v4.1.4)
                MCA coll: adapt (MCA v2.1.0, API v2.0.0, Component v4.1.4)
                MCA coll: basic (MCA v2.1.0, API v2.0.0, Component v4.1.4)
                MCA coll: han (MCA v2.1.0, API v2.0.0, Component v4.1.4)
                MCA coll: inter (MCA v2.1.0, API v2.0.0, Component v4.1.4)
                MCA coll: libnbc (MCA v2.1.0, API v2.0.0, Component v4.1.4)
                MCA coll: self (MCA v2.1.0, API v2.0.0, Component v4.1.4)
                MCA coll: sm (MCA v2.1.0, API v2.0.0, Component v4.1.4)
                MCA coll: sync (MCA v2.1.0, API v2.0.0, Component v4.1.4)
                MCA coll: tuned (MCA v2.1.0, API v2.0.0, Component v4.1.4)
                MCA coll: cuda (MCA v2.1.0, API v2.0.0, Component v4.1.4)
                MCA coll: hcoll (MCA v2.1.0, API v2.0.0, Component v4.1.4)
                MCA coll: ucc (MCA v2.1.0, API v2.0.0, Component v4.1.4)
                MCA fbtl: posix (MCA v2.1.0, API v2.0.0, Component v4.1.4)
               MCA fcoll: dynamic (MCA v2.1.0, API v2.0.0, Component v4.1.4)
               MCA fcoll: dynamic_gen2 (MCA v2.1.0, API v2.0.0, Component
                          v4.1.4)
               MCA fcoll: individual (MCA v2.1.0, API v2.0.0, Component
                          v4.1.4)
               MCA fcoll: two_phase (MCA v2.1.0, API v2.0.0, Component
                          v4.1.4)
               MCA fcoll: vulcan (MCA v2.1.0, API v2.0.0, Component v4.1.4)
                  MCA fs: lustre (MCA v2.1.0, API v2.0.0, Component v4.1.4)
                  MCA fs: ufs (MCA v2.1.0, API v2.0.0, Component v4.1.4)
                  MCA io: ompio (MCA v2.1.0, API v2.0.0, Component v4.1.4)
                  MCA io: romio321 (MCA v2.1.0, API v2.0.0, Component v4.1.4)
                  MCA op: avx (MCA v2.1.0, API v1.0.0, Component v4.1.4)
                 MCA osc: sm (MCA v2.1.0, API v3.0.0, Component v4.1.4)
                 MCA osc: pt2pt (MCA v2.1.0, API v3.0.0, Component v4.1.4)
                 MCA osc: rdma (MCA v2.1.0, API v3.0.0, Component v4.1.4)
                 MCA osc: ucx (MCA v2.1.0, API v3.0.0, Component v4.1.4)
                 MCA pml: cm (MCA v2.1.0, API v2.0.0, Component v4.1.4)
                 MCA pml: ob1 (MCA v2.1.0, API v2.0.0, Component v4.1.4)
                 MCA pml: ucx (MCA v2.1.0, API v2.0.0, Component v4.1.4)
                 MCA pml: v (MCA v2.1.0, API v2.0.0, Component v4.1.4)
                 MCA rte: orte (MCA v2.1.0, API v2.0.0, Component v4.1.4)
            MCA sharedfp: individual (MCA v2.1.0, API v2.0.0, Component
                          v4.1.4)
            MCA sharedfp: lockedfile (MCA v2.1.0, API v2.0.0, Component
                          v4.1.4)
            MCA sharedfp: sm (MCA v2.1.0, API v2.0.0, Component v4.1.4)
                MCA topo: basic (MCA v2.1.0, API v2.2.0, Component v4.1.4)
                MCA topo: treematch (MCA v2.1.0, API v2.2.0, Component
                          v4.1.4)
           MCA vprotocol: pessimist (MCA v2.1.0, API v2.0.0, Component
                          v4.1.4)

anton · 19 December 2023 00:15

Looks like openmpi will fix this, so ill make as closed

github.com/open-mpi/ompi

fs/lustre: recognize soft-links on file open

open-mpi:main ← edgargabriel:topic/lustre-link-fix

opened 07:05PM - 13 Dec 23 UTC

edgargabriel

+69 -39

the llapi_file_get_stripe() function does not accept as an input file a soft-lin…k. Check therefore whether the filename provided in File_open is a soft-link, retrieve the real file name in case it is, make sure it is also on a Lustre file system, and use the real filename in the llapi_file_get_stripe() function call instead. Fixes issue #12141

Topic		Replies	Views
Ice restart problem with payu/1.1.6 Earth System help , inscope , access-esm	12	125	7 May 2025
Payu issues for ACCESS-ESM1.5 on vk83 Earth System Model help , payu , technical	6	50	22 October 2024
Payu, archives and restarts … Technical payu	5	65	6 September 2024
Payu collation not working in ACCESS-OM2-1deg COSIMA help , cosima , inscope	12	81	28 January 2025
PAYU issues on Leonardo Technical help , payu , inscope	32	215	23 April 2025

Payu generated symlinks don't work with ParallelIO library

Related topics