Hi all.
I’d like to use some profilers on NCI, and in particular measure write speed for specific UM output files.
Has anyone here used any profilers for the UM?
The NCI Opus pages suggest Linaro DDT can be used to run a UM task.
NCI have also suggested Darsha, although they note there is some incompatibility with that and Open MPI, so you can’t use your own romio files.
Just asking if there is any experience using these before I dive in.
ping @manodeep
Hi Paul,
NCI has 256 core licenses for linaro-forge ddt and map tools - would that be sufficient for your needs? (Performance-report goes up to 2048 cores but only provides aggregate data, and not source-level info.)
I have used linaro-forge map for profiling ESM1.6 - fairly straightforward to setup through payu. Happy to assist if you would like.
Cheers!
Manodeep
Hi @manodeep
Thanks for the quick reply. I have run Linaro-forget ddt on an ACCESS-OM3 executable w/debug symbols (although that didn’t work).
I’d like to run the map on an ACCESS-AM3 suite, using a spack built executable. Will this run ‘out of the box’ or does it need recompiling/re-linking?
At this stage I only want to track I/O performance.
If you want to inspect the associated source, then you will (likely) need to build with debug symbols. Other than that, map should run out of the box.
If you are running through payu, you will need to add these lines in the config.yaml
map exe - linaro-forge/24.0.2 (under the module load section)
map exe in front of the actual mpirun command,mpi:
runcmd: map --profile --report=txt,html,summary mpirun
Once your run finishes, you can examine the generated profile (file extension .map) with the same map command, but without the --profile flag - i.e., map <path/to/mapfilename.map>. The mapfilename itself contains the executable name, the number of PBS nodes, OMP threads and the date-time string - so should be unique everytime you run the profiler.
Hope this helps!
Cheers,
Manodeep
Thanks.
The UM Atmospheric task is running in rose-cylc, so I’ll follow the guidelines suggested here:
If that doesn’t work, I’ll just build my own job submission script.
I’ll keep you posted.
On using Linaro DDT on the ACCESS models- I’m currently working on docs explaining how to do this, but the abridged version that worked for me is:
fflags='-O0 -g -traceback cflags='-O0 -g -fno-omit-frame-pointer (I know those fflags work, haven’t actually tried the cflags yet so @Manodeep may correct me?) to the model specs.spack concretize -f then spack install --keep-stage- the keep-stage is necessary to prevent Spack from cleaning up the source code used to compile the executable.At this point, you should be able to use the NCI instructions for using Linaro DDT.
@lachlanswhyborn I would recommend using the flags that you are planning to use for production runs - otherwise, the generated instructions can be dramatically different and the insights from profiling (with an un-optimised build) may not be applicable to your production exe.
The instructions look spot-on - you certainly would want to use --keep-stage to keep the pre-processed source files and point the profiler to the relevant source directory. It is good to add the -fno-omit-frame-pointer - I have not noticed any performance hits but seems to improve symbol resolution and pinpointing runtime exes into the source line of code.
This is for debugging rather than profiling- for profiling, yes I’d certainly use the same flags as used in production. I tried debugging with optimisation flags on and the debugger often seemed to get confused as to where the program was up to, relative to the source code.
Just an update. I have the DDT debugger and connected to a UM task. In this case, an ACCESS-AM3 suite.
At NCI’s suggestion, I have downloaded a Linaro Forge client which I run from my local laptop (Mac in my case).
When you launch the Linaro client, activate the “Remote Launch” Configure pull down menu and create a session for gadi. I specified the following:
connection name : gadi
Host Name : <user-id>@gadi.nci.org.au
Remote Installation Directory : /apps/linaro-forge/24.0.2/
You can then click ‘Test Remove Launch’ to check this works correctly.
Then activate your ‘gadi’ Remote Launch, and you will have a pop-up menu which states something like
”A new Reverse Connect request is available from gadi-hmem-clx-XXXX.gadi.nci.org.au for Linaro DDT.”
Click accept, and then run.
Now you’re debugging the UM inside your rose-cycle suite!
@manodeep - following on from our discussion at RSE meeting on Oct 24, the default compile flags used within the ACCESS-AM3 suite are provided in the file
/scratch/$PROJECT/$USER/UM/fcm-make/nci-x86-ifort/um-atmos-safe.cfg
include-path = $HERE/inc $HERE/../inc
$extract{?} = extract
$fcflags_level = -O2 -fp-model precise
include = um-atmos-common.cfg
Which sets the fcflags_level environment variable the in app/fcm_make_um/rose-suite.conf namelist.
The full compilation flags from fcm-make-on-success.cfg are
build.prop{class, fc.flags} = -i8 -r8 -mcmodel=medium -std08 -g -traceback -assume nosource_include -O2 -fp-model precise -qopenmp
When building an executable for debugging, I use the `debug’ config file:
/scratch/$PROJECT/$USER/UM/fcm-make/nci-x86-ifort/um-atmos-debug.cfg
include-path = $HERE/inc $HERE/../inc
$extract{?} = extract
$fcflags_level = -O0 -fp-model precise -traceback -fpe0
include = um-atmos-common.cfg
The full compilation flags then become:
build.prop{class, fc.flags} = -i8 -r8 -mcmodel=medium -std08 -g -traceback -assume nosource_include -O0 -fp-model precise -traceback -qopenmp
What were the other flags you suggested?
@Paul.Gregory Are you using the oneAPI or classic Intel compiler?
For the debugging setup, I would also add -fno-omit-frame-pointer.
@Paul.Gregory Just came by this topic again and realised I hadn’t quite answered your compiler flags question. Regardless of which compiler (classic/oneAPI) suite you use, you would want to include a architecture flag. For example, if you are building with spack and running on gadi, you would want to set a generic target = x86_64_v4 for an exe that would run on both normal and normalsr queues. If you want to be specifically using the normalsr queue, then you can set the target to sapphirerapids. Additionally, in your spack.yaml file, you would also need to set the concretizer target granularity to microarchitectures, i.e., concretizer->targets->granularity: microarchitectures (instead of generic)
There are quite a few compiler flags that could be performance boosting but you would need to test those. Some such flags for the oneAPI Fortran compiler could be:
-qopt-zmm-usage=high (which forces using the 512bit vector registers),-vec-threshold0 (which is related to the zmm flag above, and forces the compiler to vectorise every vectorisable loop, regardless of the compute cost)-unroll (unrolls the loops using compiler heuristics),-flto (for link-time optimisation, needs all compile-time flags to be repeated at link-time and -fuse-ld=lld specifically for link-time flags. Note, this will reduce compile times but can dramatically increase The first three options might change answers, the last two should not (but may). If you have many expensive math library calls, then using Intel MKL could also be another option to try.
There’s also profile-generate and profile-use, but that’s probably when the code is in a stable state.
Hope this helps!
Manodeep
Thanks Manodeep for those suggestions. I’ve built a rAM3 executable with
fcflags_overrides=-march=sapphirerapids -qopt-zmm-usage=high
as detailed here Building the UM to run on Gadi's Sapphire nodes - #2 by Paul.Gregory , and that works well.
I’ll try some of your other suggestions too.
Thanks Paul!
Might be worth trying -xsapphirerapids instead of -march=sapphirerapids(noting that that exe may not run on the normal queue, or any AMD cpus). The -x<arch> flag seems to enable some extra Intel-specific optimisations and I have seen few % performance boosts relative to the -march=<arch> flag.