Hi @cbengel - Thanks! I got it.
A further clarification on this -
How does my u-dk517 suite know about the modified SST path? Do I need to modify anything in the source code before re-running the u-dk517 on those modified SST files?
Thanks,
How does my u-dk517 suite know about the modified SST path? Do I need to modify anything in the source code before re-running the u-dk517 on those modified SST files?
Please refer to my previous suggestions on using grep to find what is happening in your source code. You can search for cm45 to see which path the suite will use.
Thanks @mlipson - I am on it now.
Thanks @mlipson and @cbengel. I found the paths and all.
I am still wondering about one other thing - my tos, ta, and hus perturbations are over the Australasia region only. I mean my perturbed values cover the very outer next of the ACCESS-rAM3 flagship model setup with 4-5 degree of buffer around the edge of the 12-km outer domain.
Should it be okay? Or is it gonna create some instabilities in the model runs?
Hi @Moulik,
It does sound like your region may benefit from being extended but you may need to talk to W21C if you want to remain part of their Flagship collection.
On an aside, apologies @Moulik, @sdahl and @mlipson I can see the issue with using the modified files. The OSTIA_ARCHIVE directory was assumed to be fixed when I put together the u-dk517 suite, so it is defined in the suite.rc file.
Please let me know if you want me to move that into the user-interface.
Oh and @Moulik it might be totally fine. It depends on the perturbations, the weather on the day and the phenomena you are studying. The perturbations may migrate into the inner nest domain during the run and should be handled fine unless there are very strong gradients. You should talk with your supervisor if that happens.
Hi @cbengel - thanks!
Now I got it using grep to find.
If so, I think it would be much easier to understand for the new user like me. However, as mentioned by @mlipson, it’s always useful to do a bit of digging under the hood and I am also learning something new along the way. So, I personally don’t have any strong opinion on that. Cheers,
Let me see how it works, otherwise we can have a meeting alongside my supervisors.
Really appreciated for the helps!
Please let me know if you want me to move that into the user-interface.
Thanks Chermelle, but I don’t see any need to move it to a GUI interface, it is easily accessible in suite.rc, and a good learning experience for those who are undertaking more complex experimentation with the suite.
Hi @cbengel @mlipson - any idea what is this error related to?
`Using the cylc session future.mm6452.if69.ps.gadi.nci.org.au
Loading cylc7/24.03
Loading requirement: mosrs-setup/2.0.1
multiprocessing.pool.RemoteTraceback:
“”"
Traceback (most recent call last):
File “/g/data/vk83/apps/conda/access-ram/2025.03.0/lib/python3.11/multiprocessing/pool.py”, line 125, in worker
result = (True, func(*args, **kwds))
^^^^^^^^^^^^^^^^^^^
File “/home/581/mm6452/cylc-run/u-dk517/bin/nci_ostia_to_pp_parallel.py”, line 75, in create_grib
var_dim.setncattr(a,d[dim_name].getncattr(a))
File “src/netCDF4/_netCDF4.pyx”, line 4709, in netCDF4._netCDF4.Variable.setncattr
AttributeError: _FillValue attribute must be set when variable is created (using fill_value keyword to createVariable)
“”"
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File “/home/581/mm6452/cylc-run/u-dk517/bin/nci_ostia_to_pp_parallel.py”, line 230, in
main()
File “/home/581/mm6452/cylc-run/u-dk517/bin/nci_ostia_to_pp_parallel.py”, line 220, in main
res.get(timeout=120)
File “/g/data/vk83/apps/conda/access-ram/2025.03.0/lib/python3.11/multiprocessing/pool.py”, line 774, in get
raise self._value
AttributeError: _FillValue attribute must be set when variable is created (using fill_value keyword to createVariable)
2026-09-10T06:45:46Z CRITICAL - failed/EXIT`
Hi @Moulik ,
The error is complaining about the netcdf variable not having a “_fillValue” attribute.
It is impossible to debug without having access to what you are trying to do.
Be very careful when modifying the netcdf files as the GRIB conversions are extremely sensitive.
I can sit with you next week if you need help.
Hi @cbengel
Yes, I would be very happy if we can meet through Zoom.
This one error comes from when I was running the OAS suite on the modified SST.
Thanks,
I have encountered another issue when using modified OSTIA SST NetCDF files during running the ACCESS-rAM3 OAS suite (u-dk517) on Gadi.
I have modified the analysed_sst field in the original OSTIA files by adding a PGW SST perturbation, while preserving the original NetCDF structure/encoding. The modified files can be opened successfully and have the same variable encodings as the original files.
With the modified SST files, the ostia_netcdf_to_pp / ancil_sst_seaice workflow successfully generates the intermediate files, including:
-
*_ostia.nc -
*_ostia_sst.pp -
*_ostia_sic.pp -
*_ostia_lsm.pp -
*_ostia_lsm.anc -
*_ostia_lsm.anc.nc
However, the final:
*_ostia.anc
files are not being generated when the suite runs, and the subsequent HOUSEKEEPING stage also does not appear to complete (but the suite successfully gets shut down!).
Interestingly, I can manually run ancil_sst_seaice.py using the generated .pp and _lsm.anc files, and it successfully creates the final .anc. I also tested nci_ancil_sst_seaice_parallel.py manually with --count 1, and it successfully produced the final .anc.
The relevant Cylc dependency is:
ostia_netcdf_to_pp => ancil_sst_seaice
and the ancil_sst_seaice task has a 20-minute walltime (I have changed it to 20 mins from 10 mins as I got an error in an earlier run indicating exceeding walltime).
I am therefore unsure why the final .anc is not being produced when the workflow is run through Cylc, despite the same conversion working successfully when executed manually.
I would appreciate any suggestions on where I should look next, particularly regarding the Cylc task environment, job execution, or interaction between the OAS tasks.
Thanks very much for your help.
Thanks Moulik, that is good progress.
@cbengel, could this be related to the python environment issue you’ve previously identified?
@mlipson - for the time being, can I proceed with manually generating the .anc files using the gnerated .pp and _lsm.anc files? It’s not a big deal for me as it’s only for 5 days.
On another note:
I think the Python environment issue could potentially be related, although the manual tests suggest that the environment itself is now working correctly. We were able to manually run ancil_sst_seaice.py using the same analysis3/21.10 environment and successfully generate the .anc file.
It looks like the issue may instead also be related to the parallel wrapper?
Thanks,
Hi Moulik, the environment issue is solution is described here by Claire Carouge ( ACCESS-rAM3: Files deleted with OSTIA ancillary suite - #8 by clairecarouge ).
I think given you are working with 5 dates, running the command manually is a quick work-around.
Perhaps true the PYTHONNOUSERSITE as well though and see if that solves your problem.
Thanks @cbengel for pointing this out to me.
I just ran the suite again after a few hours, and it worked! It generated the .anc file for the first four days, and I manually generated the file for the last day, so that seems to have solved the issue.
It looks like adding PYTHONNOUSERSITE=1 was the solution mentioned in that conversation. I’ll definitely try to remember this for future reference.
More importantly, I successfully ran the rAM3 Flagship suite with the modified SST, air temperature, and specific humidity over the weekend, and the results look good!
Thanks so much for all your help and for the quick responses.
Cheers,
Moulik