AM: inancila_rcf_inancila: Insufficient space for LOOKUP headers

lishx · 30 June 2025 06:53

OPEN: File /home/565/sl5165/cylc-run/ACCESS-AM3-N512/share/data/etc/ancil/qrclim.rivstor to be Opened on Unit 13 Exists
IO: Open: /home/565/sl5165/cylc-run/ACCESS-AM3-N512/share/data/etc/ancil/qrclim.rivstor on unit  13
CLOSE: File /home/565/sl5165/cylc-run/ACCESS-AM3-N512/share/data/etc/ancil/qrclim.rivstor Closed on Unit 13
IO: Open: /home/565/sl5165/cylc-run/ACCESS-AM3-N512/share/data/etc/ancil/qrclim.rivstor on unit  13
Ancil file num: 11
Ancil filename     : /home/565/sl5165/cylc-run/ACCESS-AM3-N512/share/data/etc/ancil/qrparm.rivseq
Stash req = 151
Stash req = 152
---------------------------------------------------------------
OPEN: File /home/565/sl5165/cylc-run/ACCESS-AM3-N512/share/data/etc/ancil/qrparm.rivseq to be Opened on Unit 13 Exists
IO: Open: /home/565/sl5165/cylc-run/ACCESS-AM3-N512/share/data/etc/ancil/qrparm.rivseq on unit  13
CLOSE: File /home/565/sl5165/cylc-run/ACCESS-AM3-N512/share/data/etc/ancil/qrparm.rivseq Closed on Unit 13
IO: Open: /home/565/sl5165/cylc-run/ACCESS-AM3-N512/share/data/etc/ancil/qrparm.rivseq on unit  13
Ancil file num: 12
Ancil filename     : /home/565/sl5165/cylc-run/ACCESS-AM3-N512/share/data/etc/ancil/qrparm.orog_radiation_parameters
Stash req = 5
Stash req = 6
---------------------------------------------------------------
OPEN: File /home/565/sl5165/cylc-run/ACCESS-AM3-N512/share/data/etc/ancil/qrparm.orog_radiation_parameters to be Opened on Unit 13 Exists
IO: Open: /home/565/sl5165/cylc-run/ACCESS-AM3-N512/share/data/etc/ancil/qrparm.orog_radiation_parameters on unit  13
CLOSE: File /home/565/sl5165/cylc-run/ACCESS-AM3-N512/share/data/etc/ancil/qrparm.orog_radiation_parameters Closed on Unit 13
rcf_ancil_atmos : nlookups -1269705575352475954
rcf_ancil_atmos : len_ancil 2432932

Ancillary File     1 : /home/565/sl5165/cylc-run/ACCESS-AM3-N512/share/data/etc/ancil/qrparm.mask
OPEN: File /home/565/sl5165/cylc-run/ACCESS-AM3-N512/share/data/etc/ancil/qrparm.mask to be Opened on Unit 13 Exists
IO: Open: /home/565/sl5165/cylc-run/ACCESS-AM3-N512/share/data/etc/ancil/qrparm.mask on unit  13
No room in LOOKUP table for Ancillary File /home/565/sl5165/cylc-run/ACCESS-AM3-N512/share/data/etc/ancil/qrparm.mask
Error in inancila_rcf_inancila
CMESSAGE inancila_rcf_inancila: Insufficient space for LOOKUP headers
ErrorStatus 14

????????????????????????????????????????????????????????????????????????????????
???!!!???!!!???!!!???!!!???!!!       ERROR        ???!!!???!!!???!!!???!!!???!!!
?  Error code: 14
?  Error from routine: RCF_ANCIL_ATMOS
?  Error message: inancila_rcf_inancila: Insufficient space for LOOKUP headers
?  Error from processor: 0
?  Error number: 1
????????????????????????????????????????????????????????????????????????????????

Does anyone have an idea how to resolve this error?
I’ve tried increasing nancil_lookupsa in [namelist:ancilcta], but that didn’t fix the issue.

The problem doesn’t appear to be specific to qrparm.orog_radiation_parameters, as the same error occurs even when this file is skipped.

I’m currently testing ACCESS-AM3 at N512 resolution, so the ancillary files are significantly larger than those for N96. These files were regridded from N96 to N512 using the script created by Martin:
/g/data/access/projects/access/apps/pythonlib/umfile_utils/interpolate_ancillary.py

Any suggestions or insights would be greatly appreciated!

Scott · 30 June 2025 23:53

Try increasing I think it’s nancil_lookupsa (use the search in rose edit to find where it’s set). The locations of each timestep and level in all of the ancil files are indexed in an array, depending on the numbers of timesteps and levels in the ancil files you’re using the default value may be insufficient.

lishx · 1 July 2025 01:19

Hi Scott,

Thanks for your reply.

I tried increasing nancil_lookupsa to a very large value in the namelist:

[namelist:ancilcta]
l_amipii_ice_processing = .true.
l_snow_assim = .false.
l_sstanom = .false.
nancil_lookupsa = 80000000
use_lookup_dates_anc_time_interp = .false.

I also updated the corresponding line in the source file ancilcta_namelist_mod.F90:
CALL chk_var(nancil_lookupsa, 'nancil_lookupsa', '[1:80000000]')

However, I still encounter the same error. Do you have any further suggestions or thoughts on what might be going wrong?

Scott · 1 July 2025 01:29

Try inspecting the files with mule-pumf to see the number of lookups in each of your ancil files to see if there’s anything odd

module use /g/data/xp65/public/modules
module load conda/analysis3-24.07

mule-pumf --components fixed_length_header /g/data/access/TIDS/UM/ancil/atmos/n512e/aerosols/naei_entec_emep/2012/v2/qrclim.murk_L70

Item ‘(152) lookup_dim2’ is the number of lookups in that ancil file

lishx · 1 July 2025 05:04

For the regridded N512 file:

File: ancil_n512/qrparm.orog_radiation_parameters

%%%%%%%%%%%%%%%%%%%%%%%
* fixed_length_header *
%%%%%%%%%%%%%%%%%%%%%%%
  (1) data_set_format_version    :     20
  (2) sub_model                  :      1
  (3) vert_coord_type            :      1
  (4) horiz_grid_type            :      0
  (5) dataset_type               :      4
  (9) grid_staggering            :      6
  (10) time_type                 :      0
  (12) model_version             :     19
  (100) integer_constants_start  :    257
  (101) integer_constants_length :     15
  (105) real_constants_start     :    272
  (106) real_constants_length    :      6
  (150) lookup_start             :    278
  (151) lookup_dim1              :     64
  (152) lookup_dim2              :      3
  (160) data_start               : 524289
  (161) data_dim1                :  82944

For the original N96 file:

File: ancil_n96/qrparm.orog_radiation_parameters

%%%%%%%%%%%%%%%%%%%%%%%
* fixed_length_header *
%%%%%%%%%%%%%%%%%%%%%%%
  (1) data_set_format_version    :     20
  (2) sub_model                  :      1
  (3) vert_coord_type            :      1
  (4) horiz_grid_type            :      0
  (5) dataset_type               :      4
  (9) grid_staggering            :      6
  (10) time_type                 :      0
  (12) model_version             :     19
  (100) integer_constants_start  :    257
  (101) integer_constants_length :     15
  (105) real_constants_start     :    272
  (106) real_constants_length    :      6
  (150) lookup_start             :    278
  (151) lookup_dim1              :     64
  (152) lookup_dim2              :      3
  (160) data_start               : 524289
  (161) data_dim1                :  82944

I’m not very familiar with UM ancillary file structure, but should I expect the data_dim1 value to differ between the N96 and N512 versions? If so then it’s surprising that both files show data_dim1 = 82944. the N512 qrparm.orog_radiation_parameters appears to load correctly in xconv:

I am wondering what is the correct way to update or regenerate the ancillary file header to reflect the appropriate grid size?

Scott · 1 July 2025 05:22

data_dim1 is an unused value, it’s not a concern.

You can find the documentation of the header values in section 3 of https://code.metoffice.gov.uk/doc/um/latest/papers/umdp_F03.pdf.

That file looks fine, do all the other ancil files in /home/565/sl5165/cylc-run/ACCESS-AM3-N512/share/data/etc/ancil/ have reasonable (> 0, < 100,000) values?

lishx · 2 July 2025 14:04

It finally turns out that the nancil_lookupsa issue was related to some ancillary files created from .nc format. Although the namelist seems to allow specifying netcdf_varname, it doesn’t work in practice — possibly due to NetCDF version or format compatibility issues (?)

But anyway, a new error has occurred:

Ancillary File    12 : /home/565/sl5165/cylc-run/ACCESS-AM3-N512/share/data/etc/ancil/qrparm.orog_radiation_parameters
OPEN: File /home/565/sl5165/cylc-run/ACCESS-AM3-N512/share/data/etc/ancil/qrparm.orog_radiation_parameters to be Opened on Unit 13 Exists
IO: Open: /home/565/sl5165/cylc-run/ACCESS-AM3-N512/share/data/etc/ancil/qrparm.orog_radiation_parameters on unit  13
CLOSE: File /home/565/sl5165/cylc-run/ACCESS-AM3-N512/share/data/etc/ancil/qrparm.orog_radiation_parameters Closed on Unit 13
Unspecified calendar type in ancillary file
OPEN: File /home/565/sl5165/cylc-run/ACCESS-AM3-N512/share/data/etc/ancil/qrparm.landfrac to be Opened on Unit 13 Exists
IO: Open: /home/565/sl5165/cylc-run/ACCESS-AM3-N512/share/data/etc/ancil/qrparm.landfrac on unit  13
CLOSE: File /home/565/sl5165/cylc-run/ACCESS-AM3-N512/share/data/etc/ancil/qrparm.landfrac Closed on Unit 13
READ IN LAND FRACTION
OPEN: File /home/565/sl5165/cylc-run/ACCESS-AM3-N512/share/data/etc/ancil/qrparm.mask to be Opened on Unit 13 Exists
IO: Open: /home/565/sl5165/cylc-run/ACCESS-AM3-N512/share/data/etc/ancil/qrparm.mask on unit  13
replanca_rcf_replanca: UPDATE REQUIRED FOR FIELD    30 : LAND MASK (No halo) (LAND=TRUE)
CLOSE: File /home/565/sl5165/cylc-run/ACCESS-AM3-N512/share/data/etc/ancil/qrparm.mask Closed on Unit 13
OPEN: File /home/565/sl5165/cylc-run/ACCESS-AM3-N512/share/data/etc/ancil/qrparm.orog_wavedrag to be Opened on Unit 13 Exists
IO: Open: /home/565/sl5165/cylc-run/ACCESS-AM3-N512/share/data/etc/ancil/qrparm.orog_wavedrag on unit  13
replanca_rcf_replanca: UPDATE REQUIRED FOR FIELD    33 : OROGRAPHY (/STRAT LOWER BC)
BUFFIN: C I/O Error - Return code = 1
****************** IO Error Report ***********************************
Unit Generating error=   13
---File States --------------------------
Unit  11 open on filename /home/565/sl5165/cylc-run/ACCESS-AM3-N512/share/data/dl594a.ainitial
--> File Type:   0 , Read Only: T , Write Only: F
--> Local: T AllLocal: F Remote: F Broadcast: T
--> Local: T AllLocal: F Remote: F Broadcast: T
Unit  12 open on filename /home/565/sl5165/cylc-run/ACCESS-AM3-N512/share/data/dl594a.astart
--> File Type:   1 , Read Only: F , Write Only: F
--> Local: T AllLocal: F Remote: F Broadcast: T
--> Local: T AllLocal: F Remote: F Broadcast: T
Unit  13 open on filename /home/565/sl5165/cylc-run/ACCESS-AM3-N512/share/data/etc/ancil/qrparm.orog_wavedrag
--> File Type:   0 , Read Only: T , Write Only: F
--> Local: T AllLocal: F Remote: F Broadcast: F
--> Local: T AllLocal: F Remote: F Broadcast: F
---End File States ----------------------

????????????????????????????????????????????????????????????????????????????????
???!!!???!!!???!!!???!!!???!!!       ERROR        ???!!!???!!!???!!!???!!!???!!!
?  Error code: 22
?  Error from routine: io:buffin
?  Error message: Error in buffin errorCode= 0.00 len=393216/786432
?  Error from processor: 0
?  Error number: 1
????????????????????????????????????????????????????????????????????????????????

I found a similar error report here: BUFFIN: Read Failed - Unified Model - NCAS Modelling Support — but the scenario there seems different.

In the case, qrparm.orog_wavedrag was interpolated using: /g/data/access/projects/access/apps/pythonlib/umfile_utils/interpolate_ancillary.py, and it can be opened and visualized using xonv without any problems. The header also looks good:

%%%%%%%%%%%%%%%%%%
* PUMF-II Report *
%%%%%%%%%%%%%%%%%%

File: /home/565/sl5165/cylc-run/ACCESS-AM3-N512/share/data/etc/ancil/qrparm.orog_wavedrag

%%%%%%%%%%%%%%%%%%%%%%%
* fixed_length_header *
%%%%%%%%%%%%%%%%%%%%%%%
  (1) data_set_format_version    :     20
  (2) sub_model                  :      1
  (3) vert_coord_type            :      1
  (4) horiz_grid_type            :      0
  (5) dataset_type               :      4
  (9) grid_staggering            :      6
  (10) time_type                 :      0
  (12) model_version             :     19
  (100) integer_constants_start  :    257
  (101) integer_constants_length :     15
  (105) real_constants_start     :    272
  (106) real_constants_length    :      6
  (150) lookup_start             :    278
  (151) lookup_dim1              :     64
  (152) lookup_dim2              :      8
  (160) data_start               : 524289
  (161) data_dim1                : 221184

Scott, do you have any suggestions on what might be going wrong here?

Scott · 3 July 2025 03:03

Can you make a copy of the file under /scratch/public/sl5165? If there are file length errors make sure that the final field in the file looks reasonable in xconv, the file may have been truncated

lishx · 3 July 2025 08:41

I have copied the file (qrparm.orog_wavedrag) to /scratch/public/sl5165. It can be read and plotted correctly using xconv.

I’m wondering if the issue could be due to a mismatch between ancillary files — specifically, I used qrparm.mask from /g/data/access/TIDS/UM/ancil/atmos/n512e/ as the original N512 ancillary, whereas qrparm.orog_wavedrag was regridded one.

Scott · 4 July 2025 00:47

I’m not sure sorry - like you say the file contents look fine.

Maybe try regridding the original file with ancil_general_regrid.py (available in module /g/data/access/ngm/modules/ants/ug-2.1.0) and see if that behaves any different?

lishx · 4 July 2025 05:59

It works!
Now this file can be read by model:

CLOSE: File /home/565/sl5165/cylc-run/ACCESS-AM3-N512/share/data/etc/ancil/qrparm.mask Closed on Unit 13
OPEN: File /home/565/sl5165/cylc-run/ACCESS-AM3-N512/share/data/etc/ancil/qrparm.orog_wavedrag to be Opened on Unit 13 Exists
IO: Open: /home/565/sl5165/cylc-run/ACCESS-AM3-N512/share/data/etc/ancil/qrparm.orog_wavedrag on unit  13
replanca_rcf_replanca: UPDATE REQUIRED FOR FIELD    33 : OROGRAPHY (/STRAT LOWER BC)
replanca_rcf_replanca: UPDATE REQUIRED FOR FIELD    34 : STANDARD DEVIATION OF OROGRAPHY
replanca_rcf_replanca: UPDATE REQUIRED FOR FIELD    35 : OROGRAPHIC GRADIENT XX COMPONENT
replanca_rcf_replanca: UPDATE REQUIRED FOR FIELD    36 : OROGRAPHIC GRADIENT XY COMPONENT
replanca_rcf_replanca: UPDATE REQUIRED FOR FIELD    37 : OROGRAPHIC GRADIENT YY COMPONENT
CLOSE: File /home/565/sl5165/cylc-run/ACCESS-AM3-N512/share/data/etc/ancil/qrparm.orog_wavedrag Closed on Unit 13
OPEN: File /home/565/sl5165/cylc-run/ACCESS-AM3-N512/share/data/etc/ancil/qrparm.soil to be Opened on Unit 13 Exists
IO: Open: /home/565/sl5165/cylc-run/ACCESS-AM3-N512/share/data/etc/ancil/qrparm.soil on unit  13

I will regrid and test for all other necessary ancil files, thanks!

lishx · 7 July 2025 07:22

Hi Scott,

Do you know if we have the aerosol ancillary files for N512 on Gadi?
For example, something similar to the files in:
/g/data/access/TIDS/UM/ancil/atmos/GC5/n96e/easyaerosol/cmip6_stratos/climatology_1850-2014/v1/?

Scott · 7 July 2025 23:19

It doesn’t look like it, but I’ve not been involved with climate ancil files for a while now so I’m not certain. Maybe one of the @access-nri folks can comment?

lishx · 9 July 2025 06:09

Just update, the AM3-N640 run was able to integrate for 10 minutes before crashing due to a physics instability:

********************************************************************************
Atm_Step: Timestep        1   Model time:   1975-01-01 00:10:00
EG_SISL_Resetcon: calculate reference profile
Initialising coarse grids...
Number of multigrid levels requested         :     3
Number of multigrid levels allowed (lon/lat) : (   5 /    5 )
Level     1:   1280 x    960 x   85 =      104,448,000
Level     2:    640 x    630 x   85 =       34,272,000
Level     3:    320 x    387 x   85 =       10,526,400
Grid complexity =   1.4289 [lower bound =   1.3125]
...Done
********************************************************************************


????????????????????????????????????????????????????????????????????????????????
???!!!???!!!???!!!???!!!???!!!       ERROR        ???!!!???!!!???!!!???!!!???!!!
?  Error code: 3
?  Error from routine: GLUE_CONV_6A
?  Error message: Mid conv went to the top of the model at point            8 in seg on call  1
?  Error from processor: 2056
?  Error number: 35
????????????????????????????????????????????????????????????????????????????????

I suspect the error may be related to the ad hoc ancillary files I generated. Can we consider this version as working?

Scott · 9 July 2025 06:17

For model instabilities inspect the ancil files particularly along coastlines to make sure there hasn’t been a masking error

Topic		Replies	Views
Regional Ancillary Suite no longer running Regional Nesting Suite help	5	99	27 August 2024
ACCESS-CM2 suite-run error Coupled Model help , access-cm2 , help-needed , error	14	123	24 February 2025
ACCESS-rAM3 Release 1.0 Feedback Regional Nesting Suite regional , atmosphere , feedback , access-ram , access-ram3	14	97	9 May 2025
Regional Ancillary Suite no longer running (again) Regional Nesting Suite	12	116	17 October 2024
UM Nesting Suite - SST ancillary not being used? Regional Nesting Suite help , um , um-ancillary , inscope , sst	14	500	17 October 2023

AM: inancila_rcf_inancila: Insufficient space for LOOKUP headers

Related topics