# Issues when changing domain in ACCESS-rAM3 **URL:** \ **Category:** rAM3/Regional Nesting Suite\ **Tags:** help\ **Created:** [19 February 2026 07:02 UTC](https://forum.access-hive.org.au/t/issues-when-changing-domain-in-access-ram3/5943 "2026-02-19T07:02:07Z")\ **Posts on this page:** 18\ **Page:** 1
**Author:** ![chenhui.jin](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/chenhui.jin/32/2550_2.png) [@chenhui.jin](https://forum.access-hive.org.au/u/chenhui.jin)\ **Post date:** [19 February 2026 07:02 UTC](https://forum.access-hive.org.au/t/issues-when-changing-domain-in-access-ram3/5943/1 "2026-02-19T07:02:07Z")
Hi, I’m new to running ACCESS-rAM3, and encouter some issues when I changed the domain. The domain is set as follows: `CENTRE=[-35,130]` `ERA_RES=[0.11,0.11]` `ERA_SIZE=[320,420]` `d1100_RES=ERA_RES` `d1100_SIZE=[300,400]` ![image](https://us1.discourse-cdn.com/flex020/uploads/access1/original/2X/6/6cfc17450d1b0ec5dfac093e4dbd9092d16b1876.png) The RAS works fine, generating some ancillary files. I used `Check_UM_ancillaries` from [UM\_configuration\_tools/notebooks at main · 21centuryweather/UM\_configuration\_tools · GitHub](https://github.com/21centuryweather/UM_configuration_tools/tree/main/notebooks) to check the ancillaries; they seem to be fine (with output of `INFO : Your ancillaries appear to be okay over land`) ![image](https://us1.discourse-cdn.com/flex020/uploads/access1/original/2X/e/ef460630b81bdfbbc87eaa88d61cdc92dee01b30.jpeg) The issue occurs when running RNS: ![Screenshot 2026-02-19 at 5.57.55 pm](https://us1.discourse-cdn.com/flex020/uploads/access1/original/2X/2/2a71fdb06b2b3b741b321ee5d1d6b908bae9d975.jpeg) I was also struggling to find `job.err` for this task. Can you help me with this? Thanks in advance! ---
**Author:** ![reyhan.respati](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/reyhan.respati/32/4344_2.png) [@reyhan.respati](https://forum.access-hive.org.au/u/reyhan.respati)\ **Post date:** [19 February 2026 07:06 UTC](https://forum.access-hive.org.au/t/issues-when-changing-domain-in-access-ram3/5943/2 "2026-02-19T07:06:53Z")
Hi @chenhui.jin, If you click the little triangle on the left side, it will expand the list and show you which specific job failed. You can right click on that specific job and view the job.err file. ---
**Author:** ![chenhui.jin](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/chenhui.jin/32/2550_2.png) [@chenhui.jin](https://forum.access-hive.org.au/u/chenhui.jin)\ **Post date:** [19 February 2026 07:27 UTC](https://forum.access-hive.org.au/t/issues-when-changing-domain-in-access-ram3/5943/3 "2026-02-19T07:27:08Z")
Fantastic! Thanks @reyhan.respati! I managed to find the `job.err` for the task. ---
**Author:** ![chenhui.jin](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/chenhui.jin/32/2550_2.png) [@chenhui.jin](https://forum.access-hive.org.au/u/chenhui.jin)\ **Post date:** [19 February 2026 07:51 UTC](https://forum.access-hive.org.au/t/issues-when-changing-domain-in-access-ram3/5943/4 "2026-02-19T07:51:26Z")
I managed to find the `job.err`. ![Screenshot 2026-02-19 at 6.28.38 pm](https://us1.discourse-cdn.com/flex020/uploads/access1/original/2X/1/194be4436fc29b7c334213e134bb1a0c9fb7403f.jpeg) The last two lines of `job.err’ `[FAIL] um-atmos # return-code=9`. `2026-02-19T05:59:27Z CRITICAL - failed/EXIT` `job.err` is attached below [job.err.txt](https://forum.access-hive.org.au/uploads/short-url/588plfMO6i4fizBUNISWyAwgptt.txt) (491.6 KB) ---
**Author:** ![Paul.Gregory](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/paul.gregory/32/2566_2.png) [@Paul.Gregory](https://forum.access-hive.org.au/u/Paul.Gregory)\ **Post date:** [19 February 2026 08:15 UTC](https://forum.access-hive.org.au/t/issues-when-changing-domain-in-access-ram3/5943/5 "2026-02-19T08:15:27Z")
Hi @chenhui.jin The error is given here in `job.err` ```auto ???????????????????????????????????????????????????????????????????????????????? ???????????????????????????????????????????????????????????????????????????????? ???!!!???!!!???!!!???!!!???!!! ERROR ???!!!???!!!???!!!???!!!???!!! ???!!!???!!!???!!!???!!!???!!! ERROR ???!!!???!!!???!!!???!!!???!!! ? Error code: 4 ? Error code: 4 ? Error from routine: DECOMPOSE_FULL ? Error from routine: DECOMPOSE_FULL ? Error message: Too many processors in the North-South direction.The maximum permitted is 16 ? Error message: Too many processors in the North-South direction.The maximum permitted is 16 ? Error from processor: 15 ? Error from processor: 272 ? Error number: 16 ? Error number: 16 ???????????????????????????????????????????????????????????????????????????????? ???????????????????????????????????????????????????????????????????????????????? ``` Can you tell me how many processors you are attempting to run with? Have a look in `~/roses/u-by395/rose-suite.conf` and search for `rg01_rs01_m01_nproc` i.e. the number of processors allocated to region 1, resolution 1, model 1. In the default setup these values are ```auto rg01_rs01_m01_nproc=18,16 ``` What are yours set to? ---
**Author:** ![chenhui.jin](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/chenhui.jin/32/2550_2.png) [@chenhui.jin](https://forum.access-hive.org.au/u/chenhui.jin)\ **Post date:** [19 February 2026 08:31 UTC](https://forum.access-hive.org.au/t/issues-when-changing-domain-in-access-ram3/5943/6 "2026-02-19T08:31:27Z")
Hi @Paul.Gregory, Mine is: ![Screenshot 2026-02-19 at 7.30.05 pm](https://us1.discourse-cdn.com/flex020/uploads/access1/original/2X/f/f29f915f2624f7b6e077afc7c0def5693550beab.png) ---
**Author:** ![Paul.Gregory](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/paul.gregory/32/2566_2.png) [@Paul.Gregory](https://forum.access-hive.org.au/u/Paul.Gregory)\ **Post date:** [19 February 2026 08:56 UTC](https://forum.access-hive.org.au/t/issues-when-changing-domain-in-access-ram3/5943/7 "2026-02-19T08:56:40Z")
Ok your domain is slightly smaller than the default Lismore outer domain which is 450x450. Try reducing your number of processors, say 16x12. So set ```auto rg01_rs01_m01_nproc=16,12 ``` in `rose-suite.conf`. Or you can use `rose edit` and change the values in the GUI. Remember to reload your suite. You will have to run the `_GAL9_LBCS` and `_GAL9_um_recon` tasks again with the new decomposition before running the forecast tasks again. Generally when running on the normal compute nodes on gadi (which have 48 cores) we want the total number of cores to be a multiple of 48. When the UM decomposes your grid across multiple cores, you need to reserve a certain number of grid rows for the `halo` - the points that are shared between each sub-domain. The UM has limits on the numbers of points in the halo region and the sub-domain itself. ---
**Author:** ![chenhui.jin](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/chenhui.jin/32/2550_2.png) [@chenhui.jin](https://forum.access-hive.org.au/u/chenhui.jin)\ **Post date:** [19 February 2026 20:03 UTC](https://forum.access-hive.org.au/t/issues-when-changing-domain-in-access-ram3/5943/8 "2026-02-19T20:03:51Z")
Thanks, reducing the number of processors fixed the problem! I noticed there is some discontinuity in the output. I assume that corresponds to the domain of d1100? ![wind_gust_20160927T00-20160928T00](https://us1.discourse-cdn.com/flex020/uploads/access1/original/2X/3/3e9fac5ebb2590d8a188ce5a8d1f2329a1d334bb.gif) For curiosity, if my domain is very large (say `CENTRE=[-42.5,92.5]; ERA_RES=[0.2,0.2]; ERA_SIZE=[196,646]; d2000_RES=ERA_RES; d20000_SIZE=[176,626]`) ![image](https://us1.discourse-cdn.com/flex020/uploads/access1/original/2X/a/a0c1a74aba932169625f2a6d6bbe259aa464629f.png) Should I keep the number of processors as 18x16, or increase it? ---
**Author:** ![Paul.Gregory](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/paul.gregory/32/2566_2.png) [@Paul.Gregory](https://forum.access-hive.org.au/u/Paul.Gregory)\ **Post date:** [20 February 2026 00:17 UTC](https://forum.access-hive.org.au/t/issues-when-changing-domain-in-access-ram3/5943/9 "2026-02-20T00:17:17Z")
There are slight ‘edge effects’ at the edges of your domain - that is normal. Yes you should increase the number of cores if the domain becomes larger. Sometimes the UM doesn’t like very rectangular domains and it prefers the regional models to be ‘squarer’. But you might be able that domain running. For larger domain sizes (i.e. \> 1000 x 1000) you should talk to someone in the Centre about using the optimised rAM3 config which runs an I/O server and contains various other optimisations. Otherwise, the standard configuration will start to hang on I/O throughput for large domain sizes when you throw lots of cores at it. I made a summary of the UM documentation which you can read through here: > **[Introduction — Unified Model Summary Documentation](https://21centuryweather.github.io/UM_summary_docs/introduction.html)** ---
**Author:** ![chenhui.jin](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/chenhui.jin/32/2550_2.png) [@chenhui.jin](https://forum.access-hive.org.au/u/chenhui.jin)\ **Post date:** [20 February 2026 00:50 UTC](https://forum.access-hive.org.au/t/issues-when-changing-domain-in-access-ram3/5943/10 "2026-02-20T00:50:15Z")
Hi @Paul.Gregory, many thanks for the information! ---
**Author:** ![Paul.Gregory](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/paul.gregory/32/2566_2.png) [@Paul.Gregory](https://forum.access-hive.org.au/u/Paul.Gregory)\ **Post date:** [20 February 2026 04:48 UTC](https://forum.access-hive.org.au/t/issues-when-changing-domain-in-access-ram3/5943/11 "2026-02-20T04:48:38Z")
Here is a better explanation of the NPROC error: > [@ACCESS-rAM3 'Flagship' Experiments](https://forum.access-hive.org.au/t/access-ram3-flagship-experiments/4580/2): > > OpenMP doesn’t require a recompile, just set environment variable $OMP\_NUM\_THREADS (normally done in the Cylc task configuration) and make sure the PBS number of CPUs is (MPI ranks) \* (OMP threads) NPROC restriction is because there’s a minimum grid size for each process, e.g. halo of cells going to the north MPI rank can’t overlap with the halo of cells going to the south MPI rank. ---
**Author:** ![chenhui.jin](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/chenhui.jin/32/2550_2.png) [@chenhui.jin](https://forum.access-hive.org.au/u/chenhui.jin)\ **Post date:** [22 February 2026 21:31 UTC](https://forum.access-hive.org.au/t/issues-when-changing-domain-in-access-ram3/5943/12 "2026-02-22T21:31:17Z")
Hi @Paul.Gregory, I started testing run ACCESS-rAM3 for the larger domain: ![Screenshot 2026-02-22 at 10.38.40 pm](https://us1.discourse-cdn.com/flex020/uploads/access1/original/2X/7/73c94b533c1f852e433a15df6b85342867e6530a.png) I managed to create some ancillary files; however, I have one bad ancillary file `canopy_height` ![image](https://us1.discourse-cdn.com/flex020/uploads/access1/original/2X/1/1eeb2a23f649e5b91b23b52f54daba110e7e2ad7.png) When running RNS, an error occurs at `nci_hres_eccb`: ![Screenshot 2026-02-22 at 10.43.20 pm](https://us1.discourse-cdn.com/flex020/uploads/access1/original/2X/2/2536c01dbba4c5724ff8359fc80cd634574c19cb.jpeg) `job.err` shows: ![Screenshot 2026-02-22 at 10.45.01 pm](https://us1.discourse-cdn.com/flex020/uploads/access1/original/2X/d/dab2e7e5119b612c2bb4282190bc5ed3a7b57973.jpeg) Is this error due to the bad ancillary file `canopa_height`, or due to the very rectangular domain? ---
**Author:** ![Paul.Gregory](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/paul.gregory/32/2566_2.png) [@Paul.Gregory](https://forum.access-hive.org.au/u/Paul.Gregory)\ **Post date:** [22 February 2026 22:47 UTC](https://forum.access-hive.org.au/t/issues-when-changing-domain-in-access-ram3/5943/13 "2026-02-22T22:47:43Z")
Hi Chenhui. Check the southernmost extents of your domain. If they exceed the limits of the BARRA-R domain - you will generate errors. The southern boundary of BARRA-R is -57.97 S. See this post [https://21centuryweather.discourse.group/t/matching-gridpoints-in-barrar-and-outer-domain-in-access-ram3/2105/19](https://21centuryweather.discourse.group/t/matching-gridpoints-in-barrar-and-outer-domain-in-access-ram3/2105/19) Can you identify where the missing value of canopy height is? The last few cells of this notebook will help you track down their location: [UM\_configuration\_tools/notebooks/Check\_UM\_ancillaries.ipynb at main · 21centuryweather/UM\_configuration\_tools · GitHub](https://github.com/21centuryweather/UM_configuration_tools/blob/main/notebooks/Check_UM_ancillaries.ipynb) Just change the lat/lon values in those cells to include the location of NaNs over land. If that NaNs are located over a small isolated island, these instructions should fix the problem. [https://21centuryweather.discourse.group/t/errors-in-ram3-set-up/2053/47](https://21centuryweather.discourse.group/t/errors-in-ram3-set-up/2053/47) ---
**Author:** ![chenhui.jin](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/chenhui.jin/32/2550_2.png) [@chenhui.jin](https://forum.access-hive.org.au/u/chenhui.jin)\ **Post date:** [23 February 2026 03:27 UTC](https://forum.access-hive.org.au/t/issues-when-changing-domain-in-access-ram3/5943/14 "2026-02-23T03:27:37Z")
Hi @Paul.Gregory, I checked the southernmost extent of `era5` and `d1100` For `era5`, the southmost is -58.900, and for `d1100`, the southmost is -57.800. I was wondering if the limit for the BARRA-R (-57.97) is for `era` or `d1100`. It seems my domain exceeds the limit? Do I have to re-run RAS to recreate ancilaries? Regarding the NaN in canopy height: - It seems NaN occurs near (-55, 160) - Zoom-in shows it at around (-55, 159) - It looks like the NaN also occurs at Macquarie Island. - I followed the instructions and edited the line of the file in `/home/565/cj0591/cylc-run/u-bu503/share/contrib _apps/CanopyHeights/canopy_heights.py` as: `loop_lim_y = index_nearest_neighbour.ydist2index(trees, 2000)` ---
**Author:** ![Paul.Gregory](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/paul.gregory/32/2566_2.png) [@Paul.Gregory](https://forum.access-hive.org.au/u/Paul.Gregory)\ **Post date:** [23 February 2026 04:45 UTC](https://forum.access-hive.org.au/t/issues-when-changing-domain-in-access-ram3/5943/15 "2026-02-23T04:45:40Z")
If you’re using the BARRA-R for the land surface, you’ll need to regenerate any ancil domains that exceed -57.97. Hence you will need to shift `era5` northwards. I would shift the southern edge of `era5` AND `d1000` northwards by the same amount to keep the same spacing between the edges of both domains. You can use these notebooks created by Scott Wales to help size your domain : [UM\_configuration\_tools/notebooks/UM\_plot\_domain.ipynb at main · 21centuryweather/UM\_configuration\_tools · GitHub](https://github.com/21centuryweather/UM_configuration_tools/blob/main/notebooks/UM_plot_domain.ipynb) The Macquarie Island canopy height issue is a known bug that can be fixed by increasing the search distance to 2,000. ---
**Author:** ![chenhui.jin](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/chenhui.jin/32/2550_2.png) [@chenhui.jin](https://forum.access-hive.org.au/u/chenhui.jin)\ **Post date:** [23 February 2026 06:30 UTC](https://forum.access-hive.org.au/t/issues-when-changing-domain-in-access-ram3/5943/16 "2026-02-23T06:30:09Z")
I double-checked the settings in RNS - I actually used ERA5-land for the land surface, when the error occurs ![Screenshot 2026-02-23 at 5.27.41 pm](https://us1.discourse-cdn.com/flex020/uploads/access1/original/2X/d/dc761d5e9864d5b94d7597f16ea732d0ad4871a7.jpeg) Do I also need to mind the southern boundary of -57.97? ---
**Author:** ![Paul.Gregory](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/paul.gregory/32/2566_2.png) [@Paul.Gregory](https://forum.access-hive.org.au/u/Paul.Gregory)\ **Post date:** [23 February 2026 06:38 UTC](https://forum.access-hive.org.au/t/issues-when-changing-domain-in-access-ram3/5943/17 "2026-02-23T06:38:33Z")
If you are using ERA5-land, your outer mesh must have a resolution of 0.1 degrees. The default rAM3 suite uses 0.11 in the outer mesh because that matches the BARRA-R land surface grids. From the NRI [rAM3 hive docs](https://docs.access-hive.org.au/models/run_a_model/run_access-ram3/#about) > Currently, ACCESS-rAM3 only supports specific nest configurations that meet the following criteria: > > The grid points of the RAS first inner nest (i.e., _Resolution 2_, because _Resolution 1_always corresponds to the outer ERA5 domain) must align with those of the [land-surface initial conditions dataset](https://docs.access-hive.org.au/models/access_models/access-ram/#land-surface-initial-conditions-source). Thus, the configuration of the RAS first inner nest (_Resolution 2_), including its position, dimension and resolution, need to be modified accordingly. Note that the position of a nest is also influenced by the [nested region position](https://docs.access-hive.org.au/models/run_a_model/run_access-ram3/#change-the-nested-region-position). Have a read here : [https://21centuryweather.discourse.group/t/matching-gridpoints-in-barrar-and-outer-domain-in-access-ram3/2105/2](https://21centuryweather.discourse.group/t/matching-gridpoints-in-barrar-and-outer-domain-in-access-ram3/2105/2) ---
**Author:** ![chenhui.jin](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/chenhui.jin/32/2550_2.png) [@chenhui.jin](https://forum.access-hive.org.au/u/chenhui.jin)\ **Post date:** [24 February 2026 05:50 UTC](https://forum.access-hive.org.au/t/issues-when-changing-domain-in-access-ram3/5943/18 "2026-02-24T05:50:09Z")
Thanks @Paul.Gregory for your suggestion. The model is now able to run and generate some output: I will be in touch with someone in the Centre later to talk about using the optimised ACCESS-rAM3 config for the large domain and my ideas in running ACCESS-rAM3 with some climatological mean ERA5 datasets. Thanks again!