# Outputting a region using MOM6 diag\_table - rather than a single file, I am getting many (one per processor?)

**URL:** <https://forum.access-hive.org.au/t/outputting-a-region-using-mom6-diag-table-rather-than-a-single-file-i-am-getting-many-one-per-processor/5262>\
**Category:** COSIMA\
**Tags:** regional, mom6, diagnostic, community-help\
**Created:** [19 September 2025 04:52 UTC](https://forum.access-hive.org.au/t/outputting-a-region-using-mom6-diag-table-rather-than-a-single-file-i-am-getting-many-one-per-processor/5262 "2025-09-19T04:52:33Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![mmr0](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/mmr0/32/2485_2.png) [@mmr0](https://forum.access-hive.org.au/u/mmr0)\
**Post date:** [19 September 2025 04:52 UTC](https://forum.access-hive.org.au/t/outputting-a-region-using-mom6-diag-table-rather-than-a-single-file-i-am-getting-many-one-per-processor/5262/1 "2025-09-19T04:52:33Z")

</div>

Hello - I am trying the option of outputting a “region” in `diag_table`. I’ve selected a 5x3 degree box and am outputting daily 3d fields. I seem to be getting multiple files (one per processor?) per variable, when I expect one, e.g.,

`access-om3.mom6.3d.region.uo.z.1day.mean._2170.nc.0250 `  
`access-om3.mom6.3d.region.uo.z.1day.mean._2170.nc.0251 `  
`access-om3.mom6.3d.region.uo.z.1day.mean._2170.nc.0252`

Is this expected? My diag\_table is here [se-aus-regional-om3-ryf/diag\_table at expt · mmr0/se-aus-regional-om3-ryf · GitHub](https://github.com/mmr0/se-aus-regional-om3-ryf/blob/expt/diag_table)

---

<div class="post-metadata">

**Author:** ![aekiss](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/aekiss/32/46_2.png) [@aekiss](https://forum.access-hive.org.au/u/aekiss)\
**Post date:** [22 September 2025 03:29 UTC](https://forum.access-hive.org.au/t/outputting-a-region-using-mom6-diag-table-rather-than-a-single-file-i-am-getting-many-one-per-processor/5262/2 "2025-09-22T03:29:55Z")

</div>

Hm, looks like they aren’t being collated automatically by payu.

What’s in the “collation” section of your config.yaml?

There may also be some clues in the `*_c.o*` and `*_c.e*` files.

---

<div class="post-metadata">

**Author:** ![angus-g](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/angus-g/32/89_2.png) [@angus-g](https://forum.access-hive.org.au/u/angus-g)\
**Post date:** [22 September 2025 14:47 UTC](https://forum.access-hive.org.au/t/outputting-a-region-using-mom6-diag-table-rather-than-a-single-file-i-am-getting-many-one-per-processor/5262/3 "2025-09-22T14:47:49Z")

</div>

> [@mmr0](#):
>
> Is this expected?

Annoyingly, yes. You have set `IO_LAYOUT = 1, 1` (as you have), which effectively collates your output when diagnostics are posted. However, your subregion doesn’t include the root processor, so FMS doesn’t have anywhere to gather all the outputs, and each rank will output its tile separately! As @aekiss alluded to, you’ll need to enable collation with a section something like the following (adjust as necessary) in your `config.yaml`:

```yaml
collate:
  mpi: true
  walltime: 1:00:00
  mem: 30GB
  ncpus: 4
  queue: expresssr
  exe: /g/data/vk83/apps/mppnccombine-fast/0.2/bin/mppnccombine-fast

```

---

<div class="post-metadata">

**Author:** ![dougiesquire](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/dougiesquire/32/16_2.png) [@dougiesquire](https://forum.access-hive.org.au/u/dougiesquire)\
**Post date:** [22 September 2025 22:46 UTC](https://forum.access-hive.org.au/t/outputting-a-region-using-mom6-diag-table-rather-than-a-single-file-i-am-getting-many-one-per-processor/5262/4 "2025-09-22T22:46:36Z")

</div>

A small note that there is now an environment module for `mppnccombine-fast` in `/g/data/vk83/modules`. We’ve yet to update all our configs to use the module, but could you instead use the following in your config:

```auto
modules:
    use:
        - /g/data/vk83/modules
    load:
        - model-tools/mppnccombine-fast/2025.07.000

collate:
  mpi: true
  walltime: 1:00:00
  mem: 30GB
  ncpus: 4
  queue: expresssr
  exe: mppnccombine-fast

```

---

<div class="post-metadata">

**Author:** ![mmr0](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/mmr0/32/2485_2.png) [@mmr0](https://forum.access-hive.org.au/u/mmr0)\
**Post date:** [22 September 2025 23:02 UTC](https://forum.access-hive.org.au/t/outputting-a-region-using-mom6-diag-table-rather-than-a-single-file-i-am-getting-many-one-per-processor/5262/5 "2025-09-22T23:02:40Z")

</div>

Thanks @aekiss @angus-g @dougiesquire for explaining what was happening! I will try this 😃

---

<div class="post-metadata">

**Author:** ![Lizzie](https://avatars.discourse-cdn.com/v4/letter/l/da6949/32.png) [@Lizzie](https://forum.access-hive.org.au/u/Lizzie)\
**Post date:** [29 October 2025 23:08 UTC](https://forum.access-hive.org.au/t/outputting-a-region-using-mom6-diag-table-rather-than-a-single-file-i-am-getting-many-one-per-processor/5262/6 "2025-10-29T23:08:52Z")

</div>

I think I have a related issue.

I have added the collate to my config file and it is working fine when;

1. I am outputting the data over the entire model domain, e.g. `“ocean_model_z”, “uo”, “uo”, “access-om3.mom6.h.z%4yr-%2mo”, “all”, “mean”, “none”, 2`

2. I am outputting the data over a very small region all within the same tile eg `“ocean_model_z”, “uo”, “uo”, “access-om3.mom6.h.nd.hourly.z%4yr-%2mo”, “all”, “mean”, “113.45 113.75 -22.55 -22.35 -1 -1”, 2`

However, if I have chosen a large subregion (one that covers multiple tiles but not all the tiles of the domain) eg `“ocean_model_z”, “uo”, “uo”, “access-om3.mom6.h.nd.hourly.z%4yr-%2mo”, “all”, “mean”, “111 124 -23 -13 -1 -1”, 2`

I get this error for the collate post processing. I think it because only some tiles are present. If I load individual tiles then the data in them looks fine, but I haven’t figured out how to combine the tiles into one file myself.

```auto
Loading access-om3/2025.08.001-tracers-from-file

  Loading requirement: access3/tracers-from-file-ycbexjc

Currently Loaded Modulefiles:

 1) access3/tracers-from-file-ycbexjc 5) openmpi/4.1.7-ushgfj4 

 2) access-om3/2025.08.001-tracers-from-file 6) pbs                   

 3) nco/5.0.5                                 

 4) model-tools/mppnccombine-fast/2025.07.000 

payu: error: Thread 0 crashed with error code 255.

 Error message:

 

Copying non-collated variables

 

Copying contiguous variables

 

Copying chunked variables

[rank 000] ERROR in HDF5 /scratch/tm70/tm70_ci/tmp/spack-stage/spack-stage-mppnccombine-fast-2025.07.000-6kyw6dwnfpnz3k74v3dkjmzkglpdxvq5/spack-src/async.c:490

 

HDF5-DIAG: Error detected in HDF5 (1.14.2) MPI-process 0:

  #000: /scratch/tm70/tm70_ci/tmp/spack-stage/spack-stage-hdf5-1.14.2-oum3uh3ht4on2t6x6wpicwhhvn3kciuj/spack-src/src/H5D.c line 403 in H5Dopen2(): unable to synchronously open dataset

    major: Dataset

    minor: Can't open object

  #001: /scratch/tm70/tm70_ci/tmp/spack-stage/spack-stage-hdf5-1.14.2-oum3uh3ht4on2t6x6wpicwhhvn3kciuj/spack-src/src/H5D.c line 364 in H5D__open_api_common(): unable to open dataset

    major: Dataset

    minor: Can't open object

  #002: /scratch/tm70/tm70_ci/tmp/spack-stage/spack-stage-hdf5-1.14.2-oum3uh3ht4on2t6x6wpicwhhvn3kciuj/spack-src/src/H5VLcallback.c line 1980 in H5VL_dataset_open(): dataset open failed

    major: Virtual Object Layer

    minor: Can't open object

  #003: /scratch/tm70/tm70_ci/tmp/spack-stage/spack-stage-hdf5-1.14.2-oum3uh3ht4on2t6x6wpicwhhvn3kciuj/spack-src/src/H5VLcallback.c line 1947 in H5VL__dataset_open(): dataset open failed

    major: Virtual Object Layer

    minor: Can't open object

  #004: /scratch/tm70/tm70_ci/tmp/spack-stage/spack-stage-hdf5-1.14.2-oum3uh3ht4on2t6x6wpicwhhvn3kciuj/spack-src/src/H5VLnative_dataset.c line 321 in H5VL__native_dataset_open(): unable to open dataset

    major: Dataset

    minor: Can't open object

  #005: /scratch/tm70/tm70_ci/tmp/spack-stage/spack-stage-hdf5-1.14.2-oum3uh3ht4on2t6x6wpicwhhvn3kciuj/spack-src/src/H5Dint.c line 1418 in H5D__open_name(): not found

    major: Dataset

    minor: Object not found

  #006: /scratch/tm70/tm70_ci/tmp/spack-stage/spack-stage-hdf5-1.14.2-oum3uh3ht4on2t6x6wpicwhhvn3kciuj/spack-src/src/H5Gloc.c line 421 in H5G_loc_find(): can't find object

    major: Symbol table

    minor: Object not found

  #007: /scratch/tm70/tm70_ci/tmp/spack-stage/spack-stage-hdf5-1.14.2-oum3uh3ht4on2t6x6wpicwhhvn3kciuj/spack-src/src/H5Gtraverse.c line 816 in H5G_traverse(): internal path traversal failed

    major: Symbol table

    minor: Object not found

  #008: /scratch/tm70/tm70_ci/tmp/spack-stage/spack-stage-hdf5-1.14.2-oum3uh3ht4on2t6x6wpicwhhvn3kciuj/spack-src/src/H5Gtraverse.c line 596 in H5G__traverse_real(): traversal operator failed

    major: Symbol table

    minor: Callback failed

  #009: /scratch/tm70/tm70_ci/tmp/spack-stage/spack-stage-hdf5-1.14.2-oum3uh3ht4on2t6x6wpicwhhvn3kciuj/spack-src/src/H5Gloc.c line 381 in H5G__loc_find_cb(): object 'xh_sub04' doesn't exist

    major: Symbol table

    minor: Object not found

/g/data/vk83/apps/spack/0.22/release/linux-rocky8-x86_64_v4/intel-2021.10.0/mppnccombine-fast-2025.07.000-6kyw6dwnfpnz3k74v3dkjmzkglpdxvq5/bin/mppnccombine-fast[0x40a841]

/g/data/vk83/apps/spack/0.22/release/linux-rocky8-x86_64_v4/intel-2021.10.0/mppnccombine-fast-2025.07.000-6kyw6dwnfpnz3k74v3dkjmzkglpdxvq5/bin/mppnccombine-fast[0x40a522]

/g/data/vk83/apps/spack/0.22/release/linux-rocky8-x86_64_v4/intel-2021.10.0/mppnccombine-fast-2025.07.000-6kyw6dwnfpnz3k74v3dkjmzkglpdxvq5/bin/mppnccombine-fast[0x405dff]

/half-root/usr/lib64/libc.so.6(__libc_start_main+0xe5)[0x148bb3cd07e5]

/g/data/vk83/apps/spack/0.22/release/linux-rocky8-x86_64_v4/intel-2021.10.0/mppnccombine-fast-2025.07.000-6kyw6dwnfpnz3k74v3dkjmzkglpdxvq5/bin/mppnccombine-fast[0x404c0e]

--------------------------------------------------------------------------

MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD

  Proc: [[26850,1],0]

  Errorcode: -1

 

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.

You may or may not see output from other processes, depending on

exactly when Open MPI kills them.

-------------------------------------------------------------------------- 

```

---

<div class="post-metadata">

**Author:** ![angus-g](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/angus-g/32/89_2.png) [@angus-g](https://forum.access-hive.org.au/u/angus-g)\
**Post date:** [31 October 2025 02:30 UTC](https://forum.access-hive.org.au/t/outputting-a-region-using-mom6-diag-table-rather-than-a-single-file-i-am-getting-many-one-per-processor/5262/7 "2025-10-31T02:30:41Z")

</div>

Thanks for the info Lizzie! I didn’t think it was an issue that tiles are missing, since that would happen on masked processors anyway. Although your subregion is somewhat more restricted than “most of the domain minus masking”, so perhaps something is going weird there. The errors being HDF5-related almost points to a file/filesystem issue? I wonder if the non-`fast` version of `mppnccombine` would treat the files any differently? I haven’t seen this specific error!

---

<div class="post-metadata">

**Author:** ![Aidan](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/aidan/32/42_2.png) [@Aidan](https://forum.access-hive.org.au/u/Aidan)\
**Post date:** [31 October 2025 03:15 UTC](https://forum.access-hive.org.au/t/outputting-a-region-using-mom6-diag-table-rather-than-a-single-file-i-am-getting-many-one-per-processor/5262/8 "2025-10-31T03:15:39Z")

</div>

> [@angus-g](#):
>
> I wonder if the non-`fast` version of `mppnccombine` would treat the files any differently?

I was thinking the same thing. Definitely worth trying.

See this recent post for how to do this:

> [@Cannot find mppncombine after compiling ACCESS-OM2-wombatlite](https://forum.access-hive.org.au/t/cannot-find-mppncombine-after-compiling-access-om2-wombatlite/5477/4):
>
> Off the shelf is fine. The ACCESS-OM3 configs give an example of how this might be done: But replace model-tools/mppnccombine-fast with fre-nctools/2024.05-1, e.g. modules: use: - /g/data/vk83/modules load: - fre-nctools/2024.05-1 And that should work out of the box. Please let us know if that didn’t fix your issue @raslekar.

---

<div class="post-metadata">

**Author:** ![Lizzie](https://avatars.discourse-cdn.com/v4/letter/l/da6949/32.png) [@Lizzie](https://forum.access-hive.org.au/u/Lizzie)\
**Post date:** [31 October 2025 03:43 UTC](https://forum.access-hive.org.au/t/outputting-a-region-using-mom6-diag-table-rather-than-a-single-file-i-am-getting-many-one-per-processor/5262/9 "2025-10-31T03:43:34Z")

</div>

Thanks @angus-g and @Aidan

I can give that a go.  
After  
`load: `  
`- fre-nctools/2024.05-1`  
Would I also need to change or delete this section?  
`collate:`  
`restart: true`  
`mpi: true`  
`walltime: 2:00:00`  
`mem: 30GB`  
`ncpus: 4`  
`queue: expresssr`  
`exe: mppnccombine-fast`

---

<div class="post-metadata">

**Author:** ![Aidan](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/aidan/32/42_2.png) [@Aidan](https://forum.access-hive.org.au/u/Aidan)\
**Post date:** [31 October 2025 04:16 UTC](https://forum.access-hive.org.au/t/outputting-a-region-using-mom6-diag-table-rather-than-a-single-file-i-am-getting-many-one-per-processor/5262/10 "2025-10-31T04:16:06Z")

</div>

> [@Lizzie](#):
>
> Would I also need to change or delete this section?

Yes.

You need to set `mpi: false` and change the name of `exe`, e.g.

```yaml
collate:
    exe: mppnccombine
    restart: true
    mpi: false
    walltime: 2:00:00
    mem: 30GB
    ncpus: 4
    queue: expresssr

```

You may need to fiddle with the memory and `ncpus ` settings depending on how much you’re collating.

The reason the high resolution models use `mppnccombine-fast` is because it is much faster, but also uses a lot less memory IIRC.

I would test this on just the subregional outputs in the first instance to see if it works. If it does then you might need fiddle to get something that works for production if you’re running a high resolution model over a large domain, e.g. don’t automatically collate but create a post-processing script that use `mppnccombine-fast` for the normal diagnostics, and `mppnccombine` for the sub-regional ones.

Note: you can specify a specific directory to collate  
e.g.

```auto
payu collate -d archive/restart000

```

---

<div class="post-metadata">

**Author:** ![Aidan](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/aidan/32/42_2.png) [@Aidan](https://forum.access-hive.org.au/u/Aidan)\
**Post date:** [31 October 2025 04:29 UTC](https://forum.access-hive.org.au/t/outputting-a-region-using-mom6-diag-table-rather-than-a-single-file-i-am-getting-many-one-per-processor/5262/11 "2025-10-31T04:29:22Z")

</div>

Also this related issue popped up .. would @angus-g fix work here too?

> [@Payu collate tiles](https://forum.access-hive.org.au/t/payu-collate-tiles/813/7):
>
> I’m not too across all the specifics, but I think there’s some kind of parallel gremlin lurking. I can reliably trigger the HDF5 abort by using 4 or 6 ranks, but I’ve been successful with 2, 3, 7, and 8. My guess is it’s some combination of the masking on tiles and the distribution of the files to process over the ranks? Will dig deeper to get a definitive answer though!

It is simple to specify a different number of cpus so maybe worth a try?

---

<div class="post-metadata">

**Author:** ![angus-g](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/angus-g/32/89_2.png) [@angus-g](https://forum.access-hive.org.au/u/angus-g)\
**Post date:** [31 October 2025 07:18 UTC](https://forum.access-hive.org.au/t/outputting-a-region-using-mom6-diag-table-rather-than-a-single-file-i-am-getting-many-one-per-processor/5262/12 "2025-10-31T07:18:02Z")

</div>

Could be worth trying at least. That error is the one that was nominally fixed by just bumping the HDF5 version.

---

<div class="post-metadata">

**Author:** ![Aidan](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/aidan/32/42_2.png) [@Aidan](https://forum.access-hive.org.au/u/Aidan)\
**Post date:** [2 November 2025 22:42 UTC](https://forum.access-hive.org.au/t/outputting-a-region-using-mom6-diag-table-rather-than-a-single-file-i-am-getting-many-one-per-processor/5262/13 "2025-11-02T22:42:43Z")

</div>

> [@angus-g](#):
>
> That error is the one that was nominally fixed by just bumping the HDF5 version.

Is this bug fix documented somewhere? I searched but couldn’t find anything.

---

<div class="post-metadata">

**Author:** ![angus-g](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/angus-g/32/89_2.png) [@angus-g](https://forum.access-hive.org.au/u/angus-g)\
**Post date:** [4 November 2025 23:08 UTC](https://forum.access-hive.org.au/t/outputting-a-region-using-mom6-diag-table-rather-than-a-single-file-i-am-getting-many-one-per-processor/5262/14 "2025-11-04T23:08:46Z")

</div>

There was a discussion about it [on Zulip](https://access-nri.zulipchat.com/#narrow/channel/470332-ocean-seaice/topic/Ice.20Shelf.20dev.20work/near/530251225) but because the solution just involved updating dependencies I don’t think it was documented otherwise…

---

<div class="post-metadata">

**Author:** ![Lizzie](https://avatars.discourse-cdn.com/v4/letter/l/da6949/32.png) [@Lizzie](https://forum.access-hive.org.au/u/Lizzie)\
**Post date:** [25 November 2025 04:46 UTC](https://forum.access-hive.org.au/t/outputting-a-region-using-mom6-diag-table-rather-than-a-single-file-i-am-getting-many-one-per-processor/5262/15 "2025-11-25T04:46:32Z")

</div>

I have tried both the non-fast and fast version of mppncombine with different number of CPUs. Changing the number of CPUs didn’t alter the error message for me.

When I used the non fast version, the error I got started like this, which is different to the error I got using the fast version.

```auto
payu: error: Thread 0 crashed with error code 9.
Error message:
ERROR: missing at least -m from the input file set. Exiting.

payu: error: Thread 1 crashed with error code 9.
Error message:
ERROR: missing at least -m from the input file set. Exiting.

payu: error: Thread 2 crashed with error code 9.
Error message:
ERROR: missing at least -m from the input file set. Exiting.

payu: error: Thread 3 crashed with error code 9.
Error message:
ERROR: missing at least -m from the input file set. Exiting.

payu: error: Thread 4 crashed with error code 9.
Error message:
ERROR: missing at least -m from the input file set. Exiting.

```

---

<div class="post-metadata">

**Author:** ![anton](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/anton/32/1836_2.png) [@anton](https://forum.access-hive.org.au/u/anton)\
**Post date:** [25 November 2025 04:56 UTC](https://forum.access-hive.org.au/t/outputting-a-region-using-mom6-diag-table-rather-than-a-single-file-i-am-getting-many-one-per-processor/5262/16 "2025-11-25T04:56:24Z")

</div>

Hi Lizze, I think

> [@Cannot find mppncombine after compiling ACCESS-OM2-wombatlite](https://forum.access-hive.org.au/t/cannot-find-mppncombine-after-compiling-access-om2-wombatlite/5477/12):
>
> When @raslekar ran this, there was an error saying -z was not found in the input files. In the end, we needed to set the collate flags through the payu config as well. In config.yaml, under the collate: section, we needed to set: flags: -n4 -m -r​ which removes the -z flag which payu includes [by default](https://github.com/payu-org/payu/blob/46fb4ac890cee6b10156d853a9f4ed11322f2ed1/payu/models/fms.py#L104)

might solve the problem with the non-fast version (mppnccombine)

---

<div class="post-metadata">

**Author:** ![Lizzie](https://avatars.discourse-cdn.com/v4/letter/l/da6949/32.png) [@Lizzie](https://forum.access-hive.org.au/u/Lizzie)\
**Post date:** [25 November 2025 07:08 UTC](https://forum.access-hive.org.au/t/outputting-a-region-using-mom6-diag-table-rather-than-a-single-file-i-am-getting-many-one-per-processor/5262/17 "2025-11-25T07:08:26Z")

</div>

Thanks Anton

That seems to give a new error!

`payu: error: Thread 0 crashed with error code -6.`  
`Error message:`  
`[gadi-cpu-spr-0182:952508:0:952508] Caught signal 11 (Segmentation fault: Sent by the kernel at address (nil))`  
`corrupted size vs. prev_size`

---

<div class="post-metadata">

**Author:** ![dougiesquire](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/dougiesquire/32/16_2.png) [@dougiesquire](https://forum.access-hive.org.au/u/dougiesquire)\
**Post date:** [27 November 2025 02:04 UTC](https://forum.access-hive.org.au/t/outputting-a-region-using-mom6-diag-table-rather-than-a-single-file-i-am-getting-many-one-per-processor/5262/18 "2025-11-27T02:04:51Z")

</div>

Thanks for sharing your files that cause the collation error @Lizzie. I am able to collate the single-variable file with `mppnccombine`, but I get the following for the mutli-variable file:

```auto
[gadi-login-04:1395861:0:1395861] Caught signal 11 (Segmentation fault: Sent by the kernel at address (nil))
BFD: Dwarf Error: Invalid abstract instance DIE ref.
BFD: Dwarf Error: Invalid abstract instance DIE ref.
BFD: Dwarf Error: Invalid abstract instance DIE ref.
BFD: Dwarf Error: Invalid abstract instance DIE ref.
BFD: Dwarf Error: Invalid abstract instance DIE ref.
BFD: Dwarf Error: Invalid abstract instance DIE ref.
BFD: Dwarf Error: Invalid abstract instance DIE ref.
BFD: Dwarf Error: Invalid abstract instance DIE ref.
BFD: Dwarf Error: Invalid abstract instance DIE ref.
BFD: Dwarf Error: Invalid abstract instance DIE ref.
BFD: Dwarf Error: Invalid abstract instance DIE ref.
==== backtrace (tid:1395861) ====
 0 0x000000000004e5b0 killpg() ???:0
 1 0x00000000001aeb81 H5FL_blk_malloc() ???:0
 2 0x00000000000930b9 H5B__cache_deserialize() H5Bcache.c:0
 3 0x00000000000af94c H5C__load_entry() H5Centry.c:0
 4 0x00000000000ae22b H5C_protect() ???:0
 5 0x0000000000089527 H5AC_protect() ???:0
 6 0x000000000008dd5b H5B_find() ???:0
 7 0x00000000000d5308 H5D__btree_idx_get_addr() H5Dbtree.c:0
 8 0x00000000000e625a H5D__chunk_lookup() ???:0
 9 0x00000000000dc5a2 H5D__chunk_read() H5Dchunk.c:0
10 0x000000000010c3c7 H5D__read() ???:0
11 0x00000000003d702f H5VL__native_dataset_read() ???:0
12 0x00000000003c0980 H5VL_dataset_read_direct() ???:0
13 0x00000000000cdbf4 H5Dread() ???:0
14 0x00000000000c2a2e NC4_get_vars() ???:0
15 0x00000000000c1c26 NC4_get_vara() ???:0
16 0x0000000000032a7f NC_get_vara() ???:0
17 0x0000000000030949 nc_get_vara() ???:0
18 0x0000000000029a36 ncvarget() ???:0
19 0x0000000000406ce7 process_vars() /scratch/tm70/tm70_ci/tmp/spack-stage/spack-stage-fre-nctools-2024.05-1-b7s7yifeqhlj3ydibuwfp3ee6zgag2vl/spack-src/src/mpp-nccombine/mppnccombine.c:1478
20 0x0000000000405d0d process_file() /scratch/tm70/tm70_ci/tmp/spack-stage/spack-stage-fre-nctools-2024.05-1-b7s7yifeqhlj3ydibuwfp3ee6zgag2vl/spack-src/src/mpp-nccombine/mppnccombine.c:1251
21 0x00000000004044e7 main() /scratch/tm70/tm70_ci/tmp/spack-stage/spack-stage-fre-nctools-2024.05-1-b7s7yifeqhlj3ydibuwfp3ee6zgag2vl/spack-src/src/mpp-nccombine/mppnccombine.c:698
22 0x000000000003a7e5 __libc_start_main() ???:0
23 0x000000000040388e _start() ???:0
=================================
Segmentation fault

```

I just wanted to check that was also your experience because I don’t recall you menioning that some were successful.

---

<div class="post-metadata">

**Author:** ![Lizzie](https://avatars.discourse-cdn.com/v4/letter/l/da6949/32.png) [@Lizzie](https://forum.access-hive.org.au/u/Lizzie)\
**Post date:** [27 November 2025 03:34 UTC](https://forum.access-hive.org.au/t/outputting-a-region-using-mom6-diag-table-rather-than-a-single-file-i-am-getting-many-one-per-processor/5262/19 "2025-11-27T03:34:53Z")

</div>

Hi Dougie, thanks for looking.

No I didn’t even get the single variable mld file to collate and I haven’t seen that error message before!

to use mppnccombine I set up like this, but I tried with and without the flags and with some difference ncpus.

```auto
modules:
use:
- /g/data/vk83/modules
- /g/data/x77/ahg157/modules
load:

    - access-om3/2025.08.001-tracers-from-file 
    - nco/5.0.5
    - fre-nctools/2024.05-1

collate:
flags: -n4 -m -r
exe: mppnccombine
restart: true
mpi: false
walltime: 0:30:00
mem: 30GB
ncpus: 1
queue: expresssr

```

---

<div class="post-metadata">

**Author:** ![dougiesquire](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/dougiesquire/32/16_2.png) [@dougiesquire](https://forum.access-hive.org.au/u/dougiesquire)\
**Post date:** [1 December 2025 03:58 UTC](https://forum.access-hive.org.au/t/outputting-a-region-using-mom6-diag-table-rather-than-a-single-file-i-am-getting-many-one-per-processor/5262/20 "2025-12-01T03:58:29Z")

</div>

Getting a bit desperate, but can you avoid the error by modifying the MOM `LAYOUT`? I think I recall you are using `AUTO_MASKTABLE = True` so you would need to turn that off and follow [these instructions](https://github.com/COSIMA/mom6-panan/wiki/Preparing-inputs-for-a-new-configuration#generating-the-computational-mask) to generate the computational mask.

The `check_mask` tool is available in the `fre-nctools` environment module:

```auto
$ module use /g/data/vk83/modules
$ module load fre-nctools/2024.05-1
$ which check_mask
/g/data/vk83/apps/spack/0.22/release/linux-rocky8-x86_64_v4/intel-2021.10.0/fre-nctools-2024.05-1-b7s7yifeqhlj3ydibuwfp3ee6zgag2vl/bin/check_mask

```

It could also be worth [opening an issue](https://github.com/NOAA-GFDL/FRE-NCtools/issues) with the people who write `mppnccombine` to see if they have any suggestions.

[Next page](https://forum.access-hive.org.au/t/outputting-a-region-using-mom6-diag-table-rather-than-a-single-file-i-am-getting-many-one-per-processor/5262.md?page=2)
