IOS Client memory cap exceeded - UM forecast fails

Atmosphere Unified Model
1

So I’m running the Centre’s rAM3 experiment (u-by395/ram3_flagship) and I’ve been asked to add some extra STASH outputs for another experiment.

The UM now fails with


????????????????????????????????????????????????????????????????????????????????
???!!!???!!!???!!!???!!!???!!!       ERROR        ???!!!???!!!???!!!???!!!???!!!
?  Error code: 99
?  Error from routine: IOS_CONSUME_CLIENT_MEM
?  Error message: IOS Client memory cap (   104857600) is too small for payload (  1024000000)
?  Error from processor: 0
?  Error number: 63
????????????????????????????????????????????????????????????????????????????????

I’ve had a guess this is related to the IO server, so I’ve updated the size of
ios_buffer_size=50000
to
ios_buffer_size=100000
in the app/um/rose-app.conf namelist section that controls the IO server, but that hasn’t fixed it.

Has anyone else found this problem before?

The internet is coming up blanks.

2

Asking copilot :robot:

What would cause the error message 'IOS Client memory cap ( 104857600) is too small for payload ( 1024000000)'

I get:

This error occurs in the IOS (IO Server) subsystem of the UM when trying to allocate memory for a data operation. Here’s what causes it:

Root Cause

The error happens when a single payload/operation tries to use more memory than the configured memory cap limit. In your case:

  • Memory cap: 104,857,600 bytes = 100 MB (set via IOS_concurrency_max_mem parameter)
  • Requested payload: 1,024,000,000 bytes ≈ 976.6 MB

The payload is nearly 10 times larger than the configured limit.

Why This Occurs

In ios_client_queue.F90, the IOS_consume_client_mem subroutine validates that each data operation doesn’t exceed the memory cap:

IF (mem > IOS_concurrency_max_mem) THEN
  ! Error: single payload exceeds the memory cap
END IF

The IOS_concurrency_max_mem is configured as a namelist parameter (in MB) and converted to bytes in ios_init.F90:

IOS_Concurrency_max_mem = IOS_Concurrency_max_mem * 1024 * 1024

Solutions

  1. Increase the memory cap in your rose configuration or namelist by setting IOS_concurrency_max_mem to a larger value (in MB). For a 1 GB payload, you’d need at least 1024 MB.

  2. Reduce the payload size by splitting large writes into multiple operations or by using different packing/compression settings.

  3. Check if the data being written is unexpectedly large — verify that your output configuration isn’t writing more fields/levels than intended.

3

Thanks Scott. I tried Gemini but it gave nonsense.

The current setting is

ios_concurrency_max_mem=100

I changed that to

ios_concurrency_max_mem=1024

And it’s worked!

4

I also just had this issue, but solved it in a slightly different way.

Increasing the ios_concurrency_max_mem is valid, but I got around it by reducing the STASH stream reinit_step from the default 6 (output every 6 hours) to 1 (output every hour). Paul only just pointed me to this thread, so I’d thought I’d add an alternative solution.

Essentially the I/O was getting overloaded with the amount of data in the stream when saving out the file, so either increasing the memory or reducing the amount of data work.