Over the last week or so I’ve been getting a lot of these errors:
RuntimeError: NetCDF: Not a valid ID
while running jupyter notebooks and using the COSIMA cookbook for loading data. They’re somewhat random, because sometimes the error is just a warning and it still calculates fine, and other times it’s fatal. And I don’t get the error repeatably. There is an example script with both warning and fatal types of these errors here.
I’m pretty sure this exact script was working fine a couple of weeks ago and I don’t think I’ve changed much since then.
I could well be wrong, but this is possibly due to multiple threads trying to read data from the same file at the same time.
Two things you could try:
Set up your dask cluster to have one thread per process. In your case: client = Client(n_workers=28)
Specify the chunking when you first query the data to be one chunk per file, though this could result in very large chunks. You can pass a chunks dictionary to cc.querying.getvar()
Thanks @dougiesquire. I agree it does sound like something like that. @angus-g@micael , has anything changed in the COSIMA cookbook recently that might be causing this behaviour?
Nope, more likely an xarray change. Kind of looks to me like something is trying to cache the files, but a worker dies so the cache is stale and the file handle is no longer valid? Can’t really say for sure.
I did a bit more testing, and found it only occurs with analysis3-22.10 and later. With analysis3-22.07 and earlier I don’t get these errors. It seems like this is going to be quite problematic going forward for many of our current analyses?
The last analysis3-22.10 update was on the 24th of January, quite a few packages that may or may not be relevant to this error were updated, including xarray, dask, distributed, netcdf4 python and hdf5. I’ve been scanning through issues in a few of those repos and there isn’t anything that really matches this. We’ve had another netCDF issue that’s come up in the last few days over on CWSHELP that may also be related. Have you tried @dougiesquire’s idea? I had a go at reproducing this error and found that everything worked as expected when I used client = Client(n_workers=28). This is problematic though, the default should just work. I’ll see if reverting xarray in a test environment helps.
In the cosima recipes examples we’ve been removing the n_workers keyword from the dask client so they work regardless how many cores the users fire up. How can we remedy this?
I thought the default for Client was processes=True, in which case I’m not sure @dougiesquire’s suggestion is doing anything beyond the default. Alternatively there’s threads_per_worker which you could set to 1 for the same effect?
That’s right, but I think that by default dask will choose the number of processes to be ~sqrt(cores). E.g. in @adele157’s linked notebook the cluster comprises 7 workers, each with 4 threads. I don’t use LocalCluster that much - is this maybe a new default behaviour?
@angus-g every test I did, if I didn’t specify n_workers it would give me 7 processes each with 4 threads on a full broadwell node. Reverting xarray to 2021.12.0 did seem to remedy the problem with @adele157’s example notebook without having to specify n_workers. I might revert xarray in analysis3-unstable while we investigate further. It seems strange to me given the huge userbase of dask + xarray + netcdf that nothing along these lines has been reported anywhere else.
Oh, I think I saw this type of errors when testing the notebooks from the COSIMA recipes. So maybe some of the notebooks that I marked as not working are actually fine. I’ll rerun them and double check.
Nevertheless, this might be problematic for the automatic tests, as they use analysis3 and we might get random failures.
Interesting! It looks like it comes from nprocesses_nthreads(). Well, for a method that is still general for CPU counts we could just do client = Client(threads_per_worker=1).
xarray has now been reverted to 2022.12.0 in analysis3-unstable. I’ve run through @adele157’s example notebook a few times now with no errors. Whilst its not a fix, it’ll hopefully get things working until an actual fix can be put in place.
I’m not convinced that this issue is related to xarray version. I created two conda environments that differ only in their version of xarray: one with v2022.12.0 and one with v2023.01.0. Adele’s warnings/errors are intermittently produced using both:
@adele157 does the reversion of the analysis3 environment seem to have fixed your problem? @dale.roberts did any other dependencies change when you reverted xarray?
So of the relevant looking stuff there, netcdf was reverted to an earlier version, and dask was updated. I’ve run @adele157’s case a few times in the current analysis environment and I didn’t get those errors. I’ve kept a copy of the previous environment, and can confirm those errors are occurring there.
