# Intake (bug?) **URL:** \ **Category:** Technical\ **Tags:** intake\ **Created:** [14 October 2025 05:54 UTC](https://forum.access-hive.org.au/t/intake-bug/5398 "2025-10-14T05:54:16Z")\ **Posts on this page:** 19\ **Page:** 1
**Author:** ![JuliaN](https://avatars.discourse-cdn.com/v4/letter/j/74df32/32.png) [@JuliaN](https://forum.access-hive.org.au/u/JuliaN)\ **Post date:** [14 October 2025 05:54 UTC](https://forum.access-hive.org.au/t/intake-bug/5398/1 "2025-10-14T05:54:16Z")
Hi! I’ve run into a silent problem in intake. I was trying to load a variable, `v`, which in the experiment I was using, comes with 2 different “cell methods”: `['time: mean', 'time: mean_pow(02)']` I was not aware and intake did not prompt me with a warning or error, it loaded it no problem and averaged them all together. The result was a very weird looking diagnostic - I noticed it was not OK, but the student I was working with had not. It would be great if intake failed and prompted an error instead of collating diagnostics without the user knowing. Or maybe a warning? “Careful, you are averaging X number of variables” Minimal example here: [intake\_problem.ipynb · GitHub](https://gist.github.com/julia-neme/8f3db08c72760205ed9aa0ff77470ff8) ---
**Author:** ![adele-morrison](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/adele-morrison/32/90_2.png) [@adele-morrison](https://forum.access-hive.org.au/u/adele-morrison)\ **Post date:** [14 October 2025 06:07 UTC](https://forum.access-hive.org.au/t/intake-bug/5398/2 "2025-10-14T06:07:23Z")
This is very scary. ---
**Author:** ![CharlesTurner](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/charlesturner/32/2518_2.png) [@CharlesTurner](https://forum.access-hive.org.au/u/CharlesTurner)\ **Post date:** [14 October 2025 06:43 UTC](https://forum.access-hive.org.au/t/intake-bug/5398/3 "2025-10-14T06:43:16Z")
---
**Author:** ![CharlesTurner](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/charlesturner/32/2518_2.png) [@CharlesTurner](https://forum.access-hive.org.au/u/CharlesTurner)\ **Post date:** [14 October 2025 06:46 UTC](https://forum.access-hive.org.au/t/intake-bug/5398/4 "2025-10-14T06:46:00Z")
I’ll get right on this and get a fix out ASAP. Thanks for the very clear example notebook. In this case, I think we can solve the problem by forcing intake to disambiguate on cell methods. Do you have any idea whether that is going to work in general - eg. do we reliably know that if two files contain: 1. The same variable - `v` in this instance 2. On the same grid 3. But with different cell methods: `['time: mean', 'time: mean_pow(02)']` in this instance, That they are in fact, and always will be (or should at least default to being) different variables? I think this should be the case, right? ---
**Author:** ![JuliaN](https://avatars.discourse-cdn.com/v4/letter/j/74df32/32.png) [@JuliaN](https://forum.access-hive.org.au/u/JuliaN)\ **Post date:** [14 October 2025 22:47 UTC](https://forum.access-hive.org.au/t/intake-bug/5398/5 "2025-10-14T22:47:06Z")
Thank you Charles!! In this case I reliably know it is the same variable in the same grid. There’s just one cell\_id, one `v`in the variable list and 3 (?) cell methods actually. One of them is just empty `’ ‘`and I have no idea what it means. ![image](https://us1.discourse-cdn.com/flex020/uploads/access1/original/2X/7/77727832e1b6bc1a7d8e6308ae94028b95ca16ca.png) I am trying to think if there would be any exceptions for this, but I don’t know. There’s another possible (but unlikely) case that comes to mind that is for the time dimension. It is possible to output diagnostics as an average in time or as a snapshot. If you happened to have a variable that has daily mean and daily snapshot available, I don’t think the `1day`argument would distinguish them? Having said that, I am not aware of any simulations right now that have such diagnostics outputted. But it could happen in the future. ---
**Author:** ![CharlesTurner](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/charlesturner/32/2518_2.png) [@CharlesTurner](https://forum.access-hive.org.au/u/CharlesTurner)\ **Post date:** [15 October 2025 00:12 UTC](https://forum.access-hive.org.au/t/intake-bug/5398/6 "2025-10-15T00:12:01Z")
So the way that the `Builder` objects that build these ESM-Datastores work is that they open every file in the directory, and extract various metadata needed for cataloguing it - all the fields you see in the datastore, basically. For `variable_cell_methods`, (and all the other `variable_xyz` fields) , these are extracted by looping over all the variables in the files & extracting their attributes. So, for eg. v `variable_cell_methods = [ds['v'].attrs.get("cell_methods", "")`. The `.get("cell_methods","")` means that if no cell methods are found for that variable, an empty string will be used as an empty value - which is where that is coming from. Potentially it would be better to just remove it. I’m not sure I asked exactly what I was trying to, reading it back. The complication is that I don’t think it’s straightforward to disambiguate based on variable cell methods - imagine the following scenario: 1. Imagine we have three files: the first with `v_mean` and `v_max`, the second with `v_max` and the third with `v_mean` (this is going to seem a bit contrived, but stay with me). 2. All are on the same grid. 3. File 1 will have `variable_cell_methods = ['time: mean','time: max']` ; file 2 `variable_cell_methods = ['time: max']`; file 3 `variable_cell_methods = ['time': mean']`. In this scenario, if we try to disambiguate on `variable_cell_methods` - that is, split up what we consider a dataset based on that - we will wind up telling `intake-esm` that none of these files can be combined. I think we would actually want to be able to combine 1 & 2 or 1 & 3, but not 2 & 3. Arguably this is the safest way of handling things, but it might also a become a bit inconvenient. I was looking through the full catalog & I found a bunch of datasets where it looks like cell methods only on differ things like `nv` and `rho2`. For eg. an Eulerian velocity field, we probably wouldn’t want to differentiate into datasets based on whether one file contains a maximum on density surfaces and another doesn’t? I _thiiink_ the behaviour that we want is something like: - If a specific variable has been searched for (ie. `esm_datastore.search(variable = 'xyz').to_dask()`), then make sure that for each file that we open in that dataset, we ensure all the variables in the dataset have compatible cell methods? Intake-ESM already has some code that handles this sort of stuff & we could extend that - if you search for variable `x` it will know to only put `x` (and necessary coordinate variables) into the dataset it returns from `to_dask`. ---
**Author:** ![adele-morrison](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/adele-morrison/32/90_2.png) [@adele-morrison](https://forum.access-hive.org.au/u/adele-morrison)\ **Post date:** [16 October 2025 04:30 UTC](https://forum.access-hive.org.au/t/intake-bug/5398/7 "2025-10-16T04:30:52Z")
That sounds good to me Charles. ---
**Author:** ![CharlesTurner](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/charlesturner/32/2518_2.png) [@CharlesTurner](https://forum.access-hive.org.au/u/CharlesTurner)\ **Post date:** [16 October 2025 04:43 UTC](https://forum.access-hive.org.au/t/intake-bug/5398/8 "2025-10-16T04:43:39Z")
Once I have a working implementation of this (turns out to irritatingly complicated to implement this) I’ll generate a demo of the updated behaviour & give you a ping to check it’s exactly what it should be if thats okay! ---
**Author:** ![JuliaN](https://avatars.discourse-cdn.com/v4/letter/j/74df32/32.png) [@JuliaN](https://forum.access-hive.org.au/u/JuliaN)\ **Post date:** [19 October 2025 23:29 UTC](https://forum.access-hive.org.au/t/intake-bug/5398/9 "2025-10-19T23:29:00Z")
Thank you Charles! Sorry I missed the messages above! ---
**Author:** ![CharlesTurner](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/charlesturner/32/2518_2.png) [@CharlesTurner](https://forum.access-hive.org.au/u/CharlesTurner)\ **Post date:** [21 October 2025 00:25 UTC](https://forum.access-hive.org.au/t/intake-bug/5398/10 "2025-10-21T00:25:35Z")
Sorry this is taking a while! I’ve rebuilt a catalog against a fork with the necessary changes here, which basically ‘explodes’ iterable columns (somewhat amusingly, that’s actually a technical term). The TLDR; is that previously, all cell methods would be attached to the **single netCDF file entry** in the catalog - this made searching on specific cell methods a bit of a faff, and if eg. `u` had `time: mean` and `v` had `time: mean_pow(02)` in the same file, there would be no way of separating the two out - because they’re attached to the same entry. Now we’ve exploded out all iterables, we have one entry per variable in each catalog, which makes the catalogs look a bit bigger, but just splits the data up in a bit more of a fine grained way. I’ve looked into changing the way that datasets are grouped, and I’m not sure it’s going to make sense to force all datasets to have the same `variable_cell_methods` as it will split up a bunch of datasets that should be together - inour tests, it seems that adding that split datastores up to be ~ 1 dataset per file - which is no real good to anyone. Due to the way our test data is constructed, the only way to really find out whether this is in fact going to be a major issue is to build a full catalog & check - which is ongoing. I’m hoping the test data is misleadingly making it look like more of an issue than it really is, as they’re designed to have minimal duplication, so it’s plausible it’s just the tests making things look bad. If it turns out that it does split things too much, we’ll have to update intake-esm to warn if multiple cell methods are merged. I’ll update when I have all the necessary rebuilt catalogs to assess what the best path forward is! ---
**Author:** ![JuliaN](https://avatars.discourse-cdn.com/v4/letter/j/74df32/32.png) [@JuliaN](https://forum.access-hive.org.au/u/JuliaN)\ **Post date:** [21 October 2025 03:19 UTC](https://forum.access-hive.org.au/t/intake-bug/5398/11 "2025-10-21T03:19:35Z")
I think this is above my understanding capacities these days 😅 but a solution underway sounds awesome! Thank you ---
**Author:** ![CharlesTurner](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/charlesturner/32/2518_2.png) [@CharlesTurner](https://forum.access-hive.org.au/u/CharlesTurner)\ **Post date:** [27 October 2025 00:05 UTC](https://forum.access-hive.org.au/t/intake-bug/5398/12 "2025-10-27T00:05:54Z")
Sorry, have replied in there! Fix will be in ASAP, workaround in the other thread. ---
**Author:** ![adele-morrison](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/adele-morrison/32/90_2.png) [@adele-morrison](https://forum.access-hive.org.au/u/adele-morrison)\ **Post date:** [4 December 2025 05:22 UTC](https://forum.access-hive.org.au/t/intake-bug/5398/13 "2025-12-04T05:22:08Z")
Hey @CharlesTurner, any progress on this? Others are running into the same issue: > [@"U" velocity incorrect when using intake in IAF runs](https://forum.access-hive.org.au/t/u-velocity-incorrect-when-using-intake-in-iaf-runs/5657/): > > Hi, Me and @Wilton_Aguiar stumbled on an error in how intake loads the uvariable in the IAF run. If we print the files it is pulling ufrom: df = cat[expt].search(variable='u', frequency='1mon').df print(df["path"]) we get: 0 /g/data/cj50/access-om2/raw-output/access-om2-01/01deg\_jra55v140\_iaf\_cycle4/output732/ocean/ocean-3d-u-1-monthly-mean-ym\_1958\_01.nc 1 /g/data/cj50/access-om2/raw-output/access-om2-01/01deg\_jra55v140\_iaf\_cycle4/output732/ocean/ocean-3d-u-1-monthly-pow02-ym\_19… Thanks! ---
**Author:** ![CharlesTurner](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/charlesturner/32/2518_2.png) [@CharlesTurner](https://forum.access-hive.org.au/u/CharlesTurner)\ **Post date:** [8 December 2025 21:18 UTC](https://forum.access-hive.org.au/t/intake-bug/5398/14 "2025-12-08T21:18:33Z")
Just to keep everyone in the loop as I’ve let this drag on far too long - I’m actively working on this right now (and it’ll stay top priority until it’s fixed). It’s got some weird & finnicky edge cases (I’d gotten a bit sidelined with issues related to these) that are taking me much longer than I’d hoped to clean up. Fingers crossed I’ll get it worked out this week. ---
**Author:** ![CharlesTurner](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/charlesturner/32/2518_2.png) [@CharlesTurner](https://forum.access-hive.org.au/u/CharlesTurner)\ **Post date:** [17 December 2025 00:14 UTC](https://forum.access-hive.org.au/t/intake-bug/5398/15 "2025-12-17T00:14:12Z")
I think I finally have a solid fix for this. I’m running a few final tests today, hopefully the catalog will be updated with the fix later today or earlier tomorrow. ---
**Author:** ![CharlesTurner](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/charlesturner/32/2518_2.png) [@CharlesTurner](https://forum.access-hive.org.au/u/CharlesTurner)\ **Post date:** [17 December 2025 03:01 UTC](https://forum.access-hive.org.au/t/intake-bug/5398/16 "2025-12-17T03:01:02Z")
Okay, it looks like the fix works. ESM Datastores will now have a `temporal_label` field which aggregates the time aggregations that `variable_cell_methods` contains into a single descriptor. I’ve updated the way that the catalogs specify what comprises a dataset such that there if is more than one `temporal_label`, intake will know it’s not a single dataset. This will change the searches necessary to get a single dataset out in some instances (hopefully not too many) - so there’ll potentially be some updating to do through cosima recipes, etc, which I’ll start working through. Again, sorry that took so long - turned out to be a surprisingly thorny issue. The updated catalog is currently building, and I’ll post a final update in here once it’s finished. ---
**Author:** ![adele-morrison](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/adele-morrison/32/90_2.png) [@adele-morrison](https://forum.access-hive.org.au/u/adele-morrison)\ **Post date:** [17 December 2025 03:46 UTC](https://forum.access-hive.org.au/t/intake-bug/5398/17 "2025-12-17T03:46:35Z")
Amazing, thanks @CharlesTurner ! ---
**Author:** ![CharlesTurner](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/charlesturner/32/2518_2.png) [@CharlesTurner](https://forum.access-hive.org.au/u/CharlesTurner)\ **Post date:** [17 December 2025 05:18 UTC](https://forum.access-hive.org.au/t/intake-bug/5398/18 "2025-12-17T05:18:02Z")
Okay, the default catalog should now take care of this. Please let me know if there are any unexpected side effects - this is sort of change that’s liable to create workflow changes. ---
**Author:** ![CharlesTurner](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.access-hive.org.au/charlesturner/32/2518_2.png) [@CharlesTurner](https://forum.access-hive.org.au/u/CharlesTurner)\ **Post date:** [18 March 2026 00:46 UTC](https://forum.access-hive.org.au/t/intake-bug/5398/19 "2026-03-18T00:46:54Z")