Adding calculated values to xarray DataArray takes too long

[kathy.gunn]

I am doing a calculation (aren’t we all!) which does not take long. However, when I try to add the calculated values to an empty DataArray, or numpy ndarray, it takes forever. I suspect that this issue is something to do with the different data types, and my unfamiliarity with xarray. Please see screenshot below.

So, my question is: how do I quickly add calculated values to an empty xarray DataArray?

P.S. I want to keep this calculation in a for loop for readability, but I could just do this in a few one liners.

Thank you, Kathy

[Aidan (Aidan Heerdegen, ACCESS-NRI Release Team Lead)]

Hey @kathy.gunn. This isn’t an answer, but just a suggestion that you can add tags to your posts to make them easier to find. In this case python, xarray and help would be good candidates.

[kathy.gunn]

Hi Aidan. Yes, I tried to but was not able to add a tag that did not already exist. (or perhaps I am missing something). Thanks, Kathy

[tammasloughran (Tammas)]

Hi,

it looks like you’re trying to do a diff operation.
In xarray you should use this: xarray.DataArray.diff
In numpy you should use this: numpy.diff — NumPy v1.23 Manual
You’re also casting from xarray to a numpy array, so there is probably some overhead there, especially if the xarray is chunked (xarrays are really just dictionaries containing numpy arrays, so it would need to traverse the dictionary, concatenate the numpy arrays and then copy everything over in the assignment). You should either do everything in numpy arrays or do everything in xarrays and avoid mixing. If your data is big, stick with xarray.

So it should look something like this:

d_yr = np.diff(decimal_years)
T_n = gridded_temperature_gamma_space.diff('time') / d_yr

Cheers,
Tam

[tammasloughran (Tammas)]

Also, xarray does calculations lazily. This means that a list of operations on the data is created at runtime, but it doesn’t actually do the operations. The operations are executed only when they are explicitly asked for. So when you execute the assignment, it does all of the operations needed to get that point. I know you said you like having the loop for readability, but I think it’s the cause of your problem.

[Aidan (Aidan Heerdegen, ACCESS-NRI Release Team Lead)]

@kathy.gunn

Yes, I tried to but was not able to add a tag that did not already exist. (or perhaps I am missing something)

Hi Kathy, no it was me who was wrong. The forum software (discourse) limits who can create tags to “Trust Level 2”, which you get to by being an awesome forum poster. I can sense you’re going to be an awesome forum poster, so I’ve set you to Trust Level 2, so you should be able to create tags now.

[kathy.gunn]

@tammasloughran Thank you for explaining! I agree about the one-liners, despite liking the readability.