I am doing a calculation (aren’t we all!) which does not take long. However, when I try to add the calculated values to an empty DataArray, or numpy ndarray, it takes forever. I suspect that this issue is something to do with the different data types, and my unfamiliarity with xarray. Please see screenshot below.
So, my question is: how do I quickly add calculated values to an empty xarray DataArray?
P.S. I want to keep this calculation in a for loop for readability, but I could just do this in a few one liners.
it looks like you’re trying to do a diff operation.
In xarray you should use this: xarray.DataArray.diff
In numpy you should use this: numpy.diff — NumPy v1.23 Manual
You’re also casting from xarray to a numpy array, so there is probably some overhead there, especially if the xarray is chunked (xarrays are really just dictionaries containing numpy arrays, so it would need to traverse the dictionary, concatenate the numpy arrays and then copy everything over in the assignment). You should either do everything in numpy arrays or do everything in xarrays and avoid mixing. If your data is big, stick with xarray.
Also, xarray does calculations lazily. This means that a list of operations on the data is created at runtime, but it doesn’t actually do the operations. The operations are executed only when they are explicitly asked for. So when you execute the assignment, it does all of the operations needed to get that point. I know you said you like having the loop for readability, but I think it’s the cause of your problem.
Hi Kathy, no it was me who was wrong. The forum software (discourse) limits who can create tags to “Trust Level 2”, which you get to by being an awesome forum poster. I can sense you’re going to be an awesome forum poster, so I’ve set you to Trust Level 2, so you should be able to create tags now.