Skip to content

Conversation

@aulemahal
Copy link
Collaborator

For some reason, two issues popped up in the conda feedstock build that did not appear in the builds here...
See conda-forge/xesmf-feedstock#51

  1. Scipy is not a requirement of sparse anymore, so add_nans_to_weights failed without it. I rewrote the function without the sparse conversion, preserving the COO object type. I didn't perform a real performance test, but my small test seems to indicate that this new version is faster!

  2. For some reason, the parallel regridder test failed with xarray 2025.6.1 on the CI. I don't remember it failing here and the upstream run seems to work fine. However, the issue looks real : xr.map_blocks requires that all xarray objects it receive can be aligned. We already had code to manage the spatial dimensions, but not really the others coords. In fact, I don't understand why it worked as the input and output dataset both had "lon_bounds" variables with different values.
    This PR goes in more depth in renaming the coordinates of the datasets as so to avoid alignment issues.

Co-authored-by: David Huard <[email protected]>
@huard
Copy link
Contributor

huard commented Nov 25, 2025

Not sure though I can comment on the change in sparse format and its impact on computations.

@aulemahal
Copy link
Collaborator Author

aulemahal commented Nov 25, 2025

Oh boy...

I compared my new code with the old one by creating a large regridder with unmapped_to_nan=False and then comparing the initial weights to the ones passed through the old and new version of add_nans_to_weights.

Well. They are different. But, the old ones seem incorrect. I tried the roundtrip sparse -> scipy -> sparse without touching anything and it seems it is not a completely reversible process:

import xesmf as xe
import sparse as sps

reg = xe.Regridder(ds1, ds2, 'bilinear', unmapped_to_nan=False)

w2 = sps.COO.from_scipy_sparse(reg.weights.data.to_scipy_sparse().tolil())

np.testing.assert_array_equal(reg.weights.data.data, w2.data)
# Fails, shapes mismatch

@raphaeldussin Do you have an idea why the round trip wouldn't be the same and is it known/correct ?

@aulemahal
Copy link
Collaborator Author

Update, sorry I was too quick to comment. The round trip removes entries in the matrix that have the same "data" as the "fill_value", which reduces the size of the matrix. A good thing I guess!

@raphaeldussin
Copy link
Contributor

@aulemahal sorry I'm late to the party, I have a 3 months old to feed and change every couple of hours :) but I'll try to keep up with this PR

@aulemahal
Copy link
Collaborator Author

No problem! xESMF is much less important than a 3 month old, please feel free to ignore any ping!

I added a manual removal of the "unnecessary" matrix entries and now numpy asserts that the new and the old outputs are exactly the same.
I tested with a large regridder (2M entries in the weights matrix) and the acceleration is 3-4x (850 ms to 250 ms).

@aulemahal aulemahal merged commit 0cd7ef6 into master Nov 25, 2025
11 checks passed
@aulemahal aulemahal deleted the no-scipy-fix-para branch November 25, 2025 19:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants