Allow regridding for very large grids

[davidhassell]

Regridding to and/or from a very large grid (in terms of number of grid points) can fail due to running out of memory.

This occurs during the call to esmpy.Regrid which we use to calculate the regridding weights (https://github.com/NCAS-CMS/cf-python/blob/v3.18.0/cf/regrid/regrid.py#L2434). The problem isn't (so much) the size of the final weights matrix (although that can be quite large), more that esmpy.Regrid uses up a lot of memory for temporary arrays.

A real use case using 'linear' regridding has destination grid size 181440000, resulting in a sparse matrix representation of the weights taking up $$8.79\ GiB \equiv 4 \times 181440000 \times 3.25 \times 4 / 2^{30}\ GiB$$ (have a poke round inside the guts of a scipy CSR sparse array to see where that number comes from). The peak memory memory required by esmpy to create this is many times that!

A work around for this is to calculate weights separately for non-overlapping partitions of the destination (not the source) grid, and then combining the weights from each of these partitions to create the final weights to partition the final weights matrix.

We can add this as an option to the regrid API:

>>> # Calculate the weights using 10 separate partitions
>>> f.regrids(g, 'linear', dst_grid_partitions=10)
>>> # Calculate the weights using the maximum possible number of partitions
>>> f.regrids(g, 'linear', dst_grid_partitions='maximum')

Why not do this always? Because it is often not required, and there can be a large speed performance hit for doing this. (Therefore it could be recommended that when using dst_grid_partitions you also save the resulting weights to a file for future use.)

---

To see how this works, consider a nearest neighbour regridding from a size 6 source grid to a size 5 destination grid. The 5 x 6 weights matrix for this could look like:

             I = 6
        + - - - - - - 
        | 1 0 0 0 0 0 
        | 0 0 0 0 1 0
  J = 5 | 0 0 1 0 0 0
        | 0 0 0 1 0 0
        | 0 0 0 0 0 1

This is clearly just a concatenation of the following 3 x 6 and 2 x 6 weights matrices:

             I = 6
        + - - - - - - 
        | 1 0 0 0 0 0 
 J0 = 3 | 0 0 0 0 1 0
        | 0 0 1 0 0 0
        + - - - - - - 
 J1 = 2 | 0 0 0 1 0 0
        | 0 0 0 0 0 1

The key is that two calls to esmpy.Regrid (one for the J0 partition, one of the J1 partition) create the correct rows of this matrix for any subset of the destination grid, provided:

1. the source grid is left un-partitioned,
2. the regridded result at destination grid cell j depends only on grid cell j, and not any other destination grid cells. If this were the case, as happens for conservative_2d and nearest_dtos methods, then some weights would be incorrect, as when they are calculated they "don't know" about destination grid cells in any of the other partitions.

[gchingjohnson]

I attempted to regrid a tripolar grid to a Lat-lon grid using cf-python and was unable to. The tripolar grid was of size (1440x1206), is this considered a "large grid" in this case? What does a failure due to size look like in this case because I had let the command run for a few hours (in hopes of receiving some sort of error message), but decided to stop as it was taking too long and it seemed like it was not working.

[davidhassell]

Hi @gchingjohnson,

Thanks for the question. I indeed hadn't really defined "large"!. Large is determined somewhat vaguely by the total number of grid points in both the source and destination grids and also the amount of memory on your machine. The main factor is the logical size of weights matrix, which has shape MxN, where M and N are the numbers of destination and source points respectively. Even though this matrix is calculated internally as a very sparse matrix, the temporary space required by ESMF for it's creation can be many times the matrix's size.

In your case M is 1736640 (= 1440 x 1206), which is ~100 times smaller than the example above which prompted this. So, depending on the source grid size, maybe not large ...?

Anyway, the symptom of "too-large" is RAM usage spiraling to 100%, your computer freezing, and eventually the python process killing itself (hopefully - I usually end up doing a hard re-boot because I'm impatient). No nice raised exceptions for this, I'm afraid, but it's easy enough see if it is the case.

I have a unit-tested PR ready to go on this, so if it is too large, we'll get that reviewed and in pronto (which I should do anyway :)).

However, if instead you've got a nice raised Exception, please post it here (along with a dump of the source and destination grids, if possible), and we can try to see where the issue is.

Cheers,
David