Skip to content

Commit 8f78e0b

Browse files
slevangslevang
authored
Aggregate all dims simultaneously in conservative (#51)
* dot product over all dims simultaneously * add benchmarking against xesmf * add changelog * add output_chunks arg, match input chunks otherwise * update changelog * add notebook to docs
1 parent d8b7b23 commit 8f78e0b

File tree

6 files changed

+424
-93
lines changed

6 files changed

+424
-93
lines changed

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/).
99
Changed:
1010
- the "most common" routine has been overhauled, thanks to [@dcherian](https://github.com/dcherian). It is now much more efficient, and can operate fully lazily on dask arrays. Users do need to provide the expected groups (i.e., unique labels in the data), and the regridder is only available for `xr.DataArray` currently ([#46](https://github.com/xarray-contrib/xarray-regrid/pull/46)).
1111
- you can now use `None` as input to the `time_dim` kwarg in the regridding methods to force regridding over the time dimension (as long as it's numeric) ([#46](https://github.com/xarray-contrib/xarray-regrid/pull/46)).
12+
- Performance of the conservative method has been improved by simultaneously aggregating over all regridding dimensions. Conservative regridding now also produces outputs with the same grid chunks as the inputs, unless explicit chunksizes are passed via the `output_chunks` argument. ([#51](https://github.com/xarray-contrib/xarray-regrid/pull/51)).
1213

1314
Added:
1415
- `.regrid.stat` for reducing datasets using statistical methods such as the variance or median ([#46](https://github.com/xarray-contrib/xarray-regrid/pull/46)).

‎docs/notebooks/benchmarks/benchmarking_xesmf.ipynb‎

Lines changed: 285 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,285 @@
1+
{
2+
"cells": [
3+
{
4+
"cell_type": "markdown",
5+
"metadata": {},
6+
"source": [
7+
"## Performance of `xesmf` vs `xarray-regrid`\n",
8+
"\n",
9+
"Compare the two conservative methods using a moderately-sized synthetic dask dataset of about 4GB."
10+
]
11+
},
12+
{
13+
"cell_type": "code",
14+
"execution_count": 18,
15+
"metadata": {},
16+
"outputs": [],
17+
"source": [
18+
"import dask.array as da\n",
19+
"import xarray as xr\n",
20+
"import xesmf\n",
21+
"\n",
22+
"import xarray_regrid\n",
23+
"\n",
24+
"bounds = dict(south=-90, north=90, west=-180, east=180)\n",
25+
"\n",
26+
"source = xarray_regrid.Grid(\n",
27+
" resolution_lat=0.25,\n",
28+
" resolution_lon=0.25,\n",
29+
" **bounds,\n",
30+
").create_regridding_dataset()\n",
31+
"\n",
32+
"target = xarray_regrid.Grid(\n",
33+
" resolution_lat=1,\n",
34+
" resolution_lon=1,\n",
35+
" **bounds,\n",
36+
").create_regridding_dataset()\n",
37+
"\n",
38+
"\n",
39+
"def source_data(source, chunks, n_times=1000):\n",
40+
" data = da.random.random(\n",
41+
" size=(n_times, source.latitude.size, source.longitude.size),\n",
42+
" chunks=chunks,\n",
43+
" ).astype(\"float32\")\n",
44+
"\n",
45+
" data = xr.DataArray(\n",
46+
" data,\n",
47+
" dims=[\"time\", \"latitude\", \"longitude\"],\n",
48+
" coords={\n",
49+
" \"time\": xr.date_range(\"2000-01-01\", periods=n_times, freq=\"D\"),\n",
50+
" \"latitude\": source.latitude,\n",
51+
" \"longitude\": source.longitude,\n",
52+
" }\n",
53+
" )\n",
54+
"\n",
55+
" return data\n"
56+
]
57+
},
58+
{
59+
"cell_type": "markdown",
60+
"metadata": {},
61+
"source": [
62+
"## Chunking\n",
63+
"\n",
64+
"Test \"pancake\" (chunked in time) and \"churro\" (chunked in space) chunks of different sizes. The \"small\" versions are about 4 MB, and the \"large\" are about 100 MB."
65+
]
66+
},
67+
{
68+
"cell_type": "code",
69+
"execution_count": 19,
70+
"metadata": {},
71+
"outputs": [],
72+
"source": [
73+
"chunk_schemes = {\n",
74+
" \"pancake_small\": (1, -1, -1),\n",
75+
" \"pancake_large\": (25, -1, -1),\n",
76+
" \"churro_small\": (-1, 32, 32),\n",
77+
" \"churro_large\": (-1, 160, 160),\n",
78+
"}"
79+
]
80+
},
81+
{
82+
"cell_type": "code",
83+
"execution_count": 20,
84+
"metadata": {},
85+
"outputs": [
86+
{
87+
"name": "stderr",
88+
"output_type": "stream",
89+
"text": [
90+
"/home/slevang/miniconda3/envs/xarray-regrid/lib/python3.12/site-packages/xesmf/backend.py:56: UserWarning: Latitude is outside of [-90, 90]\n",
91+
" warnings.warn('Latitude is outside of [-90, 90]')\n",
92+
"/home/slevang/miniconda3/envs/xarray-regrid/lib/python3.12/site-packages/xesmf/backend.py:56: UserWarning: Latitude is outside of [-90, 90]\n",
93+
" warnings.warn('Latitude is outside of [-90, 90]')\n"
94+
]
95+
}
96+
],
97+
"source": [
98+
"# For larger grids, generating weights is quite expensive\n",
99+
"xesmf_regridder = xesmf.Regridder(source, target, \"conservative\")"
100+
]
101+
},
102+
{
103+
"cell_type": "markdown",
104+
"metadata": {},
105+
"source": [
106+
"## Timings\n",
107+
"\n",
108+
"Run timings for different chunkings schemes and with NaN skipping enabled and disabled, across both libraries. Compare the ratio of `xesmf / xarray-regrid` to see the speedup factor of using this library."
109+
]
110+
},
111+
{
112+
"cell_type": "code",
113+
"execution_count": 21,
114+
"metadata": {},
115+
"outputs": [
116+
{
117+
"name": "stderr",
118+
"output_type": "stream",
119+
"text": [
120+
"/home/slevang/miniconda3/envs/xarray-regrid/lib/python3.12/site-packages/xarray/core/computation.py:320: PerformanceWarning: Regridding is increasing the number of chunks by a factor of 72.0, you might want to specify sizes in `output_chunks` in the regridder call. Default behaviour is to preserve the chunk sizes from the input (32, 32).\n",
121+
" result_var = func(*data_vars)\n",
122+
"/home/slevang/miniconda3/envs/xarray-regrid/lib/python3.12/site-packages/xarray/core/computation.py:320: PerformanceWarning: Regridding is increasing the number of chunks by a factor of 72.0, you might want to specify sizes in `output_chunks` in the regridder call. Default behaviour is to preserve the chunk sizes from the input (32, 32).\n",
123+
" result_var = func(*data_vars)\n",
124+
"/home/slevang/miniconda3/envs/xarray-regrid/lib/python3.12/site-packages/xarray/core/computation.py:320: PerformanceWarning: Regridding is increasing the number of chunks by a factor of 72.0, you might want to specify sizes in `output_chunks` in the regridder call. Default behaviour is to preserve the chunk sizes from the input (32, 32).\n",
125+
" result_var = func(*data_vars)\n",
126+
"/home/slevang/miniconda3/envs/xarray-regrid/lib/python3.12/site-packages/xarray/core/computation.py:320: PerformanceWarning: Regridding is increasing the number of chunks by a factor of 72.0, you might want to specify sizes in `output_chunks` in the regridder call. Default behaviour is to preserve the chunk sizes from the input (32, 32).\n",
127+
" result_var = func(*data_vars)\n",
128+
"/home/slevang/miniconda3/envs/xarray-regrid/lib/python3.12/site-packages/xarray/core/computation.py:320: PerformanceWarning: Regridding is increasing the number of chunks by a factor of 6.0, you might want to specify sizes in `output_chunks` in the regridder call. Default behaviour is to preserve the chunk sizes from the input (160, 160).\n",
129+
" result_var = func(*data_vars)\n",
130+
"/home/slevang/miniconda3/envs/xarray-regrid/lib/python3.12/site-packages/xarray/core/computation.py:320: PerformanceWarning: Regridding is increasing the number of chunks by a factor of 6.0, you might want to specify sizes in `output_chunks` in the regridder call. Default behaviour is to preserve the chunk sizes from the input (160, 160).\n",
131+
" result_var = func(*data_vars)\n",
132+
"/home/slevang/miniconda3/envs/xarray-regrid/lib/python3.12/site-packages/xarray/core/computation.py:320: PerformanceWarning: Regridding is increasing the number of chunks by a factor of 6.0, you might want to specify sizes in `output_chunks` in the regridder call. Default behaviour is to preserve the chunk sizes from the input (160, 160).\n",
133+
" result_var = func(*data_vars)\n",
134+
"/home/slevang/miniconda3/envs/xarray-regrid/lib/python3.12/site-packages/xarray/core/computation.py:320: PerformanceWarning: Regridding is increasing the number of chunks by a factor of 6.0, you might want to specify sizes in `output_chunks` in the regridder call. Default behaviour is to preserve the chunk sizes from the input (160, 160).\n",
135+
" result_var = func(*data_vars)\n"
136+
]
137+
}
138+
],
139+
"source": [
140+
"import time\n",
141+
"\n",
142+
"import pandas as pd\n",
143+
"\n",
144+
"pd.options.display.precision = 1\n",
145+
"\n",
146+
"\n",
147+
"def do_regrid(data, target, skipna):\n",
148+
" data.regrid.conservative(target, skipna=skipna).compute()\n",
149+
"\n",
150+
"\n",
151+
"def do_xesmf(data, target, skipna):\n",
152+
" xesmf_regridder(data, skipna=skipna).compute()\n",
153+
"\n",
154+
"\n",
155+
"def timing_grid(func, repeats=2):\n",
156+
" times = pd.DataFrame(\n",
157+
" index=chunk_schemes.keys(),\n",
158+
" columns=[\"skipna=False\", \"skipna=True\"],\n",
159+
" )\n",
160+
" for name, chunks in chunk_schemes.items():\n",
161+
" data = source_data(source, chunks)\n",
162+
" for skipna in [False, True]:\n",
163+
" execution_times = []\n",
164+
" for _ in range(repeats):\n",
165+
" start = time.perf_counter()\n",
166+
" func(data, target, skipna)\n",
167+
" end = time.perf_counter()\n",
168+
" execution_times.append(end - start)\n",
169+
" # Sometimes the first execution is a little slower\n",
170+
" times.loc[name, f\"skipna={skipna}\"] = min(execution_times)\n",
171+
"\n",
172+
" return times\n",
173+
"\n",
174+
"\n",
175+
"regrid_times = timing_grid(do_regrid)\n",
176+
"xesmf_times = timing_grid(do_xesmf)\n",
177+
"ratio = xesmf_times / regrid_times\n"
178+
]
179+
},
180+
{
181+
"cell_type": "markdown",
182+
"metadata": {},
183+
"source": [
184+
"## Results\n",
185+
"\n",
186+
"With current implementations, `xesmf` is slightly faster for large pancake-style chunks. `xarray-regrid` is much faster for small chunks, especially churro-style.\n",
187+
"\n",
188+
"These tests were run on an 8-core Intel i7 Ubuntu desktop:"
189+
]
190+
},
191+
{
192+
"cell_type": "code",
193+
"execution_count": 22,
194+
"metadata": {},
195+
"outputs": [
196+
{
197+
"data": {
198+
"text/html": [
199+
"<div>\n",
200+
"<style scoped>\n",
201+
" .dataframe tbody tr th:only-of-type {\n",
202+
" vertical-align: middle;\n",
203+
" }\n",
204+
"\n",
205+
" .dataframe tbody tr th {\n",
206+
" vertical-align: top;\n",
207+
" }\n",
208+
"\n",
209+
" .dataframe thead th {\n",
210+
" text-align: right;\n",
211+
" }\n",
212+
"</style>\n",
213+
"<table border=\"1\" class=\"dataframe\">\n",
214+
" <thead>\n",
215+
" <tr style=\"text-align: right;\">\n",
216+
" <th></th>\n",
217+
" <th>skipna=False</th>\n",
218+
" <th>skipna=True</th>\n",
219+
" </tr>\n",
220+
" </thead>\n",
221+
" <tbody>\n",
222+
" <tr>\n",
223+
" <th>pancake_small</th>\n",
224+
" <td>3.7</td>\n",
225+
" <td>7.2</td>\n",
226+
" </tr>\n",
227+
" <tr>\n",
228+
" <th>pancake_large</th>\n",
229+
" <td>0.6</td>\n",
230+
" <td>1.1</td>\n",
231+
" </tr>\n",
232+
" <tr>\n",
233+
" <th>churro_small</th>\n",
234+
" <td>14.2</td>\n",
235+
" <td>16.9</td>\n",
236+
" </tr>\n",
237+
" <tr>\n",
238+
" <th>churro_large</th>\n",
239+
" <td>1.8</td>\n",
240+
" <td>2.4</td>\n",
241+
" </tr>\n",
242+
" </tbody>\n",
243+
"</table>\n",
244+
"</div>"
245+
],
246+
"text/plain": [
247+
" skipna=False skipna=True\n",
248+
"pancake_small 3.7 7.2\n",
249+
"pancake_large 0.6 1.1\n",
250+
"churro_small 14.2 16.9\n",
251+
"churro_large 1.8 2.4"
252+
]
253+
},
254+
"execution_count": 22,
255+
"metadata": {},
256+
"output_type": "execute_result"
257+
}
258+
],
259+
"source": [
260+
"ratio"
261+
]
262+
}
263+
],
264+
"metadata": {
265+
"kernelspec": {
266+
"display_name": "xarray-regrid",
267+
"language": "python",
268+
"name": "python3"
269+
},
270+
"language_info": {
271+
"codemirror_mode": {
272+
"name": "ipython",
273+
"version": 3
274+
},
275+
"file_extension": ".py",
276+
"mimetype": "text/x-python",
277+
"name": "python",
278+
"nbconvert_exporter": "python",
279+
"pygments_lexer": "ipython3",
280+
"version": "3.12.0"
281+
}
282+
},
283+
"nbformat": 4,
284+
"nbformat_minor": 2
285+
}

‎docs/notebooks/index.rst‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@ Most notebooks compare the methods implemented in xarray-regrid against more sta
1313
benchmarks/benchmarking_bilinear.ipynb
1414
benchmarks/benchmarking_conservative.ipynb
1515
benchmarks/benchmarking_nearest.ipynb
16+
benchmarks/benchmarking_xesmf.ipynb
1617

1718
.. toctree::
1819
:maxdepth: 1

0 commit comments

Comments
 (0)