Add a callback/preprocess option to open_dataset

[ocefpaf]

It is not uncommon to find datasets with bad metadata, like gregorian_proleptic instead of the expected prolepic_gregorian [1], that will prevent users from reading the full dataset. Ideally we could have a functionality similar to iris' callbacks [2] to workaround this. In fact, it looks like xarray already does something similar in the open_mfdataset but not for the open_dataset.

Pinging @dcherian who gave the idea of using preprocess and @rsignell-usgs who is a pro on finding bad metadata everywhere.

[1] https://nbviewer.jupyter.org/gist/rsignell-usgs/27ba1fdeb934d6fd5b83abe43098a047

[2] https://scitools.org.uk/iris/docs/latest/userguide/navigating_a_cube.html?highlight=callback#adding-and-removing-metadata-to-the-cube-at-load-time

[kmuehlbauer]

@ocefpaf Sorry for letting this sit so long.

From my perspective adding preprocess would help users with this similar API. I'm often find myself using open_mfdataset([filename], preprocess=preprocess) to have preprocess available on a single dataset.

[keewis]

xr.open_dataset(filename).pipe(preprocess) would be exactly the same (minus a bit of overhead), since open_mfdataset just opens all datasets (and decodes them), applies preprocess to each of them and then combines.

[kmuehlbauer]

Yes, @keewis , that works too, thanks.

That would naturally be the easiest thing to do. It might be that it would be just nice to have the preprocess in the kwargs so that you can use the same calling convention. But maybe @ocefpaf can share more background.

[ocefpaf]

I kind of like the .pipe option. Maybe that is already well documented and with examples but it has been so long that I opened this and required such fix that I don't recall. Fell free to close this and, if you need more example/docs to show how to do this, I can try to send a PR.

[TomNicholas]

If the pipe idea works for everyone then we should add it as an example! The examples section of the docstring of open_dataset and the docs page on IO would be the two places to put an example.

[kmuehlbauer]

@ocefpaf Great, than we can just convert this to a discussion and mark @keewis answer as solution.