Can I determine if Polytope supports an item selected via `qubed`?

[ciaransweet]

Hey folks, sorry if this isn't the place to ask!

The new updates are working great! I'm hitting some interesting errors when using earthkit.data.from_source, with Polytope requests made from qubed results.

# request code
data = earthkit.data.from_source(
            "polytope",
            "destination-earth",
            request=request,
            stream=False,
            address="polytope.lumi.apps.dte.destination-earth.eu",
        )

Example polytope request:

{'request': 'activity: scenariomip\n'
            'class: d1\n'
            'dataset: climate-dt\n'
            "date: '20200120'\n"
            'experiment: ssp3-7.0\n'
            "expver: '0001'\n"
            'feature:\n'
            '  shape:\n'
            '  - - 40\n'
            '    - -14\n'
            '  - - 40\n'
            '    - -10\n'
            '  - - 44\n'
            '    - -10\n'
            '  - - 44\n'
            '    - -14\n'
            '  - - 40\n'
            '    - -14\n'
            '  type: polygon\n'
            'generation: 1\n'
            'levtype: sfc\n'
            'model: ifs-fesom\n'
            'param: 34\n'
            'realization: 2\n'
            'resolution: high\n'
            'stream: clte\n'
            "time: '1200'\n"
            'type: fc\n',
 'verb': 'retrieve'}

But Polytope will give me:

Matched datasource polytope-climate

Polytope Feature Extraction Error: Data on this grid is not supported by Polytope.

Is there anything in the qubed data that would allude to this before I go and request the data? (Sorry, I'm not super familiar with this data, but I'd like to ensure we can stop requesting data we've drilled out of qubed if it's not supported by Polytope)

[j08lue]

We could really use guidance on this. We are auto-generating Polytope requests inside our application, so when it fails the user has no means of fixing.

Probably more an issue to raise on Polytope?

I see the error we see is defined here:

https://github.com/ecmwf/polytope/blob/f95c4e8251b7b48d0815ac8ed8a71e089b130987/polytope_feature/utility/exceptions.py#L40

And raised in the context of "GribJump" request failures:

https://github.com/ecmwf/polytope/blob/f95c4e8251b7b48d0815ac8ed8a71e089b130987/polytope_feature/datacube/backends/fdb.py#L142-L147

[TomHodson]

I think @mathleur and @jameshawkes are the people to ask here, but to answer the question as best I can:

It looks like the issue is that polytope doesn't support the grid being used by some of the data you're requesting. In principle qubed should be able to tell you what grid a particular GRIB message is using but the current qubes don't have that grid information in them because some plumbing is required in various other libraries in order for qubed to extract that grid information. (Explicitly: there's no way to get grid information through remote fdb)

This is however very much on the near term road map so make it clear to @jameshawkes that it's a feature you're interested in.

[mathleur]

Hi @ciaransweet and @j08lue ,

I've just tried the request and it seems to work!

We've updated the polytope-server a few weeks ago to work on this dataset, but unfortunately probably after this issue was raised.

Please do report any such issues on the polytope or polytope-server repositories as well, as we aim to support feature extraction from all Destination Earth datasets using Polytope!

[j08lue]

Please do report any such issues on the polytope or polytope-server repositories as well, as we aim to support feature extraction from all Destination Earth datasets using Polytope!

Thanks for the guidance, @mathleur and @TomHodson. I'll raise issues on the Polytope repos - we are still frequently hitting this error these days.

This is however very much on the near term road map so make it clear to @jameshawkes that it's a feature you're interested in.

Is there a ticket for this we can track, @TomHodson or @jameshawkes? Then we can close this one here, I guess.