CQL filter returns wrong numberMatched after the first page of response

[EleniLouvari]

When using POST /search with filter-lang: cql2-json and paginating via the returned rel=next link, the first page reports the correct numberMatched, but subsequent pages report numberMatched equal to the total number of records in the catalog (ignoring the filter).

This breaks pagination UX and makes clients think the filtered result set is much larger than it is.

Example tested:

POST https://catalogue.ax3hub.xxxxx/search
{
  "filter-lang": "cql2-json",
  "filter": { 
   "op": "and",
    "args": [
      {
        "op": "=",
        "args": [
          {
            "property": "parentidentifier"
          },
          "ARD_S3"
        ]
      }
    ]
  },
  "limit": 10}
Page 1: numberMatched=268, numberReturned=10, features=10, rels=['self', 'first', 'next', 'root']
Next Page: https://catalogue.ax3hub-qa.planetek.gr/stac/search?limit=10&offset=10
Page 2: numberMatched=8001, numberReturned=10, features=10, rels=['self', 'prev', 'next', 'root']
Next Page: https://catalogue.ax3hub-qa.planetek.gr/stac/search?limit=10&offset=20
Page 3: numberMatched=8001, numberReturned=10, features=10, rels=['self', 'prev', 'next', 'root']
.....

[tomkralidis]

@EleniLouvari thanks for the report. The root cause here is that the CQL2 POST (which results in the filter of 268 results) does not get propagated to the next (GET) link. We will need to investigate options on paging when POST is used as the query/filter mechanism.

[tomkralidis]

Related issue opened in opengeospatial/ogcapi-features#1030

[tomkralidis]

Update: #1185 now includes method=POST and body=<POST request payload properties for link objects, as per https://github.com/radiantearth/stac-api-spec/tree/main/item-search#pagination. This would allow a client to page through a result set using consistent POST payloads. Having said this, I am not sure if/how pystac-client supports this approach (needs testing), if you are using same, instead of your own script(s).

[EleniLouvari]

I am not using pystac-client. I have my own scripts using POST requests.

[tomkralidis]

I am not using pystac-client. I have my own scripts using POST requests.

Then you can capture the next and/or prev links and handle accordingly (once #1185 is review/approved/merged). Below is an example next link:

        {
            "rel": "next",
            "type": "application/geo+json",
            "title": "items (next)",
            "href": "http://127.0.0.1:8000/stac/search?offset=10",
            "hreflang": "en-US",
            "method": "POST",
            "body": {
                "collections": [
                    "S2MSI1C"
                ],
                "intersects": {
                    "geometry": {
                        "type": "Polygon",
                        "coordinates": [
                            [
                                [
                                    25,
                                    40
                                ],
                                [
                                    28,
                                    40
                                ],
                                [
                                    28,
                                    42
                                ],
                                [
                                    25,
                                    42
                                ],
                                [
                                    25,
                                    40
                                ]
                            ]
                        ]
                    }
                }
            }
        }

[kalxas]

The PR has been reviewed and merged.
Let's test and close the issue once we confirm fixed.

[kalxas]

Another fix on #1186