`/search`ing by date and/or datetime and/or datetime+ms

[jgaucher-cs]

Hi, when calling the /search endpoint that is implemented here: https://github.com/stac-utils/stac-fastapi/blob/main/stac_fastapi/api/stac_fastapi/api/app.py, it seems we cannot search by date, but only by datetime.

A GET request with /search?datetime=2000-01-01 will fail with this error from https://github.com/stac-utils/stac-fastapi/blob/main/stac_fastapi/types/stac_fastapi/types/rfc3339.py

b'{"detail":"Invalid RFC3339 datetime."}'

A POST request will fail with what seems to by a pydantic error:

b'{"detail":[{"type":"datetime_parsing","loc":["body","datetime"],"msg":"Input should be a valid datetime, invalid datetime separator, expected `T`, `t`, `_` or space","input":"2000-01-01","ctx":{"error":"invalid datetime separator, expected `T`, `t`, `_` or space"}}]}'

We would like to allow our users to search by date without time, then our application (that uses stac_fastapi) would return all products for this whole day. Would this be possible ?

Then we would like to do the same for milliseconds: our products have a ms information, so if the user searches by datetime+ms, we would like to return products only for this specific ms. If he searches by datetime without ms, we would like to return all products for this whole second.

Our problem is that, if he searches for ms=.000 (to have products only for this specific ms) with a GET request e.g. /search?datetime=2000-01-01T01:01:01.000Z, stac_fastpi removes the ms=.000 information: our app receives the value datetime=2000-01-01T01:01:01Z, so it doesn't know anymore if the user wanted only products for this specific ms, or for this whole second. I think this removal is made when stac_fastapi converts the str to datetime in rfc3339_str_to_datetime https://github.com/stac-utils/stac-fastapi/blob/main/stac_fastapi/types/stac_fastapi/types/rfc3339.py#L51 (I couldn't find where this datetime is converted back to str).

To summarize, my two questions are:

1. Could a user search by date, not datetime ?
2. Could we stop stac_fastpi from removing the "ms=.000" information ?

Thank you :)

[vincentsarago]

Hi @jgaucher-cs

It would help if you state which backend you are using, some backend might have specificities.

We would like to allow our users to search by date without time, then our application (that uses stac_fastapi) would return all products for this whole day. Would this be possible ?

You can use start/end dates in form of datetime=2000-01-01T00:00:00/2000-01-01T23:59:59

Could a user search by date, not datetime ?

Well the Spec mention date in a specific format so for the implementation we are trying to follow the spec to the letter, but you can customized your own app, as Microsoft is doing: https://github.com/microsoft/planetary-computer-apis/blob/155f5f6de9ae6a09d29f53cbdd2ab4b7215a3b1f/pcstac/pcstac/search.py#L37-L41

Our problem is that, if he searches for ms=.000 (to have products only for this specific ms) with a GET request e.g. /search?datetime=2000-01-01T01:01:01.000Z, stac_fastpi removes the ms=.000 information:

I'm not sure to get this.

Let's follow the route for a datetime in the GET request:

• 1. get's converted to datetime object by https://github.com/stac-utils/stac-fastapi/blob/main/stac_fastapi/types/stac_fastapi/types/search.py#L173-L175
• 2. _datetime_converter calls str_to_interval
• 3. str_to_interval calls parse_single_date for each date
• 4. parse_single_date calls rfc3339_str_to_datetime
• 5. finally rfc3339_str_to_datetime use iso8601 to parse the string 👇

iso8601.parse_date("2000-01-01T01:01:01.001Z").microsecond

So I'm not quite sure about the statement stac_fastpi removes the ms=.000 information:

[jgaucher-cs]

Thank you very much @vincentsarago for your answer. I'm using the pgstac backend.

I will take a look at this customization for searching by date. For the ms, the str to datetime conversion with rfc3339_str_to_datetime is working well, the field microsecond has the value 0, this is OK. The problem is that datetime is converted back into a str before being passed to my app, and here the ms=.000 is lost.

Allow me to paste a screenshot from my call stack, I've called my endpoint with `/search?datetime=2000-01-01T01:01:01.000Z", then in: /stac_fastapi/pgstac/core.py

This p variable will be passed to my app, we see this is a list[str] with one element that contains "datetime":"2000-01-01T01:01:01Z" so here the ms=.000 information is lost.

A little before in the call stack, we see that self.post_request_model returns a datetime as a str (without the ms=.000) that will be used later in the call stack, and also an end_date as a datetime with the ms info, but which is not used later.

[vincentsarago]

Can you share the version of stac-fastapi, stac-fastapi-pgstac and stac-pydantic please

[jgaucher-cs]

Can you share the version of stac-fastapi, stac-fastapi-pgstac and stac-pydantic please
I have these:

pypgstac                                 0.8.6                               
pystac                                   1.11.0                              
stac-fastapi-api                         3.0.3                               
stac-fastapi-extensions                  3.0.3                               
stac-fastapi-pgstac                      3.0.0                               
stac-fastapi-types                       3.0.3                               
stac-pydantic                            3.1.3          

[jgaucher-cs]

Hello @vincentsarago, would you have any update on this ? Thank you :)

[vincentsarago]

sorry @jgaucher-cs just got back from vacations, I'll try to have a look this week

[vincentsarago]

🤔 in #771 I've added more tests about ms and I don't see anything so I'm pretty sure this is a stac-fastapi-pgstac issue

[jgaucher-cs]

Thank you @vincentsarago for your feedback. Are you still looking at this ? Should I copy this issue in https://github.com/stac-utils/stac-fastapi-pgstac/issues ?

[vincentsarago]

@jgaucher-cs transferred the issue here!

I've got other things on my todo right now but will try to go back at this as soon as I can

[vincentsarago]

@jgaucher-cs I'm still not able to reproduce this locally 🤔

Allow me to paste a screenshot from my call stack, I've called my endpoint with `/search?datetime=2000-01-01T01:01:01.000Z", then in: /stac_fastapi/pgstac/core.py

This is from #778

To me it seems normal that you lost the ms part because it is set to 0 🤷‍♂️ (this I'm able to reproduce locally)

Is your use case to know if a user used search with .000 ms ?