What is the most efficient way to query Observations of a Datastream?

[charlycou]

Hi all,
I want to query all observations of a Datastream. Currently I'm using something like staDatastream.observations().query().list().fullIterator(). However the number of observations of the datastream is quite large (around 100000000 entries) and the fullIterator() method is impacted by how the server-side pagination has been set on the STA service requested. The STA service I'm querying is returning 200 entries per request by default so this results in a lot of HTTP requests. I was wondering if there is a more efficient way of querying all observations using FROST-client?
I also read that DataArray for requesting Observations is not yet implemented.

[hylkevds]

DataArrays would still be limited by the 200 entries limit.

The best would be asking the admin to increase the limit. If that is a FROST-Server, a larger max is much more efficient, and there is no real reason to not set the maximum to 100000 or so. FROST-Server doesn't keep the entire result in memory and streams it to the client.

If the server is not a FROST-Server, that may be different.

[charlycou]

Thanks for your answer @hylkevds. Indeed, this is not a FROST-server. I'm thinking of iterating through the set of observations by making a succession of top and limit queries defining myself the size of the result list.