gdal vector sort: handle layers that don't fit in RAM

[dbaston]

it would be great if we had a way of dealing with layers that don't fit in RAM (but where the hilbertCodes vector would still fit). That would likely involve using a temporary storage like I did in the Parquet driver where when with the SORT_BY_BBOX layer creation option is used, a temporary GeoPackage file is created., for source layers that don't have efficient GetFeature() implementation.
This remark is not to be implemented in the context of this PR.

Originally posted by @rouault in #13351 (comment)

[rouault]

When researching about that, I found https://github.com/stxxl/stxxl which has stxxl::vector<> and stxxl::sort implementation , mostly API compatbile with their std versions, but using external/disk storage. The only downside is that the project is abandonned...

[dbaston]

@rouault Do you have advice for how the algorithm should decide whether to store features in memory or on disk? As far as I know we don't have a way to probe the memory usage of an OGRFeature, do we?

Maybe it's best to just always do in-memory unless the user request on-disk via a CLI flag.

[rouault]

I know we don't have a way to probe the memory usage of an OGRFeature, do we?

OGRFeature::SerializeToBinary() could potentially be used to compute an estimate. By the way that's likely what you would want to use to write into a temporary file to not be limited by the capability of file formats. That's what ogrparquetwriterlayer.cpp uses to implement SORT_BY_BBOX and using a temporary GeoPackage file that only stores a (fid, serialized_feature_as_blob) tuple

[dbaston]

OGRFeature::SerializeToBinary() could potentially be used to compute an estimate. By the way that's likely what you would want to use to write into a temporary file to not be limited by the capability of file formats

Yes, this is how I implemented the on-disk version (using raw file rather than a GeoPackage). But I don't think we want the expense of serializing for the (likely?) case where we do everything in memory...

[rouault]

Yes, this is how I implemented the on-disk version (using raw file rather than a GeoPackage). But I don't think we want the expense of serializing for the (likely?) case where we do everything in memory...

Yes, your suggestion of requesting a flag for on-disk is reasonable