gdal vsi regex (PCRE) filtering

[PostholerCom]

Feature description

Filtering data sources is one of the most common preprocessing tasks when working with large sets of files. With the ability to filter directly from gdal vsi, that all goes away.

Typically, you may get a list of files with something like:
gdal vsi list -R --abs /vsis3/bucket/path

You would save that result or parse it on the fly with grep, sed, awk, perl, etc. If you could specify regex in the requested path, you could skip the preprocessing entirely. Something like:

gdal vsi list -R --abs /vsis3/bucket/[regex]/path/dir-[regex]/file_[regex]

Everything between [] or some other delimiters, would be filtered.

Also, having vsi as an input source of one or more files to pipeline, stack or mosaic, with the ability of regex would be incredibly useful.

Thanks!

Additional context

No response

[hobu]

Why should this be in gdal vsi instead of directing people to use purpose-built commands like grep, sed, and awk which are much more flexible (and also not the responsibility of GDAL)? I get that it could be a touch more convenient, but GDAL isn't going to write its own regex engine, so a feature like this would either bring a hard dependency on PCRE or we end up taking advantage of std::regex which may have platform sensitivity discrepancies.

Is there anything to do to make output of gdal vsi ls output easier to work with by common unix filtering tools? Maybe that's a place to start.

[PostholerCom]

Is there anything to do to make output of gdal vsi ls output easier to work with by common unix filtering tools?

GDAL has always been more robust when using external system utilities.

mosaic, stack (others?) already have basic file globbing available, even on /vsi* paths. Having that available in gdal vsi ls (or anywhere) would be a plus. regex would make it a monster.

Not having the knowledge of GDAL's internal regex abilities, I'd thought I'd throw this out there. The fewer dependencies on OS system utilities, the better (within reason). :)

Thanks!

[hobu]

already have basic file globbing available

My concerns are bringing too much into GDAL's realm of responsibility, how to provide a user experience loop for regex filtering to users to know if they're succeeding or failing, and consistency between all the various utilities. Maybe the realm issue has already sailed with simple globbing already available in places, however.

[PostholerCom]

Fair enough! Thanks for the response.

[jratike80]

Maybe https://gdal.org/en/stable/programs/gdal_vsi_list.html#gdal-vsi-list could have examples about how to filter the output with grep on Linux and with findstr on Windows.

When it comes to creating a filtered list for further processing, gdal raster index and gdal vector index have --filename-filter option. Perhars you could study if this filter is usable with vsi* sources, and if there is something that could be improved.

[hobu]

Is a VRT whose SourceFilename was a regex be something we should allow?