pgstac_reader NDJSON export

[ghidalgo3]

This is an intermediate step necessary to rewrite the in-memory processing step to use Arrow.

[TomAugspurger]

Looks good overall, though some linting issues.

IMO, it'd be better to have a test that exercises the new format="ndjson" option than testing _write_ndjson directly.

[ghidalgo3]

Agreed, though offline I'm chatting with @kylebarron on some more changes. I think this PR isn't finished.

For context, I'm trying to revive the OpenPC geoparquet exports, and Kyle I think wants to switch to using Arrow instead of using geopandas to write the parquet files. This NDJSON export in the end might be just an internal intermediate step that isn't exposed to clients.

The reason why the geoparquet export broke is:

1. SFI networking rules broke reading from the collection config table, easy to fix
2. The stac-geoparquet exports kept getting OOM killed, I think because we kept the STAC query and the geodataframe in memory at the same time and that put too much memory pressure on the machine doing the export for the larger partition.

I'll eventually consume this change in PCTasks when I revive the export.

[ghidalgo3]

@bitner 👀

[kylebarron]

Is this a new dependency? I don't see it in

stac-geoparquet/pyproject.toml

Lines 13 to 27 in 57229c0

	dependencies = [
	"ciso8601",
	"deltalake",
	"geopandas",
	"packaging",
	"pandas",
	# Needed for RecordBatch.append_column
	# Below 19 b/c https://github.com/apache/arrow/issues/45283
	"pyarrow>=16,<19",
	"pyproj",
	"pystac",
	"shapely",
	"orjson",
	'typing_extensions; python_version < "3.11"',
	]

[bitner]

@ghidalgo3 Let me mess around with a few things, we can definitely make this more streamable / memory friendly on the pg access side.

[bitner]

didn't quite get there today, but I'll have something that cleans a lot of the postgres stuff up tomorrow