[REST Catalog API] Rework catalog instance rotation

[george-zubrienko]

In the current live implementation we use TrieMap to store catalog instances, and create a new one every now and then. getCatalog then emits most "fresh" instance, while old one that can still be serving workloads, is kept in TrieMap.

With increased number of Iceberg API calls, this approach leads to TrieMap growing to OOM amounts. Hence, let's replace this with a singleton cache via ZIO's Cached. Since we no longer share Iceberg catalog creation (ie it is not a singleton), we can simply close the old instance and return a new one.

There is still a chance to prematurely close within the same owner - I'd suggest we address this if needed by changing the refresh to manual.

[github-actions]

File	Coverage
All files	67%	✅
logging/ZIOLogAnnotations.scala	79%	✅
models/app/BaseStreamContext.scala	72%	✅
models/app/PluginStreamContext.scala	0%	❌
models/app/PluginStreamContext.scala	0%	❌
models/app/PluginStreamContext.scala	0%	❌
models/batches/BlobBatchCommons.scala	66%	✅
models/batches/SqlServerChangeTracking.scala	62%	✅
models/batches/UpsertBlob.scala	0%	❌
models/batches/SynapseLink.scala	79%	✅
models/batches/SynapseLink.scala	62%	✅
models/batches/SynapseLink.scala	0%	❌
models/batches/SqlServerChangeTracking.scala	79%	✅
models/batches/UpsertBlob.scala	77%	✅
models/batches/UpsertBlob.scala	0%	❌
models/cdm/SimpleCdmModel.scala	88%	✅
models/cdm/CdmParser.scala	89%	✅
models/cdm/CdmParser.scala	93%	✅
models/ddl/CreateTableRequest.scala	80%	✅
models/queries/MergeQuerySegment.scala	88%	✅
models/queries/MergeQuery.scala	85%	✅
models/queries/MergeQuerySegment.scala	85%	✅
models/schemas/ArcaneSchema.scala	83%	✅
models/schemas/ArcaneSchema.scala	50%	✅
models/schemas/ArcaneSchema.scala	65%	✅
models/schemas/DataCell.scala	91%	✅
models/schemas/ArcaneSchema.scala	65%	✅
models/schemas/DataCell.scala	0%	❌
models/schemas/ArcaneSchema.scala	0%	❌
models/schemas/DataCell.scala	0%	❌
models/schemas/ArcaneSchema.scala	71%	✅
models/serialization/ZIODurationRW.scala	0%	❌
models/serialization/S3RegionRW.scala	0%	❌
models/serialization/UriRW.scala	0%	❌
models/settings/TableNaming.scala	55%	✅
models/settings/TablePropertiesSettings.scala	85%	✅
models/settings/database/JdbcConnectionExtensions.scala	31%	❌
models/settings/iceberg/DefaultIcebergStagingSettings.scala	0%	❌
models/settings/mssql/MsSqlServerConnectionSettings.scala	0%	❌
models/settings/sink/DefaultIcebergSinkSettings.scala	0%	❌
models/settings/staging/JdbcMergeServiceClientSettings.scala	0%	❌
services/app/StreamRunnerServiceImpl.scala	0%	❌
services/app/PosixStreamLifetimeService.scala	0%	❌
services/blobsource/DefaultS3Reader.scala	0%	❌
services/blobsource/UpsertBlobBackfillOverwriteBatchFactory.scala	0%	❌
services/blobsource/UpsertBlobHookManager.scala	0%	❌
services/blobsource/providers/BlobSourceStreamingDataProvider.scala	63%	✅
services/blobsource/providers/BlobSourceDataProvider.scala	78%	✅
services/blobsource/readers/listing/BlobListingParquetSource.scala	56%	✅
services/blobsource/readers/listing/BlobListingJsonSource.scala	72%	✅
services/blobsource/readers/listing/BlobListingCsvSource.scala	0%	❌
services/blobsource/versioning/BlobSourceWatermark.scala	59%	✅
services/blobsource/versioning/BlobSourceWatermark.scala	92%	✅
services/bootstrap/DefaultStreamBootstrapper.scala	60%	✅
services/filters/ColumnSummaryFieldsFilteringService.scala	88%	✅
services/filters/FieldsFilteringService.scala	86%	✅
services/hooks/manager/DefaultHookManager.scala	0%	❌
services/hooks/manager/DefaultHookManager.scala	0%	❌
services/iceberg/SchemaConversions.scala	0%	❌
services/iceberg/SchemaConversions.scala	15%	❌
services/iceberg/IcebergCatalogCredential.scala	60%	✅
services/iceberg/SchemaConversions.scala	85%	✅
services/iceberg/IcebergEntityManager.scala	56%	✅
services/iceberg/SchemaConversions.scala	54%	✅
services/iceberg/IcebergS3CatalogWriter.scala	95%	✅
services/iceberg/IcebergTablePropertyManager.scala	62%	✅
services/iceberg/IcebergCatalogFactory.scala	99%	✅
services/iceberg/interop/JsonScanner.scala	78%	✅
services/iceberg/interop/ParquetScanner.scala	76%	✅
services/iceberg/interop/ParquetInputStreamAdapter.scala	75%	✅
services/merging/JdbcMergeServiceClient.scala	30%	❌
services/merging/maintenance/JdbcAnalyzeRequest.scala	0%	❌
services/merging/maintenance/JdbcAnalyzeRequest.scala	0%	❌
services/metrics/DeclaredMetrics.scala	98%	✅
services/metrics/ArcaneDimensionsProvider.scala	59%	✅
services/metrics/DataDog.scala	0%	❌
services/mssql/MsSqlBatch.scala	72%	✅
services/mssql/MsSqlDataProvider.scala	48%	✅
services/mssql/MsSqlBackfillOverwriteBatchFactory.scala	0%	❌
services/mssql/MsSqlStreamingDataProvider.scala	63%	✅
services/mssql/QueryProvider.scala	92%	✅
services/mssql/MsSqlHookManager.scala	0%	❌
services/mssql/base/MsSqlReader.scala	89%	✅
services/mssql/query/LazyQueryResult.scala	86%	✅
services/mssql/query/ScalarQueryResult.scala	66%	✅
services/mssql/query/LazyQueryResult.scala	66%	✅
services/mssql/versioning/MsSqlWatermark.scala	59%	✅
services/storage/models/azure/AdlsStoragePath.scala	84%	✅
services/storage/models/azure/AzureModelConversions.scala	98%	✅
services/storage/models/base/StoredBlob.scala	73%	✅
services/storage/models/s3/S3StoragePath.scala	80%	✅
services/storage/models/s3/S3ClientSettings.scala	86%	✅
services/storage/services/azure/AzureBlobStorageReader.scala	67%	✅
services/storage/services/s3/S3BlobStorageReader.scala	66%	✅
services/streaming/base/DefaultSourceDataProvider.scala	97%	✅
services/streaming/base/SourceWatermark.scala	0%	❌
services/streaming/base/StreamGraphBuilder.scala	0%	❌
services/streaming/base/SourceWatermark.scala	0%	❌
services/streaming/base/SourceWatermark.scala	0%	❌
services/streaming/base/DefaultStreamDataProvider.scala	89%	✅
services/streaming/graph_builders/GenericStreamingGraphBuilder.scala	47%	✅
services/streaming/graph_builders/backfill/GenericBackfillMergeGraphBuilder.scala	83%	✅
services/streaming/graph_builders/backfill/GenericBackfillOverwriteGraphBuilder.scala	53%	✅
services/streaming/processors/batch_processors/WatermarkProcessingExtensions.scala	0%	❌
services/streaming/processors/batch_processors/backfill/BackfillOverwriteWatermarkProcessor.scala	0%	❌
services/streaming/processors/batch_processors/backfill/BackfillApplyBatchProcessor.scala	87%	✅
services/streaming/processors/batch_processors/backfill/BackfillDisposeBatchProcessor.scala	0%	❌
services/streaming/processors/batch_processors/streaming/WatermarkProcessor.scala	97%	✅
services/streaming/processors/batch_processors/streaming/MergeBatchProcessor.scala	99%	✅
services/streaming/processors/batch_processors/streaming/DisposeBatchProcessor.scala	67%	✅
services/streaming/processors/transformers/StagingProcessor.scala	79%	✅
services/streaming/throughput/MemoryBoundShaper.scala	67%	✅
services/streaming/throughput/StaticShaper.scala	0%	❌
services/streaming/throughput/VoidShaper.scala	0%	❌
services/streaming/throughput/base/ThroughputShaperBuilder.scala	42%	✅
services/synapse/SynapseLinkStreamingDataProvider.scala	63%	✅
services/synapse/SynapseEntitySchemaProvider.scala	84%	✅
services/synapse/SynapseBackfillOverwriteBatchFactory.scala	0%	❌
services/synapse/SynapseHookManager.scala	64%	✅
services/synapse/base/SynapseLinkReader.scala	60%	✅
services/synapse/base/SynapseLinkDataProvider.scala	46%	✅
services/synapse/versioning/SynapseWatermark.scala	61%	✅
utils/SqlUtils.scala	53%	✅
utils/SqlUtils.scala	0%	❌

Minimum allowed coverage is 40%

Generated by 🐒 cobertura-action against ed1473b