New Catalog Configuration via CREATE SERVER

[sfc-gh-npuka]

Introduce a new way to configure Iceberg catalogs using PostgreSQL's native CREATE SERVER and CREATE USER MAPPING infrastructure. Each catalog becomes a named server object with its own connection settings and credentials, replacing the current global GUC-based configuration, which has some limitations (can have only single REST catalog, multiple default location prefix definitions).

Proposal

A new configuration-only FDW (iceberg_catalog, no handler) is introduced. Each catalog is represented as a CREATE SERVER with a TYPE indicating the catalog kind (postgres, object_store, or rest). Credentials are stored securely in CREATE USER MAPPING. The catalog option on Iceberg tables changes from a type string ('postgres', 'rest') to a server name reference.

Development

• Part 1 - CREATE SERVER infrastructure New Catalog Configuration via CREATE SERVER #242
• Part 2 - CREATE USER MAPPING infrastructure New Catalog Configuration Credentials #255

Defining catalogs

-- Pre-created by the extension
CREATE FOREIGN DATA WRAPPER iceberg_catalog
    VALIDATOR iceberg_catalog_validator;

CREATE SERVER postgres TYPE 'postgres'
    FOREIGN DATA WRAPPER iceberg_catalog;

CREATE SERVER object_store TYPE 'object_store'
    FOREIGN DATA WRAPPER iceberg_catalog
    OPTIONS (location_prefix '...', internal_prefix 'frompg', external_prefix 'fromsf');

-- User-defined REST catalog (e.g., Polaris / OpenCatalog)
CREATE SERVER my_open_catalog TYPE 'rest'
    FOREIGN DATA WRAPPER iceberg_catalog
    OPTIONS (
        rest_endpoint 'http://polaris:8181',
        rest_auth_type 'default',
        catalog_name 'default',
        location_prefix 's3://datalake/iceberg',
        scope 'PRINCIPAL_ROLE:ALL',
        enable_vended_credentials 'true'
    );

Credentials via USER MAPPING

Secrets are stored in USER MAPPING, which is access-controlled (unlike SERVER OPTIONS, which are publicly readable). This supports shared credentials via PUBLIC and per-user credentials.

-- Shared credentials
CREATE USER MAPPING FOR PUBLIC SERVER my_polaris
    OPTIONS (client_id 'svc-account', client_secret 'secret');

-- Per-user credentials
CREATE USER MAPPING FOR analyst SERVER my_polaris
    OPTIONS (client_id 'analyst-id', client_secret 'analyst-secret');

Creating tables

-- Default (postgres catalog, unchanged UX)
CREATE TABLE ice (id int) USING iceberg;

-- Write to a named REST catalog
CREATE TABLE events (id int, ts timestamptz) USING iceberg
    WITH (catalog = 'my_polaris');

-- Read from a REST catalog
CREATE TABLE external_data () USING iceberg
    WITH (catalog = 'my_polaris', read_only = 'true',
          catalog_namespace = 'raw', catalog_table_name = 'events');

GUC migration

Current GUC	Status	Destination	New Option Name
default_location_prefix	Still needed as GUC	--	--
default_catalog	Still needed as GUC (now holds a server name)	--	--
object_store_catalog_location_prefix	Remains for backward compat	--	--
internal_object_store_catalog_prefix	Could move to server or be removed	SERVER	internal_prefix
external_object_store_catalog_prefix	Could move to server or be removed	SERVER	external_prefix
rest_catalog_host	Becomes server property	SERVER	rest_endpoint
rest_catalog_auth_type	Becomes server property	SERVER	rest_auth_type
rest_catalog_oauth_host_path	Becomes server property	SERVER	oauth_endpoint
rest_catalog_scope	Becomes server property	SERVER	scope
rest_catalog_enable_vended_credentials	Becomes server property	SERVER	enable_vended_credentials
rest_catalog_client_id	Becomes server property	USER MAPPING	client_id
rest_catalog_client_secret	Becomes server property	USER MAPPING	client_secret

Task outline

• Support CREATE/ALTER/DROP SERVER and USER MAPPING syntax -- new iceberg_catalog FDW (validator only, no handler) with a validator that enforces allowed options per server TYPE and per context (SERVER vs USER MAPPING). Pre-create postgres and object_store servers in the extension script.
• Catalog config resolution -- new IcebergCatalogConfig struct that merges SERVER OPTIONS (non-secret) with USER MAPPING OPTIONS (credentials). A GetIcebergCatalogConfig() function resolves the full configuration from a server name. Support current settings as defaults when not specified in server.
• Rework all settings consumers to use server properties -- all functions that currently read global GUC variables (RestCatalogHost, RestCatalogClientId, etc.) accept an IcebergCatalogConfig instead. OAuth token caching becomes per-server.
• GUC fallback for backward compatibility -- when a server option is not set, fall back to the corresponding GUC value.
• Upgrade path for existing users -- extension upgrade script creates the iceberg_catalog FDW and the postgres/object_store servers. Existing tables with catalog='postgres' or catalog='object_store' resolve seamlessly to these pre-created servers.
• Post-upgrade setup -- admin runs necessary grants (e.g., GRANT USAGE ON FOREIGN DATA WRAPPER iceberg_catalog TO ...) and creates USER MAPPINGs for REST catalogs.
• Secrets and security -- credentials live exclusively in USER MAPPING (access-controlled). Understand implications in logs, pg_stat_statements.
• Location rules -- define and enforce the location_prefix resolution hierarchy: per-table location > catalog server location_prefix > global GUC. Determine whether catalogs should enforce location constraints (e.g., all tables under a catalog must share the same bucket prefix).

[sfc-gh-mslot]

Sounds good

Rework all settings consumers to use server properties -- all functions that currently read global GUC variables (RestCatalogHost, RestCatalogClientId, etc.) accept an IcebergCatalogConfig instead. OAuth token caching becomes per-server.

probably good to start with this as a refactor

[sfc-gh-abozkurt]

USER MAPPING can still be inside extension script for the new catalogs (foreign servers). We need to make sure they are created in idempotent way.