Add postgres_catalogs bundle resource#5265
Merged
Merged
Conversation
## Changes
New `postgres_catalogs` resource binding a Unity Catalog catalog to a Postgres database on a Lakebase Autoscaling branch. Supported on both direct and terraform deployment engines.
The spec fields are classified as both `recreate_on_changes` and `ignore_remote_changes: input_only`. The two cover orthogonal diffs the planner runs — recreate fires on local edits to an immutable field, and ignore_remote silences the phantom drift from GET not echoing spec back today. Lift the `input_only` entries once the backend starts returning spec.
## Tests
Acceptance coverage: `basic` and `recreate` exercise each engine, plus the existing `no_drift` and `migrate` invariants pick up the new resource. Both engines produce identical human-readable output and identical wire bodies; only the captured request streams diverge by filename (`out.requests.{direct,terraform}.json`).
Verified end to end on a live workspace: the bundle deploys a project and catalog, a row written directly into the bound Postgres database becomes visible through the UC federated view, and a follow-up write shows up on re-read.
This pull request and its description were written by Isaac.
This pull request and its description were written by Isaac.
The hand-written `branch`, `postgres_database`, and `create_database_if_missing` entries under `ignore_remote_changes` are already produced by the OpenAPI autogen (`spec:input_only`); `postgres_database` is also autogen'd under `recreate_on_changes` (`spec:immutable`). Drop the duplicates from `resources.yml`. `catalog_id` was in `ignore_remote_changes` only to mask the cosmetic `catalogs/` prefix that the old `RemapState` propagated from `remote.Name`. Source it from `remote.Status.CatalogId` instead — semantic contract from the API rather than string manipulation on the hierarchical path — and drop the entry. `catalog_id` stays in `recreate_on_changes` (synthetic hierarchical key, not in the API spec) along with `branch` and `create_database_if_missing` (no UpdateCatalog endpoint). Co-authored-by: Isaac
pietern
requested review from
denik and
janniklasrose
and removed request for
denik
denik
reviewed
May 19, 2026
| return "", nil, err | ||
| } | ||
|
|
||
| result, err := waiter.Wait(ctx) |
Contributor
There was a problem hiding this comment.
Can we put waiting into to WaitAfterCreate()? I understand we cannot use waiter convenience wrapper, but I'm sure there is API.
Splitting ensures that there is no orphaned resources if the process crashes during waiting.
The medium term plan is to move waiting into framework, splitting now would help.
Contributor
Author
There was a problem hiding this comment.
Similar discussion here: #4423 (comment)
The outcome there was to do a follow-up but that didn't happen.
I'll file a ticket for this on direct engine to solve this generically for LROs.
andrewnester
approved these changes
May 19, 2026
Contributor
andrewnester
left a comment
andrewnester
left a comment
There was a problem hiding this comment.
Do we support binding the catalogs? If so, worth adding an acceptance test as well
Direct and terraform engines produced identical output. Per the repo rule in .agent/rules/testing.md, only diverging files should be split into per-engine variants; this matches the precedent set by postgres_projects/basic. Co-authored-by: Isaac
Apply the same hoist that #5273 does for Branch/Endpoint/Project. Define PostgresCatalogRemote that embeds CatalogCatalogSpec and exposes the identifier and output-only fields at the top level. DoRead returns the new shape so state-side and remote-side paths line up, which is a prerequisite for drift detection on spec fields once the backend echoes spec on GET. Today the embedded fields are auto-classified spec:input_only from the API field behaviors in resources.generated.yml, so drift is correctly suppressed. Prompted by #5265 (comment). Should land after #5273. Co-authored-by: Isaac
# Conflicts: # NEXT_CHANGELOG.md # libs/testserver/fake_workspace.go
These annotations crept into the schema regeneration when the postgres_catalogs commit (3857f0f) was first generated; they belong to other fields and should not ride along with this PR. Keep only the postgres_catalogs additions. Co-authored-by: Isaac
# Conflicts: # acceptance/bundle/refschema/out.fields.txt
Contributor
Author
|
@andrewnester Bind/unbind is not explicitly tested yet for Postgres resources. It should work just not tested. |
pietern
added a commit
that referenced
this pull request
May 20, 2026
- Drop the duplicate postgres_catalogs block that the merge pulled in alongside the existing one in resources.yml. - Remove postgres_catalogs from knownMissingInRemoteType now that the new PostgresCatalogRemote shim from #5265 surfaces the spec fields. Co-authored-by: Isaac
pietern
added a commit
that referenced
this pull request
May 20, 2026
Adopt the same embedded-spec Remote pattern that #5273 / #5265 introduced for postgres_catalogs: PostgresSyncedTableRemote embeds SyncedTableSyncedTableSpec plus output-only fields, so every StateType path is also a valid RemoteType path. RemapState just copies the embedded shape; drift on spec fields is suppressed via the spec:input_only classifications generated from the OpenAPI schema until GET starts echoing the spec. Drop the now-empty postgres_synced_tables entry from knownMissingInRemoteType, and regenerate acceptance/bundle/refschema/out.fields.txt so the embedded spec fields show up as ALL rather than INPUT|STATE. Co-authored-by: Isaac
bernardo-rodriguez
pushed a commit
to bernardo-rodriguez/b-cli
that referenced
this pull request
May 21, 2026
…cks#5273) Define `Postgres{Branch,Endpoint,Project}Remote` types that embed the corresponding SDK `*Spec` and expose the identifier and output-only fields at the top level. `DoRead` returns the new shape so state-side and remote-side paths line up, which is a prerequisite for drift detection on spec fields once the backend starts echoing spec on GET. Today the embedded fields are auto-classified `spec:input_only` from the API field behaviors in `resources.generated.yml`, so drift is correctly suppressed. Follows the pattern set by `PipelineRemote` and `AppRemote`. Prompted by databricks#5265 (comment). This pull request and its description were written by Isaac.
deco-sdk-tagging Bot
added a commit
that referenced
this pull request
May 21, 2026
## Release v1.0.0 ### Notable Changes * The Databricks CLI is now generally available with version v1.0.0 as the first major release 🚀. From this version on, the CLI follows semantic versioning (see [README](README.md)). This change does not impact DABs or other existing commands beyond the changes listed below. * The 0.299.x line continues to receive security-critical patches through May 20, 2027; see [SECURITY](SECURITY.md) for the support policy. * Starting with v1.0.0, the CLI will use [immutable release tags](https://docs.github.com/en/code-security/concepts/supply-chain-security/immutable-releases) to increase security against supply chain attacks. * Breaking change: OAuth tokens for interactive logins (`auth_type = databricks-cli`) are now stored in the OS-native secure store by default (Keychain on macOS, Credential Manager on Windows, Secret Service on Linux) instead of `~/.databricks/token-cache.json`. After upgrading, run `databricks auth login` once per profile to re-authenticate; cached tokens from older versions are not migrated. To keep the previous file-backed storage, set `DATABRICKS_AUTH_STORAGE=plaintext` or add `auth_storage = plaintext` under `[__settings__]` in `~/.databrickscfg` (the env var takes precedence over the config setting), then re-run `databricks auth login`. On systems where the OS keyring is not reachable (e.g. Linux containers without a D-Bus session bus), the CLI transparently falls back to the file cache when reading tokens so legacy `token-cache.json` entries remain accessible without manual configuration. ### CLI * Added `databricks aitools` command group for installing Databricks skills into your coding agents (Claude Code, Cursor, Codex CLI, OpenCode, GitHub Copilot, Antigravity). Skills are fetched from [github.com/databricks/databricks-agent-skills](https://github.com/databricks/databricks-agent-skills) and either symlinked into each agent's skills directory or copied into the current project. Use `databricks aitools install` to set up, `update` to pull newer versions, `list` to see what's available, and `uninstall` to remove them. Pick where they go with `--scope=project|global` (`--scope=both` is accepted on `update` and `list`). * `[__settings__].default_profile` is now consulted as a fallback by `databricks api`, `databricks auth token`, and bundle commands when neither `--profile` nor `DATABRICKS_CONFIG_PROFILE` is set. `databricks auth token` continues to give precedence to `DATABRICKS_HOST` over `default_profile`. For bundle commands, `default_profile` only applies when the bundle does not pin its own `workspace.host`. * Fixed bug where auth commands did not load the DEFAULT profile properly during auth where type is `databricks-cli`. * `databricks workspace import-dir` now skips `.git`, `.databricks`, and `node_modules` directories during recursive imports. To import one of these directories deliberately, pass it as `SOURCE_PATH` ([#5118](#5118)). * `databricks postgres create-role --help` now documents the `--json` body shape and rejects the common mistake of wrapping the body in `{"role": ...}` client-side with a hint pointing at the correct shape ([#5111](#5111)). * `databricks aitools list` honors `--output json`, emitting a structured `{release, skills[...], summary{}}` document so coding agents and CI can consume the skill/version/installation matrix without scraping the tabular text output ([#5233](#5233)). ### Bundles * Make sure warnings asking for approval are understood by agents ([#5239](#5239)) * Support `replace_existing: true` on `postgres_branches` and `postgres_endpoints` so bundles can manage the implicitly-created production branch and primary read-write endpoint of a Lakebase project. * Add `postgres_catalogs` resource to bind a Unity Catalog catalog to a Postgres database on a Lakebase Autoscaling branch ([#5265](#5265)). * Add `postgres_synced_tables` resource to sync a Unity Catalog Delta table into a Postgres table on a Lakebase Autoscaling branch ([#5268](#5268)). * engine/direct: Changes to state file now persisted to .wal file right away instead of being saved in the end ([#5149](#5149))
TanishqDatabricks
pushed a commit
to TanishqDatabricks/cli
that referenced
this pull request
May 22, 2026
…cks#5273) Define `Postgres{Branch,Endpoint,Project}Remote` types that embed the corresponding SDK `*Spec` and expose the identifier and output-only fields at the top level. `DoRead` returns the new shape so state-side and remote-side paths line up, which is a prerequisite for drift detection on spec fields once the backend starts echoing spec on GET. Today the embedded fields are auto-classified `spec:input_only` from the API field behaviors in `resources.generated.yml`, so drift is correctly suppressed. Follows the pattern set by `PipelineRemote` and `AppRemote`. Prompted by databricks#5265 (comment). This pull request and its description were written by Isaac.
TanishqDatabricks
pushed a commit
to TanishqDatabricks/cli
that referenced
this pull request
May 22, 2026
## Changes
New `postgres_catalogs` resource binding a Unity Catalog catalog to a
Postgres database on a Lakebase Autoscaling branch. Supported on both
direct and terraform deployment engines.
## Tests
Acceptance coverage: `basic` and `recreate` exercise each engine, plus
the existing `no_drift` and `migrate` invariants pick up the new
resource. Both engines produce identical human-readable output and
identical wire bodies; only the captured request streams diverge by
filename (`out.requests.{direct,terraform}.json`).
Verified end to end on a live workspace: the bundle deploys a project
and catalog, a row written directly into the bound Postgres database
becomes visible through the UC federated view, and a follow-up write
shows up on re-read.
This pull request and its description were written by Isaac.
TanishqDatabricks
pushed a commit
to TanishqDatabricks/cli
that referenced
this pull request
May 22, 2026
## Release v1.0.0 ### Notable Changes * The Databricks CLI is now generally available with version v1.0.0 as the first major release 🚀. From this version on, the CLI follows semantic versioning (see [README](README.md)). This change does not impact DABs or other existing commands beyond the changes listed below. * The 0.299.x line continues to receive security-critical patches through May 20, 2027; see [SECURITY](SECURITY.md) for the support policy. * Starting with v1.0.0, the CLI will use [immutable release tags](https://docs.github.com/en/code-security/concepts/supply-chain-security/immutable-releases) to increase security against supply chain attacks. * Breaking change: OAuth tokens for interactive logins (`auth_type = databricks-cli`) are now stored in the OS-native secure store by default (Keychain on macOS, Credential Manager on Windows, Secret Service on Linux) instead of `~/.databricks/token-cache.json`. After upgrading, run `databricks auth login` once per profile to re-authenticate; cached tokens from older versions are not migrated. To keep the previous file-backed storage, set `DATABRICKS_AUTH_STORAGE=plaintext` or add `auth_storage = plaintext` under `[__settings__]` in `~/.databrickscfg` (the env var takes precedence over the config setting), then re-run `databricks auth login`. On systems where the OS keyring is not reachable (e.g. Linux containers without a D-Bus session bus), the CLI transparently falls back to the file cache when reading tokens so legacy `token-cache.json` entries remain accessible without manual configuration. ### CLI * Added `databricks aitools` command group for installing Databricks skills into your coding agents (Claude Code, Cursor, Codex CLI, OpenCode, GitHub Copilot, Antigravity). Skills are fetched from [github.com/databricks/databricks-agent-skills](https://github.com/databricks/databricks-agent-skills) and either symlinked into each agent's skills directory or copied into the current project. Use `databricks aitools install` to set up, `update` to pull newer versions, `list` to see what's available, and `uninstall` to remove them. Pick where they go with `--scope=project|global` (`--scope=both` is accepted on `update` and `list`). * `[__settings__].default_profile` is now consulted as a fallback by `databricks api`, `databricks auth token`, and bundle commands when neither `--profile` nor `DATABRICKS_CONFIG_PROFILE` is set. `databricks auth token` continues to give precedence to `DATABRICKS_HOST` over `default_profile`. For bundle commands, `default_profile` only applies when the bundle does not pin its own `workspace.host`. * Fixed bug where auth commands did not load the DEFAULT profile properly during auth where type is `databricks-cli`. * `databricks workspace import-dir` now skips `.git`, `.databricks`, and `node_modules` directories during recursive imports. To import one of these directories deliberately, pass it as `SOURCE_PATH` ([databricks#5118](databricks#5118)). * `databricks postgres create-role --help` now documents the `--json` body shape and rejects the common mistake of wrapping the body in `{"role": ...}` client-side with a hint pointing at the correct shape ([databricks#5111](databricks#5111)). * `databricks aitools list` honors `--output json`, emitting a structured `{release, skills[...], summary{}}` document so coding agents and CI can consume the skill/version/installation matrix without scraping the tabular text output ([databricks#5233](databricks#5233)). ### Bundles * Make sure warnings asking for approval are understood by agents ([databricks#5239](databricks#5239)) * Support `replace_existing: true` on `postgres_branches` and `postgres_endpoints` so bundles can manage the implicitly-created production branch and primary read-write endpoint of a Lakebase project. * Add `postgres_catalogs` resource to bind a Unity Catalog catalog to a Postgres database on a Lakebase Autoscaling branch ([databricks#5265](databricks#5265)). * Add `postgres_synced_tables` resource to sync a Unity Catalog Delta table into a Postgres table on a Lakebase Autoscaling branch ([databricks#5268](databricks#5268)). * engine/direct: Changes to state file now persisted to .wal file right away instead of being saved in the end ([databricks#5149](databricks#5149))
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
New
postgres_catalogsresource binding a Unity Catalog catalog to a Postgres database on a Lakebase Autoscaling branch. Supported on both direct and terraform deployment engines.Tests
Acceptance coverage:
basicandrecreateexercise each engine, plus the existingno_driftandmigrateinvariants pick up the new resource. Both engines produce identical human-readable output and identical wire bodies; only the captured request streams diverge by filename (out.requests.{direct,terraform}.json).Verified end to end on a live workspace: the bundle deploys a project and catalog, a row written directly into the bound Postgres database becomes visible through the UC federated view, and a follow-up write shows up on re-read.
This pull request and its description were written by Isaac.