Skip to content

[backend][postgres] racing SetupAndMigrateDynamic when setting up the backend can cause auth to fail to start #69315

Description

@okraport

I discovered this when working on Antithesis SUT when both auth instances start very close to each other.

Expected behavior:
already exists (SQLSTATE 42P07) is handled gracefully when one of the auth instances wins the write.

Current behavior:
The other auth instances hard exits on failure.

Bug details:

[        16.790] [               auth2] [err] 2026-08-03T15:45:46.509Z INFO [PROC:1]    Service is creating new listener. pid:1075.1 type:debug address:/var/lib/teleport/debug.sock service/signals.go:242
[        16.792] [               auth2] [err] 2026-08-03T15:45:46.511Z INFO [PGBK]      Setting up backend. pgbk/pgbk.go:187
[        16.794] [               auth1] [err] 2026-08-03T15:45:46.513Z INFO [PROC:1]    Service is creating new listener. pid:1075.1 type:debug address:/var/lib/teleport/debug.sock service/signals.go:242
[        16.796] [               auth1] [err] 2026-08-03T15:45:46.515Z INFO [PGBK]      Setting up backend. pgbk/pgbk.go:187
[        16.802] [                  pg] [err] 2026-08-03 15:45:46.521 GMT [97] ERROR:  duplicate key value violates unique constraint "pg_database_datname_index"
[        16.802] [                  pg] [err] 2026-08-03 15:45:46.521 GMT [97] DETAIL:  Key (datname)=(teleport_backend) already exists.
[        16.802] [                  pg] [err] 2026-08-03 15:45:46.521 GMT [97] STATEMENT:  CREATE DATABASE "teleport_backend" TEMPLATE template0 ENCODING UTF8 LC_COLLATE 'C' LC_CTYPE 'C'
[        16.802] [               auth1] [err] 2026-08-03T15:45:46.521Z WARN [PGBK]      Error creating database. error:[ERROR: duplicate key value violates unique constraint "pg_database_datname_index" (SQLSTATE 23505)] common/utils.go:86
[        16.808] [                  pg] [err] 2026-08-03 15:45:46.527 GMT [98] ERROR:  duplicate key value violates unique constraint "pg_type_typname_nsp_index"
[        16.808] [                  pg] [err] 2026-08-03 15:45:46.527 GMT [98] DETAIL:  Key (typname, typnamespace)=(backend_version, 2200) already exists.
[        16.808] [                  pg] [err] 2026-08-03 15:45:46.527 GMT [98] STATEMENT:  CREATE TABLE IF NOT EXISTS "backend_version" (
[        16.808] [                  pg] [err] 					version integer PRIMARY KEY CHECK (version > 0),
[        16.808] [                  pg] [err] 					created timestamptz NOT NULL DEFAULT now()
[        16.808] [                  pg] [err] 				)
[        16.809] [               auth1] [err] 2026-08-03T15:45:46.528Z INFO [PGBK]      Migrated database schema. previous_version:0 current_version:2 common/utils.go:373
[        16.809] [                  pg] [err] 2026-08-03 15:45:46.528 GMT [98] ERROR:  relation "kv" already exists
[        16.809] [                  pg] [err] 2026-08-03 15:45:46.528 GMT [98] STATEMENT:  CREATE TABLE kv (
[        16.809] [                  pg] [err] 			key bytea NOT NULL,
[        16.809] [                  pg] [err] 			value bytea NOT NULL,
[        16.809] [                  pg] [err] 			expires timestamptz,
[        16.809] [                  pg] [err] 			revision uuid NOT NULL,
[        16.809] [                  pg] [err] 			CONSTRAINT kv_pkey PRIMARY KEY (key)
[        16.809] [                  pg] [err] 		);
[        16.809] [                  pg] [err] 		CREATE INDEX kv_expires_idx ON kv (expires) WHERE expires IS NOT NULL;
[        16.810] [               auth1] [err] 2026-08-03T15:45:46.529Z INFO [PGBK]      Starting change feed stream. pgbk/background.go:103
[        16.810] [               auth2] [err] 2026-08-03T15:45:46.528Z ERRO [PROC:1]    Failed to start Teleport instance pid:1075.1 error:[
[        16.810] [               auth2] [err] ERROR REPORT:
[        16.810] [               auth2] [err] Original Error: *pgconn.PgError ERROR: relation "kv" already exists (SQLSTATE 42P07)
[        16.810] [               auth2] [err] Stack Trace:
[        16.810] [               auth2] [err] 	lib/backend/pgbk/common/utils.go:351 github.com/gravitational/teleport/lib/backend/pgbk/common.SetupAndMigrateDynamic.func2
[        16.810] [               auth2] [err] 	/go/pkg/mod/github.com/jackc/pgx/v5@v5.9.2/tx.go:435 github.com/jackc/pgx/v5.beginFuncExec
[        16.810] [               auth2] [err] 	/go/pkg/mod/github.com/jackc/pgx/v5@v5.9.2/tx.go:424 github.com/jackc/pgx/v5.BeginTxFunc
[        16.810] [               auth2] [err] 	lib/backend/pgbk/common/utils.go:223 github.com/gravitational/teleport/lib/backend/pgbk/common.RetryTx.func1
[        16.810] [               auth2] [err] 	lib/backend/pgbk/common/utils.go:120 github.com/gravitational/teleport/lib/backend/pgbk/common.retry[...]
[        16.810] [               auth2] [err] 	lib/backend/pgbk/common/utils.go:222 github.com/gravitational/teleport/lib/backend/pgbk/common.RetryTx
[        16.810] [               auth2] [err] 	lib/backend/pgbk/common/utils.go:319 github.com/gravitational/teleport/lib/backend/pgbk/common.SetupAndMigrateDynamic
[        16.810] [               auth2] [err] 	lib/backend/pgbk/common/utils.go:267 github.com/gravitational/teleport/lib/backend/pgbk/common.SetupAndMigrate
[        16.810] [               auth2] [err] 	lib/backend/pgbk/pgbk.go:196 github.com/gravitational/teleport/lib/backend/pgbk.NewWithConfig
[        16.810] [               auth2] [err] 	lib/backend/pgbk/pgbk.go:136 github.com/gravitational/teleport/lib/backend/pgbk.NewFromParams
[        16.810] [               auth2] [err] 	lib/backend/pgbk/pgbk.go:44 github.com/gravitational/teleport/lib/backend/pgbk.init.0.func1
[        16.810] [               auth2] [err] 	lib/backend/backend.go:144 github.com/gravitational/teleport/lib/backend.New
[        16.810] [               auth2] [err] 	lib/service/service.go:7209 github.com/gravitational/teleport/lib/service.(*TeleportProcess).initAuthStorage
[        16.810] [               auth2] [err] 	lib/service/service.go:2239 github.com/gravitational/teleport/lib/service.(*TeleportProcess).initAuthService
[        16.810] [               auth2] [err] 	lib/service/service.go:1612 github.com/gravitational/teleport/lib/service.NewTeleport
[        16.810] [               auth2] [err] 	lib/service/service.go:1095 github.com/gravitational/teleport/lib/service.newTeleportProcess
[        16.810] [               auth2] [err] 	lib/service/service.go:1121 github.com/gravitational/teleport/lib/service.RunWithSignalChannel
[        16.810] [               auth2] [err] 	lib/service/service.go:1109 github.com/gravitational/teleport/lib/service.Run
[        16.810] [               auth2] [err] 	tool/teleport/common/teleport.go:952 github.com/gravitational/teleport/tool/teleport/common.OnStart
[        16.810] [               auth2] [err] 	tool/teleport/common/teleport.go:747 github.com/gravitational/teleport/tool/teleport/common.Run
[        16.810] [               auth2] [err] 	tool/teleport/main.go:36 main.main
[        16.810] [               auth2] [err] 	/usr/local/go/src/runtime/proc.go:285 runtime.main
[        16.810] [               auth2] [err] 	/usr/local/go/src/runtime/asm_amd64.s:1693 runtime.goexit
[        16.810] [               auth2] [err] User Message: ERROR: relation "kv" already exists (SQLSTATE 42P07)] service/service.go:1153
[        16.810] [               auth2] [err] �[31mERROR: �[0minitialization failed
[        16.810] [               auth2] [err] 	ERROR: relation "kv" already exists (SQLSTATE 42P07)
[        16.810] [               auth2] [err] 

Metadata

Metadata

Assignees

Labels

backend/pgkbIssues related to the postgres backend

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions