Skip to content

feat(dashboard-api): supabase auth users sync background runner #2247

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ben-fornefeld wants to merge 23 commits into
base: main
Choose a base branch
from
Open

feat(dashboard-api): supabase auth users sync background runner #2247

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
c50d248
feat: supabase auth users sync background runner in dashboard api
ben-fornefeld Mar 28, 2026
68a6265
chore: auto-commit generated changes
github-actions[bot] Mar 28, 2026
9d8fe63
feat(db): enhance auth user sync triggers to retain direct operations…
ben-fornefeld Mar 30, 2026
1e5f28b
Merge branch 'feature/supabase-users-sync-worker' of https://github.c…
ben-fornefeld Mar 30, 2026
56154df
feat(sync): implement user sync queue with enhanced error handling an…
ben-fornefeld Mar 30, 2026
3a431aa
feat(sync): add supabase auth user sync configuration and secrets man…
ben-fornefeld Mar 31, 2026
eb69678
chore: remove smoke test
ben-fornefeld Mar 31, 2026
bb9fa39
add: e2e runner test
ben-fornefeld Mar 31, 2026
461d2ec
chore: change dashboard-api env variable management
ben-fornefeld Mar 31, 2026
ce0cbd1
refactor(sync): update user sync logic to utilize new database structure
ben-fornefeld Mar 31, 2026
4d622ce
fix: lint
ben-fornefeld Mar 31, 2026
84ecd06
test: apply database migrations in end-to-end test setup
ben-fornefeld Mar 31, 2026
3f56979
Merge origin/main into feature/supabase-users-sync-worker
ben-fornefeld Apr 1, 2026
386d0c6
feat(sync): enhance user sync processing and acknowledgment
ben-fornefeld Apr 1, 2026
3e3fa3c
refactor(gcp): remove auth_db_connection_string resources and update …
ben-fornefeld Apr 1, 2026
5e0972b
chore(dashboard-api): refactor environment variable management
ben-fornefeld Apr 1, 2026
6b5a5c8
refactor: use river for queue worker
ben-fornefeld Apr 1, 2026
1113654
chore: auto-commit generated changes
github-actions[bot] Apr 1, 2026
fb0afd4
chore: auto-commit generated changes
github-actions[bot] Apr 1, 2026
bd4dd80
chore: update pgx dependency to v5.9.1 and golang.org/x/mod to v0.34.…
ben-fornefeld Apr 1, 2026
97e305a
Merge branch 'feature/supabase-users-sync-worker' of https://github.c…
ben-fornefeld Apr 1, 2026
c98878f
chore: fix lint
ben-fornefeld Apr 2, 2026
e314b96
chore: auto-commit generated changes
github-actions[bot] Apr 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .env.gcp.template
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,10 @@ CLICKHOUSE_CLUSTER_SIZE=1

# Dashboard API instance count (default: 0)
DASHBOARD_API_COUNT=
# Additional non-reserved dashboard-api env vars passed directly to the Nomad job (default: {})
# Reserved keys managed by the module cannot be overridden here.
# Example: '{"SUPABASE_AUTH_USER_SYNC_ENABLED":"true"}'
DASHBOARD_API_ENV_VARS=

# Filestore cache for builds shared across cluster (default:false)
FILESTORE_CACHE_ENABLED=
Expand Down
14 changes: 3 additions & 11 deletions iac/modules/job-dashboard-api/jobs/dashboard-api.hcl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,17 +71,9 @@ job "dashboard-api" {
}

env {
GIN_MODE = "release"
ENVIRONMENT = "${environment}"
NODE_ID = "$${node.unique.id}"
PORT = "$${NOMAD_PORT_api}"
POSTGRES_CONNECTION_STRING = "${postgres_connection_string}"
AUTH_DB_CONNECTION_STRING = "${auth_db_connection_string}"
AUTH_DB_READ_REPLICA_CONNECTION_STRING = "${auth_db_read_replica_connection_string}"
CLICKHOUSE_CONNECTION_STRING = "${clickhouse_connection_string}"
SUPABASE_JWT_SECRETS = "${supabase_jwt_secrets}"
OTEL_COLLECTOR_GRPC_ENDPOINT = "${otel_collector_grpc_endpoint}"
LOGS_COLLECTOR_ADDRESS = "${logs_collector_address}"
%{ for key in sort(keys(env)) ~}
${key} = "${env[key]}"
%{ endfor ~}
}

config {
Expand Down
44 changes: 36 additions & 8 deletions iac/modules/job-dashboard-api/main.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,31 @@
locals {
base_env = {
GIN_MODE = "release"
ENVIRONMENT = var.environment
NODE_ID = "$${node.unique.id}"
PORT = "$${NOMAD_PORT_api}"
POSTGRES_CONNECTION_STRING = var.postgres_connection_string
AUTH_DB_CONNECTION_STRING = var.auth_db_connection_string
AUTH_DB_READ_REPLICA_CONNECTION_STRING = var.auth_db_read_replica_connection_string
CLICKHOUSE_CONNECTION_STRING = var.clickhouse_connection_string
SUPABASE_JWT_SECRETS = var.supabase_jwt_secrets
OTEL_COLLECTOR_GRPC_ENDPOINT = "localhost:${var.otel_collector_grpc_port}"
LOGS_COLLECTOR_ADDRESS = "http://localhost:${var.logs_proxy_port.port}"
}

extra_env = {
for key, value in var.extra_env : key => value
if value != null && trimspace(value) != ""
}

conflicting_extra_env_keys = sort(tolist(setintersection(
toset(keys(local.base_env)),
toset(keys(local.extra_env)),
)))

env = merge(local.base_env, local.extra_env)
}

Copy link
Copy Markdown

@cursor cursor bot Apr 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unused environment variable passed to HCL template

Low Severity

The environment = var.environment is still passed to templatefile() but is no longer referenced in dashboard-api.hcl. After the refactor, ENVIRONMENT is set via local.base_env through the env map loop. This is dead code left over from the migration.

Fix in Cursor Fix in Web

resource "nomad_job" "dashboard_api" {
jobspec = templatefile("${path.module}/jobs/dashboard-api.hcl", {
update_stanza = var.update_stanza
Expand All @@ -10,15 +38,15 @@ resource "nomad_job" "dashboard_api" {
memory_mb = 512
cpu_count = 1

postgres_connection_string = var.postgres_connection_string
auth_db_connection_string = var.auth_db_connection_string
auth_db_read_replica_connection_string = var.auth_db_read_replica_connection_string
clickhouse_connection_string = var.clickhouse_connection_string
supabase_jwt_secrets = var.supabase_jwt_secrets
env = local.env

subdomain = "dashboard-api"

otel_collector_grpc_endpoint = "localhost:${var.otel_collector_grpc_port}"
logs_collector_address = "http://localhost:${var.logs_proxy_port.port}"
})

lifecycle {
precondition {
condition = length(local.conflicting_extra_env_keys) == 0
error_message = "dashboard-api extra_env contains reserved keys: ${join(", ", local.conflicting_extra_env_keys)}"
}
}
}
5 changes: 5 additions & 0 deletions iac/modules/job-dashboard-api/variables.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,11 @@ variable "supabase_jwt_secrets" {
sensitive = true
}

variable "extra_env" {
type = map(string)
default = {}
}

variable "otel_collector_grpc_port" {
type = number
default = 4317
Expand Down
1 change: 1 addition & 0 deletions iac/provider-gcp/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,7 @@ tf_vars := \
$(call tfvar, LOKI_BOOT_DISK_TYPE) \
$(call tfvar, LOKI_USE_V13_SCHEMA_FROM) \
$(call tfvar, DASHBOARD_API_COUNT) \
$(call tfvar, DASHBOARD_API_ENV_VARS) \
$(call tfvar, DEFAULT_PERSISTENT_VOLUME_TYPE) \
$(call tfvar, PERSISTENT_VOLUME_TYPES) \
$(call tfvar, DB_MAX_OPEN_CONNECTIONS) \
Expand Down
3 changes: 2 additions & 1 deletion iac/provider-gcp/main.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -264,7 +264,8 @@ module "nomad" {
otel_collector_resources_cpu_count = var.otel_collector_resources_cpu_count

# Dashboard API
dashboard_api_count = var.dashboard_api_count
dashboard_api_count = var.dashboard_api_count
dashboard_api_env_vars = var.dashboard_api_env_vars

# Docker reverse proxy
docker_reverse_proxy_port = var.docker_reverse_proxy_port
Expand Down
1 change: 1 addition & 0 deletions iac/provider-gcp/nomad/main.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -135,6 +135,7 @@ module "dashboard_api" {
auth_db_read_replica_connection_string = trimspace(data.google_secret_manager_secret_version.postgres_read_replica_connection_string.secret_data)
clickhouse_connection_string = local.clickhouse_connection_string
supabase_jwt_secrets = trimspace(data.google_secret_manager_secret_version.supabase_jwt_secrets.secret_data)
extra_env = var.dashboard_api_env_vars

otel_collector_grpc_port = var.otel_collector_grpc_port
logs_proxy_port = var.logs_proxy_port
Expand Down
5 changes: 5 additions & 0 deletions iac/provider-gcp/nomad/variables.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -453,6 +453,11 @@ variable "dashboard_api_count" {
default = 0
}

variable "dashboard_api_env_vars" {
type = map(string)
default = {}
}

variable "volume_token_issuer" {
type = string
}
Expand Down
5 changes: 5 additions & 0 deletions iac/provider-gcp/variables.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -228,6 +228,11 @@ variable "dashboard_api_count" {
default = 0
}

variable "dashboard_api_env_vars" {
type = map(string)
default = {}
}

variable "docker_reverse_proxy_port" {
type = object({
name = string
Expand Down
4 changes: 2 additions & 2 deletions packages/api/go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ require (
github.com/google/uuid v1.6.0
github.com/grafana/loki/v3 v3.6.4
github.com/hashicorp/nomad/api v0.0.0-20251216171439-1dee0671280e
github.com/jackc/pgx/v5 v5.7.5
github.com/jackc/pgx/v5 v5.9.1
github.com/launchdarkly/go-sdk-common/v3 v3.3.0
github.com/launchdarkly/go-server-sdk/v7 v7.13.0
github.com/oapi-codegen/gin-middleware v1.0.2
Expand Down Expand Up @@ -381,7 +381,7 @@ require (
golang.org/x/crypto v0.48.0 // indirect
golang.org/x/exp v0.0.0-20260212183809-81e46e3db34a // indirect
golang.org/x/image v0.38.0 // indirect
golang.org/x/mod v0.33.0 // indirect
golang.org/x/mod v0.34.0 // indirect
golang.org/x/oauth2 v0.34.0 // indirect
golang.org/x/sys v0.41.0 // indirect
golang.org/x/text v0.35.0 // indirect
Expand Down
8 changes: 4 additions & 4 deletions packages/api/go.sum
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion packages/auth/go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ require (
github.com/jackc/pgerrcode v0.0.0-20250907135507-afb5586c32a6 // indirect
github.com/jackc/pgpassfile v1.0.0 // indirect
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
github.com/jackc/pgx/v5 v5.7.5 // indirect
github.com/jackc/pgx/v5 v5.9.1 // indirect
github.com/jackc/puddle/v2 v2.2.2 // indirect
github.com/jellydator/ttlcache/v3 v3.4.0 // indirect
github.com/josharian/intern v1.0.0 // indirect
Expand Down
4 changes: 2 additions & 2 deletions packages/auth/go.sum
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 2 additions & 2 deletions packages/clickhouse/go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ require (
github.com/grpc-ecosystem/grpc-gateway/v2 v2.27.3 // indirect
github.com/jackc/pgpassfile v1.0.0 // indirect
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
github.com/jackc/pgx/v5 v5.7.5 // indirect
github.com/jackc/pgx/v5 v5.9.1 // indirect
github.com/jackc/puddle/v2 v2.2.2 // indirect
github.com/joho/godotenv v1.5.1 // indirect
github.com/jonboulle/clockwork v0.5.0 // indirect
Expand Down Expand Up @@ -93,7 +93,7 @@ require (
go.uber.org/multierr v1.11.0 // indirect
golang.org/x/crypto v0.48.0 // indirect
golang.org/x/exp v0.0.0-20260212183809-81e46e3db34a // indirect
golang.org/x/mod v0.33.0 // indirect
golang.org/x/mod v0.34.0 // indirect
golang.org/x/net v0.50.0 // indirect
golang.org/x/sync v0.20.0 // indirect
golang.org/x/sys v0.41.0 // indirect
Expand Down
8 changes: 4 additions & 4 deletions packages/clickhouse/go.sum
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion packages/client-proxy/go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ require (
go.opentelemetry.io/proto/otlp v1.9.0 // indirect
go.uber.org/multierr v1.11.0 // indirect
golang.org/x/exp v0.0.0-20260212183809-81e46e3db34a // indirect
golang.org/x/mod v0.33.0 // indirect
golang.org/x/mod v0.34.0 // indirect
golang.org/x/net v0.50.0 // indirect
golang.org/x/sync v0.20.0 // indirect
golang.org/x/sys v0.41.0 // indirect
Expand Down
4 changes: 2 additions & 2 deletions packages/client-proxy/go.sum
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

10 changes: 8 additions & 2 deletions packages/dashboard-api/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,10 @@ endif
HOSTNAME := $(shell hostname 2> /dev/null || hostnamectl hostname 2> /dev/null)
$(if $(HOSTNAME),,$(error Failed to determine hostname: both 'hostname' and 'hostnamectl' failed))

define DASHBOARD_API_EXTRA_ENV
$$(printf '%s' "$${DASHBOARD_API_ENV_VARS:-}" | jq -r '(if .=="" then empty elif type=="string" then (fromjson? // empty) else . end) | to_entries? // [] | map("\(.key)=\(.value|tostring|@sh)") | join(" ")')
endef

.PHONY: generate
generate:
go generate ./...
Expand All @@ -33,12 +37,14 @@ build-and-upload:
.PHONY: run
run:
make build
./bin/dashboard-api
@EXTRA_ENV=$(DASHBOARD_API_EXTRA_ENV); \
eval "env $$EXTRA_ENV ./bin/dashboard-api"

.PHONY: run-local
run-local:
make build
NODE_ID=$(HOSTNAME) ./bin/dashboard-api
@EXTRA_ENV=$(DASHBOARD_API_EXTRA_ENV); \
eval "env NODE_ID=$(HOSTNAME) $$EXTRA_ENV ./bin/dashboard-api"

.PHONY: test
test:
Expand Down
20 changes: 15 additions & 5 deletions packages/dashboard-api/go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,9 +20,14 @@ require (
github.com/gin-contrib/cors v1.7.6
github.com/gin-gonic/gin v1.10.1
github.com/google/uuid v1.6.0
github.com/jackc/pgx/v5 v5.7.5
github.com/jackc/pgx/v5 v5.9.1
github.com/oapi-codegen/gin-middleware v1.0.2
github.com/oapi-codegen/runtime v1.1.1
github.com/riverqueue/river v0.32.0
github.com/riverqueue/river/riverdriver/riverpgxv5 v0.32.0
github.com/stretchr/testify v1.11.1
go.opentelemetry.io/otel v1.41.0
go.opentelemetry.io/otel/metric v1.41.0
go.uber.org/zap v1.27.1
)

Expand Down Expand Up @@ -112,14 +117,20 @@ require (
github.com/power-devops/perfstat v0.0.0-20240221224432-82ca36839d55 // indirect
github.com/pressly/goose/v3 v3.26.0 // indirect
github.com/redis/go-redis/v9 v9.17.3 // indirect
github.com/riverqueue/river/riverdriver v0.32.0 // indirect
github.com/riverqueue/river/rivershared v0.32.0 // indirect
github.com/riverqueue/river/rivertype v0.32.0 // indirect
github.com/segmentio/asm v1.2.0 // indirect
github.com/sethvargo/go-retry v0.3.0 // indirect
github.com/shirou/gopsutil/v4 v4.25.9 // indirect
github.com/shopspring/decimal v1.4.0 // indirect
github.com/sirupsen/logrus v1.9.3 // indirect
github.com/stretchr/testify v1.11.1 // indirect
github.com/testcontainers/testcontainers-go v0.40.0 // indirect
github.com/testcontainers/testcontainers-go/modules/postgres v0.39.0 // indirect
github.com/tidwall/gjson v1.18.0 // indirect
github.com/tidwall/match v1.2.0 // indirect
github.com/tidwall/pretty v1.2.1 // indirect
github.com/tidwall/sjson v1.2.5 // indirect
github.com/tklauser/go-sysconf v0.3.15 // indirect
github.com/tklauser/numcpus v0.10.0 // indirect
github.com/twitchyliquid64/golang-asm v0.15.1 // indirect
Expand All @@ -130,22 +141,21 @@ require (
go.opentelemetry.io/contrib/bridges/otelzap v0.14.0 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.64.0 // indirect
go.opentelemetry.io/contrib/instrumentation/runtime v0.66.0 // indirect
go.opentelemetry.io/otel v1.41.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploggrpc v0.15.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc v1.39.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.39.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.39.0 // indirect
go.opentelemetry.io/otel/log v0.15.0 // indirect
go.opentelemetry.io/otel/metric v1.41.0 // indirect
go.opentelemetry.io/otel/sdk v1.41.0 // indirect
go.opentelemetry.io/otel/sdk/log v0.15.0 // indirect
go.opentelemetry.io/otel/sdk/metric v1.41.0 // indirect
go.opentelemetry.io/otel/trace v1.41.0 // indirect
go.opentelemetry.io/proto/otlp v1.9.0 // indirect
go.uber.org/goleak v1.3.0 // indirect
go.uber.org/multierr v1.11.0 // indirect
golang.org/x/arch v0.18.0 // indirect
golang.org/x/crypto v0.48.0 // indirect
golang.org/x/mod v0.33.0 // indirect
golang.org/x/mod v0.34.0 // indirect
golang.org/x/net v0.50.0 // indirect
golang.org/x/sync v0.20.0 // indirect
golang.org/x/sys v0.41.0 // indirect
Expand Down
Loading
Loading