feat: support percentage based db limits with reload support (#2177)

## Summary
Introduce a context aware DB dial path, a new `ConnPercentage` knob to
cap Auth's share of Postgres connections, and background wiring to apply
pool changes on config reloads.

**Storage / DB**
- Add `DialContext(ctx, *conf.GlobalConfiguration)` and keep `Dial(...)`
as a thin wrapper. `serve` now passes its cancelable context so startup
can't hang indefinitely.
- `Connection` now keeps a handle to the underlying `*sql.DB` (via
`popConnToStd`) when available.
- New helpers:
- `newConnectionDetails` and `applyDBDriver` to build
`pop.ConnectionDetails` and derive driver when omitted.
- `Connection.Copy()` to retain `sqldb` reference and updated locations
that copy (`WithContext, Transaction)`.
- Runtime tuning API: `(*Connection).ApplyConfig(ctx, cfg, le)` computes
and applies connection limits to the underlying `*sql.DB`.
- Fixed limits come from `MaxPoolSize`, `MaxIdlePoolSize`,
`ConnMaxLifetime`, `ConnMaxIdleTime`.
- If `ConnPercentage` is set (1-100), compute limits from `SHOW
max_connections`, prefer percentage over fixed pool sizes, and set idle
= open.
  - Retains previous behavior when `ConnPercentage` is `0`
  - No-op (and error) if `*sql.DB` is unavailable.

**API worker**
- `apiworker.New` now accepts the DB connection.
- Split worker into three goroutines (via `errgroup`):
  - `configNotifier` fans out reload signals,
  - `templateWorker` refreshes template cache,
  - `dbWorker` applies DB connection limits on boot and each reload.

**Serve**
- Use `storage.DialContext(ctx, cfg)` and then `db =
db.WithContext(ctx)` so the DB handle participates in request/trace
context and shutdown.

**Observability**
- Add `observability.NewLogEntry(*logrus.Entry)` to construct chi
middleware log entries.
- Structured logs around applying DB limits.

**Configuration knobs** (`GOTRUE_DB_*`)
- `GOTRUE_DB_CONN_PERCENTAGE` (int, clamped to `[0,100]`):
  - `0` (default) disables percentage-based sizing.
  - `1-100` reserves that % of `max_connections` for the Auth server.

**Tests**
- `internal/storage/dial_test.go`
  - `DialContext` happy path and invalid driver/URL error path.
- Reflection bridge to `*sql.DB` (`popConnToStd`) including
`WithContext`-wrapped connection behavior.
- `ApplyConfig` end-to-end: verify pool sizing and stats reflect limits.
  - Percentage math and precedence vs fixed pools across edge cases.
- `internal/conf/configuration_test.go`
  - Validation clamps `ConnPercentage` to `[0,100]`.

## How it works
In short if `GOTRUE_DB_CONN_PERCENTAGE=0`, we use the fixed
`GOTRUE_DB_{MAX,CONN}_*` limits. If it's in the range `[1, 100]` we set
percentage based limits derived from `SHOW max_connections` and ignore
the fixed pool sizes.

### Deep Dive
The startup sequence remains the same, trying to set it _before_ we
returned from `DialContext` was a bit messy (chicken / egg: need a conn
to setup a conn). I also didn't want to delay startup time during
failure scenarios (db is unavailable, db is blocking, etc).

So after `DialContext` we have a connection which is configured
initially with only the existing DB settings:
```bash
GOTRUE_DB_MAX_POOL_SIZE="50"
GOTRUE_DB_MAX_IDLE_POOL_SIZE="10"
GOTRUE_DB_CONN_MAX_IDLE_TIME="60s"
GOTRUE_DB_CONN_MAX_LIFETIME="0"
```

Next the server starts the `apiworker` which immediatelly creates a new
[dbWorker](https://github.com/supabase/auth/pull/2177/files#diff-b20c1e9d1c21d077494cf5ff490de301a864d3d1812538cf594a687f620a7175R122)
goroutine concurrently while the rest of the startup sequence continues.
Before entering the config update loop the `dbWorker` will call the
newly added
[ApplyConfig(...)](https://github.com/supabase/auth/pull/2177/files#diff-5b7e4f0f03bfbc3a58168e58eb88386b9e683241c1ebcb57f6764c38308f2257R179)
method on the `*storage.Connection`.

The `ApplyConfig` method is where the logic for obtaining the best
values to call the `sql.DB.Set*(...)` methods below lives:
* [SetConnMaxIdleTime(d
time.Duration)](https://pkg.go.dev/database/sql#DB.SetConnMaxIdleTime)
* [SetConnMaxLifetime(d
time.Duration)](https://pkg.go.dev/database/sql#DB.SetConnMaxLifetime)
* [SetMaxIdleConns(n
int)](https://pkg.go.dev/database/sql#DB.SetMaxIdleConns)
* [SetMaxOpenConns(n
int)](https://pkg.go.dev/database/sql#DB.SetMaxOpenConns)

Right now
[ApplyConfig](https://github.com/supabase/auth/blob/4b5bc8d08fb4fbdf778f504f226da29246d34e84/internal/storage/dial.go#L179)
works like this:
1. Checks that we were able to
[reflect](https://github.com/supabase/auth/blob/4b5bc8d08fb4fbdf778f504f226da29246d34e84/internal/storage/dial.go#L140)
a `*sql.DB` during `DialContext`, if not we do nothing since we can't
call `sql.DB.Set*(...)`.
* If we can't access *sql.DB or fetch max_connections, we leave the
prior limits untouched and log a warning. Always all-or-nothing, no
partial application of limits.
2. Calls the new
[getConnLimits](https://github.com/supabase/auth/blob/4b5bc8d08fb4fbdf778f504f226da29246d34e84/internal/storage/dial.go#L209)
method.
3. `getConnLimits` calls
[newConnLimitsFromConfig](https://github.com/supabase/auth/blob/4b5bc8d08fb4fbdf778f504f226da29246d34e84/internal/storage/dial.go#L287)
which returns a `ConnLimits` setup with `GOTRUE_DB_{MAX,CONN}_*`
settings.
4. Check if `GOTRUE_DB_CONN_PERCENTAGE` is zero, if so [it
returns](https://github.com/supabase/auth/blob/4b5bc8d08fb4fbdf778f504f226da29246d34e84/internal/storage/dial.go#L218)
the `GOTRUE_DB_{MAX,CONN}_*` from `newConnLimitsFromConfig`.
    * This means the limits are set exactly as they are today.
5. Percentage config is non-zero so we make a call to
[showMaxConns](https://github.com/supabase/auth/blob/4b5bc8d08fb4fbdf778f504f226da29246d34e84/internal/storage/dial.go#L266)
which just returns an integer from `"SHOW max_connections;"`. In my
testing this value always seems to be available for the auth server:
	* This value cannot change without postgres restarts.
	* Postgres will not start if it is 0.
	* Being in recovery mode still shows the maximum connections.
6. As long as `showMaxConns` does not return an error we attempt to
apply percentage based connection limits in
[applyPercentageLimits](https://github.com/supabase/auth/blob/4b5bc8d08fb4fbdf778f504f226da29246d34e84/internal/storage/dial.go#L235).
7. If max conns is <= 0 we return [an
error](https://github.com/supabase/auth/blob/4b5bc8d08fb4fbdf778f504f226da29246d34e84/internal/storage/dial.go#L245)
which prevents any config changes from being applied. Leaving the
connection in its prior state.
* max_connections > 0 is guaranteed if postgres is running, this is a
defensive check to prevent applying a clamp to 1 max conns on 0.
8. We perform a simple [bounds
check](https://github.com/supabase/auth/blob/4b5bc8d08fb4fbdf778f504f226da29246d34e84/internal/storage/dial.go#L252)
and then set the [`MaxOpenConns` and
`MaxIdleConns`](https://github.com/supabase/auth/blob/4b5bc8d08fb4fbdf778f504f226da29246d34e84/internal/storage/dial.go#L257)
to the values derived from the `ConnPercentage` and `maxConns`.
* Note that we preserve the existing behavior of IdleConns == MaxConns.
I believe the aim is to minimize connection churn (latency) at the cost
of more Postgres slots when idle. It's worth thinking about making this
a bit more considerate in the future, something simple like (open/2) or
more advanced heuristics using
[sql.DBStats](https://pkg.go.dev/database/sql#DBStats).

	```Go
	pct := float64(dbCfg.ConnPercentage)
	cl.MaxOpenConns = int(max(1, (pct/100)*float64(maxConns)))
	cl.MaxIdleConns = cl.MaxOpenConns
	```

9. The values set from the call to `getConnLimits` are logged before
being [applied via the
`sql.DB.Set*(...)`](https://github.com/supabase/auth/blob/4b5bc8d08fb4fbdf778f504f226da29246d34e84/internal/storage/dial.go#L202)
calls.

We fail strictly and quickly on derivation errors to keep the last known
good settings. By supporting config reloading my hope is that when under
high load users may balance this setting without taking down the auth
server. This tight feedback loop should help rule out (or resolve) the
auth server as a potential root cause to connection timeouts and similar
downstream effects.

@stojan `apiworker` approach also gives a good place for your [stats
tracking](#2167) to live, adding a
simple ticker in the `dbWorker` to poll stats between config updates.
This stats polling could be used to form additional heuristics in our
connect limit tuning if we would like to explore that in the future. For
example use the mean connection time as an additional weight to further
increase the pool size.


## Some notes:
I tested this extensively but please give a thorough review, I made some
judgement calls on non-happy paths. I'm also not sure how reliable the
sqldb reference is as it seems the composition of the *pop.Store can
change based on inputs, context, dialect, driver, etc. The entire
feature will not work if I can't reflect out the sqldb.

---------

Co-authored-by: Chris Stockton <[email protected]>

Author: cstockton

cmd/serve_cmd.go

@@ -44,7 +44,9 @@ func serve(ctx context.Context) {
 		logrus.WithError(err).Fatal("unable to load config")
 	}
 
-	db, err := storage.Dial(config)
+	// Include serve ctx which carries cancelation signals so DialContext does
+	// not hang indefinitely at startup.
+	db, err := storage.DialContext(ctx, config)
 	if err != nil {
 		logrus.Fatalf("error opening database: %+v", err)
 	}
@@ -53,6 +55,10 @@ func serve(ctx context.Context) {
 	baseCtx, baseCancel := context.WithCancel(context.Background())
 	defer baseCancel()
 
+	// Add the base context to the db, this is so during the shutdown sequence
+	// the DB will be available while connections drain.
+	db = db.WithContext(ctx)
+
 	var wg sync.WaitGroup
 	defer wg.Wait() // Do not return to caller until this goroutine is done.
 
@@ -79,7 +85,7 @@ func serve(ctx context.Context) {
 	log := logrus.WithField("component", "api")
 
 	wrkLog := logrus.WithField("component", "apiworker")
-	wrk := apiworker.New(config, mrCache, wrkLog)
+	wrk := apiworker.New(config, mrCache, db, wrkLog)
 	wg.Add(1)
 	go func() {
 		defer wg.Done()

internal/api/apiworker/apiworker.go

@@ -9,12 +9,15 @@ import (
 	"github.com/sirupsen/logrus"
 	"github.com/supabase/auth/internal/conf"
 	"github.com/supabase/auth/internal/mailer/templatemailer"
+	"github.com/supabase/auth/internal/storage"
+	"golang.org/x/sync/errgroup"
 )
 
 // Worker is a simple background worker for async tasks.
 type Worker struct {
 	le *logrus.Entry
 	tc *templatemailer.Cache
+	db *storage.Connection
 
 	// Notifies worker the cfg has been updated.
 	cfgCh chan struct{}
@@ -31,12 +34,14 @@ type Worker struct {
 func New(
 	cfg *conf.GlobalConfiguration,
 	tc *templatemailer.Cache,
+	db *storage.Connection,
 	le *logrus.Entry,
 ) *Worker {
 	return &Worker{
 		le:    le,
 		cfg:   cfg,
 		tc:    tc,
+		db:    db,
 		cfgCh: make(chan struct{}, 1),
 	}
 }
@@ -63,14 +68,85 @@ func (o *Worker) ReloadConfig(cfg *conf.GlobalConfiguration) {
 	}
 }
 
-// Work will periodically reload the templates in the background as long as the
-// system remains active.
+// Work will run background workers.
 func (o *Worker) Work(ctx context.Context) error {
 	if ok := o.workMu.TryLock(); !ok {
 		return errors.New("apiworker: concurrent calls to Work are invalid")
 	}
 	defer o.workMu.Unlock()
 
+	var (
+		eg        errgroup.Group
+		notifyTpl = make(chan struct{}, 1)
+		notifyDb  = make(chan struct{}, 1)
+	)
+	eg.Go(func() error {
+		return o.configNotifier(ctx, notifyTpl, notifyDb)
+	})
+	eg.Go(func() error {
+		return o.templateWorker(ctx, notifyTpl)
+	})
+	eg.Go(func() error {
+		return o.dbWorker(ctx, notifyDb)
+	})
+	return eg.Wait()
+}
+
+func (o *Worker) configNotifier(
+	ctx context.Context,
+	notifyCh ...chan<- struct{},
+) error {
+	le := o.le.WithFields(logrus.Fields{
+		"worker_type": "apiworker_config_notifier",
+	})
+	le.Info("apiworker: config notifier started")
+	defer le.Info("apiworker: config notifier exited")
+
+	for {
+		select {
+		case <-ctx.Done():
+			return ctx.Err()
+		case <-o.cfgCh:
+
+			// When we get a config update, notify each worker to wake up
+			for _, ch := range notifyCh {
+				select {
+				case ch <- struct{}{}:
+				default:
+				}
+			}
+		}
+	}
+}
+
+func (o *Worker) dbWorker(ctx context.Context, cfgCh <-chan struct{}) error {
+	le := o.le.WithFields(logrus.Fields{
+		"worker_type": "apiworker_db_worker",
+	})
+	le.Info("apiworker: db worker started")
+	defer le.Info("apiworker: db worker exited")
+
+	if err := o.db.ApplyConfig(ctx, o.getConfig(), le); err != nil {
+		le.WithError(err).Error(
+			"failure applying config connection limits to db")
+	}
+
+	for {
+		select {
+		case <-ctx.Done():
+			return ctx.Err()
+		case <-cfgCh:
+			if err := o.db.ApplyConfig(ctx, o.getConfig(), le); err != nil {
+				le.WithError(err).Error(
+					"failure applying config connection limits to db")
+			}
+		}
+	}
+}
+
+// templateWorker will periodically reload the templates in the background as
+// long as the system remains active.
+func (o *Worker) templateWorker(ctx context.Context, cfgCh <-chan struct{}) error {
 	le := o.le.WithFields(logrus.Fields{
 		"worker_type": "apiworker_template_cache",
 	})
@@ -91,7 +167,7 @@ func (o *Worker) Work(ctx context.Context) error {
 		select {
 		case <-ctx.Done():
 			return ctx.Err()
-		case <-o.cfgCh:
+		case <-cfgCh:
 			tr.Reset(ival())
 		case <-tr.C:
 		}

internal/conf/configuration.go

@@ -104,6 +104,11 @@ type DBConfiguration struct {
 	Driver    string `json:"driver" required:"true"`
 	URL       string `json:"url" envconfig:"DATABASE_URL" required:"true"`
 	Namespace string `json:"namespace" envconfig:"DB_NAMESPACE" default:"auth"`
+
+	// Percentage of DB conns the auth server may use in
+	// integer form i.e.: [1, 100] -> [1%, 100%]
+	ConnPercentage int `json:"conn_percentage" split_words:"true"`
+
 	// MaxPoolSize defaults to 0 (unlimited).
 	MaxPoolSize       int           `json:"max_pool_size" split_words:"true"`
 	MaxIdlePoolSize   int           `json:"max_idle_pool_size" split_words:"true"`
@@ -117,6 +122,7 @@ type DBConfiguration struct {
 }
 
 func (c *DBConfiguration) Validate() error {
+	c.ConnPercentage = min(max(c.ConnPercentage, 0), 100)
 	return nil
 }
 

internal/conf/configuration_test.go

@@ -262,6 +262,30 @@ func TestGlobal(t *testing.T) {
 		err := populateGlobal(cfg)
 		require.NoError(t, err)
 	}
+
+	// ConnPercentage
+	{
+		tests := []struct {
+			from int
+			exp  int
+		}{
+			{-2, 0},
+			{-1, 0},
+			{0, 0},
+			{1, 1},
+			{25, 25},
+			{99, 99},
+			{100, 100},
+			{101, 100},
+			{102, 100},
+		}
+		for _, test := range tests {
+			cfg := &DBConfiguration{ConnPercentage: test.from}
+			err := cfg.Validate()
+			require.NoError(t, err)
+			require.Equal(t, test.exp, cfg.ConnPercentage)
+		}
+	}
 }
 
 func TestPasswordRequiredCharactersDecode(t *testing.T) {

internal/observability/request-logger.go

@@ -73,6 +73,11 @@ type logEntry struct {
 	Entry *logrus.Entry
 }
 
+// NewLogEntry returns a new chimiddleware.LogEntry from a *logrus.Entry.
+func NewLogEntry(le *logrus.Entry) chimiddleware.LogEntry {
+	return &logEntry{le}
+}
+
 func (e *logEntry) Write(status, bytes int, header http.Header, elapsed time.Duration, extra interface{}) {
 	fields := logrus.Fields{
 		"status":   status,