Skip to content

SQLite engine ignores db_pool_size/db_max_overflow/db_pool_timeout, pinning the pool at QueuePool 5+10 #366

Description

@VascoSch92

Summary

_create_sqlite_engine() (openhands/automation/db.py:151) never forwards the
db_pool_* settings to create_async_engine:

engine = create_async_engine(
    db_url,
    # SQLite-specific settings
    connect_args={"check_same_thread": False},
    # No pooling for SQLite - it handles this internally
    pool_pre_ping=True,
)

The Postgres branch (db.py:136-146) and the GCP Cloud SQL branch
(db.py:198-206) both pass pool_size, max_overflow and pool_timeout. The
SQLite branch passes none of them, so db_pool_size (10), db_max_overflow (5)
and db_pool_timeout (30) from ServiceSettings (config.py:480-499) are
silently ignored on SQLite deployments — and so is the AUTOMATION_DB_POOL_SIZE
environment variable that the AUTOMATION_ env prefix implies.

The comment on line 167 — "No pooling for SQLite - it handles this internally"
is not correct for file-backed aiosqlite.

Verification

Against the versions actually deployed (openhands-automation 1.8.0,
SQLAlchemy 2.0.52):

>>> from sqlalchemy.ext.asyncio import create_async_engine
>>> e = create_async_engine("sqlite+aiosqlite:////tmp/probe.db",
...                         connect_args={"check_same_thread": False},
...                         pool_pre_ping=True)
>>> type(e.pool).__name__, e.pool.size(), e.pool._max_overflow, e.pool._timeout
('AsyncAdaptedQueuePool', 5, 10, 30.0)

SQLAlchemy hands file-backed sqlite+aiosqlite a real AsyncAdaptedQueuePool
with its own defaults (5 + 10, 30s), not a NullPool. Only :memory: gets
pool-less treatment.

Impact

This is an active defect, not a latent one. On the OSS automation VM
(dev-oss-automation-vm, SQLite backend) the resulting exhaustion fires
regularly and surfaces as 500s from the automation API:

sqlalchemy.exc.TimeoutError: QueuePool limit of size 5 overflow 10 reached,
connection timed out, timeout 30.00

722 occurrences in 7 days, in bursts that line up exactly with the minutes when
several 5-minute-cron automations fire simultaneously (Aug 17 ~346, Aug 19 ~203,
Aug 21 ~95). The 5/10 in the message are SQLAlchemy's defaults, confirming
the configured 10/5 never took effect.

The operator-facing part is the sharpest edge: db.py:143 justifies the
fail-fast pool_timeout as

Fail fast if pool is exhausted rather than waiting indefinitely. This surfaces
pool exhaustion issues as errors instead of timeouts, making it easier to
diagnose and fix (e.g., by increasing pool_size).

but on the one path that actually hits this, increasing pool_size does nothing.

Proposed fix

Either honour the settings:

engine = create_async_engine(
    db_url,
    connect_args={"check_same_thread": False},
    pool_size=settings.db_pool_size,
    max_overflow=settings.db_max_overflow,
    pool_timeout=settings.db_pool_timeout,
    pool_pre_ping=True,
)

(_create_sqlite_engine currently takes only db_url, so it would need the
settings object.)

Or make the existing comment true:

from sqlalchemy.pool import NullPool
engine = create_async_engine(
    db_url,
    connect_args={"check_same_thread": False},
    poolclass=NullPool,
    pool_pre_ping=True,
)

Either way, the current state — settings that exist, are documented by their env
prefix, and are silently dropped — is the worst of the three.

Relation to other issues

Distinct from #290, which covers Postgres pool size multiplying across
replicas × workers and is explicitly filed as a latent scaling hazard
("Nothing has broken because of this yet"). This one is SQLite-only, the
settings are ignored outright rather than mis-scaled, and it is causing
failures today. Likely belongs under the #291 hardening tracker.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions