[BUG] Exporter goes unresponsive after not able to ping the DB during maintenance downtime

[jens-ho]

Description

This may be related to "Invalid database configuration, will not attempt reconnection" error after DB maintenance #427 , but logs and behavior is somewhat different.

During the scheduled maintenance window, the database is shut down, and both the DB host as well as the host the Exporter is running on are rebooted at random times.
The Exporter is configured as a systemd unit, and starts automatically shortly after the reboot
of its own host, regardless of the database state.

After successfully scraping metrics for some time, the Exporter was suddenly unsuccessful to ping the database, and logged that. The process stayed running, but stopped logging thereafter and was
unresponsive to any further scrape attempts until manually stopped and restarted, although the database was long accessible again.

Expected Behavior

The Exporter should retry pinging the database periodically and start responding to metric scrapes once a connection can be established again.
A timeout period after which the process shuts down cleanly when unable to ping the DB may be an additional option.

Actual Behavior

The exporter process stayed up and running, but did not respond to scrapes,
or log anything, until manually stopped and restarted.

Environment

• OS:

$ cat /etc/redhat-release
Red Hat Enterprise Linux release 8.10 (Ootpa)

• Oracle AI Database version:

Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.24.0.0.0

• Exporter version: oracledb_exporter-2.2.1.linux-arm64-glibc-2.28.tar.gz

$ ./oracledb_exporter/oracledb_exporter --version
oracledb_exporter, version  (branch: , revision: e1c0bcf32548dbbcadd400ace46cb3eb90e0fc62)
  build user:
  build date:
  go version:       go1.24.11
  go version:       go1.24.11
  platform:         linux/amd64
  tags:             godror

Screenshots / Logs

[...]
time=2026-02-14T21:07:14.624-05:00 level=DEBUG source=collector.go:312 msg="Successfully scraped metric" Context=XXXXX MetricDesc="XXXXX" duration=XXXXX database=prod
time=2026-02-14T21:09:54.158-05:00 level=DEBUG source=collector.go:370 msg="Checking modifications in following metrics definition file:XXXXX"
time=2026-02-14T21:09:54.159-05:00 level=DEBUG source=collector.go:370 msg="Checking modifications in following metrics definition file:XXXXX"
time=2026-02-14T21:09:54.159-05:00 level=DEBUG source=collector.go:370 msg="Checking modifications in following metrics definition file:XXXXX"
time=2026-02-14T21:09:54.159-05:00 level=DEBUG source=collector.go:370 msg="Checking modifications in following metrics definition file:XXXXX"
time=2026-02-14T21:09:54.159-05:00 level=DEBUG source=collector.go:370 msg="Checking modifications in following metrics definition file:XXXXX"
time=2026-02-14T21:12:57.522-05:00 level=ERROR source=collector.go:246 msg="Error pinging database" error="context deadline exceeded" database=prod
time=2026-02-14T21:14:54.158-05:00 level=DEBUG source=collector.go:370 msg="Checking modifications in following metrics definition file:XXXXX"
time=2026-02-14T21:14:54.159-05:00 level=DEBUG source=collector.go:370 msg="Checking modifications in following metrics definition file:XXXXX"
time=2026-02-14T21:14:54.159-05:00 level=DEBUG source=collector.go:370 msg="Checking modifications in following metrics definition file:XXXXX"
time=2026-02-14T21:14:54.159-05:00 level=DEBUG source=collector.go:370 msg="Checking modifications in following metrics definition file:XXXXX"
time=2026-02-14T21:14:54.159-05:00 level=DEBUG source=collector.go:370 msg="Checking modifications in following metrics definition file:XXXXX"
time=2026-02-14T21:17:58.108-05:00 level=ERROR source=collector.go:246 msg="Error pinging database" error="context deadline exceeded" database=prod
time=2026-02-14T21:25:29.836-05:00 level=INFO source=main.go:74 msg="FREE_INTERVAL end var is not present, will not periodically attempt to release memory"
time=2026-02-14T21:25:30.035-05:00 level=INFO source=main.go:81 msg="RESTART_INTERVAL env var is not present, so will not restart myself periodically"
time=2026-02-14T21:25:30.081-05:00 level=INFO source=collector.go:67 msg="Initializing database" database=prod
time=2026-02-14T21:25:30.081-05:00 level=DEBUG source=connect_godror.go:19 msg="Launching connection to (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=XXXXX)(PORT=1521))(ADDRESS=(PROTOCOL=TCP)(HOST=XXXXX)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=XXXXX)))" database=prod
time=2026-02-14T21:25:30.081-05:00 level=DEBUG source=connect_godror.go:26 msg="external authentication set to false" database=prod
time=2026-02-14T21:25:30.081-05:00 level=INFO source=connect_godror.go:32 msg="Using Username/Password Authentication." database=prod
time=2026-02-14T21:25:30.081-05:00 level=DEBUG source=database.go:139 msg="set max idle connections to 824636408656" database=prod
time=2026-02-14T21:25:30.081-05:00 level=DEBUG source=database.go:141 msg="set max open connections to 824636408648" database=prod
time=2026-02-14T21:25:30.081-05:00 level=DEBUG source=database.go:144 msg="Successfully configured connection to (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=XXXXX)(PORT=1521))(ADDRESS=(PROTOCOL=TCP)(HOST=XXXXX)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=XXXXX)))" database=prod
time=2026-02-14T21:25:32.643-05:00 level=INFO source=database.go:152 msg="Could not set CLIENT_INFO." database=prod
time=2026-02-14T21:25:32.661-05:00 level=ERROR source=database.go:157 msg="error checking my database role" error="user=\"XXXXX\" standalone params={authMode:0 connectionClass:<nil> connectionClassLength:0 purity:0 newPassword:<nil> newPasswordLength:0 appContext:<nil> numAppContext:0 externalAuth:0 externalHandle:<nil> pool:<nil> tag:<nil> tagLength:0 matchAnyTag:0 outTag:<nil> outTagLength:0 outTagFound:0 shardingKeyColumns:<nil> numShardingKeyColumns:0 superShardingKeyColumns:<nil> numSuperShardingKeyColumns:0 outNewSession:0}: ORA-12541: Cannot connect. No listener at host XXXXX port 1521.\nHelp: https://docs.oracle.com/error-help/db/ora-12541/" database=prod
time=2026-02-14T21:25:32.661-05:00 level=INFO source=database.go:159 msg="Connected as SYSDBA? " database=prod
time=2026-02-14T21:25:32.788-05:00 level=ERROR source=database.go:90 msg="Failed warmup database connection pool" conn=0 error="user=\"XXXXX\" standalone params={authMode:0 connectionClass:<nil> connectionClassLength:0 purity:0 newPassword:<nil> newPasswordLength:0 appContext:<nil> numAppContext:0 externalAuth:0 externalHandle:<nil> pool:<nil> tag:<nil> tagLength:0 matchAnyTag:0 outTag:<nil> outTagLength:0 outTagFound:0 shardingKeyColumns:<nil> numShardingKeyColumns:0 superShardingKeyColumns:<nil> numSuperShardingKeyColumns:0 outNewSession:0}: ORA-12541: Cannot connect. No listener at host XXXXX port 1521.\nHelp: https://docs.oracle.com/error-help/db/ora-12541/" database=prod
time=2026-02-14T21:25:32.788-05:00 level=DEBUG source=database.go:96 msg="Warmed connection pool" total=0 database=prod
time=2026-02-14T21:25:32.825-05:00 level=DEBUG source=collector.go:370 msg="Checking modifications in following metrics definition file:XXXXX/oracledb_exporter_custom-metrics.toml"
time=2026-02-14T21:25:32.884-05:00 level=INFO source=collector.go:378 msg="XXXXX has been changed. Reloading metrics..."
time=2026-02-14T21:25:32.886-05:00 level=INFO source=data_loader.go:29 msg="Successfully loaded custom metrics from XXXXX"
time=2026-02-14T21:25:32.888-05:00 level=INFO source=data_loader.go:29 msg="Successfully loaded custom metrics from XXXXX"
time=2026-02-14T21:25:32.929-05:00 level=INFO source=data_loader.go:29 msg="Successfully loaded custom metrics from XXXXX"
time=2026-02-14T21:25:32.940-05:00 level=INFO source=data_loader.go:29 msg="Successfully loaded custom metrics from XXXXX"
time=2026-02-14T21:25:32.943-05:00 level=INFO source=data_loader.go:29 msg="Successfully loaded custom metrics from XXXXX"
time=2026-02-14T21:25:32.985-05:00 level=ERROR source=collector.go:246 msg="Error pinging database" error="user=\"XXXXX\" standalone params={authMode:0 connectionClass:<nil> connectionClassLength:0 purity:0 newPassword:<nil> newPasswordLength:0 appContext:<nil> numAppContext:0 externalAuth:0 externalHandle:<nil> pool:<nil> tag:<nil> tagLength:0 matchAnyTag:0 outTag:<nil> outTagLength:0 outTagFound:0 shardingKeyColumns:<nil> numShardingKeyColumns:0 superShardingKeyColumns:<nil> numSuperShardingKeyColumns:0 outNewSession:0}: ORA-12541: Cannot connect. No listener at host XXXXX port 1521.\nHelp: https://docs.oracle.com/error-help/db/ora-12541/" database=prod

No further logs until the manual stop (systemctl --user stop ...) and restart:

time=2026-02-15T19:25:59.627-05:00 level=INFO source=main.go:74 msg="FREE_INTERVAL end var is not present, will not periodically attempt to release memory"
time=2026-02-15T19:25:59.627-05:00 level=INFO source=main.go:81 msg="RESTART_INTERVAL env var is not present, so will not restart myself periodically"
time=2026-02-15T19:25:59.628-05:00 level=INFO source=collector.go:67 msg="Initializing database" database=prod
[...]

[anders-swanson]

Hi @jens-ho, thanks for reporting this bug. Until a proper fix is provided, we recommend configuring the RESTART_INTERVAL to periodically restart the exporter, or configuring a periodic restart through other means.