ORA 3137 - malformed TTC packets - Timeouts with Oracle DBs

[promadmin86]

Hi we are facing a bit of a show stopper at the moment and would be glad if anyone can help.

Describe the bug
When using the exporter on Oracle DBs we are facing regular timeouts.
Log from Oracle shows
"DDE: Problem Key 'ORA 3137 [3120]' was flood controlled
ORA-03137: malformed TTC packet from client rejected"

This happens on all the Oracle DBs we tested

This happens with version 17.1. -17.3 of sql exporter (Docker and binary)

From description by Oracle it is an issue with the driver, the connection appears to be closed too soon, then there are some packets left which the DB cant handle (TTC packets) and disables the connection for a moment for the client

To Reproduce
Steps to reproduce the behavior:

1. Add Oracle DB in config
2. Set Log level to Debug
3. Wait a few minutes
4. See "Timeout error" and "context deadline exceeded" in exporter log
5. Look up Oracle Alerts, see above noted error

Expected behavior
No timeouts

Configuration

sql_exporter.yml
global:
scrape_timeout_offset: 500ms
min_interval: 0s
max_connections: 4
max_idle_connections: 4
max_connection_lifetime: 30m

Additional context
with version 2.8.25 (April 2025) of go-ora driver there should be a new option conn.EnableGracefulDisconnect(true), maybe this can help?
This driver is not added yet right? We couldnt find anything related in the changelogs

Kind regards

[burningalchemist]

Hi @promadmin86, thanks for reporting the issue. I wasn't able to find the new version (v2.8.25) nor the information about EnableGracefulDisconnect option. Could you please point me at the source?

I'd suggest opening an issue at the driver level (https://github.com/sijms/go-ora/issues). The maintainer most likely has access to Oracle DB instances, and might suggest some ideas on how to debug it.

Here is the list of previously solved ORA-03137 issues - https://github.com/sijms/go-ora/issues?q=is%3Aissue%20state%3Aclosed%2003137. So this might give you some ideas on the root cause.

[burningalchemist]

@promadmin86 JFYI, there is a new version of the driver released, v2.9.0. I will use this version for the new release of sql_exporter. Please have a look (once it's released).

[burningalchemist]

@promadmin86, v0.18.0 is released, give it a try.

[itd-fsc]

We are using v0.18.0 and still receive the error:
ORA-03137: malformed TTC packet from client rejected

This is the issue of the driver for this error message:
sijms/go-ora#697

[burningalchemist]

Hey @itd-fsc, thanks for letting me know, also for sharing the issue on the driver side. 👍 I'd be interested to debug it, however I don't have any access to Oracle databases at the moment. So I guess the best bet for now is to wait for the developer's response.

Just curious, what's your Oracle DB version?

[itd-fsc]

We are also using Oracle 19, like the original creator of the issue.

As an idea for a workaround:
Would it be an option to adjust the health endpoint to return unhealthy if no metric can be loaded?
Then we could simply have the liveness probe restart the pod, if it gets stuck in this behavior.

It's a bit difficult to notice automatically otherwise, as the health endpoint always returns healthy and the metrics page still contains the scrape_errors_total metric, even if all other metrics fail due to the db connection. So even prometheus lists up = 1 for the target, as there is still a metric available.

[burningalchemist]

Hey @promadmin86, I'll convert this issue into a discussion, as this is not something we can fix in this particular project nor can I investigate since I don't have access to the Oracle database instance.

Let me know if the new version has improved the situation. 👍