DuckDB enters safe mode after crash and keeps returning fatal error

[srimal-randoli]

Hi there,

We’ve been using GizmoSQL with a DuckDB backend and have encountered an issue where DuckDB occasionally crashes. When this happens, the database server enters safe mode and returns the following error:

FATAL Error:
Failed: database has been invalidated because of a previous fatal error.
The database must be restarted prior to being used again.

DuckDB Doc: https://duckdb.org/docs/stable/guides/troubleshooting/crashes.html#types-of-crashes

As the error message indicates, the database must be restarted before it can be used again. However, my application server continues to receive this error because GizmoSQL propagates the underlying error back to the client.

Is there a way to automatically trigger a restart when this error occurs? Alternatively, can GizmoSQL be configured to reinitialize DuckDB in such cases?

Any guidance would be greatly appreciated.

Thanks in advance!

[prmoore77]

hi @srimal-randoli - thanks for raising this issue. Is there any way to trigger the crash manually that we could use to verify that we can handle this?

[prmoore77]

I had a thought, @srimal-randoli - if you are running GizmoSQL in Kubernetes - you could put a livenessProbe perhaps that would run gizmosql_client with SQL like: SELECT 1, or something - if it got that error - we should fail the liveness probe check - and it would then restart the GizmoSQL container...

[prmoore77]

(I just wanted to make that prior suggestion as a workaround until we handle this error with GizmoSQL)

[srimal-randoli]

@prmoore77 Thanks for the reply. Yes, the liveness probe should do the trick.

I haven’t been able to reproduce the exact issue yet, but I’ll share the steps once I can.

In the meantime, I found this in the DuckDB docs that shows how to reproduce a crash in an older version:
https://duckdb.org/docs/stable/dev/internal_errors.html

[srimal-randoli]

Hi @prmoore77 .

It looks like libgflags is missing in the container image. I'm getting the following error when I execute the client CLI.

Logs:

Liveness probe failed: gizmosql_client: error while loading shared libraries: libgflags.so.2.2: cannot open shared object file: No such file or directory

[prmoore77]

Ah good point - I'll statically compile the library into the client soon...

[prmoore77]

@srimal-randoli - I've just released a new version of the server/client - and the client now has gflags statically compiled into the executable. Could you retry?

[srimal-randoli]

@prmoore77 Sure. I'll give it a try.

We tried with v1.11.5 a week ago, and it was crashing intermittently. Let me get the log and create a separate issue for that.

[prmoore77]

hi @srimal-randoli - any luck?

[srimal-randoli]

Hi @prmoore77. Sorry I could not get back to you. I'm gonna update our dev envs today. Will keep you posted.

[srimal-randoli]

Hi @prmoore77

We are seeing this error in the latest version too. The database is being invalidated and not usable until it is restarted.

" kind=sql peer=ipv4:127.0.0.1:50394 role=admin session_id=f68293da-8832-4dda-b193-b22e84f4e76d sql="SELECT database_name, database_size, wal_size, memory_usage, memory_limit FROM pragma_database_size();" statement_handle=366f0755-5b17-4f2c-bc07-9d9f4c233620 status=failure user=admin - Client SQL command failed execution
2025-10-23T12:24:27.792Z INFO pid=9 tid=139640271009472 component=gizmosql_server kind=sql peer=ipv4:127.0.0.1:47634 role=admin session_id=eec53cff-469c-416a-90f6-c401ff5bd102 sql="SELECT database_name, database_size, wal_size, memory_usage, memory_limit FROM pragma_database_size();" statement_handle=02524b51-85b6-459d-b083-72d10565d336 status=attempt user=admin - Client is attempting to run a SQL command
2025-10-23T12:24:27.792Z WARN pid=9 tid=139640271009472 component=gizmosql_server error="FATAL Error: Failed: database has been invalidated because of a previous fatal error. The database must be restarted prior to being used again.
Original error: "Unaligned fetch in validity and main column data for update"

Stack Trace:

gizmosql_server(+0x17a812a) [0x55a13025512a]
gizmosql_server(+0x17a8218) [0x55a130255218]
gizmosql_server(+0x17abc6d) [0x55a130258c6d]
gizmosql_server(+0x1aabf52) [0x55a130558f52]
gizmosql_server(+0x32d827) [0x55a12edda827]
gizmosql_server(+0x1acaa96) [0x55a130577a96]
gizmosql_server(+0x1acad25) [0x55a130577d25]
gizmosql_server(+0x1acb138) [0x55a130578138]
gizmosql_server(+0x1acb1f0) [0x55a1305781f0]
gizmosql_server(+0x1acb366) [0x55a130578366]
gizmosql_server(+0x1acb87b) [0x55a13057887b]
gizmosql_server(+0x1acbab3) [0x55a130578ab3]
gizmosql_server(+0x731560) [0x55a12f1de560]
gizmosql_server(+0x733da2) [0x55a12f1e0da2]
gizmosql_server(+0xe20666) [0x55a12f8cd666]
gizmosql_server(+0xe1e953) [0x55a12f8cb953]
gizmosql_server(+0xe997ac) [0x55a12f9467ac]
gizmosql_server(+0xe29d64) [0x55a12f8d6d64]
gizmosql_server(+0xe3ad6e) [0x55a12f8e7d6e]
gizmosql_server(+0xe6a776) [0x55a12f917776]
gizmosql_server(+0x1034ea9) [0x55a12fae1ea9]
gizmosql_server(+0x115645c) [0x55a12fc0345c]
gizmosql_server(+0x11565a0) [0x55a12fc035a0]
gizmosql_server(+0xf34671) [0x55a12f9e1671]
/lib/x86_64-linux-gnu/libc.so.6(+0x92b7b) [0x7f00d00d8b7b]
/lib/x86_64-linux-gnu/libc.so.6(+0x1107b8) [0x7f00d01567b8]
" kind=sql peer=ipv4:127.0.0.1:47634 role=admin session_id=eec53cff-469c-416a-90f6-c401ff5bd102 sql="SELECT database_name, database_size, wal_size, memory_usage, memory_limit FROM pragma_database_size();" statement_handle=02524b51-85b6-459d-b083-72d10565d336 status=failure user=admin - Client SQL command failed execution
2025-10-23T12:24:42.394Z INFO pid=9 tid=139640271009472 component=gizmosql_server kind=sql peer=ipv4:127.0.0.1:50394 role=admin session_id=f68293da-8832-4dda-b193-b22e84f4e76d sql="SELECT database_name, database_size, wal_size, memory_usage, memory_limit FROM pragma_database_size();" statement_handle=b944e9f8-5040-4ec7-89a0-586b257b3f2f status=attempt user=admin - Client is attempting to run a SQL command
2025-10-23T12:24:42.394Z WARN pid=9 tid=139640271009472 component=gizmosql_server error="FATAL Error: Failed: database has been invalidated because of a previous fatal error. The database must be restarted prior to being used again.
Original error: "Unaligned fetch in validity and main column data for update"

Stack Trace:

gizmosql_server(+0x17a812a) [0x55a13025512a]
gizmosql_server(+0x17a8218) [0x55a130255218]
gizmosql_server(+0x17abc6d) [0x55a130258c6d]
gizmosql_server(+0x1aabf52) [0x55a130558f52]
gizmosql_server(+0x32d827) [0x55a12edda827]
gizmosql_server(+0x1acaa96) [0x55a130577a96]
gizmosql_server(+0x1acad25) [0x55a130577d25]
gizmosql_server(+0x1acb138) [0x55a130578138]
gizmosql_server(+0x1acb1f0) [0x55a1305781f0]
gizmosql_server(+0x1acb366) [0x55a130578366]
gizmosql_server(+0x1acb87b) [0x55a13057887b]
gizmosql_server(+0x1acbab3) [0x55a130578ab3]
gizmosql_server(+0x731560) [0x55a12f1de560]
gizmosql_server(+0x733da2) [0x55a12f1e0da2]
gizmosql_server(+0xe20666) [0x55a12f8cd666]
gizmosql_server(+0xe1e953) [0x55a12f8cb953]
gizmosql_server(+0xe997ac) [0x55a12f9467ac]
gizmosql_server(+0xe29d64) [0x55a12f8d6d64]
gizmosql_server(+0xe3ad6e) [0x55a12f8e7d6e]
gizmosql_server(+0xe6a776) [0x55a12f917776]
gizmosql_server(+0x1034ea9) [0x55a12fae1ea9]
gizmosql_server(+0x115645c) [0x55a12fc0345c]
gizmosql_server(+0x11565a0) [0x55a12fc035a0]
gizmosql_server(+0xf34671) [0x55a12f9e1671]
/lib/x86_64-linux-gnu/libc.so.6(+0x92b7b) [0x7f00d00d8b7b]
/lib/x86_64-linux-gnu/libc.so.6(+0x1107b8) [0x7f00d01567b8]
" kind=sql peer=ipv4:127.0.0.1:50394 role=admin session_id=f68293da-8832-4dda-b193-b22e84f4e76d sql="SELECT database_name, database_size, wal_size, memory_usage, memory_limit FROM pragma_database_size();" statement_handle=b944e9f8-5040-4ec7-89a0-586b257b3f2f status=failure user=admin - Client SQL command failed execution
2025-10-23T12:24:42.806Z INFO pid=9 tid=139640405214912 component=gizmosql_server kind=sql peer=ipv4:127.0.0.1:47634 role=admin session_id=eec53cff-469c-416a-90f6-c401ff5bd102 sql="SELECT database_name, database_size, wal_size, memory_usage, memory_limit FROM pragma_database_size();" statement_handle=77e1d0c8-b705-4285-a7e5-94c7391f2797 status=attempt user=admin - Client is attempting to run a SQL command
2025-10-23T12:24:42.806Z WARN pid=9 tid=139640405214912 component=gizmosql_server error="FATAL Error: Failed: database has been invalidated because of a previous fatal error. The database must be restarted prior to being used again.
Original error: "Unaligned fetch in validity and main column data for update"

Stack Trace:

gizmosql_server(+0x17a812a) [0x55a13025512a]
gizmosql_server(+0x17a8218) [0x55a130255218]
gizmosql_server(+0x17abc6d) [0x55a130258c6d]
gizmosql_server(+0x1aabf52) [0x55a130558f52]
gizmosql_server(+0x32d827) [0x55a12edda827]
gizmosql_server(+0x1acaa96) [0x55a130577a96]
gizmosql_server(+0x1acad25) [0x55a130577d25]
gizmosql_server(+0x1acb138) [0x55a130578138]
gizmosql_server(+0x1acb1f0) [0x55a1305781f0]
gizmosql_server(+0x1acb366) [0x55a130578366]
gizmosql_server(+0x1acb87b) [0x55a13057887b]
gizmosql_server(+0x1acbab3) [0x55a130578ab3]
gizmosql_server(+0x731560) [0x55a12f1de560]
gizmosql_server(+0x733da2) [0x55a12f1e0da2]
gizmosql_server(+0xe20666) [0x55a12f8cd666]
gizmosql_server(+0xe1e953) [0x55a12f8cb953]
gizmosql_server(+0xe997ac) [0x55a12f9467ac]
gizmosql_server(+0xe29d64) [0x55a12f8d6d64]
gizmosql_server(+0xe3ad6e) [0x55a12f8e7d6e]
gizmosql_server(+0xe6a776) [0x55a12f917776]
gizmosql_server(+0x1034ea9) [0x55a12fae1ea9]
gizmosql_server(+0x115645c) [0x55a12fc0345c]
gizmosql_server(+0x11565a0) [0x55a12fc035a0]
gizmosql_server(+0xf34671) [0x55a12f9e1671]
/lib/x86_64-linux-gnu/libc.so.6(+0x92b7b) [0x7f00d00d8b7b]
/lib/x86_64-linux-gnu/libc.so.6(+0x1107b8) [0x7f00d01567b8]
" kind=sql peer=ipv4:127.0.0.1:47634 role=admin session_id=eec53cff-469c-416a-90f6-c401ff5bd102 sql="SELECT database_name, database_size, wal_size, memory_usage, memory_limit FROM pragma_database_size();" statement_handle=77e1d0c8-b705-4285-a7e5-94c7391f2797 status=failure user=admin - Client SQL command failed execution
stream closed EOF for randoli-agents-dev1/randoli-agents-dev1-0 (gizmosql)

[prmoore77]

hi @srimal-randoli - is there a relatively easy way to trigger the problem? I would like to try to re-create the issue, so I can put a handler and test in place...

Thanks!

[srimal-randoli]

@prmoore77 To be honest, I haven’t been able to reproduce this issue consistently. It occurs randomly, about once every 24 hours, which makes it difficult to reproduce. I’ll keep trying and let you know.

[srimal-randoli]

For the time being, I have a liveness probe running that immediately restarts the DB container when this happens, which gives us some breathing room. 🙂

[prmoore77]

Maybe it's related to this? duckdb/duckdb#19343

It looks like there may be a fix for that one coming to DuckDB...

It isn't quite clear to me how to reproduce it, however - I deleted .wal files in a test instance - and I can't reproduce the issue..