leveldb read error after disk full incident. (exposes non-atomic writes across DBs)

[dan-da]

I had a disk fill up, which caused neptune-core to crash/exit, due to a write error.

After freeing up space, the node starts again OK and reports correct balance. But once the node tries to sync from a peer I consistently get this error:

2025-12-10T02:10:23.053244099Z  INFO ThreadId(01) neptune_cash::application::loops::main_loop: Creating new sync request

thread 'main' (952697) panicked at /home/user/neptune-core/neptune-core/src/application/database/storage/storage_schema/dbtvec_private.rs:50:9:
Out-of-bounds. Got 37455 but length was 37455. persisted vector name: archival_block_mmr
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

So:

1. Clearly the database(s) are in an inconsistent state. this indicates that the disk writes are somehow not atomic, or in a wrong order. It should always be that raw-truth data is written first, and then summary data that references it, not vice versa.

2. In particular "Got 37455 but length was 37455" is indicating that some code believes there is a DbtVec with index 37455, but we only have 37455 actual entries, so off by 1. If the len is correct, then where is the invalid 37455 index coming from, and why was it written before the write to append to the DbtVec? My guess is that it is coming (somehow) from a different DB. Let's check...

3. In MainLoopHandler::block_sync() we call ArchivalBlockMmr::get_leaf_async(), passing it a height, which comes from Self::batch_request_uca_candidate_heights(own_tip_height). That fn only subtracts from the tip height, so we must look at where the tip height came from, and we find it is global_state.chain.light_state().kernel.header.height, which ultimately comes from ArchivalState::block_index_db.

4. Thus it appears that a record for height 37455 was written to block_index_db before corresponding record gets written to archival_block_mmr. But when reading we also go to the block_index_db first, which means we expose the inconsistent state. If we instead wrote to archival_block_mmr first then the read pattern would have been fine in this case, as it is no problem if the height from block_index_db is lower than the vec length in archival_block_mmr.

5. In general, this exposes how brittle our many-db files solution is. In my view, it would be preferable to use a db engine such as redb that supports atomic writes across multiple logical tables cleanly. That would likely be an easier change to make than fixing all possible ordering problems (also good practice).

6. This issue is a direct "in practice" result of the analysis in Atomicity of database updates across block updates #79 that remains open. It is worth reading Pieter Wiulle's answer to my question about Bitcoin's approach to atomicity. tl;dr: they use fewer DBs and are careful about ordering, ie "In this list, the later entries contain references to the previous ones".

https://bitcoin.stackexchange.com/a/121561/49396

7. how can a user recover from this? I tried "--scan-blocks 19000.." but that didn't help. I have not yet tried rebuilding all databases from scratch, but that seems like perhaps what is necessary.

Full stack trace follows.

2025-12-10T02:35:48.621408432Z DEBUG ThreadId(63) neptune_cash::protocol::consensus::block::validity::block_program: ** Call to triton_vm::verify to verify block proof completed; verdict: true.
2025-12-10T02:35:48.636932858Z DEBUG ThreadId(01) neptune_cash::application::loops::main_loop: Running sync
2025-12-10T02:35:48.636961693Z  INFO ThreadId(01) neptune_cash::application::loops::main_loop: Creating new sync request

thread 'main' (962750) panicked at /home/danda/neptune-core/neptune-core/src/application/database/storage/storage_schema/dbtvec_private.rs:50:9:
Out-of-bounds. Got 37455 but length was 37455. persisted vector name: archival_block_mmr
stack backtrace:
   0: __rustc::rust_begin_unwind
   1: core::panicking::panic_fmt
   2: <neptune_cash::application::database::storage::storage_schema::dbtvec::DbtVec<V> as neptune_cash::application::database::storage::storage_vec::traits::StorageVecBase<V>>::get::{{closure}}
   3: neptune_cash::util_types::archival_mmr::ArchivalMmr<Storage>::get_leaf_async::{{closure}}
   4: neptune_cash::application::loops::main_loop::MainLoopHandler::run::{{closure}}
   5: <core::pin::Pin<P> as core::future::future::Future>::poll
   6: <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll
   7: tokio::runtime::context::runtime::enter_runtime
   8: tokio::runtime::runtime::Runtime::block_on
   9: neptune_core::main
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
2025-12-10T02:35:48.642519225Z DEBUG ThreadId(01) neptune_cash: executed run()::select::block_sync_interval in 0.005594934 secs.  exceeds slow fn threshold of 0.001 secs.  location: /home/danda/neptune-core/neptune-core/src/application/loops/main_loop.rs:1858:21
2025-12-10T02:35:48.642602375Z  WARN ThreadId(62) neptune_cash::application::loops::peer_loop: Main loop broadcast channel closed. Exiting peer loop.
2025-12-10T02:35:48.642638824Z DEBUG ThreadId(03) neptune_cash::application::job_queue::job_handle: JobHandle dropping for job: 4ed3797f6c8311c613102fb4
2025-12-10T02:35:48.642647494Z DEBUG ThreadId(62) neptune_cash::application::loops::peer_loop: Exited peer loop for 139.162.193.206:9798
2025-12-10T02:35:48.64267519Z DEBUG ThreadId(62) neptune_cash::application::loops::connect_to_peers: Fetched peer info standing 0 for peer 139.162.193.206:9798
2025-12-10T02:35:48.708337487Z DEBUG ThreadId(63) neptune_cash::protocol::consensus::block::validity::block_program: ** Calling triton_vm::verify to verify block proof ...

thread 'tokio-runtime-worker' (962815) panicked at /home/danda/neptune-core/neptune-core/src/protocol/proof_abstractions/verifier.rs:63:14:
should be able to verify proof in new tokio task: JoinError::Cancelled(Id(37931))
stack backtrace:
   0: __rustc::rust_begin_unwind
   1: core::panicking::panic_fmt
   2: core::result::unwrap_failed
   3: neptune_cash::protocol::proof_abstractions::verifier::verify::{{closure}}
   4: neptune_cash::protocol::consensus::block::Block::is_valid::{{closure}}
   5: neptune_cash::protocol::peer::SyncChallengeResponse::is_valid::{{closure}}
   6: neptune_cash::application::loops::peer_loop::PeerLoopHandler::handle_peer_message::{{closure}}
   7: neptune_cash::application::loops::peer_loop::PeerLoopHandler::run::{{closure}}
   8: neptune_cash::application::loops::peer_loop::PeerLoopHandler::run_wrapper::{{closure}}
   9: neptune_cash::application::loops::connect_to_peers::call_peer::{{closure}}::{{closure}}
  10: neptune_cash::application::loops::connect_to_peers::call_peer::{{closure}}
  11: <core::pin::Pin<P> as core::future::future::Future>::poll
  12: <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll
  13: tokio::runtime::task::core::Core<T,S>::poll
  14: tokio::runtime::task::harness::Harness<T,S>::poll
  15: tokio::runtime::scheduler::multi_thread::worker::Context::run_task
  16: tokio::runtime::scheduler::multi_thread::worker::Context::run
  17: tokio::runtime::context::scoped::Scoped<T>::set
  18: tokio::runtime::context::runtime::enter_runtime
  19: tokio::runtime::scheduler::multi_thread::worker::run
  20: <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll
  21: tokio::runtime::task::core::Core<T,S>::poll
  22: tokio::runtime::task::harness::Harness<T,S>::poll
  23: tokio::runtime::blocking::pool::Inner::run
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
2025-12-10T02:35:48.709210641Z DEBUG ThreadId(63) neptune_cash: executed handle_peer_message()::PeerMessage::SyncChallengeResponse in 0.847406868 secs.  exceeds slow fn threshold of 0.001 secs.  location: /home/danda/neptune-core/neptune-core/src/application/loops/peer_loop.rs:773:17
2025-12-10T02:35:48.709434918Z ERROR ThreadId(63) neptune_cash::application::loops::connect_to_peers: Peer task (outgoing) for 162.219.87.217:9798 panicked. Invoking close connection callback
2025-12-10T02:35:48.709453596Z DEBUG ThreadId(63) neptune_cash::application::loops::connect_to_peers: Fetched peer info standing 0 for peer 162.219.87.217:9798

thread 'tokio-runtime-worker' (962815) panicked at /home/danda/neptune-core/neptune-core/src/application/database/neptune_leveldb.rs:260:60:
called `Result::unwrap()` on an `Err` value: JoinError::Cancelled(Id(37932))
stack backtrace:
   0: __rustc::rust_begin_unwind
   1: core::panicking::panic_fmt
   2: core::result::unwrap_failed
   3: neptune_cash::application::database::neptune_leveldb::NeptuneLevelDb<Key,Value>::get::{{closure}}
   4: neptune_cash::application::loops::connect_to_peers::close_peer_connected_callback::{{closure}}
   5: neptune_cash::application::loops::connect_to_peers::call_peer::{{closure}}
   6: <core::pin::Pin<P> as core::future::future::Future>::poll
   7: <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll
   8: tokio::runtime::task::core::Core<T,S>::poll
   9: tokio::runtime::task::harness::Harness<T,S>::poll
  10: tokio::runtime::scheduler::multi_thread::worker::Context::run_task
  11: tokio::runtime::scheduler::multi_thread::worker::Context::run
  12: tokio::runtime::context::scoped::Scoped<T>::set
  13: tokio::runtime::context::runtime::enter_runtime
  14: tokio::runtime::scheduler::multi_thread::worker::run
  15: <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll
  16: tokio::runtime::task::core::Core<T,S>::poll
  17: tokio::runtime::task::harness::Harness<T,S>::poll
  18: tokio::runtime::blocking::pool::Inner::run
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
2025-12-10T02:35:48.710066993Z DEBUG ThreadId(63) neptune_cash::state: spawning flush db thread
2025-12-10T02:35:48.713175621Z  INFO ThreadId(171) neptune_cash::state: GlobalState is dropping. flushing database
2025-12-10T02:35:48.735421404Z DEBUG ThreadId(171) neptune_cash::state: Flushed all databases

[dan-da]

I added the loss-of-funds label. While the funds are still present and not at risk, they are effectively unavailable to the user if one cannot access them because the node keeps crashing, and there are no clear instructions in README how to deal with such issue(s). (except perhaps rm -rf blocks databases and sync from genesis).

btw, this is not the fist time I've seen something like this. It's the first time I've taken the time to investigate root cause and write up an issue clear enough to be be acted upon.

[dan-da]

Here is an AI summary of how RocksDB and/or redb could solve this issue for us, and how each could be integrated.

---

The RocksDB Column Family Solution

The core reason our node is crashing is the lack of inter-database atomicity: state updates like the block height reference (block_index_db) and the corresponding heavy data (archival_mms) are stored in separate, independent LevelDB instances. A crash between the two write operations leaves the system in an unrecoverable, inconsistent state where the index points to data that was never fully written.

RocksDB, a high-performance fork of LevelDB, solves this by introducing Column Families. While we currently use multiple physical database directories to separate data, Column Families allow us to logically partition the entire state (e.g., separating the Block Index from the Archival Mutator Set) within a single physical database instance. This is conceptually similar to tables in a relational database, but at the key-value store level.

Because all logical partitions (Column Families) share the same underlying Write-Ahead Log (WAL) and are managed by a single RocksDB process, a single Write Batch can be created that targets keys in multiple Column Families simultaneously. When this batch is committed, the update to the block height and the update to the MMR data are guaranteed to succeed or fail as a single, atomic operation. This entirely eliminates the risk of "torn writes" and corruption issues stemming from power loss or node crashes during block processing.

The Redb Atomic Transaction Solution

The alternative approach to achieving critical inter-database atomicity is to adopt redb, a modern, high-performance database engine built natively in Rust. Unlike LevelDB/RocksDB, redb is explicitly designed around ACID (Atomic, Consistent, Isolated, Durable) transactions that inherently operate across multiple storage domains.

With redb, the entire state is managed by a single database file, partitioned into named Tables (e.g., a table for the Block Index and a separate table for the Archival Mutator Set). To update the state, we open a single WriteTransaction. All modifications to multiple tables within that transaction—such as appending to the MMR table and then updating the block height in the index table—are treated as a single unit of work.

The system only writes these changes to the disk upon a successful commit() call. If the application crashes, the power fails, or an internal error occurs before the transaction commits, the underlying database remains at its last known valid, complete state. This transaction-first architecture eliminates the need for manual write ordering or complex prefixing, providing a cleaner, more robust, and fully atomic guarantee across all critical state updates.

Summary Statement on Code Integration

Both RocksDB (using Column Families) and redb (using Tables/Transactions) solve the atomicity problem, but they require different levels of refactoring and offer different developer experiences in Rust:

• RocksDB offers the most drop-in integration due to its API compatibility with LevelDB. If your existing code uses LevelDB calls like db.put(key, value) and db.get(key), the move to RocksDB's wrapper is relatively seamless. The primary effort would be refactoring the database instantiation to open a single instance and correctly define and use the Column Families to separate your data namespaces, allowing the critical atomic WriteBatch to span all state components.

• redb, being native Rust, requires a more significant refactor due to its fundamentally different, transactional API. You would need to rewrite every database interaction to use explicit read_transaction() or write_transaction() calls and work with explicitly defined, typed Tables (e.g., TableDefinition<Key, Value>). However, this refactoring pays dividends by introducing strong type safety at the database layer, eliminating the manual byte-prefixing required by LevelDB/RocksDB, and greatly simplifying concurrent access management, leading to more robust and idiomatic Rust code in the long term.

Feature	RocksDB (via Rust Wrapper)	redb (Native Rust)
API/Drop-in Ease	High. It is LevelDB API-compatible. The majority of put, get, and WriteBatch calls would require minimal signature changes once the core data structure is updated to handle Column Families.	Low. Requires switching to a completely different, transaction-based API (begin_write, open_table, commit). Existing database logic must be rewritten to fit the new transactional model.
Type Safety	Low. It remains a byte-to-byte store. Key/Value serialization (like bincode) is handled manually by the application layer, which is prone to mismatch errors (like the Digest mismatch we fixed).	High. It is inherently type-safe. Tables are defined with specific Rust types (e.g., TableDefinition<u64, BlockHeader>), and the compiler enforces consistency, preventing serialization errors from reaching production.
Build/Runtime	Adds C++ dependency complexity and slower compile times due to FFI.	Pure Rust dependency, simple build, faster compile times.

[skaunov]

I wonder if it is possible for a DB to acquire the needed space from the system before an operation which is non-reproducible (like send)? And if it can't store some random stub beforehand, then deny the whole operation. Is it a lame approach, or maybe it's already used to some extent?

[dan-da]

I wonder if it is possible for a DB to acquire the needed space from the system before an operation which is non-reproducible (like send)? And if it can't store some random stub beforehand, then deny the whole operation. Is it a lame approach, or maybe it's already used to some extent?

1. Internally DBs use various techniques to achieve atomic behavior and/or ACID compliance. But that doesn't help for operations that span databases, which is our situation.

2. I think you might be suggesting a meta layer to solve the inter-db problem. I found a rust solution for that last year, but have not tried it out.

3. While solutions such as that provided in (2) are interesting, it seems easier to just replace leveldb with rocks-db, since both largely share the same API. Still if you or anyone would like to experiment with the"flawless" durable execution engine and prototype an integration, that might be worthwhile.

4. I don't think it's a great idea to roll our own from scratch, unless done by someone with a deep database background. Lots of gotchas.

[skaunov]

Thank you for the reply! It's interesting without any irony, but feels more about #79 you linked earlier. And I just have no opinion on that lacking any knowledge of Level. So totally trust you on this. (A cool solution linked, btw; if I got it they're more logging/journaling based.)

My question/wonder was about the current issue and the workaround. The down-side for it I can see is that for some time the most recent record will be filled with a place holder instead of data which would take cleaning/collecting if the operation won't go that way. But I'd be glad to hear issues of this I overlooked.

[dan-da]

My question/wonder was about the current issue and workaround.

My view is that the current issue is just the first symptom exposed by the underlying root cause. So "fixing" it in isolation would be like plugging a hole in a sieve. The codebase is too inter-related to confidently say that we have write and read ordering correct in all or even most situations (in order to ensure atomicity across power cuts, etc)