Skip to content

RocksDB storage#37

Open
kriszyp wants to merge 172 commits intomainfrom
rocksdb
Open

RocksDB storage#37
kriszyp wants to merge 172 commits intomainfrom
rocksdb

Conversation

@kriszyp
Copy link
Member

@kriszyp kriszyp commented Nov 18, 2025

Adding support for running on our new Rocksdb-js storage system.
The major changes in this PR are:

  • Updates to interaction with DB stores that normalize appropriately across LMDB and RocksDB
  • Ability to find and open LMDB or RocksDB databases
  • Updates to auditStore/transaction-log to wrap RocksDB's log writing and querying functionality, and handle its own encoding and deleting
  • Blobs are deleted when their record is deleted so that transaction logs can be deleted without scanning before deletion
  • New DatabaseTransaction class for handling distinctions of RocksDB transactions; RocksDB can immediately write to transactions, which can be concurrent with other open transactions.
  • Resource API updates for v5 that interact with Rocksdb, including new save() method (for saving data in the midst of a transaction).

@kriszyp kriszyp marked this pull request as ready for review February 10, 2026 23:19
@kriszyp kriszyp requested a review from a team as a code owner February 10, 2026 23:19
Copy link
Member

@cap10morgan cap10morgan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK this was fun! ;)

I don't think there's any showstoppers in my review, so I'm approving. But I do have a few questions to consider and requests for a few small / nit-picky changes here and there. Feel free to ignore / delay / question any of it!

This is awesome BTW! 🪨

let db_size;
try {
db_size = (await fs.stat(tableObj.primaryStore.env.path)).size;
db_size = (await fs.stat(tableObj.primaryStore.path ?? tableObj.primaryStore.env.path)).size;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it make sense to move this into a method?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, lmdb-js sets path on the database instance, so this could be simplified to:

db_size = (await fs.stat(tableObj.primaryStore.path)).size;

storeMetric(analyticsTable, metric);
let metric;
if (firstTable.primaryStore instanceof RocksDatabase) {
const dbPath = firstTable.primaryStore.store.path;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like in schemaDescribe we're using tableObj.primaryStore.path but here it's .primaryStore.store.path. Is one correct and the other not?

Should we have a method that abstracts the Rocks / LMDB difference and centralizes & standardizes this?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In rocksdb-js, db.store.path is the source of truth. db.path is a getter that returns the store's path. With that said, while store is publicly accessible, it wasn't meant to be public. The correct usage in this case should be firstTable.primaryStore.path.

dev: { path: dbPath },
test: { path: dbPath },
test2: { path: dbPath },
});
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps not for this PR, but I've had lots of issues with this not being hermetic w/r/t ~/hdb. Sometimes also setting ROOTPATH seems to help, but other times it doesn't. 🤷🏻

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, this change was intended to help with this. Previously the dev, test, and test2 folders were being created/used from ~/hdb, but now they should be a part of the process specific folder for unit test execution (isolated to each test run).

@socket-security
Copy link

socket-security bot commented Feb 12, 2026

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff Package Supply Chain
Security
Vulnerability Quality Maintenance License
Added@​harperfast/​rocksdb-js@​0.1.67510010096100

View full report

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants