Fulcrum 2.0 Pre-release 1 (beta test)

[cculianu]

Fulcrum 2.0 Pre-Release 1 (Beta)

This is a beta test and/or pre-release version of Fulcrum 2.0.

While it has been thoroughly tested by me here on several machines -- and should be very stable -- there have been massive modifications to the database format and/or other Fulcrum internals. As such, it is best if cautious admins don't run this version until it is officially released, and more adventuresome admins that are willing to test beta software do.

Note that, like I said, it should be very stable.

Brief Summary: What's New in 2.0

A brief summary of what Fulcrum 2.0 offers vs the 1.x series:

• Database Reliability - one of the main weaknesses of Fulcrum 1.x was that should the process be killed at an inopportune time, or should abrupt powerloss or kernel-level crash occur at an inopportune time, there was always a risk that the datadir would get corrupted and the admin would be forced to do a full database resynch. This is no longer the case!
  • Fulcrum 2.x series will no longer suffer from this problem and the process can be killed at any time (including abrupt powerloss), without any database corruption. At worst the last few blocks worth of data is rolled-back and Fulcrum will re-synch from the rollback point.
  • It is still recommended that admins gracefully shut-down Fulcrum, however, via SIGINT/SIGTERM/SIGQUIT (either from their init/systemd scripts or via kill -TERM <pid>, etc), and wait for it to terminate. This maximizes performance and minimizes startup times for Fulcrum the next time it is restarted.
• Database format is now 100% platform-neutral. This means it should be possible to always copy datadirs around regardless if transferring them from Windows, Linux, BSD, macOS, and regardless of platform architecture (including big endian platforms).
• Improved performance.

Upgrading from the Fulcrum 1.x Series

Upgrade Instructions

Fulcrum 2.0 has a different database format than the 1.x series. As such, you have two options for upgrading:

• Create a blank new datadir and just re-synch from block 0 (slow, but 100% reliable)
• Upgrade your existing 1.x datadir (faster, but irreversible and destructice)

To upgade your existing database, be sure to pass the one-time --db-upgrade flag to Fulcrum. It will refuse to start up if it detects that your datadir is in the old format and you did not pass this flag. This ensures that admins know what they are getting into.

Note

The --utxo-cache option has been completely removed. If you want to temporarily leverage memory to help out initial synch, use a temporary --db_mem (CLI) or db_mem= setting in the conf file to give Fulcrum more memory for initial synch, stop Fulcrum, then set it down again to a smaller value for operations.

Critical Note On Upgrading the DB from 1.x -> 2.0

Let it run to completion!

While Fulcrum 2.0 is 100% reliable and may be killed at any time during its operation -- there is one exception to this rule: the upgrade process that converts 1.x databases -> 2.0. As such please let the upgrade process run to completion and do not kill, stop, molest, or otherwise abuse the Fulcrum process while it's upgrading the DB!!. If you do, the old 1.x database will be lost and your 2.0 database will also be corrupted.

You have been warned!

---

This discussion was created from the release Fulcrum 2.0 Pre-release 1 (beta test).

[eugenesan]

Finally, thank you very much!

A couple questions:

1. How long does the 1.x -> 2.x migration process takes compared to the full resync?
2. How big is the new format compared to the old one and how much space should be allocated during the migration?

[cculianu]

(1) Depends on the chain and the machine's disk speed . I presume you are on BTC so on my machine it took a good 2 hours or so for my BTC node. On BCH it took about 20 minutes. Haven't tried LTC yet.

[cculianu]

(2) Oh sorry I am listening to a podcast now and I didn't read your second part of your question and thus neglected to answer it.

Good question!

So it should take about the same amount of space after the upgrade is done. However, during the upgrade it needs to temporarily duplicate some data.

Fulcrum does check disk space ahead of time before it really starts the upgrade, and it refuses to continue you if it thinks you don't have enough extra temp space, and even tells you how much space to free up. So the upgrade does use some some extra space temporarily.

The amount of temporary space it uses is precisely the largest "database element" in the old 1.x DB. Each database element is a subdirectory in the old format. So If you do du -h /path/to/datadir for the old 1.x DB, you will see a bunch of directories and sizes. Pick the largest size you see there and that's the amount of temp space utilized. So for my BTC mainnet data dir, it looks like this:

$ du -h /media/calin/T4_Ext4/btc/fulcrum.btc.main
84G	/media/calin/T4_Ext4/btc/fulcrum.btc.main/scripthash_history
7.0G	/media/calin/T4_Ext4/btc/fulcrum.btc.main/scripthash_unspent
1.3G	/media/calin/T4_Ext4/btc/fulcrum.btc.main/undo
13M	/media/calin/T4_Ext4/btc/fulcrum.btc.main/meta
23G	/media/calin/T4_Ext4/btc/fulcrum.btc.main/txhash2txnum
14G	/media/calin/T4_Ext4/btc/fulcrum.btc.main/utxoset
39M	/media/calin/T4_Ext4/btc/fulcrum.btc.main/blkinfo
12M	/media/calin/T4_Ext4/btc/fulcrum.btc.main/rpa
166G	/media/calin/T4_Ext4/btc/fulcrum.btc.main/

In the above list.. every subdir is a database element. The largest is the scripthash_history which is 84G. That needs to be duplicated into the new 2.0 format before being deleted. So you need ~84 G extra temp space to perform the upgrade.

And then after the upgrade completes you'll be back to about the same amount of space as you had before (maybe even a little bit less, maybe a little bit more -- it's like +/- 1% here so on a 166G DB, that's maybe 1.6G difference after the upgrade completes either up or down. RocksDB is weird that way due to how it tries to cache stuff).

I hope this long-winded explanation I just gave is sufficiently clear. If not, let me know and I can clarify.

[eugenesan]

Thank you for such a prompt response.
The answers are very detailed and cover both my questions and then some ;-)

Now I have another one.
With the new DB layout, you still enforce kNoCompression for RocksDB.
Should I assume that there is no change in how you estimate the data can be compressed?

I ask because me being a contrarian, I was enforcing kZSTD for local instances to save a few GBs and prevent some disk wear.
I wonder if I should expect similar, better or worse results?
I understand why my solution makes little sense in most cases but my electricity is free while SSDs are not ;-)

[cculianu]

Hey seriously -- You can also try running btrfs or some other FS that has optional native compression as well?

But yes you can turn on compression in Fulcrum by modifying the sources in the function openOrCreateDB in Storage.cpp..

[eugenesan]

It's up to you -- you can modify the sources if you really want compression -- in all of my experiments that performed much worse and I wanted Fulcrum to be fast. But you can modify the sources and do whatever you want man.. the beauty of open source!

But yes the estimates assume default everything and not you changing compression :)

I think I'll use this opportunity to benchmark both options. Maybe the juice is not worth squeeze even for me ;-)

Hey seriously -- You can also try running btrfs or some other FS that has optional native compression as well?

But yes you can turn on compression in Fulcrum by modifying the sources in the function openOrCreateDB in Storage.cpp..

Yeap, as simple as sed -i 's/kNoCompression/kZSTD/g' src/Storage.cpp ;-)

Thank you for taking time to answer my questions.
And I believe I speak on behalf of the whole BTC community, thank for the best electrum server implementation!

[cculianu]

Yeap, as simple as sed -i 's/kNoCompression/kZSTD/g' src/Storage.cpp ;-)

100% that will work. Just make sure to only molest files in src/.. because in staticlibs/ you may be molesting rocksdb itself.. but yes that should work.

I think I'll use this opportunity to benchmark both options. Maybe the juice is not worth squeeze even for me ;-)

If you have time to do that I would really appreciate your datapoint with that. Also what the compressed vs non-compressed datadir size is.

Note you can actually just --db-upgrade both the Fulcrum binary built with and without compression from the same 1.x input dir.. since re-encoding the db will apply the compression option to the new rebuilt 2.0 db...

And I believe I speak on behalf of the whole BTC community, thank for the best electrum server implementation!

I'm flattered thank you. And thanks to all that contributed seriously I could not have devoted this much time to this project were it not from the donations. 🙏

[eugenesan]

If you have time to do that I would really appreciate your datapoint with that. Also what the compressed vs non-compressed datadir size is.

I finished testing and here are some stats:

1. Original DB - v1.12.0 - kZSTD
   Actual disk size - 132 GB (137,672,676 KB)

2. Original DB - v1.12.0 - kNoCompression (Based on my memory since I don't have one)
   Actual disk size - 145 GB

3. Upgrade run 1 - v2.0.0 - kZSTD
   Imported 10 tables, 4242774530 rows (of which 168939656 were converted), 7687 write batches, 153.5 GB in 11384.6 seconds.
   Actual disk size 144 GB (150,173,060 KB)

4. Upgrade run 2 - v2.0.0 - kZSTD (I did rerun kZSTD upgrade 3 times due to unexpected results but they were consistent)
   Imported 10 tables, 4276144851 rows (of which 168487400 were converted), 7761 write batches, 154.5 GB in 11403.1 seconds.
   Actual disk size 142 GB (148,749,132 KB)

5. Upgrade run 3 - v2.0.0 - kNoCompression
   Imported 10 tables, 4276144851 rows (of which 168487400 were converted), 7761 write batches, 154.5 GB in 11281.4 seconds.
   Actual disk size - 173 GB (181,190,692 KB)

Summary:

1. Disk size differs quite a bit. For example your v2.0.0-kNoCompression DB is 166 GB mine is 173 GB. Not sure why. I am running rocksdb: version 9.10.0-0.

2. v2 DB is somewhat bigger:

   • 7.1% for kZSTD -> kZSTD
   • 20.1% for kZSTD-> kNoCompression at best based on your results from Fulcrum 2.0 Pre-release 1 (beta test) #297 (reply in thread) and 23.6% at worst based on my results
   • 12.5& (at best) and 16% (at worst) for kNoCompression -> kNoCompression

3. v2 DB kNoCompression is 12.6% bigger at best and 16.1% bigger at worst

4. Judging by upgrade time variation (+/-0.524%), performance difference between kNoCompression and kZSTD is negligible especially since I was running the tests on an older hardware (Intel Skylake). Conversion used only two threads. The situation might be even better for regular operations on a newer/faster hardware.

5. Max temporary space required during upgrade was +32% of the final DB for kZSTD and +26% for kNoCompression.

Questions:

1. Why are we seeing so much variation in disk size of different runs of upgrade to v2 DB?
2. Why v2 DB is so much bigger?
3. What are your though on the results? Especially on virtually no difference in performance between kZSTD and kNoCompression and up to 23% difference in size?

[cculianu]

Thanks for all these metrics. Very detailed and very helpful. Super job!

As for your questions:

1. The way rocksdb decides to juggle data around as you do writes is not entirely clear to me and I also have observed variation in the space requirements. As the database is run, it auto-compacts so sometimes size jumps back down, only to climb again as you continue to write. Rocksdb is complex.
2. The database format changed slightly to be faster, but that should only add at most 1GB for a BTC mainnet. I suspect it's bigger now because everything lives in a single database now, using this rocksdb-only concept called "column families" and I suspect rocksdb needs to maintain more information to partition the tables in this way. That is not entirely under my control as a client-code programmer, how rocksdb manages space requirements given the same raw data.
3. I don't think the speed of the V1.x -> V2.x upgrade is necessarily illustrative of runtime performance of Fulcrum in general. It's one thing to continually write to the end of a table using kZSTD compression, it's another to do random access as Fulcrum runs and queries come in from clients. In my experience (with my fast disks and slow CPUs), I would get a 50-100% slowdown for the most complex/expensive queries when using kZSTD or kSnappyCompression, while only "gaining" 16% or so in space savings. To me this was not a reasonable tradeoff. But to some users it might be. I encourage you to try some of the more expensive queries on your two setups, the kZSTD one vs kNoCompression, and see -- maybe on your setup the penalty is very small or non-existant. In which case that would be VERY helpful to know! Issue Fund raiser: Help fund Fulcrum 2.0 #287 has some discussion about what the "slow" queries are that you can try.

One thing I may wish to add to Fulcrum is a flag to turn on/off compression. Maybe even a flag to "force" the DB to recompress/uncompress if the flag is changed. This may be a knob users such as yourself might want to play with to trade performance for space savings and vice versa.

Thanks again for these numbers. They are very helpful to me and to the community of admins!

[linkinparkrulz]

Thank you Calin for making this happen!

Fulcrum 2.0 is available for testing on Start9:

https://github.com/linkinparkrulz/fulcrum-startos/actions/runs/17881598558

[kingspam77]

Thanks Calin! Excited for the upgrade. But still no mac installer :( Can you start making them available with 2.0 please?

[cculianu]

There is so little interest in the mac as a server platform. It's just me, you, and this 1 friend of mine in the BCH space. Is there more interest than just us 3 for this? I don't even use the mac for production just for developer testing.. ha.

Anybody else want to chime in here? Maybe if there's more than 5 people that care I can spend the 10+ hours to figure out how to get the mac to build and run on various older and newer macos's..

[status21]

I'd be interested too! But also no problem building myself.

[twofaktor]

Thanks, @cculianu, you did a spectacular job. I tested the new Fulcrum 2.0-pre1 on testnet4 in a Raspberry Pi 4 - 4GB, with db_mem=1024.0, and the migration took only 4 minutes. So far, it's more speedy at startup and, as always, at deep wallet scanning. I haven't been able to test it in more scenarios yet, but it looks incredible. Congrats!

This is the trace it has left on the CPU, not bad, it doesn't seem to have required percentages even close to 100%:

I wanted to let you know that I encountered this when trying to start. I corrected it by replacing utxo-cache with db_mem, and it was solved:

---

3 questions:

• Do you think it would be appropriate to include that change (the utxo-cache removal) in the release notes as part of the relevant changes? (There might be quite a few people who had utxo-cache set in their configuration previously)
• Can you confirm to me that db_mem not only takes effect only on initial sync, and always has an effect, both in initial synchronization and in daily operation? not like the old utxo-cache?
• Do you think that a db_mem=1024.0 is appropriate to keep for 4GB RAM and 59% available (2.20GB) without counting Fulcrum? Or could I set it to db_mem=2048.0 without any worries?

Thanks and again, congrats!

[cculianu]

Thanks for the feedback. Awesome that it's working well for you and yes the --utxo-cache thing was intentional.

• Yes, good idea. I'll add it to release notes now. I sorta wanted to have people hit that wall so they remember to remove --utxo-cache from all documentation that's floating out there, etc. But yes probably should be in the release notes. Will do.
• Correct, it always stays in place. As such, maybe don't max it out. Just use a reasonable value.
• I would be conservative on that system and use 1024. Maybe use 2048 temporarily on initial synch on mainnet or something then drop it down to 1024.