Logs should be visible only after commit succeeds

[kriszyp]

Currently logs were being made visible after a commit finishes, regardless of whether or not it actually committed the data. Specifically if the commit is "busy" (conflicted), the log position was being updated even though the corresponding data was not committed and visible/readable yet.
With this PR, the logs won't be made immediately visible if the commit is conflicted and does not actually complete. In order for the logs to be made visible and the position to be properly updated on the retry, I believe we need to preserve the existing transaction instance/handle to continue tracking the log position and a reset the transaction so it can be properly re-executed and the preserved log position can be used when the commit does eventually succeed for accurate/consistent reporting of log visibility. (I know we have previously discussed a preference for creating entirely new transaction wrappers to reset, but I am not sure how we would do that and continue to track the internal log position without exposing more complexity).

[github-actions]

📊 Benchmark Results

get-sync.bench.ts

getSync() > random keys - small key size (100 records)

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 lmdb	1	23.27K ops/sec	42.98	41.43	491.89	0.113	116,326
🥈 rocksdb	2	11.41K ops/sec	87.61	83.68	2,928.621	0.120	57,074

getSync() > sequential keys - small key size (100 records)

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 lmdb	1	27.10K ops/sec	36.90	35.77	413.539	0.101	135,488
🥈 rocksdb	2	12.34K ops/sec	81.07	79.01	429.151	0.049	61,677

ranges.bench.ts

getRange() > small range (100 records, 50 range)

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 lmdb	1	25.26K ops/sec	39.59	37.39	719.273	0.130	126,289
🥈 rocksdb	2	3.40K ops/sec	293.81	258.172	1,801.924	0.542	17,018

realistic-load.bench.ts

Realistic write load with workers > write variable records with transaction log

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 rocksdb	1	183.19 ops/sec	5,458.898	63.13	166,780.49	49.15	384
🥈 lmdb	2	26.87 ops/sec	37,210.179	89.11	1,171,951.473	137.536	64.00

transaction-log.bench.ts

Transaction log > read 100 iterators while write log with 100 byte records

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 rocksdb	1	35.06K ops/sec	28.52	15.19	14,249.548	0.780	175,295
🥈 lmdb	2	439.88 ops/sec	2,273.333	428.06	10,354.248	1.18	2,200

Transaction log > read one entry from random position from log with 1000 100 byte records

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 rocksdb	1	614.49K ops/sec	1.63	1.43	3,861.393	0.199	3,072,426
🥈 lmdb	2	416.65K ops/sec	2.40	1.43	3,713.164	0.358	2,083,266

worker-put-sync.bench.ts

putSync() > random keys - small key size (100 records, 10 workers)

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 rocksdb	1	865.72 ops/sec	1,155.113	999.651	2,193.52	0.278	1,732
🥈 lmdb	2	1.01 ops/sec	988,633.076	805,934.17	1,036,689.315	5.03	10.00

worker-transaction-log.bench.ts

Transaction log with workers > write log with 100 byte records

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 rocksdb	1	17.85K ops/sec	56.01	31.98	5,900.108	0.831	35,705
🥈 lmdb	2	844.55 ops/sec	1,184.056	136.728	13,343.115	5.16	1,690

---

Results from commit c5763c8

[kriszyp]

I have had some weird flakiness with the benchmarks, but it is green right now 🤔 .

[cb1kenobi]

I have had some weird flakiness with the benchmarks, but it is green right now 🤔 .

Me too. I restarted the GitHub runner on my server, but I don't think it helped. Not sure what's up.