Persistent to Hasql #1938

Cmdv · 2025-01-30T17:07:53Z

Hasql Migration

Summary

The migration from Persistent to Hasql delivered a 3-4x performance improvement in epoch processing time, with some gains in cache efficiency.

Processing Time Comparison

Before (Persistent)

Epoch 432: 29 minutes
Epoch 433: 32 minutes
Average: ~30.5 minutes per epoch

After (Hasql)

Epoch 441: 8m 14s
Epoch 442: 9m 19s
Average: ~8.7 minutes per epoch

Improvement: 73% reduction in processing time (3.5x faster)

Resource Utilisation

Database Connections

Before: ORM connection pooling with abstraction overhead
After: Direct PostgreSQL connection management
Benefit: Reduced connection overhead and more efficient resource usage

Technical Factors Contributing to Performance

1. Eliminated ORM Overhead

No runtime type reflection
No query generation overhead
Direct SQL statement preparation

2. Optimised SQL Generation

Hand-crafted SQL statements
Better query plans
Reduced parsing overhead

3. Type-Safe Operations

Compile-time validation reduces runtime checks
Efficient encoder/decoder patterns
Zero-copy data transformations where possible

4. Connection Management

Direct PostgreSQL protocol usage
Reduced middleware layers
Better connection reuse

Epoch Context

Based on Cardano network data, each epoch processes approximately 21,600 blocks over a 5-day period. The performance improvement scales linearly with blockchain growth, making this migration critical for long-term sustainability.

Conclusion

The Hasql migration represents a significant infrastructure improvement, providing both immediate performance gains and better long-term scalability. The 73% reduction in processing time, positions the system for handling increased blockchain throughput.

1938 - 1 - Variant/Tests Name Changes

kderme

I did a pass of the cardano-db-sync diffs

cardano-db/src/Cardano/Db/Run.hs

cardano-db-sync/src/Cardano/DbSync.hs

kderme · 2025-07-26T08:29:19Z

cardano-db-sync/src/Cardano/DbSync.hs

-      then logWarning trce $ Db.renderMigrationValidateError unknown
-      else logError trce $ Db.renderMigrationValidateError unknown
+      then logWarning trce $ DB.renderMigrationValidateError unknown
+      else logError trce $ DB.renderMigrationValidateError unknown

  logInfo trce "Schema migration files validated"

  let runMigration mode = do


Avoid duplication between runMigration and runIndexesMigration. This can be done by defining a single function in runDbSyncNode with type RunMigration and passing it to both functions.

cardano-db-sync/src/Cardano/DbSync/Api.hs

cardano-db-sync/src/Cardano/DbSync/Era/Universal/Insert/Other.hs

kderme · 2025-07-26T16:00:08Z

cardano-db-sync/src/Cardano/DbSync/OffChain.hs

    isFetchError :: OffChainVoteResult -> Bool
    isFetchError = \case
      OffChainVoteResultMetadata {} -> False
      OffChainVoteResultError {} -> True

+    processResultsBatched :: MonadIO m => [OffChainVoteResult] -> DB.DbAction m ()


Good idea to use pipeline here. Personal note to review it again.

kderme · 2025-07-26T16:04:02Z

cardano-db-sync/src/Cardano/DbSync/Rollback.hs

-      pure False
+          mBlockNo <- lift $ DB.queryBlockHashBlockNo (SBS.fromShort . getOneEraHash $ blockPointHash blk)
+          case mBlockNo of
+            Nothing -> throwError $ SNErrRollback "Rollback.prepareRollback: queryBlockHashBlockNo: Block hash not found"


In general the whole purpose of ExceptT is to have better "flow" and to avoid case patterns for errors. I think we lose this in many places.

cardano-db-sync/src/Cardano/DbSync/Default.hs

kderme · 2025-08-04T09:46:34Z

cardano-db-sync/src/Cardano/DbSync/Era/Universal/Insert/Grouped.hs

+
+  -- Parallel preparation of independent data
+  (preparedTxIn, preparedMetadata, preparedMint, txOutChunks) <- liftIO $ do
+    a1 <- async $ pure $ prepareTxInProcessing syncEnv grouped


This spawns threads, but they do so minimal work that I think it's not worth it. Due to laziness, they may even do no work at all.

cardano-db-sync/src/Cardano/DbSync/Era/Universal/Insert/Grouped.hs

cardano-db/src/Cardano/Db/Statement/Types.hs

cardano-db/src/Cardano/Db/Schema/Core/StakeDeligation.hs

doc/database-monad.md

cardano-db-sync/src/Cardano/DbSync/Default.hs

cardano-db/src/Cardano/Db/Operations/Delete.hs

cardano-db/src/Cardano/Db/Statement/Base.hs

kderme · 2025-08-21T14:24:14Z

cardano-db/src/Cardano/Db/Schema/MinIds.hs

+    maTxOutIdAddressToText _ = "" -- Skip non-variant IDs
+
+--------------------------------------------------------------------------------
+textToMinIds :: TxOutVariantType -> Text -> Maybe MinIdsWrapper


This doesn't match minIdsCoreToText. Should revert the old parser in Cardano.Db.Operations.Other.MinId, except toSqlKey.

cardano-db-sync/src/Cardano/DbSync/OffChain.hs

kderme · 2025-08-21T15:28:53Z

cardano-db-sync/src/Cardano/DbSync/Database.hs

+    updateBlockMetrics :: IO ()
+    updateBlockMetrics = do
+      let metricsSetters = envMetricSetters syncEnv
+      void $ async $ DB.runDbDirectLogged (fromMaybe mempty $ DB.dbTracer $ envDbEnv syncEnv) (envDbEnv syncEnv) $ do


Avoid using async, or use it with a different connection from the connection pool

Cmdv self-assigned this Jan 30, 2025

Cmdv force-pushed the Persistent-to-Hasql branch from f80c67b to 96182d8 Compare March 14, 2025 09:16

Cmdv force-pushed the Persistent-to-Hasql branch 2 times, most recently from 89285ee to 2db7312 Compare April 3, 2025 12:16

Cmdv force-pushed the Persistent-to-Hasql branch from 7c34694 to f434016 Compare May 8, 2025 15:09

Cmdv force-pushed the Persistent-to-Hasql branch 5 times, most recently from a67562c to 94f48af Compare June 30, 2025 22:20

Cmdv marked this pull request as ready for review June 30, 2025 22:21

Cmdv requested a review from a team as a code owner June 30, 2025 22:21

Cmdv force-pushed the Persistent-to-Hasql branch 5 times, most recently from cdab159 to 8ce46c6 Compare July 11, 2025 12:41

Cmdv requested a review from a team as a code owner July 11, 2025 12:41

Cmdv force-pushed the Persistent-to-Hasql branch 5 times, most recently from 08d56ec to f237b25 Compare July 11, 2025 18:15

Cmdv mentioned this pull request Jul 16, 2025

1938 - 1 - Variant Name Change #1989

Merged

9 tasks

Cmdv added a commit that referenced this pull request Jul 17, 2025

Merge pull request #1989 from IntersectMBO/#1938-1-Variant-Name-Change

0c4fffa

1938 - 1 - Variant/Tests Name Changes

Cmdv force-pushed the Persistent-to-Hasql branch 2 times, most recently from ec1082f to fe197d6 Compare July 17, 2025 11:11

Cmdv added 3 commits July 17, 2025 12:18

Persistent to Hasql

5f1c1f2

finalise the bulk inserts

e81deee

add HasDbInfo class and instances

c69d867

kderme requested changes Jul 26, 2025

View reviewed changes

Cmdv added 2 commits July 30, 2025 22:45

update the running function to be clearer

422bf43

start adding pool queries

19ba860

Cmdv force-pushed the Persistent-to-Hasql branch from 6ad7d4d to 87c8789 Compare July 30, 2025 21:46

kderme reviewed Aug 1, 2025

View reviewed changes

cardano-db-sync/src/Cardano/DbSync/Default.hs Outdated Show resolved Hide resolved

kderme reviewed Aug 4, 2025

View reviewed changes

cardano-db-sync/src/Cardano/DbSync/Era/Universal/Insert/Grouped.hs Outdated Show resolved Hide resolved

change DbAction monad to not have ExceptT

7738702

Cmdv force-pushed the Persistent-to-Hasql branch 3 times, most recently from b2df816 to 80cc82b Compare August 7, 2025 13:56

kderme reviewed Aug 11, 2025

View reviewed changes

Cmdv force-pushed the Persistent-to-Hasql branch 3 times, most recently from 779de8b to 54e8f11 Compare August 15, 2025 22:20

Cmdv added 2 commits August 19, 2025 13:36

convert DB.DbAction to DB.DbM

42a6bfe

typo stake deligation -> delegation

c486d58

Cmdv force-pushed the Persistent-to-Hasql branch from d1362d8 to 1e41b5a Compare August 19, 2025 12:36

update errors

1143664

Cmdv force-pushed the Persistent-to-Hasql branch from 1e41b5a to 1143664 Compare August 19, 2025 13:19

Cmdv added 2 commits August 19, 2025 15:36

fixes

018a0bf

add custom buffer sizes & additional pipelines

1fb3c0e

Cmdv force-pushed the Persistent-to-Hasql branch from f3b215c to 1fb3c0e Compare August 20, 2025 13:56

add envDbIsolationState for when syncing and following

1d8476b

kderme reviewed Aug 21, 2025

View reviewed changes

cardano-db/src/Cardano/Db/Operations/Delete.hs Show resolved Hide resolved

kderme requested changes Aug 21, 2025

View reviewed changes

cardano-db/src/Cardano/Db/Statement/Base.hs Outdated Show resolved Hide resolved

kderme requested changes Aug 21, 2025

View reviewed changes

cardano-db-sync/src/Cardano/DbSync/OffChain.hs Outdated Show resolved Hide resolved

don't delete rewards on rollback, fix insertBulkOffChainVoteData

900bb76

kderme reviewed Aug 21, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Persistent to Hasql #1938

Persistent to Hasql #1938

Cmdv commented Jan 30, 2025 •

edited

Loading

Uh oh!

kderme left a comment

Uh oh!

Uh oh!

Uh oh!

kderme Jul 26, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kderme Jul 26, 2025

Uh oh!

kderme Jul 26, 2025

Uh oh!

Uh oh!

kderme Aug 4, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kderme Aug 21, 2025

Uh oh!

Uh oh!

kderme Aug 21, 2025

Uh oh!

Uh oh!

Persistent to Hasql #1938

Are you sure you want to change the base?

Persistent to Hasql #1938

Conversation

Cmdv commented Jan 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Hasql Migration

Summary

Processing Time Comparison

Before (Persistent)

After (Hasql)

Resource Utilisation

Database Connections

Technical Factors Contributing to Performance

1. Eliminated ORM Overhead

2. Optimised SQL Generation

3. Type-Safe Operations

4. Connection Management

Epoch Context

Conclusion

Uh oh!

kderme left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

kderme Jul 26, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kderme Jul 26, 2025

Choose a reason for hiding this comment

Uh oh!

kderme Jul 26, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kderme Aug 4, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kderme Aug 21, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kderme Aug 21, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Cmdv commented Jan 30, 2025 •

edited

Loading