Skip to content

Commit 911cfb0

Browse files
dnadalesdnadales
authored
Full rework of the BlockFetch logic for bulk sync mode (#1179)
Integrates a new implementation of the BulkSync mode, where blocks are downloaded from alternative peers as soon as the node has no more blocks to validate while there are longstanding requests in flight. This PR depends on the new implementation of the BulkSync mode (IntersectMBO/ouroboros-network#4919). `cabal.project` is made to point to a back-port of the BulkSync implementation on `ouroboros-network-0.16.1.1`. ### CSJ Changes CSJ is involved because the new BulkSync mode requires to change the dynamo if it is also serving blocks, and it is not sending them promptly enough. The dynamo choice has an influence in the blocks that are chosen to be downloaded by BlockFetch. For this sake, b93c379 gives the ability to order the ChainSync clients, so the dynamo role can be rotated among them whenever BlockFetch requests it. b1c0bf8 provides the implementation of the rotation operation. ### BlockFetch tests c4bfa37 allows to specify in tests in which order to start the peers, which has an effect on what peer is chosen as initial dynamo. c594c09 in turn adds a new BlockFetch test to show that syncing isn't slowed down by peers that don't send blocks. ### Integration of BlockFetch changes The collection of ChainSync client handles now needs to be passed between BlockFetch and ChainSync so dynamo rotations can be requested by BlockFetch. The parameter `bfcMaxConcurrencyBulkSync` has been removed since blocks are not coordinated to be downloaded concurrently. These changes are in 6926278. ### ChainSel changes Now BlockFetch requires the ability to detect if ChainSel has run out of blocks to validate. This motivates 73187ba, which implements a mechanism to measure if ChainSel is waiting for more blocks (starves), and determines for how long. The above change is not sufficient to measure starvation. The queue to send blocks for validation used to allow only for one block to sit in the queue. This would interfere with the ability to measure starvation since BlockFetch would block waiting for the queue to become empty, and the queue would quickly become empty after taking just 1 block. For download delays to be amortized, a larger queue capacity was needed. This is the reason why a fix similar to IntersectMBO/ouroboros-network#2721 is part of 0d3fc28. ### Miscellaneous fixes #### CSJ jump size adjustment When syncing from mainnet, we discovered that CSJ wouldn't sync the blocks from the Byron era. This was because the jump size was set to the length of the genesis window of the Shelley era, which is much larger than Byron's. When the jump size is larger than the genesis window, the dynamo will block on the forecast horizon before offering a jump that allows the chain selection to advance. In this case, CSJ and chain selection will deadlock. For this reason we set the default jump size to the size of Byron's genesis window in 028883a. This didn't show an impact on syncing time in our measures. Future work (as part of deploying Genesis) might involve allowing the jump size to vary between different eras. #### GDD rate limit GDD evaluation showed an overhead of 10% if run after every header arrives via ChainSync. Therefore, in b7fa122 we limited how often it could run, so multiple header arrivals could be handled by a single GDD evaluation. #### Candidate fragment comparison in the ChainSync client We stumbled upon a test case where the candidate fragments of the dynamo and an objector were no longer than the current selection (both peers were adversarial). This was problematic because BlockFetch would refuse to download blocks from these candidates, and ChainSync in turn would wait for the selection to advance in order to download more headers. The fix in e27a73c is to have the ChainSync client disconnect a peer which is about to block on the forecast horizon if its candidate isn't better than the selection. #### Candidate fragment truncations At the moment, it is possible for a candidate fragment to be truncated by CSJ when a jumper jumps to a point that is not younger than the tip of its current candidate fragment. We encountered tests where the jump point could be so old that it would fall behind the immutable tip, and GDD would ignore the peer when computing the Limit on Eagerness. This in turn would cause the selection to advance into potentially adversarial chains. The fix in dc5f6f7 is to have GDD never drop candidates. When the candidate does not intersect the current selection, the LoE is not advanced. This is a situation guaranteed to be unblocked by the ChainSync client since it will either disconnect the peer or bring the candidate to intersect with the current selection.
2 parents a020b7b + 6333df0 commit 911cfb0

File tree

44 files changed

+1413
-517
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

44 files changed

+1413
-517
lines changed

‎cabal.project‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,3 +47,13 @@ if(os(windows))
4747

4848
-- https://github.com/ulidtko/cabal-doctest/issues/85
4949
constraints: Cabal < 3.13
50+
51+
source-repository-package
52+
type: git
53+
location: https://github.com/IntersectMBO/ouroboros-network
54+
tag: bb0a7d0ff41e265a8ec47bc94377cb4d65e0b498
55+
--sha256: sha256-P7m+nsjtogNQsdpXQnaH1kWxYibEWa0UC6iNGg0+bH4=
56+
subdir:
57+
ouroboros-network
58+
ouroboros-network-api
59+
ouroboros-network-protocols

‎ouroboros-consensus-diffusion/changelog.d/20240807_100458_alexander.esgen_milestone_1.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
### Breaking
2+
3+
- Adapted to Genesis-related changes in `ouroboros-consensus` ([#1179](https://github.com/IntersectMBO/ouroboros-consensus/pull/1179)).

‎ouroboros-consensus-diffusion/src/ouroboros-consensus-diffusion/Ouroboros/Consensus/Node/Genesis.hs‎

Lines changed: 131 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -1,24 +1,32 @@
1+
{-# LANGUAGE DeriveGeneric #-}
12
{-# LANGUAGE DeriveTraversable #-}
23
{-# LANGUAGE DerivingStrategies #-}
34
{-# LANGUAGE LambdaCase #-}
45
{-# LANGUAGE NamedFieldPuns #-}
56
{-# LANGUAGE NumericUnderscores #-}
7+
{-# LANGUAGE RecordWildCards #-}
68
{-# LANGUAGE ScopedTypeVariables #-}
79

810
module Ouroboros.Consensus.Node.Genesis (
911
-- * 'GenesisConfig'
1012
GenesisConfig (..)
13+
, GenesisConfigFlags (..)
1114
, LoEAndGDDConfig (..)
15+
, defaultGenesisConfigFlags
1216
, disableGenesisConfig
1317
, enableGenesisConfigDefault
18+
, mkGenesisConfig
1419
-- * NodeKernel helpers
1520
, GenesisNodeKernelArgs (..)
21+
, LoEAndGDDNodeKernelArgs (..)
1622
, mkGenesisNodeKernelArgs
1723
, setGetLoEFragment
1824
) where
1925

2026
import Control.Monad (join)
27+
import Data.Maybe (fromMaybe)
2128
import Data.Traversable (for)
29+
import GHC.Generics (Generic)
2230
import Ouroboros.Consensus.Block
2331
import Ouroboros.Consensus.MiniProtocol.ChainSync.Client
2432
(CSJConfig (..), CSJEnabledConfig (..),
@@ -34,57 +42,143 @@ import Ouroboros.Consensus.Util.Args
3442
import Ouroboros.Consensus.Util.IOLike
3543
import Ouroboros.Network.AnchoredFragment (AnchoredFragment)
3644
import qualified Ouroboros.Network.AnchoredFragment as AF
45+
import Ouroboros.Network.BlockFetch
46+
(GenesisBlockFetchConfiguration (..))
3747

3848
-- | Whether to en-/disable the Limit on Eagerness and the Genesis Density
3949
-- Disconnector.
4050
data LoEAndGDDConfig a =
4151
LoEAndGDDEnabled !a
4252
| LoEAndGDDDisabled
43-
deriving stock (Show, Functor, Foldable, Traversable)
53+
deriving stock (Eq, Generic, Show, Functor, Foldable, Traversable)
4454

4555
-- | Aggregating the various configs for Genesis-related subcomponents.
46-
data GenesisConfig = GenesisConfig {
47-
gcChainSyncLoPBucketConfig :: !ChainSyncLoPBucketConfig
56+
--
57+
-- Usually, 'enableGenesisConfigDefault' or 'disableGenesisConfig' can be used.
58+
-- See the haddocks of the types of the individual fields for details.
59+
data GenesisConfig = GenesisConfig
60+
{ gcBlockFetchConfig :: !GenesisBlockFetchConfiguration
61+
, gcChainSyncLoPBucketConfig :: !ChainSyncLoPBucketConfig
4862
, gcCSJConfig :: !CSJConfig
49-
, gcLoEAndGDDConfig :: !(LoEAndGDDConfig ())
63+
, gcLoEAndGDDConfig :: !(LoEAndGDDConfig LoEAndGDDParams)
5064
, gcHistoricityCutoff :: !(Maybe HistoricityCutoff)
65+
} deriving stock (Eq, Generic, Show)
66+
67+
-- | Genesis configuration flags and low-level args, as parsed from config file or CLI
68+
data GenesisConfigFlags = GenesisConfigFlags
69+
{ gcfEnableCSJ :: Bool
70+
, gcfEnableLoEAndGDD :: Bool
71+
, gcfEnableLoP :: Bool
72+
, gcfBlockFetchGracePeriod :: Maybe Integer
73+
, gcfBucketCapacity :: Maybe Integer
74+
, gcfBucketRate :: Maybe Integer
75+
, gcfCSJJumpSize :: Maybe Integer
76+
, gcfGDDRateLimit :: Maybe DiffTime
77+
} deriving stock (Eq, Generic, Show)
78+
79+
defaultGenesisConfigFlags :: GenesisConfigFlags
80+
defaultGenesisConfigFlags = GenesisConfigFlags
81+
{ gcfEnableCSJ = True
82+
, gcfEnableLoEAndGDD = True
83+
, gcfEnableLoP = True
84+
, gcfBlockFetchGracePeriod = Nothing
85+
, gcfBucketCapacity = Nothing
86+
, gcfBucketRate = Nothing
87+
, gcfCSJJumpSize = Nothing
88+
, gcfGDDRateLimit = Nothing
5189
}
5290

53-
-- TODO justification/derivation from other parameters
5491
enableGenesisConfigDefault :: GenesisConfig
55-
enableGenesisConfigDefault = GenesisConfig {
56-
gcChainSyncLoPBucketConfig = ChainSyncLoPBucketEnabled ChainSyncLoPBucketEnabledConfig {
57-
csbcCapacity = 100_000 -- number of tokens
58-
, csbcRate = 500 -- tokens per second leaking, 1/2ms
59-
}
60-
, gcCSJConfig = CSJEnabled CSJEnabledConfig {
61-
csjcJumpSize = 3 * 2160 * 20 -- mainnet forecast range
62-
}
63-
, gcLoEAndGDDConfig = LoEAndGDDEnabled ()
64-
-- Duration in seconds of one Cardano mainnet Shelley stability window
65-
-- (3k/f slots times one second per slot) plus one extra hour as a
66-
-- safety margin.
67-
, gcHistoricityCutoff = Just $ HistoricityCutoff $ 3 * 2160 * 20 + 3600
68-
}
92+
enableGenesisConfigDefault = mkGenesisConfig $ Just defaultGenesisConfigFlags
6993

7094
-- | Disable all Genesis components, yielding Praos behavior.
7195
disableGenesisConfig :: GenesisConfig
72-
disableGenesisConfig = GenesisConfig {
73-
gcChainSyncLoPBucketConfig = ChainSyncLoPBucketDisabled
96+
disableGenesisConfig = mkGenesisConfig Nothing
97+
98+
mkGenesisConfig :: Maybe GenesisConfigFlags -> GenesisConfig
99+
mkGenesisConfig Nothing = -- disable Genesis
100+
GenesisConfig
101+
{ gcBlockFetchConfig = GenesisBlockFetchConfiguration
102+
{ gbfcGracePeriod = 0 -- no grace period when Genesis is disabled
103+
}
104+
, gcChainSyncLoPBucketConfig = ChainSyncLoPBucketDisabled
74105
, gcCSJConfig = CSJDisabled
75106
, gcLoEAndGDDConfig = LoEAndGDDDisabled
76107
, gcHistoricityCutoff = Nothing
77108
}
109+
mkGenesisConfig (Just GenesisConfigFlags{..}) =
110+
GenesisConfig
111+
{ gcBlockFetchConfig = GenesisBlockFetchConfiguration
112+
{ gbfcGracePeriod
113+
}
114+
, gcChainSyncLoPBucketConfig = if gcfEnableLoP
115+
then ChainSyncLoPBucketEnabled ChainSyncLoPBucketEnabledConfig
116+
{ csbcCapacity
117+
, csbcRate
118+
}
119+
else ChainSyncLoPBucketDisabled
120+
, gcCSJConfig = if gcfEnableCSJ
121+
then CSJEnabled CSJEnabledConfig
122+
{ csjcJumpSize
123+
}
124+
else CSJDisabled
125+
, gcLoEAndGDDConfig = if gcfEnableLoEAndGDD
126+
then LoEAndGDDEnabled LoEAndGDDParams{lgpGDDRateLimit}
127+
else LoEAndGDDDisabled
128+
, -- Duration in seconds of one Cardano mainnet Shelley stability window
129+
-- (3k/f slots times one second per slot) plus one extra hour as a
130+
-- safety margin.
131+
gcHistoricityCutoff = Just $ HistoricityCutoff $ 3 * 2160 * 20 + 3600
132+
}
133+
where
134+
-- The minimum amount of time during which the Genesis BlockFetch logic will
135+
-- download blocks from a specific peer (even if it is not performing well
136+
-- during that period).
137+
defaultBlockFetchGracePeriod = 10 -- seconds
138+
139+
-- LoP parameters. Empirically, it takes less than 1ms to validate a header,
140+
-- so leaking one token per 2ms is conservative. The capacity of 100_000
141+
-- tokens corresponds to 200s, which is definitely enough to handle long GC
142+
-- pauses; we could even make this more conservative.
143+
defaultCapacity = 100_000 -- number of tokens
144+
defaultRate = 500 -- tokens per second leaking, 1/2ms
145+
146+
-- The larger Shelley forecast range (3 * 2160 * 20) works in more recent
147+
-- ranges of slots, but causes syncing to block in Byron. A future
148+
-- improvement would be to make this era-dynamic, such that we can use the
149+
-- larger (and hence more efficient) larger CSJ jump size in Shelley-based
150+
-- eras.
151+
defaultCSJJumpSize = 2 * 2160 -- Byron forecast range
152+
153+
-- Limiting the performance impact of the GDD.
154+
defaultGDDRateLimit = 1.0 -- seconds
155+
156+
gbfcGracePeriod = fromInteger $ fromMaybe defaultBlockFetchGracePeriod gcfBlockFetchGracePeriod
157+
csbcCapacity = fromInteger $ fromMaybe defaultCapacity gcfBucketCapacity
158+
csbcRate = fromInteger $ fromMaybe defaultRate gcfBucketRate
159+
csjcJumpSize = fromInteger $ fromMaybe defaultCSJJumpSize gcfCSJJumpSize
160+
lgpGDDRateLimit = fromMaybe defaultGDDRateLimit gcfGDDRateLimit
161+
162+
newtype LoEAndGDDParams = LoEAndGDDParams
163+
{ -- | How often to evaluate GDD. 0 means as soon as possible.
164+
-- Otherwise, no faster than once every T seconds, where T is the
165+
-- value of the field.
166+
lgpGDDRateLimit :: DiffTime
167+
} deriving stock (Eq, Generic, Show)
78168

79169
-- | Genesis-related arguments needed by the NodeKernel initialization logic.
80170
data GenesisNodeKernelArgs m blk = GenesisNodeKernelArgs {
171+
gnkaLoEAndGDDArgs :: !(LoEAndGDDConfig (LoEAndGDDNodeKernelArgs m blk))
172+
}
173+
174+
data LoEAndGDDNodeKernelArgs m blk = LoEAndGDDNodeKernelArgs {
81175
-- | A TVar containing an action that returns the 'ChainDB.GetLoEFragment'
82176
-- action. We use this extra indirection to update this action after we
83177
-- opened the ChainDB (which happens before we initialize the NodeKernel).
84178
-- After that, this TVar will not be modified again.
85-
gnkaGetLoEFragment :: !(LoEAndGDDConfig (StrictTVar m (ChainDB.GetLoEFragment m blk)))
179+
lgnkaLoEFragmentTVar :: !(StrictTVar m (ChainDB.GetLoEFragment m blk))
180+
, lgnkaGDDRateLimit :: DiffTime
86181
}
87-
88182
-- | Create the initial 'GenesisNodeKernelArgs" (with a temporary
89183
-- 'ChainDB.GetLoEFragment' that will be replaced via 'setGetLoEFragment') and a
90184
-- function to update the 'ChainDbArgs' accordingly.
@@ -95,20 +189,24 @@ mkGenesisNodeKernelArgs ::
95189
, Complete ChainDbArgs m blk -> Complete ChainDbArgs m blk
96190
)
97191
mkGenesisNodeKernelArgs gcfg = do
98-
gnkaGetLoEFragment <- for (gcLoEAndGDDConfig gcfg) $ \() ->
99-
newTVarIO $ pure $
192+
gnkaLoEAndGDDArgs <- for (gcLoEAndGDDConfig gcfg) $ \p -> do
193+
loeFragmentTVar <- newTVarIO $ pure $
100194
-- Use the most conservative LoE fragment until 'setGetLoEFragment'
101195
-- is called.
102196
ChainDB.LoEEnabled $ AF.Empty AF.AnchorGenesis
103-
let updateChainDbArgs = case gnkaGetLoEFragment of
197+
pure LoEAndGDDNodeKernelArgs
198+
{ lgnkaLoEFragmentTVar = loeFragmentTVar
199+
, lgnkaGDDRateLimit = lgpGDDRateLimit p
200+
}
201+
let updateChainDbArgs = case gnkaLoEAndGDDArgs of
104202
LoEAndGDDDisabled -> id
105-
LoEAndGDDEnabled varGetLoEFragment -> \cfg ->
203+
LoEAndGDDEnabled lgnkArgs -> \cfg ->
106204
cfg { ChainDB.cdbsArgs =
107205
(ChainDB.cdbsArgs cfg) { ChainDB.cdbsLoE = getLoEFragment }
108206
}
109207
where
110-
getLoEFragment = join $ readTVarIO varGetLoEFragment
111-
pure (GenesisNodeKernelArgs {gnkaGetLoEFragment}, updateChainDbArgs)
208+
getLoEFragment = join $ readTVarIO $ lgnkaLoEFragmentTVar lgnkArgs
209+
pure (GenesisNodeKernelArgs{gnkaLoEAndGDDArgs}, updateChainDbArgs)
112210

113211
-- | Set 'gnkaGetLoEFragment' to the actual logic for determining the current
114212
-- LoE fragment.
@@ -124,9 +222,10 @@ setGetLoEFragment readGsmState readLoEFragment varGetLoEFragment =
124222
where
125223
getLoEFragment :: ChainDB.GetLoEFragment m blk
126224
getLoEFragment = atomically $ readGsmState >>= \case
127-
-- When the Honest Availability Assumption cannot currently be guaranteed, we should not select
128-
-- any blocks that would cause our immutable tip to advance, so we
129-
-- return the most conservative LoE fragment.
225+
-- When the Honest Availability Assumption cannot currently be
226+
-- guaranteed, we should not select any blocks that would cause our
227+
-- immutable tip to advance, so we return the most conservative LoE
228+
-- fragment.
130229
GSM.PreSyncing ->
131230
pure $ ChainDB.LoEEnabled $ AF.Empty AF.AnchorGenesis
132231
-- When we are syncing, return the current LoE fragment.

‎ouroboros-consensus-diffusion/src/ouroboros-consensus-diffusion/Ouroboros/Consensus/Node/Tracers.hs‎

Lines changed: 10 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -32,14 +32,17 @@ import Ouroboros.Consensus.MiniProtocol.BlockFetch.Server
3232
(TraceBlockFetchServerEvent)
3333
import Ouroboros.Consensus.MiniProtocol.ChainSync.Client
3434
(TraceChainSyncClientEvent)
35+
import qualified Ouroboros.Consensus.MiniProtocol.ChainSync.Client.Jumping as CSJumping
3536
import Ouroboros.Consensus.MiniProtocol.ChainSync.Server
3637
(TraceChainSyncServerEvent)
3738
import Ouroboros.Consensus.MiniProtocol.LocalTxSubmission.Server
3839
(TraceLocalTxSubmissionServerEvent (..))
3940
import Ouroboros.Consensus.Node.GSM (TraceGsmEvent)
4041
import Ouroboros.Network.Block (Tip)
41-
import Ouroboros.Network.BlockFetch (FetchDecision,
42-
TraceFetchClientState, TraceLabelPeer)
42+
import Ouroboros.Network.BlockFetch (TraceFetchClientState,
43+
TraceLabelPeer)
44+
import Ouroboros.Network.BlockFetch.Decision.Trace
45+
(TraceDecisionEvent)
4346
import Ouroboros.Network.KeepAlive (TraceKeepAliveClient)
4447
import Ouroboros.Network.TxSubmission.Inbound
4548
(TraceTxSubmissionInbound)
@@ -54,7 +57,7 @@ data Tracers' remotePeer localPeer blk f = Tracers
5457
{ chainSyncClientTracer :: f (TraceLabelPeer remotePeer (TraceChainSyncClientEvent blk))
5558
, chainSyncServerHeaderTracer :: f (TraceLabelPeer remotePeer (TraceChainSyncServerEvent blk))
5659
, chainSyncServerBlockTracer :: f (TraceChainSyncServerEvent blk)
57-
, blockFetchDecisionTracer :: f [TraceLabelPeer remotePeer (FetchDecision [Point (Header blk)])]
60+
, blockFetchDecisionTracer :: f (TraceDecisionEvent remotePeer (Header blk))
5861
, blockFetchClientTracer :: f (TraceLabelPeer remotePeer (TraceFetchClientState (Header blk)))
5962
, blockFetchServerTracer :: f (TraceLabelPeer remotePeer (TraceBlockFetchServerEvent blk))
6063
, txInboundTracer :: f (TraceLabelPeer remotePeer (TraceTxSubmissionInbound (GenTxId blk) (GenTx blk)))
@@ -69,6 +72,7 @@ data Tracers' remotePeer localPeer blk f = Tracers
6972
, consensusErrorTracer :: f SomeException
7073
, gsmTracer :: f (TraceGsmEvent (Tip blk))
7174
, gddTracer :: f (TraceGDDEvent remotePeer blk)
75+
, csjTracer :: f (CSJumping.TraceEvent remotePeer)
7276
}
7377

7478
instance (forall a. Semigroup (f a))
@@ -92,6 +96,7 @@ instance (forall a. Semigroup (f a))
9296
, consensusErrorTracer = f consensusErrorTracer
9397
, gsmTracer = f gsmTracer
9498
, gddTracer = f gddTracer
99+
, csjTracer = f csjTracer
95100
}
96101
where
97102
f :: forall a. Semigroup a
@@ -123,6 +128,7 @@ nullTracers = Tracers
123128
, consensusErrorTracer = nullTracer
124129
, gsmTracer = nullTracer
125130
, gddTracer = nullTracer
131+
, csjTracer = nullTracer
126132
}
127133

128134
showTracers :: ( Show blk
@@ -157,6 +163,7 @@ showTracers tr = Tracers
157163
, consensusErrorTracer = showTracing tr
158164
, gsmTracer = showTracing tr
159165
, gddTracer = showTracing tr
166+
, csjTracer = showTracing tr
160167
}
161168

162169
{-------------------------------------------------------------------------------

0 commit comments

Comments
 (0)