Skip to content

Commit 4ce1e01

Browse files
charlesHetterich0xlukemnhussein11
authored
Add async backing technical reference (#1179)
* first pass async backing technical reference - still need to finalize diagrams and give better context of them * split "async backing" into "async backing" & "inclusion pipeline" sections - improved timeline graphs of inclusion pipeline * renamed parachain language to rollup * committing all difs generated by workflow script on commits * replaced all references of *rollup* --> *parachain/parablock* * added description to inclusion-pipeline.md * fixed typos pointed out by copilot and updated diagrams * committing generated content * added links to definitions for , , and * removed unnecessary from internal link * improve `Generation` and `Backing` definitions within the Inclusion Pipeline * adding generated files * added intro to inclusion pipeline page * add ai generated pages * improved backing definition in the inclusion pipeline reference * improved sync vs async section of async backing * fixed mermaid chart * Apply suggestions from code review Co-authored-by: Nicolás Hussein <[email protected]> * WIP inclusion-pipeline * styling feedback * removed comments from async backing docs * better link for PoV --------- Co-authored-by: Lucas Malizia <[email protected]> Co-authored-by: Nicolás Hussein <[email protected]> Co-authored-by: 0xlukem <[email protected]>
1 parent 86cc8e2 commit 4ce1e01

File tree

2 files changed

+230
-1
lines changed

2 files changed

+230
-1
lines changed

‎.reference/parachains/consensus/async-backing.md‎

Lines changed: 169 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1,169 @@
1-
TODO
1+
---
2+
title: Asynchronous Backing
3+
description: Understand how asynchronous backing pipelines parachain block production, the protocol changes it introduces on the Relay Chain, and how parachains participate safely and efficiently.
4+
categories: Polkadot Protocol
5+
---
6+
7+
# Asynchronous Backing
8+
9+
## Introduction
10+
11+
Asynchronous backing often shortened to _Async Backing_ is a parachain protocol feature that significantly improves performance, enabling parachains to produce blocks twice as fast (every 6 seconds instead of every 12) and to provide 4x more execution time per block (2 seconds instead of 0.5).
12+
13+
Technically, async backing is a parachain [configuration](https://paritytech.github.io/polkadot-sdk/master/cumulus_primitives_core/relay_chain/struct.AsyncBackingParams.html){target=\_blank} that allows collators and validators to build blocks ahead of the relay chain during the generation and backing stages of the [Inclusion Pipeline](/reference/parachains/consensus/inclusion-pipeline){target=\_blank} by using unincluded segments, which are chains of parachain blocks that have not yet been fully included in the relay chain. This decouples parachain block production from relay chain inclusion, improves coretime efficiency, and enables the parallel processing required for parachains to further scale throughput using [Elastic Scaling](/reference/parachains/consensus/elastic-scaling){target=\_blank}.
14+
15+
## Configurations
16+
The following configurations can be set by on-chain governance, dictating how many blocks ahead of the relay chain a given parachain's collators can run:
17+
18+
- [**`max_candidate_depth`**](https://paritytech.github.io/polkadot-sdk/master/cumulus_primitives_core/relay_chain/struct.AsyncBackingParams.html#structfield.max_candidate_depth){target=\_blank}: the number of parablocks a collator can produce that are not yet included in the relay chain. A value of `2` means that there can be a maximum of 3 unincluded parablocks at any given time.
19+
- [**`allowed_ancestry_len`**](https://paritytech.github.io/polkadot-sdk/master/cumulus_primitives_core/relay_chain/struct.AsyncBackingParams.html#structfield.allowed_ancestry_len){target=\_blank}: the oldest relay parent a parablock can be built on top of. A value of `1` means collators can start building blocks 6 seconds in advance.
20+
21+
## Synchronous VS. Asynchronous Processing
22+
23+
The Polkadot-parachain protocol originally operated in synchronous mode, where both collators and validators drew context exclusively from the relay parent of the prior parablock, which lives on the relay chain. This made the Backing and Generation steps tightly coupled to the prior parablock completing the entire inclusion pipeline. As a result, one parablock could only be processed every other relay block, with just 0.5 seconds assigned for execution.
24+
25+
```mermaid
26+
---
27+
displayMode: compact
28+
config:
29+
themeCSS: "
30+
#item1 { fill: #450693; stroke: #450693; } \n
31+
#item2 { fill: #8C00FF; stroke: #8C00FF; } \n
32+
#item3 { fill: #FFC400; stroke: #FFC400; } \n
33+
#r { fill: #eb4172; stroke:none; font-size: 20px; } \n
34+
svg text { font-size: 20px !important; } \n
35+
svg .sectionTitle { font-size: 20px !important; } \n #p1padTop { display: none; } \n
36+
37+
/* Hide ALL task labels by default */
38+
text.taskText,
39+
text.taskTextOutside,
40+
[class*='taskText'] tspan { display: none !important; } \n
41+
42+
/* Show labels for the 'r' group (inside or outside, incl. tspans) */
43+
text.taskText[id^='r'],
44+
text.taskTextOutside[id^='r'],
45+
text[id^='r'] tspan { display: inline !important; font-size: 20px; color: #000 !important; } \n
46+
47+
/* Keep section titles styled */
48+
.sectionTitle { fill: #000 !important; font-weight: 700; font-size: 18px; } \n
49+
50+
/* Hide the first two section titles (F1, F2). Change indexes if needed. */
51+
.sectionTitle:nth-of-type(1),
52+
.sectionTitle:nth-of-type(2) { display: none !important; } \n
53+
54+
/* Also hide SPACING row labels on the left */
55+
text.taskTextOutside[id^='p1padTop'] { display: none !important; } \n
56+
"
57+
themeVariables:
58+
sectionBkgColor: '#fff'
59+
gantt:
60+
numberSectionStyles: 1
61+
barHeight: 70
62+
gridLineStartPadding: 100
63+
---
64+
%%{init: {"gantt": {"barHeight": 70 }}}%%
65+
gantt
66+
dateFormat YYYY
67+
axisFormat %y
68+
tickInterval '10year'
69+
70+
section F1
71+
R1 : r, 1905, 1907
72+
R2 : r, 1911, 1913
73+
R3 : r, 1917, 1919
74+
R4 : r, 1923, 1925
75+
76+
section F2
77+
SPACING : p1padTop, 1901, 1924
78+
79+
section P1
80+
X : item1, 1900, 1901
81+
Backing : item2, 1901, 1906
82+
Inclusion : item3, 1906, 1912
83+
84+
section P2
85+
X : item1, 1912, 1913
86+
Backing : item2, 1913, 1918
87+
Inclusion : item3, 1918, 1924
88+
89+
90+
```
91+
92+
The modern protocol now uses asynchronous backing, where both collators and validators have access to [unincluded segments](/reference/parachains/consensus/inclusion-pipeline){target=\_blank} as an additional context source. The Backing and Generation steps are no longer coupled to the prior block completing the full inclusion pipeline. Instead, the prior parablock only needs to complete the generation step and be added to the Unincluded Segments before the next parablock can begin the Backing and Generation steps.
93+
94+
This results in one parablock being processed every relay block (instead of every other relay block), and allows for more time to execute during the Generation step (0.5s → 2s).
95+
96+
```mermaid
97+
---
98+
displayMode: compact
99+
config:
100+
themeCSS: "
101+
#item1 { fill: #450693; stroke: #450693; } \n
102+
#item2 { fill: #8C00FF; stroke: #8C00FF; } \n
103+
#item3 { fill: #FFC400; stroke: #FFC400; } \n
104+
#r { fill: #eb4172; stroke:none; font-size: 20px; } \n
105+
svg text { font-size: 20px !important; } \n
106+
svg .sectionTitle { font-size: 20px !important; } \n #p1padTop { display: none; } \n
107+
108+
/* Hide ALL task labels by default */
109+
text.taskText,
110+
text.taskTextOutside,
111+
[class*='taskText'] tspan { display: none !important; } \n
112+
113+
/* Show labels for the 'r' group (inside or outside, incl. tspans) */
114+
text.taskText[id^='r'],
115+
text.taskTextOutside[id^='r'],
116+
text[id^='r'] tspan { display: inline !important; font-size: 20px; color: #000 !important; } \n
117+
118+
/* Keep section titles styled */
119+
.sectionTitle { fill: #000 !important; font-weight: 700; font-size: 18px; } \n
120+
121+
/* Hide the first two section titles (F1, F2). Change indexes if needed. */
122+
.sectionTitle:nth-of-type(1),
123+
.sectionTitle:nth-of-type(2) { display: none !important; } \n
124+
125+
/* Also hide SPACING row labels on the left */
126+
text.taskTextOutside[id^='p1padTop'] { display: none !important; } \n
127+
"
128+
themeVariables:
129+
sectionBkgColor: '#fff'
130+
gantt:
131+
numberSectionStyles: 1
132+
barHeight: 70
133+
gridLineStartPadding: 100
134+
---
135+
%%{init: {"gantt": {"barHeight": 70 }}}%%
136+
gantt
137+
dateFormat YYYY
138+
axisFormat %y
139+
tickInterval '10year'
140+
141+
section F1
142+
R1 : r, 1905, 1907
143+
R2 : r, 1911, 1913
144+
R3 : r, 1917, 1919
145+
R4 : r, 1923, 1925
146+
R5 : r, 1929, 1931
147+
148+
section F2
149+
SPACING : p1padTop, 1901, 1930
150+
151+
section P1
152+
X : item1, 1900, 1902
153+
Backing : item2, 1902, 1912
154+
Inclusion : item3, 1912, 1918
155+
156+
section P2
157+
X : item1, 1906, 1908
158+
Backing : item2, 1908, 1918
159+
Inclusion : item3, 1918, 1924
160+
161+
section P3
162+
X : item1, 1912, 1914
163+
Backing : item2, 1914, 1924f
164+
Inclusion : item3, 1924, 1930
165+
166+
section P4
167+
X : item1, 1918, 1920
168+
Backing : item2, 1920, 1930
169+
```

‎.reference/parachains/consensus/inclusion-pipeline.md‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
---
2+
title: Inclusion Pipeline
3+
description: Learn how Polkadot validates parachain blocks through the Inclusion Pipeline, detailing context, generation, backing, and inclusion stages.
4+
categories: Polkadot Protocol
5+
---
6+
7+
# Inclusion Pipeline
8+
9+
## Introduction
10+
11+
The inclusion pipeline is the multi-stage process through which every parachain block (parablock) is validated and secured before being finalized in the Polkadot relay chain. This pipeline ensures that all parachain blocks meet validity requirements through progressive verification by multiple sets of validators.
12+
13+
The pipeline exists to provide Polkadot's security guarantees, rather than relying on a single validator group. Each parablock passes through multiple validation stages with different validator sets, ensuring that invalid blocks cannot be finalized even if some validators are malicious or compromised.
14+
15+
Whether a parachain uses synchronous or [asynchronous backing](/reference/parachains/consensus/async-backing){target=\_blank}, all parablocks follow the same inclusion pipeline. The difference is in the timing: asynchronous backing allows multiple blocks to be at different stages of the pipeline simultaneously (pipelining), while synchronous backing processes one block through the entire pipeline before starting the next.
16+
17+
## Pipeline Stages
18+
19+
The inclusion pipeline consists of three main stages:
20+
21+
```mermaid
22+
%%{init: {"flowchart": {"nodeSpacing": 40, "rankSpacing": 60}}}%%
23+
flowchart LR
24+
%% Keep the pipeline on one row (container is hidden)
25+
subgraph Row[" "]
26+
direction LR
27+
G["Generation"] --> B["Backing"] --> I["Inclusion"]
28+
end
29+
style Row fill:none,stroke:none
30+
31+
%% Context: plain text (no box) pointing to both G and B
32+
C["Context"]:::nobox
33+
C -.-> G
34+
C -.-> B
35+
36+
classDef nobox fill:none,stroke:none,color:inherit;
37+
```
38+
### Context
39+
40+
To build a parablock during the generation and backing stages, collators and validators require access to the state context of the parachain. This context is derived from two sources:
41+
42+
- **Relay Parent**: The relay chain block to which the parablock is anchored. Note that the relay parent of a parablock is always different from the relay chain block that eventually includes it. This context source resides on the relay chain.
43+
44+
- **Unincluded Segments**: Chains of candidate parablocks that have not yet been included in the relay chain. These segments represent sequences of block ancestors and may contain candidates at any stage pre-inclusion. A key feature enabled by [async backing](/reference/parachains/consensus/async-backing){target=\_blank} is that collators can build new parablocks on top of these unincluded ancestors rather than being limited to ancestors already included in the relay chain state. This context source resides on the collators.
45+
46+
### Generation
47+
48+
Collators execute their blockchain core functionality to generate a new block, producing a [proof-of-validity](https://wiki.polkadot.com/general/glossary/#proof-of-validity){target=\_blank} (PoV), which is passed to validators selected for backing. The PoV is composed of:
49+
50+
- A list of state transitions called the **block candidate**
51+
- The values in the parachain's database that the block modifies
52+
- The hashes of the unaffected points in the Merkle tree
53+
54+
55+
### Backing
56+
57+
A subset of active validators verify that the parablock follows the state transition rules of the parachain and sign a [validity statement](https://paritytech.github.io/polkadot-sdk/book/types/backing.html?#validity-attestation){target=\_blank} about the PoV which can have a positive or negative outcome. With enough positive statements (at least 2/3 of assigned validators), the candidate is considered backable. It is then noted in a fork on the relay chain, at which point it is considered backed, ready for the next stage of the pipeline.
58+
59+
### Inclusion
60+
61+
Validators gossip [erasure code chunks](https://paritytech.github.io/polkadot-sdk/book/types/availability.html#erasure-chunk){target=\_blank} and put the parablock through the final [approval process](https://paritytech.github.io/polkadot-sdk/book/protocol-approval.html){target=\_blank} before it is considered *included* in the relay chain.

0 commit comments

Comments
 (0)