Skip to content

Commit ad408e9

Browse files
kwxmkwxm
authored
Kwxm/doc/release builtins (#7312)
* Add document about releasing builtins * Add comments about not changing a test * Change title * Change title * Change title again
1 parent b2a72b8 commit ad408e9

File tree

3 files changed

+221
-0
lines changed

3 files changed

+221
-0
lines changed

‎doc/notes/release-builtins/release-builtins.md‎

Lines changed: 219 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,219 @@
1+
# Enabling new Plutus Core builtins in `plutus-ledger-api`
2+
3+
The process of releasing new builtins (or other Plutus Core features such as new
4+
AST nodes, like `constr` and `case`) is quite complex and has to take place in
5+
several stages.
6+
7+
1. We add new builtins to the Plutus codebase and test and cost them, and
8+
eventually merge the code with the main branch. At this point the new builtins
9+
are available for use locally via eg `uplc` or Plinth.
10+
11+
2. We also have to expose them to the ledger by adding code in
12+
`plutus-ledger-api`. This process will be described in greater detail later.
13+
14+
3. Eventually a Plutus release including the new builtins will be integrated
15+
with the Cardano node, and at some point a new version of the node software will
16+
be released for public use and will be adopted by node operators. It may take
17+
some time until a majority of the nodes in the network are using the new
18+
software, and until that happens it must be impossible to run scripts which use
19+
the new builtins on the network since nodes may disagree about the validity of
20+
scripts. Once a majority of nodes are using the updated software a hard fork
21+
will take place and at that point it will in principle become possible to use
22+
the new builtins. There are two factors which determine whether it's possible to
23+
use them:
24+
25+
* During script deserialisation there's a check (see later) which checks if a
26+
builtin is allowed based on the (major) Protocol Version, which is incremented
27+
during the hard fork. If an invalid builtin is found then there will be a
28+
Phase 1 error and the script will not be executed and will not be stored on
29+
the chain.
30+
31+
* If the script deserialises correctly then it will be executed, and any uses of
32+
a builtin will be charged for as usual, based on the cost model parameters in
33+
the protocol parameters; however, it's possible that the cost model parameters
34+
may not contain an entry for a particular builtin, and if this happens then
35+
the charge for that builtin will be set to the maximum possible, and if the
36+
builtin is used then the script budget will be exhausted immediately, causing
37+
a Phase 2 error.
38+
39+
Thus two things have to happen before a builtin becomes usable:
40+
41+
* A: a protocol parameter update has to happen which updates the cost model
42+
parameters with costs for the builtin.
43+
44+
* B: a hard fork has to occur which enables the builtin.
45+
46+
Because these events have to be voted upon separately it's not currently
47+
possible to ensure that they happen simultaneously.
48+
49+
If the parameter update happens first (ie, A happens before B) then until the
50+
hard fork, scripts using the new builtin(s) will fail during Phase 1 validation;
51+
when the hard fork happens such scripts will immediately pass both Phase 1 and
52+
Phase 2 validation.
53+
54+
If the hard fork happens before the parameter update (ie, B happens before A)
55+
then until the parameter update, scripts using the new builtin(s) will pass the
56+
Phase 1 checks but will fail during execution; after the parameter update they
57+
will succeed.
58+
59+
## Updating `plutus-ledger-api`
60+
61+
When a new buitin is added, the `plutus-ledger-api` code and tests must be
62+
updated to prepare for both A and B.
63+
64+
### Cost model parameters
65+
66+
After a costing function has been determined for a new builtin (see [this
67+
document](https://github.com/IntersectMBO/plutus/blob/master/plutus-core/cost-model/CostModelGeneration.md)),
68+
costing information for the builtin will be stored in the files
69+
`builtinCostModelA.json`, `builtinCostModelB.json`, and `builtinCostModelC.json`
70+
in `plutus-core/cost-model/data`. Once this has been done it will be possible
71+
(inside the `plutus` repository) to run scripts using the new builtin using the
72+
`uplc` command, and script execution will be correctly costed. However, before
73+
the builtin can be used on the chain the code in `plutus-ledger-api` must be
74+
updated to make it aware of the parameters determining the relevant costing
75+
function.
76+
77+
#### Example
78+
The current costing function for the `multiplyInteger` builtin is represented
79+
by the following entry in `builtinCostModelC.json`.
80+
81+
```
82+
"multiplyInteger": {
83+
"cpu": {
84+
"arguments": {
85+
"intercept": 90434,
86+
"slope": 519
87+
},
88+
"type": "multiplied_sizes"
89+
},
90+
"memory": {
91+
"arguments": {
92+
"intercept": 0,
93+
"slope": 1
94+
},
95+
"type": "added_sizes"
96+
}
97+
}
98+
```
99+
100+
This says that the CPU cost of calling `multiplyInteger` with arguments `m` and
101+
`n` is `90434 + 519 * size(m) * size(n)` and the memory cost is `size(m) +
102+
size(n)`, where sizes are measured in units of 8-byte words. There are similar
103+
entries for all of the other builtins, and also for the basic operations of the
104+
CEK machine (see `cekMachineCosts[ABC].json`). The Plutus Core cost model
105+
consists of all of this information, and the A/B/C variants account for
106+
differences in costs for different ledger languages (ie, PlutusV1, PlutusV2, and
107+
PlutusV3) and some historic changes in costing functions. The ledger's view of
108+
the cost model is simply as three lists of numbers (one list for each variant)
109+
stored in the protocol parameters, and there is code in `plutus-ledger-api`
110+
which is used to convert these numbers into our more structured internal
111+
representation. The numbers are referred to as "cost model parameters".
112+
113+
### Cost model parameter names
114+
Before executing a script the the ledger retrieves the relevant set of cost
115+
model parameters from the protocol parameters and calls code in
116+
`plutus-ledger-api` to update the evaluator's internal representation of the
117+
cost model (and some caching is done to avoid repeated updates). The update
118+
process involves a flattened representation of the cost model where each parameter
119+
has a name obtained from the JSON structure: for example, the four numbers in
120+
the costing function for `multiplyInteger` are called
121+
`multiplyInteger-cpu-arguments-intercept`,
122+
`multiplyInteger-cpu-arguments-slope`,
123+
`multiplyInteger-memory-arguments-intercept`, and `multiplyInteger-memory-arguments-slope` (see
124+
[this file](https://github.com/IntersectMBO/plutus/blob/master/doc/notes/cost-model-representations/cost-model-representations.md)
125+
for more on the different cost model representations).
126+
127+
The three modules `PlutusLedgerAPI.V[123].ParamName` define `ParamName` types
128+
which enumerate the cost model parameter names for each Plutus ledger language.
129+
These have one constructor for each cost model parameter, and the names of the
130+
constructors are obtained from the flattened cost model parameter names by capitalising
131+
the first letter of the name and replacting all occurrences of `-` by `'`. Thus the
132+
parameter names for `multiplyInteger` are represented by the following constructors
133+
134+
```
135+
| MultiplyInteger'cpu'arguments'intercept
136+
| MultiplyInteger'cpu'arguments'slope
137+
| MultiplyInteger'memory'arguments'intercept
138+
| MultiplyInteger'memory'arguments'slope
139+
```
140+
141+
Before a new builtin can be enabled on the chain the `ParamName` types must be
142+
updated by adding constructors for the parameter names for the costing functions
143+
for the new builtin **at the end** of the `ParamName` type. Do **NOT** change
144+
the order of the existing constructors or add any new entries in the middle.
145+
The order of new cost model parameters doesn't matter, but it's easiest to add
146+
them in the order in which they appear in the JSON files.
147+
148+
Note that we don't add the actual values of the parameters, which are instead
149+
obtained from the ledger via the protocol parameters. Values for new cost model
150+
parameters must be added to the protocol parameters during a parameter update,
151+
and after the parameter names have been added to the `ParamName` types a
152+
representation of the cost model parameters suitable for the parameter update
153+
can be obtained using the `dump-cost-model-parameters` executable. See the
154+
`--help` option for information about available output formats; the default
155+
`--json` option should usually be used.
156+
157+
#### Tests
158+
After extending the `ParamName` types also update the "length" tests in
159+
`Spec.CostModelParams` and `Spec.Data.CostModelParams` to use the new numbers of
160+
constructors in the `ParamName` types.
161+
162+
### Available builtins and Plutus Core language versions
163+
164+
Builtins are introduced in batches and each batch is enabled on the chain at a
165+
hard fork (equivalent to a new major protocol version (PV)). We have also
166+
introduced different Plutus Code ledge languages (or LLs: these are currently
167+
PlutusV1, PlutusV2, and PlutusV3) at various hard forks and up to and including
168+
PV10 it was only possible to enable new builtins for the most recent ledger
169+
language: for example the `verifyEcdsaSecp256k1Signature` function was
170+
introduced in PlutusV2 at PV8 and also became available in PlutusV3 when that
171+
was introduced at PV9, but at the time of writing (PV10) it has never been
172+
available in PlutusV1. This situation has led to a rather complicated system of
173+
checks for which builtins are available in which (LL,PV) combinations.
174+
175+
Fortunately, from PV11 onwards all builtins will become available in all Plutus
176+
ledger languages, and this simplifies the process of adding new builtins to the
177+
availability check (remember that this is required to account for the fact that
178+
at a given time different nodes may be running different versions of the node
179+
software, which may have different builtins available).
180+
181+
The on-chain builtin availability check is performed by the
182+
`builtinsAvailableIn` function in `PlutusLedgerApi.Common.Versions`. This makes
183+
use of lists defined in the same module which contain the different batches of
184+
builtins . Once a batch of builtins has been released, the corresponding list
185+
must **NOT** subsequently be altered in any way.
186+
187+
When new builtins are added to Plutus Core but we don't yet intend to release
188+
them, add them to a new batch (say `batch<N>`) and associate them with
189+
`futurePV` in all Plutus ledger languages in the function `builtinsIntroducedIn`
190+
in `PlutusLedgerApi.Common.Version`. More builtins can be added to this batch
191+
until we get to the point where we're preparing to release the entire batch.
192+
193+
When we get to the point where we intend to release a new batch of builtins in
194+
an approaching hard fork, do the following:
195+
* Add a variable corresponding to the new PV, for example `somethingPV`, in `PlutusLedgerApi.Common.ProtocolVersions`.
196+
* Update `newestPV` to be equal to `somethingPV`.
197+
* Also update `knownPVs` to include `somethingPV`.
198+
* Update `builtinsIntroducedIn` say that `batch<N>` will be available in `somethingPV`.
199+
* Update the tests in `Spec.Versions` and `Spec.Data.Versions` to include the
200+
names of the new builtins in the `testPermittedBuiltins` functions.
201+
202+
It is possible that we may have added a number of new builtins to Plutus Core
203+
but decide to enable only some subset at the next hard fork. In this case,
204+
create a new batch `batch<N+1>`, move the builtins aren't to be released into it
205+
and and associate `batch<N+1>` with `futurePV` in `builtinsIntroducedIn`. Leave
206+
the builtins which we _do_ plan to release in `batch<N>` and proceed as above.
207+
Also make sure that the constructors for the builtins in `batch<N>` come before
208+
the ones for `batch<N+1>` in the `ParamNames` types: this may require
209+
re-ordering the constructors (but **NOT** the ones for batches before
210+
`batch<N>`).
211+
212+
**DO NOT** change `testBuiltinAvailabilityCompatibility` in `Spec.Versions` and
213+
`Spec.Data.Versions`. This checks that old and new versions of
214+
builtinsAvailableIn agree up to the Plomin hard fork (the start of PV10) and it
215+
should never need to be changed in future.
216+
217+
There are also some tests in `PlutusLedgerApi.Test.V3.EvaluationContext` that
218+
probably don't make sense any more and if nothing goes wrong it's probably safe
219+
to ignore them. These will be reworked or removed at a later date.

‎plutus-ledger-api/test/Spec/Data/Versions.hs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -266,6 +266,7 @@ testPermittedBuiltins =
266266
the new version returns the empty set in this case: to avoid this we only
267267
test pairs where LL was available in PV.
268268
-}
269+
{- DON'T CHANGE THIS: it tests only up to PV10 and should never need to be extended. -}
269270
testBuiltinAvailabilityCompatibility :: TestTree
270271
testBuiltinAvailabilityCompatibility =
271272
testCase "Old and new versions of builtinsAvailableIn are compatible" $

‎plutus-ledger-api/test/Spec/Versions.hs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -267,6 +267,7 @@ testPermittedBuiltins =
267267
the new version returns the empty set in this case: to avoid this we only
268268
test pairs where LL was available in PV.
269269
-}
270+
{- DON'T CHANGE THIS: it tests only up to PV10 and should never need to be extended. -}
270271
testBuiltinAvailabilityCompatibility :: TestTree
271272
testBuiltinAvailabilityCompatibility =
272273
testCase "Old and new versions of builtinsAvailableIn are compatible" $

0 commit comments

Comments
 (0)