Skip to content

Commit 07c686a

Browse files
NathanRebNathanReb
authored
Merge pull request ocaml-ppx#627 from NathanReb/document-future-nodes-manipulation
Document future features support
2 parents 132fbd7 + 24acbc7 commit 07c686a

File tree

5 files changed

+166
-62
lines changed

5 files changed

+166
-62
lines changed

‎doc/compatibility.mld‎

Lines changed: 138 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,138 @@
1+
{%html: <div style="display: flex; justify-content:space-between"><div>%}{{!page-"driver"}< The Driver}{%html: </div><div>%}{{!"writing-ppxs"}Writing PPXs >}{%html: </div></div>%}
2+
3+
{1:compat_mult_ver Compatibility With Multiple OCaml Versions}
4+
5+
One of the important issues with working with the
6+
{{!Ppxlib.Parsetree}[Parsetree]} is that the API is not stable. For instance, in
7+
the {{:https://ocaml.org/releases/4.13.0}OCaml 4.13 release}, the following
8+
{{:https://github.com/ocaml/ocaml/pull/9584/files#diff-ebecf307cba2d756cc28f0ec614dfc57d3adc6946eb4faa9825eb25a92b2596d}two}
9+
{{:https://github.com/ocaml/ocaml/pull/10133/files#diff-ebecf307cba2d756cc28f0ec614dfc57d3adc6946eb4faa9825eb25a92b2596d}changes}
10+
were made to the {{!Ppxlib.Parsetree}[Parsetree]} type. Although they are small changes, they may
11+
break any PPX that is written to directly manipulate the (evolving) type.
12+
13+
This instability causes a maintenance issue. PPX authors wish to maintain
14+
a single version of their PPX, not one per OCaml version, and ideally not have
15+
to update their code when seemingly irrelevant changes to the OCaml Parsetree happen.
16+
17+
[ppxlib] helps to solve both issues. PPX authors only have to support
18+
ppxlib's version of the {{!Ppxlib.Parsetree}[Parsetree]}. The
19+
{{!page-driver.driver}driver} will take care of converting from OCaml's
20+
Parsetree to ppxlib's, applying all PPX transformations and then converting
21+
back to the compiler's Parsetree. This means PPX code will always only deal
22+
with the {{!Ppxlib.Parsetree}[Parsetree]} types as defined in ppxlib, regardless
23+
of the version of the compiler being used.
24+
25+
Note that [ppxlib]'s {{!Ppxlib.Parsetree}[Parsetree]} is just a frozen version of
26+
the compiler's that we only update once in a while, instead of every minor
27+
compiler release, to limit breakage and maintenance burden.
28+
Such [ppxlib]'s AST bumps only happen on ppxlib major releases from
29+
[ppxlib.0.36.0] onward.
30+
31+
Even when [ppxlib] does update its AST representation to a more recent one,
32+
less breakage is caused thanks to:
33+
+ its stable APIs to {{!page-"generating-code"}generate} and
34+
{{!page-"matching-code"}destruct} AST nodes
35+
+ the reduced API surface between a PPX and the [Parsetree] types provided
36+
by {{!page-"writing-ppxs".context_free}context-free transformations}
37+
38+
{2:future_compilers Supporting latests language features}
39+
40+
Since [ppxlib]'s AST remains frozen for a while, there are times at which it has
41+
to co-exist with newer compilers.
42+
43+
Those newer compilers have different Parsetrees with new representations of
44+
existing AST nodes or even entirely new features that cannot be represented
45+
with older [Parsetree] types such as the ones [ppxlib] uses.
46+
47+
To circumvent this limitation while still providing a certain amount of
48+
stability, [ppxlib] "encodes" those newer nodes into extension points with the
49+
[ppxlib.migration] namespace. When working with a newer compiler, the AST is
50+
migrated down to [ppxlib]'s version, encoding all unsupported nodes. Once the AST
51+
has been transformed, it is migrated back up to its original version and the
52+
nodes are automatically decoded back into their proper representation by the
53+
[ppxlib] {{!page-driver.driver}driver}.
54+
55+
This approach allows not only to compile and preprocess old code with new
56+
compilers but also to use new language features alongside ppx-es.
57+
58+
The only limitation is that using those features inside parts of the source code
59+
that are interpreted by a ppx can cause a failure as the ppx might try to
60+
interpret an encoded node and won't know what to do with it. We cannot reasonably
61+
expect an old ppx to properly handle a language feature it doesn't even know
62+
existed anyway.
63+
64+
We do provide ways for ppx-es to build and destruct such encoded nodes so that
65+
a ppx author can easily update and add support for new features when it makes
66+
sense. {{!page-"generating-code".ast_builder}[Ast_builder]} functions can be
67+
used to generate AST nodes, eventually encoding them so they are properly
68+
migrated upward.
69+
Be aware that generating code this way will make it compatible only with
70+
compilers that support said feature.
71+
Conversely {{!page-"matching-code".ast_pattern}[Ast_pattern]} provides an
72+
interface for destructing any nodes, including ones that don't have a proper
73+
representation in [ppxlib]'s {{!Ppxlib.Parsetree}[Parsetree]}.
74+
75+
{3 Example: labeled tuples}
76+
77+
Let's take labeled tuples as an example to illustrate this.
78+
79+
At the time of writing, [ppxlib]'s AST is frozen at 5.2. Labeled tuples have been
80+
introduced in 5.4, meaning a ppx-author won't be able to consume or produce a
81+
labeled tuple expression using the regular [Parsetree] types since the 5.2
82+
version of the `Pexp_tuple` variant won't allow it, see for yourself:
83+
84+
In 5.2's AST:
85+
{[
86+
| Pexp_tuple of expression list
87+
]}
88+
89+
In 5.4's AST:
90+
{[
91+
| Pexp_tuple of (string option * expression) list
92+
]}
93+
94+
If I want to generate a labeled tuple expression I can simply use
95+
{!Ppxlib.Ast_builder.Default.pexp_labeled_tuple}:
96+
97+
{[
98+
val pexp_labeled_tuple :
99+
((string option * expression) list -> expression) with_loc
100+
]}
101+
102+
Not much more to say here.
103+
104+
Now imagine my ppx has the following code to expand expressions it receives
105+
as a payload:
106+
{[
107+
let expand_expression expr =
108+
let loc = expr.pexp_loc in
109+
match expr with
110+
| { pexp_desc = Pexp_tuple elist; _ } -> expand_tuple ~loc elist
111+
| _ -> unsupported_payload_error ~loc ()
112+
]}
113+
114+
Now let's say I want to allow users to pass labeled tuples as well, I can
115+
support that thanks to {!Ppxlib.Ast_pattern.pexp_labeled_tuple}:
116+
117+
{[
118+
val pexp_labeled_tuple :
119+
((string option * expression) list, 'a, 'b) t -> (expression, 'a, 'b) t
120+
]}
121+
122+
I simply need to update my code like this:
123+
{[
124+
let expand_expression expr =
125+
let loc = expr.pexp_loc in
126+
match expr with
127+
| { pexp_desc = Pexp_tuple elist; _ } -> expand_tuple ~loc elist
128+
| _ ->
129+
let labeled =
130+
Ast_pattern.(parse_res (pexp_labeled_tuple __)) pexp_loc expr
131+
(fun x -> x)
132+
in
133+
(match labeled with
134+
| Ok l_and_elist -> expand_labeled_tuple ~loc l_and_elist
135+
| Error _ -> unsupported_payload_error ~loc ())
136+
]}
137+
138+
{%html: <div style="display: flex; justify-content:space-between"><div>%}{{!page-"driver"}< The Driver}{%html: </div><div>%}{{!"writing-ppxs"}Writing PPXs >}{%html: </div></div>%}

‎doc/driver.mld‎

Lines changed: 3 additions & 38 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,10 @@
1-
{%html: <div style="display: flex; justify-content:space-between"><div>%}{{!"quick_intro"}< Introduction}{%html: </div><div>%}{{!"writing-ppxs"}Writing PPXs >}{%html: </div></div>%}
1+
{%html: <div style="display: flex; justify-content:space-between"><div>%}{{!"quick_intro"}< Introduction}{%html: </div><div>%}{{!"compatibility"}Compatibility With Multiple OCaml Versions >}{%html: </div></div>%}
22

33
{0 How It Works}
44

55
{1 General Concepts}
66

7-
{2 The Driver}
7+
{2:driver The Driver}
88

99
[ppxlib] sits in between the PPXs authors and the compiler toolchain. For the PPX
1010
author, it provides an API to define the transformation and register it to
@@ -177,41 +177,6 @@ promotion, for instance when using {{!"writing-ppxs"."inlining-transformations"}
177177
generate its own promotion suggestion using the
178178
{{!Ppxlib.Driver.register_correction}[Driver.register_correction]} function.
179179

180-
{1:compat_mult_ver Compatibility With Multiple OCaml Versions}
181-
182-
One of the important issues with working with the
183-
{{!Ppxlib.Parsetree}[Parsetree]} is that the API is not stable. For instance, in
184-
the {{:https://ocaml.org/releases/4.13.0}OCaml 4.13 release}, the following
185-
{{:https://github.com/ocaml/ocaml/pull/9584/files#diff-ebecf307cba2d756cc28f0ec614dfc57d3adc6946eb4faa9825eb25a92b2596d}two}
186-
{{:https://github.com/ocaml/ocaml/pull/10133/files#diff-ebecf307cba2d756cc28f0ec614dfc57d3adc6946eb4faa9825eb25a92b2596d}changes}
187-
were made to the {{!Ppxlib.Parsetree}[Parsetree]} type. Although they are small changes, they may
188-
break any PPX that is written to directly manipulate the (evolving) type.
189-
190-
This instability causes an issue with maintenance. PPX authors wish to maintain
191-
a single version of their PPX, not one per OCaml version, and ideally not have
192-
to update their code when an irrelevant (for them) field is changed in the
193-
{{!Ppxlib.Parsetree}[Parsetree]}.
194-
195-
[ppxlib] helps to solve both issues. The first one, having to maintain a single
196-
PPX version working for every OCaml version, is done by migrating the
197-
{{!Ppxlib.Parsetree}[Parsetree]}. The PPX author only maintains a version
198-
working with the latest version, and the [ppxlib] driver will convert the values from one version to another.
199-
200-
For example, say a deriver is applied in the context of OCaml 4.08. After the
201-
4.08 {{!Ppxlib.Parsetree}[Parsetree]} has been given to it, the [ppxlib] driver
202-
will migrate this value into the latest {{!Ppxlib.Parsetree}[Parsetree]}
203-
version, using the {!Astlib} module. The "latest" here depends on the version of
204-
[ppxlib], but at any given time, the latest released version of [ppxlib] will always
205-
use the latest released version of the {{!Ppxlib.Parsetree}[Parsetree]}.
206-
207-
After the migration to the latest {{!Ppxlib.Parsetree}[Parsetree]},
208-
the driver runs all transformations on it, which ends with a rewritten
209-
{{!Ppxlib.Parsetree}[Parsetree]} of the latest version. However, since the
210-
context of rewriting is OCaml 4.08 (in this example), the driver needs to
211-
migrate back the rewritten {{!Ppxlib.Parsetree}[Parsetree]} to an OCaml 4.08
212-
version. Again, [ppxlib] uses the {!Astlib} module for this migration. Once the
213-
driver has rewritten the AST for OCaml 4.08, the compilation can continue as usual.
214-
215180
{1:derivers-and-extenders Context-Free Transformations}
216181

217182
[ppxlib] defines several kinds of transformations whose core property is that they
@@ -410,4 +375,4 @@ one of them. Thus, only register your transformation in this phase if it is
410375
absolutely vital to be the last transformation, as your PPX will become
411376
incompatible with any other that registers a transformation during this phase.
412377

413-
{%html: <div style="display: flex; justify-content:space-between"><div>%}{{!"quick_intro"}< Introduction}{%html: </div><div>%}{{!"writing-ppxs"}Writing PPXs >}{%html: </div></div>%}
378+
{%html: <div style="display: flex; justify-content:space-between"><div>%}{{!"quick_intro"}< Introduction}{%html: </div><div>%}{{!"compatibility"}Compatibility With Multiple OCaml Versions >}{%html: </div></div>%}

‎doc/generating-code.mld‎

Lines changed: 21 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ constructors directly:
1111
- The AST type might change at a version bump. In this case, the types used in
1212
the PPX would become incompatible with the types of the new OCaml version.
1313

14-
The second point is important: since [ppxlib] {{!page-driver.compat_mult_ver}translates} the AST to
14+
The second point is important: since [ppxlib] {{!page-compatibility.compat_mult_ver}translates} the AST to
1515
the newest OCaml AST available before rewriting, your PPX would not
1616
only become incompatible with the new OCaml version, but also with all [ppxlib]
1717
versions released after the new AST type is introduced.
@@ -169,7 +169,7 @@ changes when a new feature modifies the function in use.
169169
If a feature that was introduced in some recent version of OCaml is essential
170170
for your PPX to work, it might imply that you need to restrict the OCaml version
171171
on your opam dependencies.
172-
{{!page-driver.compat_mult_ver}Remember} that
172+
{{!page-compatibility.compat_mult_ver}Remember} that
173173
[ppxlib] will rewrite using the latest [Parsetree] version, {e but} it will then migrate the
174174
[Parsetree] back to the OCaml version of the switch, possibly losing the information
175175
given by the new feature.
@@ -226,16 +226,16 @@ other kind of nodes, such as {{!Ppxlib.Parsetree.expression}[expressions]} or {{
226226
{{!Ppxlib.Parsetree.structure}[structure]}
227227
and {{!Ppxlib.Parsetree.signature}[signature]}.
228228
{[
229-
let str =
230-
[%str
231-
let x = 5
232-
let y = 6.3]
233-
234-
let sig_ =
235-
[%sig:
236-
val x : int
237-
val y : float]
238-
]}
229+
let str =
230+
[%str
231+
let x = 5
232+
let y = 6.3]
233+
234+
let sig_ =
235+
[%sig:
236+
val x : int
237+
val y : float]
238+
]}
239239

240240
Note the replacement work when the extension node is an "expression"
241241
extension node: Indeed, the [payload] is a {e value} (of [Parsetree] type) that would not fit
@@ -329,18 +329,18 @@ The syntax for anti-quotation depends on the type of the node you wish to insert
329329
{{!Ppxlib.Parsetree.structure_item}[structure_item]} or
330330
{{!Ppxlib.Parsetree.signature_item}[signature_item]}. Note that the syntax for structure/signature item extension nodes uses two [%%]:
331331
{[
332-
let f some_structure_item_node =
333-
[%str
334-
let a = 1
332+
let f some_structure_item_node =
333+
[%str
334+
let a = 1
335335

336-
[%%i some_structure_item_node]]
336+
[%%i some_structure_item_node]]
337337

338-
let f some_signature_item_node =
339-
[%sig:
340-
val a : int
338+
let f some_signature_item_node =
339+
[%sig:
340+
val a : int
341341

342-
[%%i some_signature_item_node]]
343-
]}
342+
[%%i some_signature_item_node]]
343+
]}
344344

345345
If an anti-quote extension node is in the wrong context, it won't be
346346
rewritten by {{!Ppxlib_metaquot}[Metaquot]}. For instance, in [[%expr match [] with [%e some_value] -> 1]]

‎doc/index.mld‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@ The manual consists of several sections. It can be read linearly, but you can al
1616
{ol
1717
{li {{!page-"quick_intro"}An introduction to [ppxlib]}}
1818
{li {{!page-"driver"}How [ppxlib] works internally}}
19+
{li {{!page-compatibility}Compatibility with Multiple OCaml Versions}}
1920
{li {{!page-"writing-ppxs"}Registering a transformation}}
2021
{li {{!page-"generating-code"}Generating AST nodes}}
2122
{li {{!page-"matching-code"}Destructing AST nodes}}

‎doc/writing-ppxs.mld‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
{%html: <div style="display: flex; justify-content:space-between"><div>%}{{!"driver"}< The Driver}{%html: </div><div>%}{{!"generating-code"}Generating AST nodes >}{%html: </div></div>%}
1+
{%html: <div style="display: flex; justify-content:space-between"><div>%}{{!"compatibility"}< Compatibility With Multiple OCaml Versions}{%html: </div><div>%}{{!"generating-code"}Generating AST nodes >}{%html: </div></div>%}
22

33
{0 Writing a Transformation}
44

@@ -26,7 +26,7 @@ In order to register a transformation to the [ppxlib] driver, one should use the
2626
rewriter types in every different phase, except derivers, which are abstracted
2727
away in {{!Ppxlib.Deriving}[Deriving]}.
2828

29-
{1 Context-Free Transformation}
29+
{1:context_free Context-Free Transformation}
3030

3131
In [ppxlib], the type for context-free transformation is
3232
{{!Ppxlib.Context_free.Rule.t}[Context_free.Rule.t]}. Rules will be applied during the AST's top-down traverse
@@ -625,4 +625,4 @@ We encourage you to read and follow it to produce high quality PPXs:
625625
- A section on how to {{!page-"good-practices"."testing-your-ppx"}test} your PPX
626626
- A section on how to collaborate with Merlin effectively by being careful with {{!page-"good-practices"."testing-your-ppx"}locations}
627627

628-
{%html: <div style="display: flex; justify-content:space-between"><div>%}{{!"driver"}< The Driver}{%html: </div><div>%}{{!"generating-code"}Generating AST nodes >}{%html: </div></div>%}
628+
{%html: <div style="display: flex; justify-content:space-between"><div>%}{{!"compatibility"}< Compatibility With Multiple OCaml Versions}{%html: </div><div>%}{{!"generating-code"}Generating AST nodes >}{%html: </div></div>%}

0 commit comments

Comments
 (0)