WIP: Provide mechanism for Julia syntax evolution

[Keno]

There are several corner cases in the Julia syntax that are essentially bugs or mistakes that we'd like to possibly remove, but can't due to backwards compatibility concerns.

Similarly, when adding new syntax features, there are often cases that overlap with valid (but often nonsensical) existing syntax. In the past, we've mostly done judegement calls of these being "minor changes", but as the package ecosystem grows, so does the chance of someone accidentally using these anyway and our "minor changes" have (subjectively) resulted in more breakages recently.

Fortunately, all the recent work on making the parser replacable, combined with the fact that JuliaSyntax already supports parsing multiple revisions of Julia syntax provides a solution here: Just let packages declare what version of the Julia syntax they are using. That way, packages would not break if we make changes to the syntax and they can be upgraded at their own pace the next time the author of that particular package upgrades to a new julia version.

The way this works is simple. Right now, the parser function is always looked up in Core._parse. With this PR, it is instead looked up as rootmodule(mod)._internal_julia_parse (slightly longer name to avoid conflicting with existing bindings of the name in downstream packages), or Core._parse if no such binding exists. Similar for _lower. A little more work is required to propagate down mod from the various places that use it, but it's still relatively straightforward.

At the moment, the supported way to make this election is to write @Base.Experimental.set_syntax_version v"1.14" (or whatever the version is that you're writing your syntax against).

However, to make this truly smooth, I think this should happen automatically through a Project.toml opt-in specifying the expected syntax version. My preference would be to use #59995 if that is merged, but this is a separate feature (with similar motivations around API evolution of course) and there could be a different opt-in mechanism.

I should emphasize that I'm not proposing using this for any big syntax revolutions or anything. I would just like to start cleaning up a few corners of the syntax that I think are universally agreed to be bad but that we've kept for backwards compatibility. This way, by the time we get around to making a breaking revision, our entire ecosystem will have already upgraded to the new syntax.

WIP because there needs to be a (small) corresponding change in JuliaSyntax that I haven't wired you yet - waiting on #59870 for that.

[KristofferC]

Can you compare this to https://doc.rust-lang.org/edition-guide/editions/ which on the surface looks fairly similar.

How would macros be handled? The rust edition docs has a special section about that https://doc.rust-lang.org/edition-guide/editions/advanced-migrations.html#migrating-macros

[KristofferC]

It's also common to directly include package files in the REPL for e.g. interactive debugging. To me that would mean the project file syntax version would be required (so you know how to parse things based on the current active project). and the @Base.Experimental.set_syntax_version v"1.14" is not really workable. Alternatively, all files need a marker that says how they are parsed.

[Keno]

https://doc.rust-lang.org/edition-guide/editions/

Yes, it's a substantially similar mechanism with the same goals.

How would macros be handled?

Macros are expanded according to the lowering version of the calling module. This may of course mean that the macro sees syntax that is not part of the syntax revision that the defining module expects, but the user of the macro can decide how to deal with that at usage time - the resolution will not retroactively change.

It's also common to directly include package files in the REPL for e.g. interactive debugging

If the REPL context module is switched to the package, the REPL will use the syntax version of the package. If the file is included in Main, then of course the environment may be different. However, I don't think this is all that different from e.g. loading a different version of a package because the project wasn't activated. That said, I think it would be reasonable and useful to have the REPL use the syntax revision of the activated project, even for the main module (and switch this when the project is switched).

[Keno]

Alternatively, all files need a marker that says how they are parsed.

I don't want to do this at file (or even module) granularity, at least without explicit opt-in - I think it would be very confusing if it was a common situation that different files within the same package did not have the same syntax revision.

[tecosaur]

However, to make this truly smooth, I think this should happen automatically through a Project.toml opt-in specifying the expected syntax version.

I do wonder if in the future it would be reasonable to have certain minimum Julia versions imply a certain minimum syntax version?

[mlechu]

I read the discussion above, but I still think macros are going to be tricky. This shouldn't block starting work on the mechanism, but it might become a problem when we start trying to do the evolution part. Some questions and suggestions are below, hopefully more helpful than distracting.

Here are the guarantees I think we should be providing with this mechanism:

1. Within a syntax version, the same AST has the same semantics (or lack thereof). Requiring a constant syntax version here is a weakening of what we currently have, but it should be made OK by (3) and (4) below.
2. Within a syntax version, the same text parses to the same AST (or error). The "weakening" comment above applies here since we haven't realistically been able to change parsing without macro breakage, but it's a bit of a strengthening too: we don't promise this stability right now, and it would be nice to. It would eliminate issues like 59911.
3. The compiler and runtime can handle code written using the current syntax version or any earlier one.
4. (from rust) No splitting the ecosystem; modules can interact as they do now no matter what syntax version each uses

I'm the least sure about how to achieve the last one with macros. If we're designating the caller responsible for knowing what syntax version it's running in and what syntax version its callees take, I think we might as well make syntax evolution a parsing-only thing and not worry about breaking the AST. I fear either would just produce our current "change breaks macros" situation with extra steps. Some examples:

• Assume we want to change parsing so that global a=1, b=2 becomes (global (= a 1) (= b 2)) instead of (global (= a (= (tuple 1 b) 2))). What syntax should the macro in DifferentVersion.@foo global a=1, b=2 see? Should a macro anticipate being called from other syntax versions?
• On the output end, would @eval $(M.@produce_different_version_syntax) work?
• 57368 is an example of a potential change of meaning to an existing AST. Only a few packages needed their macros updated, but ancient versions of those packages show up in dependency chains everywhere. Unless I'm missing something, "caller responsibility" would mean views.jl needs to provide different macro implementations per syntax version and have the caller choose between them.

My suggestion is to run an AST conversion on other-version macro inputs and outputs. I wrote down a few thoughts in the "attempt to define the AST" PR (c42f/JuliaLowering.jl#93), but I'll keep thinking about this

Another suggestion: if we're able to convert to the latest version at the AST level, could we define "syntax version" to end between macro expansion and desugaring rather than after lowering? Old syntax would be converted to new syntax before lowering. This way we can avoid tying up the lowering implementation (and the version of CodeInfo it produces) into the definition of a syntax version, so we can get new lowering changes in all versions without worrying about syntax other than the latest version.