Add hook for testing JuliaLowering in core

[mlechu]

This change allows us to activate JuliaLowering as the core lowering implementation via @activate JuliaLowering (InteractiveUtils) or JuliaLowering.activate!().

This is a simplified version of the hook originally in #10. The previous version attempted to swap out the parser hook for one that produced SyntaxTree, but that was broken in many cases, since parser output is expected to be Expr by consumers other than lowering. This change just uses the Expr->SyntaxTree conversion before lowering, which doesn't solve our provenance problems, but is helpful for testing both the lowering implementation and the tree conversion itself.

In the future, we can update Core._parse to take an argument specifying the output type. This needs to happen after JuliaSyntax changes, and both JuliaLowering and base julia updating to use them, which will take some time.

[c42f]

Looks good, thanks!

[c42f]

Very cool that we can do this now, with expr_to_syntaxtree(). Do we get a benefit from the following?

Suggested change

	st0 = code isa Expr ? expr_to_syntaxtree(code) : code
	st0 = code isa Expr ? expr_to_syntaxtree(code, LineNumberNode(line, file)) : code

[mlechu]

Yes, I forgot to pass that in. That will be the source attribute for the root and all nodes until we traverse a linenode.

[c42f]

Ok, so I see that in JuliaLang/julia#58207 you allowed for the trailing elements of this to be anything and we'll specify it later (seems reasonable). Are st5 and ctx5 used by JETLS somehow? (I see they're not used by the C code as expected ... who knows what the ctx5 API even is 😅 )

[mlechu]

Not used by JETLS yet, and even if it was, you're free to "break" this non-API. JETLS does use ctx objects outside of Core._lower, though, I think mostly for the binding table.

I see they're not used by the C code as expected

The C code unfortunately has no idea what a SyntaxTree is yet. I ran into this when trying to make Core._parse produce a syntax tree---toplevel.c had no way to positively check the thing it parsed was a SyntaxTree, and I had to smuggle it through by wrapping it in Expr in the parse hook, which is pretty goofy.

[c42f]

Oh, this also needs tests:

• A few to cover the important branches in the hook function (doesn't need to be integration tests)
• An integration test which calls activate!() and checks that eval(Meta.lower()) works ... or something? (we already overload Core.eval(::SyntaxTree) so can't check that entry point)

[c42f]

Looks really nice now, please merge when ready :)