Compiler frontend API and Core integration sketch

[c42f]

Here I've generalized and refactored the incremental lowering code from
PR #84, splitting out the interface from the implementation and
factoring the parts which should go into Core into a _Core module.

_Core.eval() is implemented in terms of a new _Core.simple_eval()
which can evaluate only CodeInfo and the new types
TopLevelCodeIterator, BeginModule and EndModule. Evaluation of
TopLevelCodeIterator is implemented in terms of the incremental
lowering interface which must be provided by a compiler frontend.

A compiler frontend (subtype of CompilerFrontend) must come with
implementations of the functions lower_init and parseall and
lower_step must be being defined on the return type of lower_init.
Having a type for the frontend solves the issue of how to tie parsing
and lowering together without needing to convert to Expr.

parseall(frontend, code, ...) returns a syntax tree of the preferred
expression type for the frontend which can be fed into lower_init.
This function is just a placeholder - it needs to be generalized to
allow Meta.parse(), Meta.parseatom() and Meta.parseall() to be
implemented in terms of it.

Parser and lowering diagnostics are handled by throwing an exception
from the frontend but we may want a fuller diagnostics API instead.

[mlechu]

Apologies for the delayed review. I'm unsure about the complexity of this API for a few reasons:

• The main division being between "compiler frontend" and "not compiler frontend" feels out of place, as much as it's necessary with our flisp setup. I think it's more likely that future lowering and type inference will be tightly integrated than lowering and parsing (or even lowering and macro expansion for that matter). Once JuliaLowering achieves compatibility, I think people want to start changing the definition of "lowering" itself, where parsing and macro expansion will stay roughly the same.
• I don't think we need to anticipate another lowering implementation or AST type before a 2.0
• We can't see how well this API will work in practice, how easy it will be to implement fully, or how much old complexity we can delete with it until we move JuliaLowering into the julia source tree and start hacking there.

I'm not against using it though, especially if you believe this is the right abstraction, or if getting this up and running is less work than the "simple" alternative (storing AST type in a global and changing Core._parse to take a type). I'd go with whichever one puts less effort between now and lowering compatibility.

Implementing this API after a simple integration (even the five-line round-tripping through Expr way) is also an option we have.

[mlechu]

We're good to drop 1.12 support (and even 1.13 once that branches, as decided on the julialang slack, where a quicker feature freeze was preferred). Impressive that you've found a way to make it work on 1.12 though.