Bug report from JETLS: `:scope_layer` interfering with user code

[mlechu]

aviatesk/JETLS.jl#292

As far as I'm aware, this is working as intended. We may want to reconsider the :scope_layer injection behaviour if it becomes an issue frequently.

[aviatesk]

MRE is:

julia> JL.include_string(Main, """
       macro do_stuff(expr)
           expr isa Symbol || error("")
           expr
       end

       function do_stuff()
           some_data = []
           GC.@preserve some_data begin
               @do_stuff nothing
           end
       end
       """)
ERROR: MacroExpansionError while expanding $(Expr(:scope_layer, :do_stuff, 1)) in module JuliaLowering:
    some_data = []
    GC.@preserve some_data begin
        @do_stuff nothing
#       └───────────────┘ ── Error expanding macro
    end
end
Caused by:

Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:44
  [2] var"@do_stuff"(__source__::LineNumberNode, __module__::Module, expr::Any)
    @ Main ./REPL[1]:2
  [3] expand_macro(ctx::JuliaLowering.MacroExpansionContext{JuliaLowering.SyntaxGraph{Dict{Symbol, Any}}}, ex::JuliaLowering.SyntaxTree{JuliaLowering.SyntaxGraph{Dict{Symbol, Any}}})
    @ JuliaLowering ~/.julia/packages/JuliaLowering/LXBBb/src/macro_expansion.jl:316
  [4] expand_forms_1(ctx::JuliaLowering.MacroExpansionContext{JuliaLowering.SyntaxGraph{Dict{…}}}, ex::JuliaLowering.SyntaxTree{JuliaLowering.SyntaxGraph{Dict{…}}})
    @ JuliaLowering ~/.julia/packages/JuliaLowering/LXBBb/src/macro_expansion.jl:442
  [5] #expand_forms_1##4
    @ ~/.julia/packages/JuliaLowering/LXBBb/src/macro_expansion.jl:519 [inlined]
  [6] mapchildren(f::JuliaLowering.var"#expand_forms_1##4#expand_forms_1##5"{…}, ctx::JuliaLowering.MacroExpansionContext{…}, ex::JuliaLowering.SyntaxTree{…}, do_map_child::JuliaLowering.var"#38#39"; extra_attrs::@Kwargs{})
    @ JuliaLowering ~/.julia/packages/JuliaLowering/LXBBb/src/ast.jl:403
  [7] mapchildren(f::Function, ctx::JuliaLowering.MacroExpansionContext{…}, ex::JuliaLowering.SyntaxTree{…}, do_map_child::Function)
    @ JuliaLowering ~/.julia/packages/JuliaLowering/LXBBb/src/ast.jl:395
  [8] mapchildren(f::Function, ctx::JuliaLowering.MacroExpansionContext{…}, ex::JuliaLowering.SyntaxTree{…}; extra_attrs::@Kwargs{})
    @ JuliaLowering ~/.julia/packages/JuliaLowering/LXBBb/src/ast.jl:442
  [9] mapchildren
    @ ~/.julia/packages/JuliaLowering/LXBBb/src/ast.jl:441 [inlined]
 [10] expand_forms_1(ctx::JuliaLowering.MacroExpansionContext{JuliaLowering.SyntaxGraph{Dict{…}}}, ex::JuliaLowering.SyntaxTree{JuliaLowering.SyntaxGraph{Dict{…}}})
    @ JuliaLowering ~/.julia/packages/JuliaLowering/LXBBb/src/macro_expansion.jl:519
 [11] #expand_forms_1##4
    @ ~/.julia/packages/JuliaLowering/LXBBb/src/macro_expansion.jl:519 [inlined]
 [12] mapchildren(f::JuliaLowering.var"#expand_forms_1##4#expand_forms_1##5"{…}, ctx::JuliaLowering.MacroExpansionContext{…}, ex::JuliaLowering.SyntaxTree{…}, do_map_child::JuliaLowering.var"#38#39"; extra_attrs::@Kwargs{})
    @ JuliaLowering ~/.julia/packages/JuliaLowering/LXBBb/src/ast.jl:403
 [13] mapchildren(f::Function, ctx::JuliaLowering.MacroExpansionContext{…}, ex::JuliaLowering.SyntaxTree{…}, do_map_child::Function)
    @ JuliaLowering ~/.julia/packages/JuliaLowering/LXBBb/src/ast.jl:395
 [14] mapchildren(f::Function, ctx::JuliaLowering.MacroExpansionContext{…}, ex::JuliaLowering.SyntaxTree{…}; extra_attrs::@Kwargs{})
    @ JuliaLowering ~/.julia/packages/JuliaLowering/LXBBb/src/ast.jl:442
 [15] mapchildren
    @ ~/.julia/packages/JuliaLowering/LXBBb/src/ast.jl:441 [inlined]
 [16] expand_forms_1(ctx::JuliaLowering.MacroExpansionContext{JuliaLowering.SyntaxGraph{Dict{…}}}, ex::JuliaLowering.SyntaxTree{JuliaLowering.SyntaxGraph{Dict{…}}})
    @ JuliaLowering ~/.julia/packages/JuliaLowering/LXBBb/src/macro_expansion.jl:519
 [17] #expand_forms_1##4
    @ ~/.julia/packages/JuliaLowering/LXBBb/src/macro_expansion.jl:519 [inlined]
 [18] mapchildren(f::JuliaLowering.var"#expand_forms_1##4#expand_forms_1##5"{…}, ctx::JuliaLowering.MacroExpansionContext{…}, ex::JuliaLowering.SyntaxTree{…}, do_map_child::JuliaLowering.var"#38#39"; extra_attrs::@Kwargs{})
    @ JuliaLowering ~/.julia/packages/JuliaLowering/LXBBb/src/ast.jl:403
 [19] mapchildren(f::Function, ctx::JuliaLowering.MacroExpansionContext{…}, ex::JuliaLowering.SyntaxTree{…}, do_map_child::Function)
    @ JuliaLowering ~/.julia/packages/JuliaLowering/LXBBb/src/ast.jl:395
 [20] mapchildren(f::Function, ctx::JuliaLowering.MacroExpansionContext{…}, ex::JuliaLowering.SyntaxTree{…}; extra_attrs::@Kwargs{})
    @ JuliaLowering ~/.julia/packages/JuliaLowering/LXBBb/src/ast.jl:442
 [21] mapchildren
    @ ~/.julia/packages/JuliaLowering/LXBBb/src/ast.jl:441 [inlined]
 [22] expand_forms_1(ctx::JuliaLowering.MacroExpansionContext{JuliaLowering.SyntaxGraph{Dict{…}}}, ex::JuliaLowering.SyntaxTree{JuliaLowering.SyntaxGraph{Dict{…}}})
    @ JuliaLowering ~/.julia/packages/JuliaLowering/LXBBb/src/macro_expansion.jl:519
 [23] expand_macro(ctx::JuliaLowering.MacroExpansionContext{JuliaLowering.SyntaxGraph{Dict{Symbol, Any}}}, ex::JuliaLowering.SyntaxTree{JuliaLowering.SyntaxGraph{Dict{Symbol, Any}}})
    @ JuliaLowering ~/.julia/packages/JuliaLowering/LXBBb/src/macro_expansion.jl:341
 [24] expand_forms_1(ctx::JuliaLowering.MacroExpansionContext{JuliaLowering.SyntaxGraph{Dict{…}}}, ex::JuliaLowering.SyntaxTree{JuliaLowering.SyntaxGraph{Dict{…}}})
    @ JuliaLowering ~/.julia/packages/JuliaLowering/LXBBb/src/macro_expansion.jl:442
 [25] #expand_forms_1##4
    @ ~/.julia/packages/JuliaLowering/LXBBb/src/macro_expansion.jl:519 [inlined]
 [26] mapchildren(f::JuliaLowering.var"#expand_forms_1##4#expand_forms_1##5"{…}, ctx::JuliaLowering.MacroExpansionContext{…}, ex::JuliaLowering.SyntaxTree{…}, do_map_child::JuliaLowering.var"#38#39"; extra_attrs::@Kwargs{})
    @ JuliaLowering ~/.julia/packages/JuliaLowering/LXBBb/src/ast.jl:403
 [27] mapchildren(f::Function, ctx::JuliaLowering.MacroExpansionContext{…}, ex::JuliaLowering.SyntaxTree{…}, do_map_child::Function)
    @ JuliaLowering ~/.julia/packages/JuliaLowering/LXBBb/src/ast.jl:395
 [28] mapchildren(f::Function, ctx::JuliaLowering.MacroExpansionContext{…}, ex::JuliaLowering.SyntaxTree{…}; extra_attrs::@Kwargs{})
    @ JuliaLowering ~/.julia/packages/JuliaLowering/LXBBb/src/ast.jl:442
 [29] mapchildren
    @ ~/.julia/packages/JuliaLowering/LXBBb/src/ast.jl:441 [inlined]
 [30] expand_forms_1(ctx::JuliaLowering.MacroExpansionContext{JuliaLowering.SyntaxGraph{Dict{…}}}, ex::JuliaLowering.SyntaxTree{JuliaLowering.SyntaxGraph{Dict{…}}})
    @ JuliaLowering ~/.julia/packages/JuliaLowering/LXBBb/src/macro_expansion.jl:519
 [31] #expand_forms_1##4
    @ ~/.julia/packages/JuliaLowering/LXBBb/src/macro_expansion.jl:519 [inlined]
 [32] mapchildren(f::JuliaLowering.var"#expand_forms_1##4#expand_forms_1##5"{…}, ctx::JuliaLowering.MacroExpansionContext{…}, ex::JuliaLowering.SyntaxTree{…}, do_map_child::JuliaLowering.var"#38#39"; extra_attrs::@Kwargs{})
    @ JuliaLowering ~/.julia/packages/JuliaLowering/LXBBb/src/ast.jl:403
 [33] mapchildren(f::Function, ctx::JuliaLowering.MacroExpansionContext{…}, ex::JuliaLowering.SyntaxTree{…}, do_map_child::Function)
    @ JuliaLowering ~/.julia/packages/JuliaLowering/LXBBb/src/ast.jl:395
 [34] mapchildren(f::Function, ctx::JuliaLowering.MacroExpansionContext{…}, ex::JuliaLowering.SyntaxTree{…}; extra_attrs::@Kwargs{})
    @ JuliaLowering ~/.julia/packages/JuliaLowering/LXBBb/src/ast.jl:442
 [35] mapchildren
    @ ~/.julia/packages/JuliaLowering/LXBBb/src/ast.jl:441 [inlined]
 [36] expand_forms_1(ctx::JuliaLowering.MacroExpansionContext{JuliaLowering.SyntaxGraph{Dict{…}}}, ex::JuliaLowering.SyntaxTree{JuliaLowering.SyntaxGraph{Dict{…}}})
    @ JuliaLowering ~/.julia/packages/JuliaLowering/LXBBb/src/macro_expansion.jl:519
 [37] lower_step(iter::JuliaLowering.LoweringIterator{JuliaLowering.SyntaxGraph{Dict{Symbol, Any}}}, push_mod::Nothing)
    @ JuliaLowering ~/.julia/packages/JuliaLowering/LXBBb/src/eval.jl:74
 [38] _eval(mod::Module, iter::JuliaLowering.LoweringIterator{JuliaLowering.SyntaxGraph{Dict{Symbol, Any}}}, new_mod::Nothing)
    @ JuliaLowering ~/.julia/packages/JuliaLowering/LXBBb/src/eval.jl:501
 [39] _eval
    @ ~/.julia/packages/JuliaLowering/LXBBb/src/eval.jl:498 [inlined]
 [40] #eval#170
    @ ~/.julia/packages/JuliaLowering/LXBBb/src/eval.jl:461 [inlined]
 [41] eval
    @ ~/.julia/packages/JuliaLowering/LXBBb/src/eval.jl:457 [inlined]
 [42] #include_string#172
    @ ~/.julia/packages/JuliaLowering/LXBBb/src/eval.jl:562 [inlined]
 [43] include_string
    @ ~/.julia/packages/JuliaLowering/LXBBb/src/eval.jl:560 [inlined]
 [44] include_string(mod::Module, code::String)
    @ JuliaLowering ~/.julia/packages/JuliaLowering/LXBBb/src/eval.jl:560
 [45] top-level scope
    @ REPL[5]:1
Some type information was truncated. Use `show(err)` to see complete types.

, which just runs as expected with the native lowerer:

julia> macro do_stuff(expr)
           expr isa Symbol || error("")
           expr
       end
@do_stuff (macro with 2 methods)

julia> function do_stuff()
           some_data = []
           GC.@preserve some_data begin
               @do_stuff nothing
           end
       end
do_stuff (generic function with 1 method)

julia> do_stuff()

[c42f]

Oh this is a tricky one. The issue arises here in a subtle interplay of mixing old and new style macros with the two different hygiene representations.

If we just had @do_stuff implemented as an old-style macro without the @preserve wrapping, we currently don't attach scope layer information to the arguments. So nothing would be a plain symbol. But if we have the outer macro call (implemented in JuliaLowering as a new-style macro) we do attach scope layer information to the arguments of the @preserve. Then when @do_stuff gets called later, it gets scope layer information it doesn't expect.

We can probably fix this by removing the scope layer information from the arguments in this particular case, but we'll need to think how to make that correct and general. (In some circumstances it's not possible - and in those cases existing macros would normally see Expr(:escape) and likely break or produce incorrect results. But we shouldn't break existing cases which work correctly and it seems this is one such example.)