New invalidations (#418)

This is a rewrite of the invalidations infrastructure based on Julia 1.12+. Julia 1.12 separates the data into two logging streams, one for immediate method insertion/deletion and the other for edge-validation during package loading. This allows a cleaner implementation of the parsing. I've also invested in a more extensive and systematic test suite, attempting to cover all code paths, and added tests for binding invalidation. Finally, this expands the documentation on developer topics.

Author: timholy

.github/workflows/ci.yml

@@ -20,8 +20,7 @@ jobs:
       fail-fast: false
       matrix:
         version:
-          - '1.10'
-          - '1'
+#          - 'min'
           - 'pre'
           - 'nightly'
         os:
@@ -44,7 +43,7 @@ jobs:
             ${{ runner.os }}-test-${{ env.cache-name }}-
             ${{ runner.os }}-test-
             ${{ runner.os }}-
-      - run: julia --project -e 'using Pkg; Pkg.develop([PackageSpec(path="SnoopCompileCore")])'
+      # - run: julia --project -e 'using Pkg; Pkg.develop([PackageSpec(path="SnoopCompileCore")])'
       - uses: julia-actions/julia-buildpkg@latest
       - uses: julia-actions/julia-runtest@latest
       - run: julia  --check-bounds=yes --project -e 'using Pkg; Pkg.test(; test_args=["cthulhu"], coverage=true)'

Project.toml

@@ -1,7 +1,7 @@
 name = "SnoopCompile"
 uuid = "aa65fe97-06da-5843-b5b1-d5d13cad87d2"
+version = "3.2.0"
 author = ["Tim Holy <[email protected]>", "Shuhei Kadowaki <[email protected]>"]
-version = "3.1.3"
 
 [deps]
 AbstractTrees = "1520ce14-60c1-5f80-bbc7-55ef81b5835c"
@@ -20,6 +20,9 @@ Cthulhu = "f68482b8-f384-11e8-15f7-abe071a5a75f"
 JET = "c3a54625-cd67-489e-a8e7-0a5a0ff4e31b"
 PyPlot = "d330b81b-6aea-500a-939a-2ce795aea3ee"
 
+[sources]
+SnoopCompileCore = {path = "SnoopCompileCore"}
+
 [extensions]
 CthulhuExt = "Cthulhu"
 JETExt = ["JET", "Cthulhu"]
@@ -32,7 +35,7 @@ Cthulhu = "2"
 FlameGraphs = "1"
 InteractiveUtils = "1"
 JET = "0.9"
-MethodAnalysis = "0.4"
+MethodAnalysis = "1"
 OrderedCollections = "1"
 Pkg = "1"
 PrettyTables = "2, 3"
@@ -42,10 +45,10 @@ PyPlot = "2"
 REPL = "1"
 Random = "1"
 Serialization = "1"
-SnoopCompileCore = "3"
+SnoopCompileCore = "3.1"
 Test = "1"
 YAML = "0.4"
-julia = "1.10"
+julia = "1.12"
 
 [extras]
 Cthulhu = "f68482b8-f384-11e8-15f7-abe071a5a75f"

SnoopCompileCore/Project.toml

@@ -1,10 +1,10 @@
 name = "SnoopCompileCore"
 uuid = "e2b509da-e806-4183-be48-004708413034"
 author = ["Tim Holy <[email protected]>"]
-version = "3.0.0"
+version = "3.1.0"
 
 [deps]
 Serialization = "9e88b42a-f829-5b0c-bbe9-9e923198166b"
 
 [compat]
-julia = "1"
+julia = "1.12"

SnoopCompileCore/src/snoop_invalidations.jl

@@ -1,5 +1,10 @@
 export @snoop_invalidations
 
+struct InvalidationLists
+    logedges::Vector{Any}
+    logmeths::Vector{Any}
+end
+
 """
     invs = @snoop_invalidations expr
 
@@ -26,12 +31,19 @@ Method insertion results in the sequence
 The authoritative reference is Julia's own `src/gf.c` file.
 """
 macro snoop_invalidations(expr)
-    quote
-        local invs = ccall(:jl_debug_method_invalidation, Any, (Cint,), 1)
-        Expr(:tryfinally,
-            $(esc(expr)),
+    # It's a little unclear why this is better than a quoted try/finally, but it seems to be
+    # Guessing it's a lack of a block around `expr`
+    exoff = Expr(:tryfinally,
+        esc(expr),
+        quote
+            Base.StaticData.debug_method_invalidation(false)
             ccall(:jl_debug_method_invalidation, Any, (Cint,), 0)
-        )
-        invs
+        end
+    )
+    return quote
+        local logedges = Base.StaticData.debug_method_invalidation(true)
+        local logmeths = ccall(:jl_debug_method_invalidation, Any, (Cint,), 1)
+        $exoff
+        $InvalidationLists(logedges, logmeths)
     end
 end

docs/make.jl

@@ -15,7 +15,7 @@ makedocs(
     pages = ["index.md",
              "Basic tutorials" => ["tutorials/invalidations.md", "tutorials/snoop_inference.md", "tutorials/snoop_llvm.md", "tutorials/pgdsgui.md", "tutorials/jet.md"],
              "Advanced tutorials" => ["tutorials/snoop_inference_analysis.md", "tutorials/snoop_inference_parcel.md"],
-             "Explanations" => ["explanations/tools.md", "explanations/gotchas.md", "explanations/fixing_inference.md"],
+             "Explanations" => ["explanations/tools.md", "explanations/gotchas.md", "explanations/fixing_inference.md", "explanations/invalidation_classes.md", "explanations/devs.md"],
              "reference.md",
     ]
 )

docs/src/explanations/devs.md

@@ -0,0 +1,51 @@
+# Information for SnoopCompile developers
+
+## Invalidations
+
+### Capturing invalidation logs
+
+Julia itself handles (in)validation when you define (or delete) methods and load packages. Julia's internal machinery provides the option of recording these invalidation decisions to a log, which is just a `Vector{Any}`. Currently (as of Julia 1.12) there are two independent logs:
+
+- for method insertion and deletion (i.e., new methods invalidating old code), logging is handled in Julia's `src/gf.c`. You enable it with `logmeths = ccall(:jl_debug_method_invalidation, Any, (Cint,), true)` and pass a final argument of `false` to turn it off.
+- for validating precompiled code during package loading (i.e., "new" code being invalidated by old methods), logging is handled in Julia's `base/staticdata.jl`. You enable it with `logedges = Base.StaticData.debug_method_invalidation(true)` and pass `false` to turn it off.
+
+In both cases, the log will initially be empty, but subsequent activity (defining or deleting methods, or loading packages) may add entries.
+
+SnoopCompileCore's `@snoop_invalidation` just turns on these logging streams, executes the user's block of code, turns off logging, and returns the captured log streams.
+
+### Interpreting invalidation logs
+
+The definitive source for interpreting these two logging streams is Julia's own code; the documentation below may be outdated by future changes in Julia. (Such changes have happened repeatedly over the course of Julia's development.) If you have even a shred of doubt about whether any of this is (still) correct, check Julia's code.
+
+For both logging streams, a single decision typically results in appending multiple entries to the log. These decisions come with a string (the *tag*) documenting the origin of each entry. In general, each distinct mechanism by which invalidations can occur should have its own unique tag. Often these correspond to specific lines in the source code.
+
+#### method logs
+
+Let `trigger::Method` indicate an added or deleted method for function `f`. If defining/deleting this method would change how one or more `caller::MethodInstance`s of the corresponding function would dispatch, those `caller`s must be invalidated. Such events can result in a cascade of invalidations of code that directly or indirectly called `trigger` or less-specific methods of the same function. The order in which these invalidations appear in the log stream is as follows:
+
+2. Backedges of `callee` below, encoded as a tree where links are specified as `(caller::MethodInstance, depth::Int32)` pairs.
+  `depth=1` typically corresponds to an inferrable caller. `depth=0` corresponds to a potentially-missing callee (at the time of compilation), and will be followed by `calleesig::DataType`. (If the called function had potentially-applicable methods, `calleesig` will not be a subtype of any of their signatures.) corresponds to the root (though no entry with `depth=0` is written), and sequential increases in `depth` indicate a traversal through branches. If `depth` decreases, this indicates the start of a new branch from the parent with depth `depth-1`.
+1. `(callee::MethodInstance, tag)` pairs that were directly affected by change in dispatch.
+3. Possibly,
+
+After all such `callee` branches are complete, the `(trigger::Method, tag)` event that initiated the entire set of invalidations pair is logged.
+
+The interpretation of the tags is as follows:
+
+- `"jl_method_table_disable"`: the `trigger` with the same tag was deleted (`Base.delete_method`)
+- `"jl_method_table_insert"`: the `trigger` with the same tag was added (`function f(...) end`)
+- `(callee::MethodInstance, "invalidate_mt_cache")`: a method-table cache for runtime dispatch was invalidated by a method insertion. At sites of runtime dispatch, Julia will maintain local method tables of the most common call targets to make dispatch more efficient. Since runtime dispatch involves real-time method lookup anyway, this form of invalidation is not serious, and a detailed listing is suppressed by SnoopCompile's printing behavior. These are always followed (eventually) by a `(trigger::Method, "jl_method_table_insert")` pair.
+
+#### edge logs
+
+Since edge logs are populated during package loading, we'll use `PkgDep` to indicate a package that is a dependency for `PkgUser`. (`PkgUser`'s `Project.toml` might list `PkgDep` in its `[deps]` section, or it might be an indirect dependency.)
+Invalidation events result in the insertion of 3 or 4 items in `logedges`. The tag is always the second item. They take one of the following forms:
+
+- `(def::Method, "method_globalref", codeinst::CodeInstance, nothing)`: method `def` in `PkgUser` references `PkgDep.SomeObject` (which might be `const` data, a type, etc.), but the binding for `SomeObject` has been modified since `PkgUser` was compiled. `codeinst`, which holds a compiled specialization of `def`, needs to be recompiled.
+- `(edge::Union{MethodInstance,DataType,Core.Binding}, "insert_backedges_callee", codeinst::CodeInstance, matches::Union{Vector{Any},Nothing})`: `edge` was selected as a dispatch target (a "callee") of `codeinst`, but new method(s) listed in `matches` now supersede it in dispatch specificity. There are 3 or 4 sub-cases:
+  * `edge::MethodInstance` indicates a known target at the time of compilation
+  * `edge::DataType` represents either
+    + `Tuple{typeof(f), argtypes...}` for a poorly-inferred or `invoke`d call for which the target selected at compilation time is no longer valid (`matches` will be `nothing`)
+    + a signature of a known function for which no appropriate method had yet been defined at the time of compilation. `matches` lists methods that now apply.
+  * `edge::Core.Binding` indicates a target that was unknown at the time of compilation, and `matches` will be `nothing`.
+- `(caller::CodeInstance, "verify_methods", callee::CodeInstance)`: `callee` is an invalidated dependency of `caller`. These encode invalidations that cascade from the proximal source.

docs/src/explanations/invalidation_classes.md

@@ -0,0 +1,22 @@
+# Invalidation classes
+
+[`invalidation_trees`](@ref) returns two broad classes of invalidated targets in `backedges` and `mt_backedges`.
+To understand the difference, let's introduce a new term: we say that a callee method *covers* a call if it can accept all possible types used in that call. Consider this example:
+
+```julia
+f(x::Integer) = false
+g1(x::Signed) = f(x)    # `f(::Integer)` always covers this call
+g2(x::Number) = f(x)    # `f(::Integer)` may not cover this call
+```
+
+`g1` will only ever be called for `Signed` inputs, and because `Signed <: Integer`, the method of `f` fully covers the call in `g1`. In contrast, `g2` can be called for any `Number` type, and since `Number` is not a subtype of `Integer`, `f` may not cover the entire call.
+
+With this understanding, the difference is straightforward: `backedges`-class invalidations are when there is exactly one applicable method and it fully covers the call. `mt_backedges`-class invalidations are for anything else. In such cases, Julia may need to scan the method table (the `mt` in `mt_backedges`) of the function in order to determine which method, if any, might be applicable.
+
+This helps explain why `mt_backedges` invalidations are more likely to arise from poor inference: poor inference "widens" the argument types and thus makes it more likely that a call is unlikely to be covered by exactly one method. It's still possible to get a `backedges`-class invalidation from poor inference:
+
+```julia
+g3(x::Ref{Any}) = f(x[]::Signed)
+```
+
+guarantees that our method of `f` covers the call, even though we can't predict with precision what type `x[]` will return. Thus if you invalidate the compiled code of `g3` by defining a new method for `f(x::Signed)`, you'll get a `backedges`-class invalidation.

docs/src/tutorials/invalidations.md

@@ -163,7 +163,7 @@ sig, victim = tree.mt_backedges[end];
 ```
 
 !!! note
-    `mt_backedges` stands for "MethodTable backedges." In other cases you may see a second type of invalidation, just called `backedges`. With these, there is no `sig`, and so you'll use just `victim = tree.backedges[i]`.
+    `mt_backedges` stands for "MethodTable backedges." In other cases you may see a second kind of invalidation, just called `backedges`. With these, there is no `sig`, and so you'll use just `victim = tree.backedges[i]`. For those curious about the reasons for these two kinds of invalidation, see [Invalidation classes](@ref).
 
 First let's look at the the problematic method `sig`nature:
 
@@ -223,7 +223,9 @@ The first and simplest technique is to ensure that the full range of possibiltie
 
 #### Method 2: improve inferability
 
-The second way to prevent invalidations is to improve the inferability of the victim(s). If `Int` and `Char` really are the only possible kinds of cards, then in `playgame` it would be better to declare
+The second way to prevent invalidations is to improve the inferability of the victim(s). This approach is often applicable to `mt_backedges` invalidations,  but it can sometimes fix `backedges` invalidations too. [Invalidation classes](@ref) explains the differences in detail and why inference failures tend to be affiliated with `mt_backedges` invalidations.
+
+In our blackjack example, if `Int` and `Char` really are the only possible kinds of cards, then in `playgame` it would be better to declare
 
 ```julia
 myhand = Union{Int,Char}[]

src/SnoopCompile.jl

@@ -33,9 +33,10 @@ you should prefer them above the more limited tools available on earlier version
 module SnoopCompile
 
 using SnoopCompileCore
+using SnoopCompileCore: InvalidationLists
 # More exports are defined below in the conditional loading sections
 
-using Core: MethodInstance, CodeInfo
+using Core: MethodInstance, CodeInstance, Binding, CodeInfo
 using InteractiveUtils
 using Serialization
 using Printf
@@ -82,8 +83,8 @@ export read_snoop_llvm
 include("invalidations.jl")
 export uinvalidated, invalidation_trees, filtermod, findcaller
 
-include("invalidation_and_inference.jl")
-export precompile_blockers
+# include("invalidation_and_inference.jl")
+# export precompile_blockers
 
 # Write
 include("write.jl")