Merge pull request #172 from JuliaDebug/teh/refresh_docs

Enhance the documentation

Author: timholy

docs/make.jl

@@ -1,5 +1,7 @@
 using Documenter, JuliaInterpreter, Test, CodeTracking
 
+remove()   # ensure there are no activate breakpoints
+
 makedocs(
     modules = [JuliaInterpreter],
     clean = false,

docs/src/ast.md

@@ -1,6 +1,22 @@
+!!! note
+    This page and the next are designed to teach a little more about the internals.
+    Depending on your interest, you may be able to skip them.
+
 # Lowered representation
 
-Let's start with a demonstration on simple function:
+JuliaInterpreter uses the lowered representation of code.
+The key advantage of lowered representation is that it is fairly well circumscribed:
+
+- There are only a limited number of legal statements that can appear in lowered code
+- Each statement is "unpacked" to essentially do one thing
+- Scoping of variables is simplified via the slot mechanism, described below
+- Names are fully resolved by module
+- Macros are expanded
+
+[Julia AST](https://docs.julialang.org/en/latest/devdocs/ast/) describes the kinds of
+objects that can appear in lowered code.
+
+Let's start with a demonstration on a simple function:
 
 ```julia
 function summer(A::AbstractArray{T}) where T
@@ -14,69 +30,136 @@ end
 A = [1, 2, 5]
 ```
 
-JuliaInterpreter uses the lowered representation of code:
+To interpret lowered representation, it maybe be useful to rewrite the body of `summer` in the following ways.
+First let's use an intermediate representation that expands the `for a in A ... end` loop:
+
+```julia
+    s = zero(T)
+    temp = iterate(A)         # `for` loops get lowered to `iterate/while` loops
+    while temp !== nothing
+        a, state = temp
+        s += a
+        temp = iterate(A, state)
+    end
+    return s
+```
+
+The lowered code takes the additional step of resolving the names by module and turning all the
+branching into `@goto/@label` equivalents:
 
 ```julia
-julia> code = @code_lowered summer(A)
+    # Code starting at line 2 (the first line of the body)
+    s = Main.zero(T)       # T corresponds to the first parameter, i.e., $(Expr(:static_parameter, 1))
+
+    # Code starting at line 3
+    temp = Base.iterate(A) # here temp = @_4
+    if temp === nothing    # this comparison gets stored as %4, and %5 stores !(temp===nothing)
+        @goto block4
+    end
+
+    @label block2
+        ## BEGIN block2
+        a, state = temp[1], temp[2]  # these correspond to the `getfield` calls, state is %9
+
+        # Code starting at line 4
+        s = s + a
+
+        # Code starting at line 5
+        temp = iterate(A, state)     # A is also %2
+        if temp === nothing
+            @goto block4             # the `while` condition was false
+        end
+        ## END block2
+
+    @goto block2           # here the `while` condition is still true
+
+    # Code starting at line 6
+    @label block4
+        ## BEGIN block4
+        return s
+        ## END block4
+```
+
+This has very close correspondence to the lowered representation:
+
+```julia
+julia> code = @code_lowered debuginfo=:source summer(A)
 CodeInfo(
-1 ─       s = (Main.zero)($(Expr(:static_parameter, 1)))
+    @ REPL[1]:2 within `summer'
+1 ─       s = Main.zero($(Expr(:static_parameter, 1)))
+│   @ REPL[1]:3 within `summer'
 │   %2  = A
-│         #temp# = (Base.iterate)(%2)
-│   %4  = #temp# === nothing
-│   %5  = (Base.not_int)(%4)
+│         @_4 = Base.iterate(%2)
+│   %4  = @_4 === nothing
+│   %5  = Base.not_int(%4)
 └──       goto #4 if not %5
-2 ┄ %7  = #temp#
-│         a = (Core.getfield)(%7, 1)
-│   %9  = (Core.getfield)(%7, 2)
+2 ┄ %7  = @_4
+│         a = Core.getfield(%7, 1)
+│   %9  = Core.getfield(%7, 2)
+│   @ REPL[1]:4 within `summer'
 │         s = s + a
-│         #temp# = (Base.iterate)(%2, %9)
-│   %12 = #temp# === nothing
-│   %13 = (Base.not_int)(%12)
+│         @_4 = Base.iterate(%2, %9)
+│   %12 = @_4 === nothing
+│   %13 = Base.not_int(%12)
 └──       goto #4 if not %13
 3 ─       goto #2
+    @ REPL[1]:6 within `summer'
 4 ┄       return s
 )
 ```
+!!! note
+    Not all Julia versions support `debuginfo`. If the command above fails for you,
+    just omit the `debuginfo=:source` portion.
 
 To understand this package's internals, you need to familiarize yourself with these
-`CodeInfo` objects. The numbers on the left correspond to [basic blocks](https://en.wikipedia.org/wiki/Basic_block);
-when used in statements these are printed with a hash, e.g., in `goto #4 if not %6`, the
+`CodeInfo` objects.
+The lines that start with `@ REPL[1]:n` indicate the source line of the succeeding
+block of statements; here we defined this method in the REPL, so the source file is `REPL[1]`;
+the number after the colon is the line number.
+
+The numbers on the left correspond to [basic blocks](https://en.wikipedia.org/wiki/Basic_block),
+as we annotated with `@label block2` above.
+When used in statements these are printed with a hash, e.g., in `goto #4 if not %5`, the
 `#4` refers to basic block 4.
-The numbers in the next column--e.g., `%1`, refer to [single static assignment (SSA) values](https://en.wikipedia.org/wiki/Static_single_assignment_form).
+The numbers in the next column--e.g., `%2`, refer to
+[single static assignment (SSA) values](https://en.wikipedia.org/wiki/Static_single_assignment_form).
 Each statement (each line of this printout) corresponds to a single SSA value,
 but only those used later in the code are printed using assignment syntax.
-Wherever a previous SSA value is used, it's referenced by an `SSAValue` and printed as `%6`;
-for example, in `goto #4 if not %6`, the `%6` is the result of evaluating the 6th statement,
-which is `(Base.not_int)(%5)`, which in turn refers to the result of statement 5.
-Together lines 5 and 6 correspond to `!(#temp# === nothing)`.
-(The `#temp#` means that this was a generated variable name not present explicitly in the original source code.)
+Wherever a previous SSA value is used, it's referenced by an `SSAValue` and printed as `%5`;
+for example, in `goto #4 if not %5`, the `%5` is the result of evaluating the 5th statement,
+which is `(Base.not_int)(%4)`, which in turn refers to the result of statement 4.
+Finally, temporary variables here are shown as `@_4`; the `_` indicates a *slot*, either
+one of the input arguments or a local variable, and the 4 means the 4th one.
+Together lines 4 and 5 correspond to `!(@_4 === nothing)`, where `@_4` has been assigned the
+result of the call to `iterate` occurring on line 3. (In some Julia versions, this may be printed as `#temp#`,
+similar to how we named it in our alternative implementation above.)
 
-Before diving into the details, let's first look at the statements themselves:
+Let's look at a couple of the fields of the `CodeInfo`. First, the statements themselves:
 
 ```julia
 julia> code.code
 16-element Array{Any,1}:
- :(_3 = (Main.zero)($(Expr(:static_parameter, 1))))
+ :(_3 = Main.zero($(Expr(:static_parameter, 1))))
  :(_2)
- :(_4 = (Base.iterate)(%2))
+ :(_4 = Base.iterate(%2))
  :(_4 === nothing)
- :((Base.not_int)(%4))
+ :(Base.not_int(%4))
  :(unless %5 goto %16)
  :(_4)
- :(_5 = (Core.getfield)(%7, 1))
- :((Core.getfield)(%7, 2))
+ :(_5 = Core.getfield(%7, 1))
+ :(Core.getfield(%7, 2))
  :(_3 = _3 + _5)
- :(_4 = (Base.iterate)(%2, %9))
+ :(_4 = Base.iterate(%2, %9))
  :(_4 === nothing)
- :((Base.not_int)(%12))
+ :(Base.not_int(%12))
  :(unless %13 goto %16)
  :(goto %7)
  :(return _3)
 ```
 
 You can see directly that the SSA assignments are implicit; they are not directly
 present in the statement list.
-The most noteworthy change here is the appearance of objects like `_3`, which are
+The most noteworthy change here is the appearance of more objects like `_3`, which are
 references that index into local variable slots:
 
 ```julia
@@ -85,9 +168,9 @@ julia> code.slotnames
  Symbol("#self#")
  :A
  :s
- Symbol("#temp#")
+ Symbol("")
  :a
 ```
 
-When printing the whole `CodeInfo` object, these `slotnames` are substituted in.
-The types of objects that can be in `code.code` is well-described in the [Julia AST](https://docs.julialang.org/en/latest/devdocs/ast/) documentation.
+When printing the whole `CodeInfo` object, these `slotnames` are substituted in
+(unless they are empty, as was the case for `@_4` above).

docs/src/index.md

@@ -5,22 +5,133 @@ Normally, Julia compiles your code when you first execute it; using JuliaInterpr
 avoid compilation and execute the expressions that define your code directly.
 Interpreters have a number of applications, including support for stepping debuggers.
 
-At a pure user level, there is not much to know:
+## Use an an interpreter
 
-```jldoctest
+Using this package as an interpreter is straightforward:
+
+```jldoctest demo1
 julia> using JuliaInterpreter
 
-julia> a = [1, 2, 5]
+julia> list = [1, 2, 5]
 3-element Array{Int64,1}:
  1
  2
  5
 
-julia> sum(a)
+julia> sum(list)
+8
+
+julia> @interpret sum(list)
 8
+```
+
+## Breakpoints
+
+You can interrupt execution by setting breakpoints.
+You can set breakpoints via packages that explicitly target debugging,
+like [Juno](http://junolab.org/), [Debugger](https://github.com/JuliaDebug/Debugger.jl), and
+[Rebugger](https://github.com/timholy/Rebugger.jl).
+But all of these just leverage the core functionality defined in JuliaInterpreter,
+so here we'll illustrate it without using any of these other packages.
+
+Let's set a conditional breakpoint, to be triggered any time one of the elements in the
+argument to `sum` is bigger than 4:
+
+```jldoctest demo1; filter = r"in Base at .*$"
+julia> @breakpoint sum([1, 2]) any(x->x>4, a)
+breakpoint(sum(a::AbstractArray) in Base at reducedim.jl:648, line 648)
+```
+
+Note that in writing the condition, we used `a`, the name of the argument to the relevant
+method of `sum`. Conditionals should be written using a combination of argument and parameter
+names of the method into which you're inserting a breakpoint; you can also use any
+globally-available name (as used here with the `any` function).
+
+Now let's see what happens:
+
+```jldoctest demo1; filter = r"in Base at .*$"
+julia> @interpret sum([1,2,3])  # no element bigger than 4, breakpoint should not trigger
+6
+
+julia> frame, bp = @interpret sum([1,2,5])  # should trigger breakpoint
+(Frame for sum(a::AbstractArray) in Base at reducedim.jl:648
+c 1* 648  1 ─      nothing
+  2  648  │   %2 = (Base.#sum#550)(Colon(), #self#, a)
+  3  648  └──      return %2
+a = [1, 2, 5], breakpoint(sum(a::AbstractArray) in Base at reducedim.jl:648, line 648))
+```
 
-julia> @interpret sum(a)
+`frame` is described in more detail on the next page; for now, suffice it to say
+that the `c` in the leftmost column indicates the presence of a conditional breakpoint
+upon entry to `sum`. `bp` is a reference to the breakpoint. You can manipulate these
+at the command line:
+
+```jldoctest demo1; filter = r"in Base at .*$"
+julia> disable(bp)
+false
+
+julia> @interpret sum([1,2,5])
 8
+
+julia> enable(bp)
+true
+
+julia> @interpret sum([1,2,5])
+(Frame for sum(a::AbstractArray) in Base at reducedim.jl:648
+c 1* 648  1 ─      nothing
+  2  648  │   %2 = (Base.#sum#550)(Colon(), #self#, a)
+  3  648  └──      return %2
+a = [1, 2, 5], breakpoint(sum(a::AbstractArray) in Base at reducedim.jl:648, line 648))
+```
+
+[`disable`](@ref) and [`enable`](@ref) allow you to turn breakpoints off and on without losing any
+conditional statements you may have provided; [`remove`](@ref) allows a permanent removal of
+the breakpoint. You can use `remove()` to remove all breakpoints in all methods.
+
+[`@breakpoint`](@ref) allows you to optionally specify a line number at which the breakpoint
+is to be set. You can also use a functional form, [`breakpoint`](@ref), to specify file/line
+combinations or that you want to break on entry to *any* method of a particular function.
+At present, note that some of this functionality requires that you be running
+[Revise.jl](https://github.com/timholy/Revise.jl).
+
+Finally, you can set breakpoints using [`@bp`](@ref):
+
+```jldoctest demo1
+julia> function myfunction(x, y)
+           a = 1
+           b = 2
+           x > 3 && @bp
+           return a + b + x + y
+       end
+myfunction (generic function with 1 method)
+
+julia> @interpret myfunction(1, 2)
+6
+
+julia> @interpret myfunction(5, 6)
+(Frame for myfunction(x, y) in Main at none:2
+⋮
+  3  4  │   %3 = (>)(x, 3)
+  4  4  └──      goto #3 if not %3
+b 5* 4  2 ─      nothing
+  6  4  └──      goto #3
+  7  5  3 ┄ %7 = (+)(a, b, x, y)
+⋮
+x = 5
+y = 6
+a = 1
+b = 2, breakpoint(myfunction(x, y) in Main at none:2, line 4))
 ```
 
-Those who want to dive deeper should continue reading.
+Here the breakpoint is marked with a `b` indicating that it is an unconditional breakpoint.
+Because we placed it inside the condition `x > 3`, we've achieved a conditional outcome.
+
+When using `@bp` in source-code files, the use of Revise is recommended,
+since it allows you to add breakpoints, test code, and then remove the breakpoints from the
+code without restarting Julia.
+
+## `debug_command`
+
+You can control execution of frames via [`debug_command`](@ref).
+Authors of debugging applications should target `debug_command` for their interaction
+with JuliaInterpreter.

docs/src/internals.md

@@ -24,7 +24,7 @@ as `code = frame.framecode.src`. (It's a slightly modified form of one returned
 in that it has been processed by [`JuliaInterpreter.optimize!`](@ref) to speed up run-time execution.)
 
 `frame` has another field, `framedata`, that holds values needed for or generated by execution.
-The input arguments are in `locals`:
+The input arguments and local variables are in `locals`:
 
 ```julia
 julia> frame.framedata.locals