Merge branch 'master' of https://github.com/JuliaTeachingCTU/Scientific-Programming-in-Julia

Author: pevnak

docs/make.jl

@@ -115,6 +115,7 @@ makedocs(;
     pages = [
         "Home" => "index.md",
         "Installation" => "installation.md", 
+        "Projects" => "projects.md",
         "1: Introduction" => lecture_01,
         "2: The power of Type System & multiple dispatch" => lecture_02,
         "3: Design patterns" => lecture_03,

docs/src/index.md

@@ -59,9 +59,7 @@ Hint: The **first few homeworks are easier**. Use them to fill up your points.
 The final project will be individually agreed on for each student. Ideally you
 can use this project to solve a problem you have e.g. in your thesis, but don't
 worry - if you cannot come up with an own project idea, we will suggest one to
-you.
-
-The evaluation criteria of the final project are ...
+you. More info and project suggestion can be found [here](@ref projects).
 
 
 ## Grading

docs/src/lecture_06/hw.md

@@ -15,7 +15,7 @@ x + 2*y*z - c*x
 ```
 return an array of *unique alphabetically sorted symbols* representing variables in an expression.
 ```julia
-[:x, :y, :z, :c]
+[:c, :x, :y, :z]
 ```
 Implement this in a function called `find_variables`. Note that there may be some edge cases that you may have to handle in a special way, such as 
 - variable assignments `r = x*x` should return the variable on the left as well (`r` in this case)

docs/src/lecture_06/lab.md

@@ -59,6 +59,7 @@ function implicit_len()
 end
 nothing #hide
 ```
+For now do not try to understand the details, but focus on the overall differences such as length of the code.
 
 !!! info "Redirecting `stdout`"
     If the output of the method introspection tools is too long you can use a general way of redirecting standard output `stdout` to a file
@@ -179,9 +180,36 @@ function polynomial(a, x)
 end
 ```
 
+!!! info "Splatting/slurping operator `...`"
+    The operator `...` serves two purposes inside function calls [^3][^4]:
+    - combines multiple arguments into one
+    ```@repl lab06_splat
+    function printargs(args...)
+        println(typeof(args))
+        for (i, arg) in enumerate(args)
+            println("Arg #$i = $arg")
+        end
+    end
+    printargs(1, 2, 3)
+    ```
+    - splits one argument into many different arguments
+    ```@repl lab06_splat
+    function threeargs(a, b, c)
+        println("a = $a::$(typeof(a))")
+        println("b = $b::$(typeof(b))")
+        println("c = $c::$(typeof(c))")
+    end
+    threeargs([1,2,3]...) # or with a variable threeargs(x...)
+    ```
+
+    [^3]: [https://docs.julialang.org/en/v1/manual/faq/#What-does-the-...-operator-do?](https://docs.julialang.org/en/v1/manual/faq/#What-does-the-...-operator-do?)
+    [^4]: [https://docs.julialang.org/en/v1/manual/functions/#Varargs-Functions](https://docs.julialang.org/en/v1/manual/functions/#Varargs-Functions)
+
 **HINTS**:
-- define two methods `_polynomial(x, a...)` and `_polynomial(x, a)`
-- recall that these kind of optimization are run just after type inference
+- define two methods `_polynomial!(ac, x, a...)` and `_polynomial!(ac, x, a)` for the case of ≥2 coefficients and the last coefficient
+- use splatting together with range indexing `a[1:end-1]...`
+- the correctness can be checked using the built-in `evalpoly`
+- recall that these kind of optimization are possible just around the type inference stage
 - use container of known length to store the coefficients
 
 ```@raw html
@@ -200,8 +228,8 @@ a = Tuple(ones(Int, 21)) # everything less than 22 gets inlined
 x = 2
 polynomial(a,x) == evalpoly(x,a) # compare with built-in function
 
+# @code_llvm polynomial(a,x)    # seen here too, but code_typed is a better option
 @code_lowered polynomial(a,x) # cannot be seen here as optimizations are not applied
-@code_llvm polynomial(a,x)    # seen here too, but code_typed is a better option
 nothing #hide
 ```
 
@@ -216,7 +244,7 @@ nothing #hide
 ## AST manipulation: The first steps to metaprogramming
 Julia is so called homoiconic language, as it allows the language to reason about its code. This capability is inspired by years of development in other languages such as Lisp, Clojure or Prolog.
 
-There are two easy ways to extract/construct the code structure [^3]
+There are two easy ways to extract/construct the code structure [^5]
 - parsing code stored in string with internal `Meta.parse`
 ```@repl lab06_meta
 code_parse = Meta.parse("x = 2")    # for single line expressions (additional spaces are ignored)
@@ -317,15 +345,14 @@ Given a function `replace_i`, which replaces variables `i` for `k` in an express
 ex = :(i + i*i + y*i - sin(z))
 @test replace_i(ex) == :(k + k*k + y*k - sin(z))
 ```
-write function `sreplace_i(s)` which does the same thing but instead of a parsed expression (AST) it manipulates a string, such as
+write a different function `sreplace_i(s)`, which does the same thing but instead of a parsed expression (AST) it manipulates a string, such as
 ```@repl lab06_meta
 s = string(ex)
 ```
-Think of some corner cases, that the method may not handle properly.
-
 **HINTS**:
 - Use `Meta.parse` in combination with `replace_i` **ONLY** for checking of correctness.
-- You can use the `replace` function.
+- You can use the `replace` function in combination with regular expressions.
+- Think of some corner cases, that the method may not handle properly.
 
 ```@raw html
 </div></div>
@@ -394,9 +421,8 @@ eval(ex)
 
 This kind of manipulation is at the core of some pkgs, such as aforementioned [`IntervalArithmetics.jl`](https://github.com/JuliaIntervals/IntervalArithmetic.jl) where every number is replaced with a narrow interval in order to find some bounds on the result of a computation.
 
-
-[^3]: Once you understand the recursive structure of expressions, the AST can be constructed manually like any other type.
 ---
+[^5]: Once you understand the recursive structure of expressions, the AST can be constructed manually like any other type.
 
 ## Resources
 - Julia's manual on [metaprogramming](https://docs.julialang.org/en/v1/manual/metaprogramming/)

docs/src/lecture_06/lecture.md

@@ -153,7 +153,8 @@ We can see that
 - The expression `a + b` has been also replaced with its implementation in terms of the `add_int` intrinsic and its result type annotated as Int64. 
 - And the return type of the entire function body has been annotated as `Int64`.
 - The phi-instruction `%2 = φ (#1 => 1, #3 => %6)` is a **selector function**, which returns the value depending on from which branch do you come from. In this case, variable `%2` will have value 1, if the control was transfered from block `#1` and it will have value copied from variable `%6` if the control was transferreed from block `3` [see also](https://llvm.org/docs/LangRef.html#phi-instruction). The `φ` stands from *phony* variable.
-	When we have called `@code_lower`, the role of types of the argument was in selecting the approapriate function body, they are needed for multiple dispatch. Contrary in `@code_typed`, the types of parameters determine the choice if inner methods that needs to be called (again the multiple dispatch), which can trigger other optimization, such as inlining, which seen in `One(n)`. 
+
+When we have called `@code_lower`, the role of types of arguments was in selecting - via multiple dispatch - the appropriate function body among different methods. Contrary in `@code_typed`, the types of parameters determine the choice of inner methods that need to be called (again with multiple dispatch). This process can trigger other optimization, such as inlining, as seen in the case of `one(n)` being replaced with `1` directly, though here this replacement is hidden in the `φ` function. 
 
 Note that the same view of the code is offered by the `@code_warntype` macro, which we have seen in the previous [lecture](@ref perf_lecture). The main difference from `@code_typed` is that it highlights type instabilities with red color and shows only unoptimized view of the code. You can view the unoptimized code with a keyword argument `optimize=false`:
 ```julia
@@ -247,7 +248,7 @@ and the output is used mainly for debugging / inspection.
 Language introspection is very convenient for investigating, how things are implemented and how they are optimized / compiled to the native code.
 
 !!! note "Reminder `@which`"
-	Though we have already used it quite a few times, recall the very useful macro `@which`, which identifies the concrete function called in the function call. For example `@which mapreduce(sin, +, [1,2,3,4])`. Note again that the macro here is a convenience macro to obtain types of arguments from the expression. Under the hood, it calls `InteractiveUtils.which(function_name, (Base.typesof)(args...))`. Funnily enough, you can call `@which InteractiveUtils.which(+, (Base.typesof)(1,1))` to inspect, where `which` is defined.
+	Though we have already used it quite a few times, recall the very useful macro `@which`, which identifies the concrete function called in a function call. For example `@which mapreduce(sin, +, [1,2,3,4])`. Note again that the macro here is a convenience macro to obtain types of arguments from the expression. Under the hood, it calls `InteractiveUtils.which(function_name, (Base.typesof)(args...))`. Funnily enough, you can call `@which InteractiveUtils.which(+, (Base.typesof)(1,1))` to inspect, where `which` is defined.
 
 ### Broadcasting
 Broadcasting is not a unique concept in programming languages (Python/Numpy, MATLAB), however its implementation in Julia allows to easily fuse operations. For example 
@@ -349,7 +350,7 @@ The type returned by the quotation depends on what is quoted. Observe the return
 ```julia
 :(1)			|> typeof
 :(:x)		 	|> typeof
-:(1 + x) 	|> typeof
+:(1 + x) 		|> typeof
 quote
     1 + x
     x + 1
@@ -428,9 +429,7 @@ The parsed code `p` is of type `Expr`, which according to Julia's help[^2] is *a
 
 [^3]: An example provided by Stefan Karpinski [https://stackoverflow.com/questions/23480722/what-is-a-symbol-in-julia](https://stackoverflow.com/questions/23480722/what-is-a-symbol-in-julia)
 
-!!! info 
-	### Expressions
-
+!!! info "`Expr`essions"
 	From Julia's help[^2]:
 
 	`Expr(head::Symbol, args...)`

docs/src/projects.md

@@ -0,0 +1,69 @@
+# [Projects](@id projects)
+
+We want you to use Julia for something that is acutally useful for you.
+Therefore you can choose your final, graded project very freely.
+We will discuss your ideas with you individually and come up with a sufficiently
+extensive project together.
+
+For you inspiration of what such a project could look like we have four
+suggestions for you (which you can of course choose to work on as well).
+
+## The Equation Learner And Its Symbolic Representation
+
+In many scientific and engineering one searches for interpretable (i.e.
+human-understandable) models instead of the black-box function approximators
+that neural networks provide.
+The [*equation learner*](http://proceedings.mlr.press/v80/sahoo18a.html) (EQL)
+is one approach that can identify concise equations that describe a given
+dataset.
+
+The EQL is essentially a neural network with different unary or binary
+activation functions at each indiviual unit. The network weights are
+regularized during training to obtain a sparse model which hopefully results in
+a model that represents a simple equation.
+
+The goal of this project is to implement the EQL, and if there is enough time
+the [*improved equation learner*](https://arxiv.org/abs/2105.06331) (iEQL).
+The equation learners should be tested on a few toy problems (possibly inspired
+by the tasks in the papers).  Finally, you will implement functionality that
+can transform the learned model into a symbolic, human readable, and exectuable
+Julia expression.
+
+## An Evolutionary Algorithm Applied To Julia's AST
+
+Most of the approaches to equation learning have to be differentiable by default
+in order to use the traditional machinery of stochastic gradient descent with
+backpropagation. This often leads to equations with too many terms, requiring 
+special techniques for enforcing sparsity for terms with low weights.
+
+In Julia we can however use a different learning paradigm of evolutionary 
+algorithms, which can work on discrete set of expressions. The goal is to 
+write mutation and recombination - the basic operators of a genetic algorithm,
+but applied on top of Julia AST.
+
+Data: AI Feyman [database](https://space.mit.edu/home/tegmark/aifeynman.html) on symbolic regression (from [article](https://arxiv.org/pdf/1905.11481.pdf)/[code](https://github.com/SJ001/AI-Feynman))
+Inspiration: 
+- Logic Guided Genetic Algorithms [article](https://arxiv.org/pdf/2010.11328.pdf)/[code](https://github.com/DhananjayAshok/LGGA)
+- AI Feynman 2.0: Pareto-optimal symbolic regression exploiting graph modularity [article](https://arxiv.org/pdf/2006.10782)
+- Genetic Programming for Julia: fast performance and parallel island model implementation [report](http://courses.csail.mit.edu/18.337/2015/projects/MorganFrank/projectReport.pdf)
+
+## Distributed Optimization Package
+
+One click distributed optimization is at the heart of other machine learning 
+and optimization libraries such as pytorch, however some equivalents are 
+missing in the Julia's Flux ecosystem. The goal of this project is to explore,
+implement and compare at least two state-of-the-art methods of distributed 
+gradient descent on data that will be provided for you.
+
+Some of the work has already been done in this area by one of our former students, 
+see [link](https://dspace.cvut.cz/handle/10467/97057).
+
+## A Rule Learning Algorithm
+
+[Rule-based models](https://christophm.github.io/interpretable-ml-book/rules.html)
+are simple and very interpretable models that have been around for a long time
+and are gaining popularity again.
+The goal of this project is to implement a
+[sequential covering](https://christophm.github.io/interpretable-ml-book/rules.html#sequential-covering)
+algorithm called [`RIPPER`](http://www.cs.utsa.edu/~bylander/cs6243/cohen95ripper.pdf)
+and evaluate it on a number of datasets.