Load rules in parallel

[jariji]

Currently each rule is loaded sequentially during compilation, which takes a long time. The rules could be loaded independently in parallel, and then merged into a combined list afterwards: Map then reduce. The key is avoiding global state---pure functions only.

Conceptually:

load_rule(path) = [evaluate(read(path))]
load_rules(paths) = parallel_mapreduce(load_rule, vcat, paths)
all_rules = load_rules(PATHS)

This could probably be done in threads, or by splitting off the files into separate extension packages and letting Pkg.jl take care of parallelizing across them.

[ChrisRackauckas]

It can probably be done at precompilation stage too?

[Bumblebee00]

@jariji this seems a good idea, did you also try it in practice?

[Bumblebee00]

It can probably be done at precompilation stage too?

I think right now is not done at precompilation, is just a function call in a file included in the main file SymbolicIntegration.jl

SymbolicIntegration.jl/src/methods/rule_based/rules_loader.jl

Lines 122 to 123 in 21f239e

	# TODO PrecompileTools.jl?
	load_rules()

but I noticed this: if you

• precompile it once (so you execute using SymbolicIntegration in the repl and it loads all the rules, ~5 minutes)
• then close the repl
• do not change any code
• reopen the repl
• type using SymbolicIntegration

then it will load all the rules instantly, like if they were in precompilation. I remember talking about this with @AayushSabharwal, is quite strange but handy 😅, and also the reasons i didn't use PrecompileTools.jl yet

[jariji]

this seems a good idea, did you also try it in practice?

I just looked at the code a little bit; I didn't try changing it.

At the moment include is being used to load files; I'm not sure if include is legal to do in parallel or not, but I imagine there is some parallel decomposition of these operations that is legal.

[AayushSabharwal]

I'm not aware of existing parallel loading techniques to reduce compile times, but the loading of rules should be cached by precompilation. There's almost no point using PrecompileTools on this, since SymbolicUtils' rule matching code is not in a state where it will be able to cache anything meaningful.