[ORC][llvm-jitlink] Add SimpleLazyReexportsSpeculator, use in llvm-ji…

[lhames]

…tlink.

Also adds a new IdleTask type and updates DynamicThreadPoolTaskDispatcher to schedule IdleTasks whenever the total number of threads running is less than the maximum number of MaterializationThreads.

A SimpleLazyReexportsSpeculator instance maintains a list of speculation suggestions ((JITDylib, Function) pairs) and registered lazy reexports. When speculation opportunities are available (having been added via addSpeculationSuggestions or when lazy reexports were created) it schedules an IdleTask that triggers the next speculative lookup as soon as resources are available. Speculation suggestions are processed first, followed by lookups for lazy reexport bodies. A callback can be registered at object construction time to record lazy reexport executions as they happen, and these executions can be fed back into the speculator as suggestions on subsequent executions.

The llvm-jitlink tool is updated to support speculation when lazy linking is used via three new arguments:

-speculate=[none|simple] : When the 'simple' value is specified a
SimpleLazyReexportsSpeculator instances is used
for speculation.

-speculate-order : Specifies a path to a CSV containing
(jit dylib name, function name) triples to use
as speculative suggestions in the current run.

-record-lazy-execs : Specifies a path in which to record lazy function
executions as a CSV of (jit dylib name, function
name) pairs, suitable for use with
-speculate-order.

The same path can be passed to -speculate-order and -record-lazy-execs, in which case the file will be overwritten at the end of the execution.

No testcase yet: Speculative linking is difficult to test (since by definition execution behavior should be unaffected by speculation) and this is an new prototype of the concept*. Tests will be added in the future once the interface and behavior settle down.

• An earlier implementation of the speculation concept can be found in llvm/include/llvm/ExecutionEngine/Orc/Speculation.h. Both systems have the same goal (hiding compilation latency) but different mechanisms. This patch relies entirely on information available in the controller, where the old system could receive additional information from the JIT'd runtime via callbacks. I aim to combine the two in the future, but want to gain more practical experience with speculation first.