docs: Simplify and streamline quickstart

The docs were far too wordy for what's supposed to be an introduction to
Dagger. This commit moves most of that information to separate deep-dive
pages, and sets up the quickstart page to have simple examples of how to
do common tasks. The hope is that new users will start by adapting the
simple example, and read more into the deep dive when they want to do
something fancy or learn how the example works.

Author: jpsamaroo

docs/Project.toml

@@ -1,2 +1,5 @@
 [deps]
+Dagger = "d58978e5-989f-55fb-8d15-ea34adc7bf54"
+DaggerWebDash = "cfc5aa84-1a2a-41ab-b391-ede92ecae40c"
 Documenter = "e30172f5-a6a5-5a46-863b-614d45cd2de4"
+TimespanLogging = "a526e669-04d3-4846-9525-c66122c55f63"

docs/make.jl

@@ -1,8 +1,8 @@
-using Dagger
+using Dagger, TimespanLogging, DaggerWebDash
 using Documenter
 
 makedocs(;
-    modules = [Dagger],
+    modules = [Dagger, TimespanLogging, DaggerWebDash],
     authors = "JuliaParallel and contributors",
     repo = "https://github.com/JuliaParallel/Dagger.jl/blob/{commit}{path}#L{line}",
     sitename = "Dagger.jl",
@@ -13,20 +13,29 @@ makedocs(;
     ),
     pages = [
         "Home" => "index.md",
-        "Processors" => "processors.md",
+        "Task Spawning" => "task-spawning.md",
+        "Data Management" => "data-management.md",
         "Scopes" => "scopes.md",
-        "Mutation and Shards" => "mutation.md",
+        "Processors" => "processors.md",
         "Dynamic Scheduler Control" => "dynamic.md",
         "Option Propagation" => "propagation.md",
         "Logging and Graphing" => "logging.md",
+        "Checkpointing" => "checkpointing.md",
         "Scheduler Visualization" => "scheduler-visualization.md",
         "Benchmarking" => "benchmarking.md",
-        "Checkpointing" => "checkpointing.md",
-        "API" => [
-            "Types" => "api/types.md",
-            "Functions and Macros" => "api/functions.md",
-        ],
         "Scheduler Internals" => "scheduler-internals.md",
+        "Dagger API" => [
+            "Types" => "api-dagger/types.md",
+            "Functions and Macros" => "api-dagger/functions.md",
+        ],
+        "TimespanLogging API" => [
+            "Types" => "api-timespanlogging/types.md",
+            "Functions and Macros" => "api-timespanlogging/functions.md",
+        ],
+        "DaggerWebDash API" => [
+            "Types" => "api-daggerwebdash/types.md",
+            "Functions and Macros" => "api-daggerwebdash/functions.md",
+        ],
     ]
 )
 

docs/src/api/functions.md renamed to docs/src/api-dagger/functions.md

@@ -2,44 +2,49 @@
 CurrentModule = Dagger
 ```
 
-# Functions
+# Dagger Functions
 ```@index
 Pages = ["functions.md"]
 ```
 
-## General Functions
+## Task Functions/Macros
 ```@docs
-delayed
+@spawn
 spawn
+delayed
+@par
+```
+
+## Task Options Functions/Macros
+```@docs
+with_options
+get_options
+@option
+default_option
+```
+
+## Data Management Functions
+```@docs
 tochunk
-domain
-compute
-dependents
-noffspring
-order
-treereduce
+@mutable
+@shard
+shard
 ```
 
-## Table Functions
+## Scope Functions
 ```@docs
-tabletype
-tabletype!
-trim
-trim!
-map
-filter
-reduce
-groupby
-leftjoin
-innerjoin
-keys
-getindex
+scope
+constrain
 ```
 
-## Array Functions
+## Lazy Task Functions
 ```@docs
-alignfirst
-view
+domain
+compute
+dependents
+noffspring
+order
+treereduce
 ```
 
 ## Processor Functions
@@ -50,26 +55,16 @@ default_enabled
 get_processors
 get_parent
 move
-capacity
 get_tls
 set_tls!
 ```
 
-## Shard Functions
-[`Dagger.@shard`](@ref)
-[`Dagger.shard`](@ref)
-
 ## Context Functions
 ```@docs
 addprocs!
 rmprocs!
 ```
 
-## Logging Functions
-```@docs
-get_logs!
-```
-
 ## Thunk Execution Environment Functions
 
 These functions are used within the function called by a `Thunk`.
@@ -92,19 +87,3 @@ Sch.exec!
 Sch.halt!
 Sch.get_dag_ids
 ```
-
-## File IO Functions
-
-!!! warning
-    These APIs are currently untested and may be removed or modified.
-
-```@docs
-save
-load
-```
-
-# Macros API
-```@docs
-@par
-@spawn
-```

docs/src/api/types.md renamed to docs/src/api-dagger/types.md

@@ -2,24 +2,28 @@
 CurrentModule = Dagger
 ```
 
-# Types
+# Dagger Types
 ```@index
 Pages = ["types.md"]
 ```
 
-## General Types
+## Task Types
 ```@docs
 Thunk
 EagerThunk
-Chunk
-UnitDomain
 ```
 
-## Array Types
-```@docs
-DArray
-Blocks
-ArrayDomain
+## Task Options Types
+```
+Options
+Sch.ThunkOptions
+Sch.SchedulerOptions
+```
+
+## Data Management Types
+```
+Chunk
+Shard
 ```
 
 ## Processor Types
@@ -34,6 +38,8 @@ ThreadProc
 AnyScope
 NodeScope
 ProcessScope
+ProcessorTypeScope
+TaintScope
 UnionScope
 ExactScope
 ```
@@ -43,25 +49,17 @@ ExactScope
 Context
 ```
 
-## Logging Types
-```@docs
-NoOpLog
-LocalEventLog
-```
-
-## Scheduling Types
+## Array Types
 ```@docs
-Sch.SchedulerOptions
-Sch.ThunkOptions
-Sch.MaxUtilization
-Sch.DynamicThunkException
+DArray
+Blocks
+ArrayDomain
+UnitDomain
 ```
 
-## File IO Types
-
-!!! warning
-    These APIs are currently untested and may be removed or modified.
-
+## Logging Event Types
 ```@docs
-FileReader
+Events.BytesAllocd
+Events.ProcessorSaturation
+Events.WorkerSaturation
 ```

docs/src/api-daggerwebdash/functions.md

@@ -0,0 +1,9 @@
+```@meta
+CurrentModule = DaggerWebDash
+```
+
+# DaggerWebDash Functions
+```@index
+Pages = ["functions.md"]
+```
+

docs/src/api-daggerwebdash/types.md

@@ -0,0 +1,15 @@
+```@meta
+CurrentModule = DaggerWebDash
+```
+
+# DaggerWebDash Types
+```@index
+Pages = ["types.md"]
+```
+
+## Logging Event Types
+```@docs
+D3Renderer
+TableStorage
+ProfileMetrics
+```

docs/src/api-timespanlogging/functions.md

@@ -0,0 +1,21 @@
+```@meta
+CurrentModule = TimespanLogging
+```
+
+# TimespanLogging Functions
+```@index
+Pages = ["functions.md"]
+```
+
+## Basic Functions
+```@docs
+timespan_start
+timespan_finish
+get_logs!
+make_timespan
+```
+
+## Logging Metric Functions
+```@docs
+init_similar
+```

docs/src/api-timespanlogging/types.md

@@ -0,0 +1,34 @@
+```@meta
+CurrentModule = TimespanLogging
+```
+
+# TimespanLogging Types
+```@index
+Pages = ["types.md"]
+```
+
+## Log Sink Types
+```@docs
+MultiEventLog
+LocalEventLog
+NoOpLog
+```
+
+## Event Types
+```@docs
+Event
+```
+
+## Built-in Event Types
+```@docs
+Events.CoreMetrics
+Events.IDMetrics
+Events.TimelineMetrics
+Events.FullMetrics
+Events.CPULoadAverages
+Events.MemoryFree
+Events.EventSaturation
+Events.DebugMetrics
+Events.LogWindow
+```
+```

docs/src/mutation.md renamed to docs/src/data-management.md

@@ -1,4 +1,32 @@
-# Mutation Support
+# Data Management
+
+Dagger is not just a computing platform - it also has awareness of where each
+piece of data resides, and will move data between workers and perform
+conversions as necessary to satisfy the needs of your tasks.
+
+## Chunks
+
+Dagger often needs to move data between workers to allow a task to execute. To
+make this efficient when communicating potentially large units of data, Dagger
+uses a remote reference, called a `Chunk`, to refer to objects which may
+exist on another worker. `Chunk`s are backed by a distributed refcounting
+mechanism provided by MemPool.jl, which ensures that the referenced data is not
+garbage collected until all `Chunk`s referencing that object are GC'd from all
+workers.
+
+Conveniently, if you pass in a `Chunk` object as an input to a Dagger task,
+then the task's payload function will get executed with the value contained in
+the `Chunk`. The scheduler also understands `Chunk`s, and will try to schedule
+tasks close to where their `Chunk` inputs reside, to reduce communication
+overhead.
+
+`Chunk`s also have a cached type, a "processor", and a "scope", which are
+important for identifying the type of the object, where in memory (CPU RAM, GPU
+VRAM, etc.) the value resides, and where the value is allowed to be transferred
+and dereferenced. See [Processors](@ref) and [Scopes](@ref) for more details on
+how these properties can be used to control scheduling behavior around `Chunk`s.
+
+## Mutation
 
 Normally, Dagger tasks should be functional and "pure": never mutating their
 inputs, always producing identical outputs for a given set of inputs, and never
@@ -74,4 +102,4 @@ per shard "piece", and each result is considered immutable. `map` is an easy
 way to make a copy of each piece of the shard, to be later reduced, scanned,
 etc.
 
-Further details about what arguments can be passed to `@shard`/`shard` can be found in [Shard Functions](@ref).
+Further details about what arguments can be passed to `@shard`/`shard` can be found in [Data Management Functions](@ref).