Checkpoint, many content changes/formatting

Author: erikhatcher

docs/10_About_workshop/1_intro.mdx

@@ -19,10 +19,18 @@ To begin, navigate to the [Atlas Search Playground](https://search-playground.mo
 In the next section, you'll work through the first exercise to get familiar with the
 Playground's Code Sandbox.
 
-Let's dive into the world of Atlas Search using this convenient and powerful playground!
+### Run button
+
+After you make changes to any of the Playground areas, press the `Run` button to execute
+the aggregation pipeline.
+
+![Run button](/img/playground_run.png)
 
 ## Resources
-* https://www.mongodb.com/developer/products/atlas/search-playground-intro/
+  * https://www.mongodb.com/developer/products/atlas/search-playground-intro/
+
+------
+
+Let's dive into the world of Atlas Search using this convenient and powerful playground environment!
 
 
-* `index` parameter to $search!!!

docs/10_About_workshop/2_lets_go.mdx

@@ -12,9 +12,7 @@ we'll get there next. See if you can solve it with what you already know or expe
 
 The objective with this exercise is to adjust the `FIX_ME` in the `$search` aggregation pipeline stage so that the document (in the Data Source pane) matches the query and appears in the Results pane.
 
-![Playground intro exercise](/img/playground_intro_exercise.png)
-
-![Run button](/img/playground_run.png)
+[![Playground intro exercise](/img/playground_intro_exercise.png)](/img/playground_intro_exercise.png)
 
 <details>
 <summary>Here's a solution...</summary>

docs/20_Intro_to_Atlas_Search/1_system.mdx

@@ -4,9 +4,26 @@ Atlas Search provides powerful findability capabilities to your data collections
 A flexible index configuration allows mapping and indexing only the fields needed,
 or dynamically mapping any and all fields supported. 
 
+## The Big Picture
+
+Applications communicate to Atlas Search through the same mechanism as
+all other requests: an aggregation pipeline through a MongoDB driver.
+
 ![big picture](/img/big_picture.png)
 
-![system diagram](/img/system_diagram.png)
+## System architecture
+
+The Atlas Search `mongot` process, built on Apache Lucene, interfaces with the `mongod`
+database process to create and manage full-text (and vector search) indexes and queries.
+
+The `mongot` process performs the following tasks:
+  * Creates Atlas Search indexes based on the index definition.
+  * Monitors change streams for the current state of the documents.
+  * Processes Atlas Search queries and returns the document IDs and other search metadata for
+    the matching documents to `mongod`, which then does a full document lookup and returns the
+    results to the client.
+
+[![system diagram](/img/system_diagram.png)](/img/system_diagram.png)
 
 Changes to a collection via updates, deletes, or additions are *eventually consistent*, meaning the
 index is updated independently of changes to the collection in a separate process, asynchronously.

docs/20_Intro_to_Atlas_Search/2_aggregation_stages.mdx

@@ -1,5 +1,9 @@
 # 👐 Aggregation pipeline search stages
 
+The search stages must be the first stage in a pipeline, as they do not accept incoming
+documents but rather only emit documents. There are two stages available, depending on 
+the particular needs.
+
 ## $search
 
 Returns matching documents.
@@ -10,59 +14,57 @@ Results are returned in descending **score** order or in an optional `sort` orde
 ## $searchMeta
 
 Returns a single document of search result metadata including count of matching documents
-and facets requested. No actual collection documents are returned.
+and any facets requested. No actual collection documents are returned.
 
 The `$searchMeta` stage performs the same search that `$search` does,
 but only returns the results metadata, not actual matching documents.
 Results metadata includes the count of matching results and facets.
 This same metadata is available when using `$search` too, 
 accessible in the $$SEARCH_META context variable.
 
-
-
 ## Exercises: search pipeline stages
 
 ### Step 1
-1. Navigate to the original Playground used in the last section's exercise
-   https://search-playground.mongodb.com/tools/code-sandbox/snapshots/6782aea0667feaaf06324b87
-2. Press Run. Got the empty `[]` array of results?
-3. Change `$search` to `$searchMeta` (in the Query pane), and press Run again.
-
-<details>
-<summary>Here's the expected results...</summary>
-<div>
-```js
-[
-  {
-    "count": {
-      "lowerBound": 0
+  1. Navigate to the original Playground used in the last section's exercise
+     https://search-playground.mongodb.com/tools/code-sandbox/snapshots/6782aea0667feaaf06324b87
+  2. Press Run. Got the empty `[]` array of results?
+  3. Change `$search` to `$searchMeta` (in the Query pane), and press Run again.
+
+  <details>
+  <summary>Here's the expected results...</summary>
+  <div>
+  ```js
+  [
+    {
+      "count": {
+        "lowerBound": 0
+      }
     }
-  }
-]
-```
-</div>
-</details>
+  ]
+  ```
+  </div>
+  </details>
 
 ### Step 2
 
-1. Now fix the query to match the document as you did previously
-2. Press Run again
-3. Did the `$searchMeta` results change?
-
-<details>
-<summary>Here's the expected results...</summary>
-<div>
-```js
-[
-  {
-    "count": {
-      "lowerBound": 1
+  1. Now fix the query to match the document as you did previously
+  2. Press Run again
+  3. Did the `$searchMeta` results change?
+
+  <details>
+  <summary>Here's the expected results...</summary>
+  <div>
+  ```js
+  [
+    {
+      "count": {
+        "lowerBound": 1
+      }
     }
-  }
-]
-```
-</div>
-</details>
+  ]
+  ```
+  </div>
+  </details>
 
 ## Post $search-stages
 

docs/20_Intro_to_Atlas_Search/3_lucene.mdx

@@ -1,27 +1,63 @@
 # 📘 Powered by Lucene
 
-https://lucene.apache.org
+[Lucene](https://lucene.apache.org), is a Java library providing powerful indexing and search
+features, as well as spellchecking, hit highlighting, and advanced analysis/tokenization
+capabilities.  Without a doubt, you've already used Lucene perhaps without even realizing it.
+It powers the search facilities of countless websites and applications, both public and private.
 
 ## Anatomy of a Lucene index
 
-A Lucene index encapsulates specialized data structures unique to each type of data indexed.
+A Lucene index encapsulates specialized data structures unique to each type of data indexed. 
 
-  * Numbers and dates: ...
-  * Geo-spatial: ...
+  * Numbers, dates, geo-spatial points: Indexed into a k-d structure
+  * Vectors: Hierarchical Navigable Small Worlds (HNSW) data structure
   * Text: via inverted indexes
 
-Each field is indexed independently.
+What Lucene, and Atlas Search, call an "index" is really a collection of separate individual
+per-field data structures.
 
-Segmented architecture, append-only, for fast indexing. Background processes to optimize the index
-segments.
+Lucene is designed for both fast searches and speedy indexing. The indexing speed derives from its
+append-only segmented architecture. When new docuemnts are indexed, they are added to a new segments.
+When that indexing session is complete, the new segments are opened and blended with all the other 
+active segments. Background processes optimize the index segments by combining them to form larger,
+and less, segments over time.
+
+A single Lucene index can handle up to 2 billion documents. There is generally a 1-1 correspondence
+between documents in your collection to Lucene documents, with the exception of nested documents
+mapped as `embeddedDocuments` (a topic covered later). To differentiate the terminology, Atlas Search 
+calls the documents in Lucene index "index objects". See
+[index size and configuration doc](https://www.mongodb.com/docs/atlas/atlas-search/performance/index-performance/#index-size-and-configuration)
+for more details.
 
 ## Inverted Index
 
+Textual content is the heart and soul of Lucene. `string` fields are analzyed. The output of the
+analysis process is a series of **terms**. Terms are generally a normalized version of the individual
+words of the text. These terms are then organized into an **inverted index** data structure.
+This data structure is lexicographically (or alphabetically) ordered. The following image illustrates
+an inverted index built from 3 documents, each with a single string field.
+
 ![inverted index](/img/analysis_lucene_standard.png)
 
+Along with an ordered dictionary of terms, corpus and document-level statistics are also collected into
+the inverted index structure. These statistics include:
+
+  * term frequency (`tf`): the number of times a term occurs in the field
+  * document frequency (`df`): how many documents contain the term
+  * field length: how many terms are there in each field
+
 ## Search algorithms
 
-  * "index intersection" using skip lists
-  * link to Adrien's presentation
+Lucene queries leverage the data structures built at index-time to quickly find, and rank, matching
+documents. The synergy "index intersection" shines when searching across multiple fields in a single
+query. 
+
+Atlas Search translates its search operators directly to Lucene's `Query` API.
+
+## Resources
 
-Atlas Search translates its search operators to Lucene's `Query` API.
+  * ["What is in a Lucene index"](https://www.youtube.com/watch?v=T5RmMNDR5XI) - a very
+    educational presentation delivered by Lucene project committer Adrien Grand.
+  * ["Visualizing Lucene's segment merges"](https://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-merges.html)
+    - an illustrative set of animations on the how Lucene keeps itself optimized, balancing both
+    indexing and searching needs.

docs/30_Index_configuration/1_index_config.mdx

@@ -1,24 +1,88 @@
 # 📘 Index configuration
 
-Documents are mapped to an index through a flexible configuration. 
-
-* https://www.mongodb.com/docs/atlas/atlas-search/define-field-mappings/
+Documents are indexed in Lucene using a flexible configuration specification.
 
 ## Supported types
 
-## Dynamic mapping
+The type of a field determines how it can be indexed, and thus how it can be searched.
+
+Most field types are supported, including all the basic types such as booleans, dates, numerics,
+and strings, as well as ObjectID, UUID, GeoJSON. Null values are also supported implicitly.
+
+## Mapping
+
+A JSON-formatted index configuration mapping specifies which fields are indexed.
+
+Fields can be mapped explicitly, by name (and path), or dynamically, or a combination of
+both. 
+
+### Dynamic mapping 
+
+An index configuration defaults a fully dynamic mapping:
+
+```
+{
+  "mappings": {
+    "dynamic": true
+  }
+}
+```
+
+A dynamic mapping indexes all *dynamically supported* fields automatically. Dynamic mapping alleviates
+having to specify every field explicitly, which would be arduous in situations where there are many
+fields, or where new fields could be added to documents over time.
 
-You can configure an entire index to use dynamic mappings, or specify individual fields,
-such as fields of type `document`, to be dynamically mapped.
+You can configure an entire index to use dynamic mappings, or have fields only within nested documents,
+be dynamically mapped.
 
-## Configuring a real Atlas Search index
+Not all data types are supported with dynamic mapping. Most notably, GeoJSON fields require explicit
+mapping.
+
+If `mappings.dynamic` is set to `false`, at least one field must be explicitly mapped.
+
+## Static/explicit field mapping
+
+To explicitly specify a fields mapping, it is listed in a `mappings.fields` section of the configuration.
+
+An example:
+
+```
+{
+  "mappings": {
+    "dynamic": false,
+    "fields": {
+      "in_stock": [
+        {
+          "type": "boolean"
+        }
+      ]
+    }
+  }
+}
+```
+Using that mapping, this document would only have the `in_stock` field indexed, not the `name` field.
+
+```
+{
+  _id: 1,
+  name: "Product One",
+  in_stock: true
+}
+```
+
+## Configuring an Atlas Search index
+
+Outside of the Playground, you have several options to set up and configure a persistent
+Atlas Search index.
 
   * Atlas Search Visual Editor or JSON Editor
   * via Compass
   * Atlas CLI
   * Driver commands
 
-## Configuration options
+The Atlas Search Visual Editor is a good place to start, to become familiar with the syntax
+and options available.
+
+## Resources
 
-  * `storedSource`
-  * `synonyms`
+  * https://www.mongodb.com/docs/atlas/atlas-search/define-field-mappings/

docs/30_Index_configuration/2_basic_types.mdx

@@ -28,3 +28,31 @@ mapped type.
 ## ObjectId
   * `equals`: https://search-playground.mongodb.com/tools/code-sandbox/snapshots/678506c2b6487c1cfd0bb540
   * `in`: 
+
+## Exercises
+
+### boolean equals
+
+Why doesn't this playground match as expected?
+  * https://search-playground.mongodb.com/tools/code-sandbox/snapshots/6787a56ed8892198a2a8f3f4
+
+  <details>
+  <summary>Explanation</summary>
+  <div>
+  Because $search.equals.value is a string of "true", not an actual boolean. 
+  Here's a corrected pipeline:
+  ```js
+  [
+    {
+      $search: {
+        index: "default",
+        equals: {
+          value: true,
+          path: "in_stock"
+        }
+      }
+    }
+  ]
+  ```
+  </div>
+  </details>