Add storage uri and metastore uri (#1066)

fmassot · web-flow · commit d6bbdf359455 · 2022-01-11T12:03:59.000+09:00
* Add storage uri and metastore uri

* Fix broken links
diff --git a/docs/reference/index-config.md b/docs/reference/index-config.md
@@ -1,6 +1,6 @@
 ---
 title: Index configuration
-position: 3
+position: 4
 ---
 
 This page describes how to configure an index.
diff --git a/docs/reference/metastore-config.md b/docs/reference/metastore-config.md
@@ -0,0 +1,94 @@
+---
+title: Metastore configuration
+position: 3
+---
+
+Quickwit needs a place to store meta-information about its indexes.
+
+For instance:
+
+- The index configuration.
+- Meta-information about its splits. For instance, their IDs, the number of documents they contain, their sizes, their min/max timestamp, and the set of tags present in the split.
+- The different sources checkpoints.
+- Some extra information such as the index creation time.
+
+The metastore is entirely defined by a single URI. One can set it by editing the `metastore_uri` parameter of the [Quickwit configuration file](https://www.notion.so/Quickwit-configuration-MERGED-3fab5a181a9a43cba83db2fb25b46729) (often named `quickwit.yaml`).
+
+Currently, Quickwit offers two implementations:
+
+- **PostgreSQL**: recommended for distributed usage.
+- **File-backed implementation**.
+
+# PostgreSQL Metastore
+
+We recommend the PostgreSQL metastore for any distributed usage.
+
+The PostgreSQL metastore can be configured by setting a PostgreSQL URI in the `metastore_uri` parameter of the Quickwit configuration file. The URI takes the following format:
+
+```
+postgres://[user]:[password]@[host]:[port]/[dbname]
+```
+
+Some of those parameters can be omitted. The following PostgreSQL URIs are for instance valid:
+
+```
+postgres://localhost/mydb
+postgres://user@localhost
+postgres://user:secret@localhost
+postgres://host1:123,host2:456/mydb
+```
+
+The database has to be created in advance.
+
+On its first execution, Quickwit will transparently create the necessary tables.
+
+Likewise, if you upgrade Quickwit to a version that includes some changes in the PostgreSQL schema, Quickwit will transparently operate the migration startup.
+
+# File-backed metastore
+
+For convenience, Quickwit also makes it possible to store its metadata in files using a file-backed metastore. In that case, Quickwit will write one file per index.
+
+The metastore is then configured by passing a [Storage URI](https://www.notion.so/Storage-URI-APPROVED-176d8befb8d144fb820bcd0df077a728) that will serve as the root of the metastore storage.
+
+The metadata file associated with a given index will then be stored under 
+
+  `[storage_uri]/[index_id]/metastore.json`  
+
+For the moment, Quickwit supports two types of storage types:
+
+- a local file system URI (e.g., `file:///opt/toto`). It is also valid to pass a file path directly (without file://). `/var/quickwit`. Relative paths will be resolved with respect to the current working directory.
+- S3-compatible storage URI (e.g. `s3://my-bucket/some-path`] ). See the [Storage URI](https://www.notion.so/Storage-URI-APPROVED-176d8befb8d144fb820bcd0df077a728) documentation to configure S3 or S3-compatible storage.
+
+### Polling configuration
+
+By default, the File-Backed Metastore is only read once when you start a Quickwit process (searcher, indexer,...).
+
+You can also configure it to poll the File-Backed Metastore periodically to keep a fresh view of it. This is useful for a Searcher instance that needs to be aware of new splits published by an Indexer running in parallel.
+
+To configure the polling interval (in seconds only), add a URI fragment to the storage URI like this: `s3://quickwit/my-indexes#polling_interval=30s`
+
+<aside>
+👌 Amazon S3 charges $0.0004 per 1000 GET requests. Polling a metastore every 30 seconds will induce a cost of $0.04 per month and per index.
+
+</aside>
+
+### Examples
+
+The following file-backed metastore URIs for instance are valid:
+
+```markdown
+s3://my-indexes
+s3://quickwit/my-indexes
+s3://quickwit/my-indexes#polling_interval=30s
+file:///local/indices
+file:///local/indices#polling_interval=30s
+/local/indices
+./quickwit-metastores
+```
+
+<aside>
+⛔ The file-backed metastore does not allow concurrent writes. For this reason, it should not be used in distributed settings. 
+Running several indexer services on the same file-backed metastore can lead to the corruption of the metastore.
+Running several search services, on the other hand, is perfectly safe.
+
+</aside>
diff --git a/docs/reference/ports.md b/docs/reference/ports.md
@@ -1,6 +1,6 @@
 ---
 title: Quickwit's hosts and ports
-position: 7
+position: 8
 ---
 
 When starting a quickwit search server, one important parameter that can be configured is
diff --git a/docs/reference/query-language.md b/docs/reference/query-language.md
@@ -1,6 +1,6 @@
 ---
 title: Query language
-position: 6
+position: 7
 ---
 
 Quickwit uses a query mini-language which is used by providing a `query` parameter to the search endpoints.
diff --git a/docs/reference/quickwit-config.md b/docs/reference/quickwit-config.md
@@ -27,7 +27,7 @@ A commented example is accessible here: [quickwit.yaml]([link](https://github.co
 
 ## Indexer configuration
 
-This section contains the configuration options for an indexer. The split store is documented in the  [indexing document](indexing.md#split-store).
+This section contains the configuration options for an indexer. The split store is documented in the  [indexing document](../design/indexing.md#split-store).
 
 | Property | Description | Default value |
 | --- | --- | --- |
diff --git a/docs/reference/rest-api.md b/docs/reference/rest-api.md
@@ -1,6 +1,6 @@
 ---
 title: Search REST API
-position: 5
+position: 9
 ---
 
 ## API version
diff --git a/docs/reference/source-config.md b/docs/reference/source-config.md
@@ -1,6 +1,6 @@
 ---
 title: Source configuration
-position: 4
+position: 5
 ---
 
 Quickwit can insert data into an index from one or multiple sources. When creating an index, sources are declared in the [index config](index-config.md). Additional sources can be added later using the [CLI command](cli.md#source) `quickwit source add`.
@@ -56,7 +56,7 @@ sources:
 quickwit source add --index my-index-id --source my-source-id --type file --params '{"filepath": "path/to/file.json"}'
 ```
 
-Finally, note that the [CLI command](clid.md#index) `quickwit index ingest` allows ingesting data directly from a file or the standard input without creating a source beforehand.
+Finally, note that the [CLI command](cli.md#index) `quickwit index ingest` allows ingesting data directly from a file or the standard input without creating a source beforehand.
 
 ## Kafka source
 
diff --git a/docs/reference/storage-uri.md b/docs/reference/storage-uri.md
@@ -0,0 +1,76 @@
+---
+title: Storage URI
+position: 6
+---
+
+In Quickwit, Storage URIs refer to different kinds of storage.
+
+Generally speaking, you can use a storage URI or a regular file path wherever you would have expected a file path.
+
+ 
+For instance
+
+- when configuring the index storage. (Passed as the `index_uri` in the index command line.)
+- when configuring a file-backed metastore. (`metastore_uri` in the QuickwitConfig).
+- when passing a config file in the command line. (you can store your `quickwit.yaml` on Amazon S3 if you want)
+
+Right now, only two types of storage are supported.
+
+## Local file system
+
+One can refer to the file system storage by using a file path directly, or a URI with the `file://` protocol. Relative file paths are allowed and are resolved relatively to the current working directory (CWD). `~` can be used as a shortcut to refer to the current user directory.
+
+The following are valid local file system URIs
+
+```markdown
+- /var/quickwit
+- file:///var/quickwit
+- /home/quickwit/data
+- ~/data 
+- ./quickwit
+```
+
+<aside>
+⚠️ When using the `file://` protocol, a third `/` is necessary to express an absolute path.
+
+For instance, the following URI `file://home/quickwit/` is interpreted as `./home/quickwit`
+
+</aside>
+
+## Amazon S3
+
+It is also possible to refer to Amazon S3 using a S3 URI. S3 URIs must have to follow the following format:
+
+```markdown
+s3://<bucket name>/<key>
+```
+
+For instance
+
+```markdown
+s3://quickwit-prod/quickwit-indexes
+```
+
+The credentials, as well as the region or the custom endpoint, have to be configured separately, using the methods described below.
+
+### S3 credentials
+
+Quickwit will detect the S3 credentials using the first successful method in this list (order matters)
+
+- check for environment variables (`AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`)
+- check for the configuration in the  `~/.aws/credentials` filepath.
+- check for the [Amazon ECS environment](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html)
+- check the [EC2 instance metadata API](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html)
+
+### Region
+
+The region will be detected using the first successful method in this list (order matters)
+
+- `AWS_DEFAULT_REGION` environment variable
+- `AWS_REGION` environment variable
+- Amazon’s instance metadata API [https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html)
+
+<aside>
+⚠️ Custom endpoints are not supported yet.
+
+</aside>
diff --git a/docs/reference/telemetry.md b/docs/reference/telemetry.md
@@ -1,6 +1,6 @@
 ---
 title: Telemetry
-position: 8
+position: 10
 ---
 
 Quickwit Inc. collects anonymous data regarding general usage to help us drive our development. Privacy and transparency are at the heart of Quickwit values and we only collect the minimal useful data and don't use any third party tool for the collection.
