You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/reference/index-config.md
-18Lines changed: 0 additions & 18 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -85,13 +85,11 @@ Today, only the s3 storage is available when running several searcher nodes.
85
85
86
86
The doc mapping defines how a document and the fields it contains are stored and indexed for a given index. A document is a collection of named fields, each having its own data type (text, binary, date, i64, u64, f64).
87
87
88
-
```markdown
89
88
| Variable | Description | Default value |
90
89
| ------------- | ------------- | ------------- |
91
90
| `field_mappings` | Collection of field mapping, each having its own data type (text, binary, date, i64, u64, f64). | [] |
92
91
| `tag_fields` | Collection of fields already defined in `field_mappings` whose values will be stored in a dedicated `tags` (1) | [] |
93
92
| `store_source` | Whether or not the original JSON document is stored or not in the index. | false |
94
-
```
95
93
96
94
(1) [Learn more on the tags usage](../design/querying.md).
97
95
@@ -119,33 +117,27 @@ record: position
119
117
120
118
**Parameters for text field**
121
119
122
-
```markdown
123
120
| Variable | Description | Default value |
124
121
| ------------- | ------------- | ------------- |
125
122
| `stored` | Whether value is stored in the document store | `true` |
126
123
| `tokenizer` | Name of the `Tokenizer`, choices between `raw`, `default` and `stem_en` | `default` |
127
124
| `record` | Describes the amount of information indexed, choices between `basic`, `freq` and `position` | `basic` |
128
-
```
129
125
130
126
**Description of available tokenizers**
131
127
132
-
```markdown
133
128
| Tokenizer | Description |
134
129
| ------------- | ------------- |
135
130
| `raw` | Does not process nor tokenize text |
136
131
| `default` | Chops the text on according to whitespace and punctuation, removes tokens that are too long, and lowercases tokens |
137
132
| `stem_en` | Like `default`, but also applies stemming on the resulting tokens |
138
-
```
139
133
140
134
**Description of record options**
141
135
142
-
```markdown
143
136
| Record option | Description |
144
137
| ------------- | ------------- |
145
138
| `basic` | Records only the `DocId`s |
146
139
| `freq` | Records the document ids as well as the term frequency |
147
140
| `position` | Records the document id, the term frequency and the positions of occurences. |
148
-
```
149
141
150
142
Indexing with position is required to run phrase queries.
151
143
@@ -167,13 +159,11 @@ fast: true
167
159
168
160
**Parameters for i64, u64 and f64 field**
169
161
170
-
```markdown
171
162
| Variable | Description | Default value |
172
163
| ------------- | ------------- | ------------- |
173
164
| `stored` | Whether value is stored in the document store | `true` |
174
165
| `indexed` | Whether value is indexed | `true` |
175
166
| `fast` | Whether value is stored in a fast field | `false` |
176
-
```
177
167
178
168
#### `date` type
179
169
@@ -191,13 +181,11 @@ fast: true
191
181
192
182
**Parameters for date field**
193
183
194
-
```markdown
195
184
| Variable | Description | Default value |
196
185
| ------------- | ------------- | ------------- |
197
186
| `stored` | Whether value is stored in the document store | `true` |
198
187
| `indexed` | Whether value is indexed | `true` |
199
188
| `fast` | Whether value is stored in a fast field | `false` |
200
-
```
201
189
202
190
#### `bytes` type
203
191
The `bytes` type accepts a binary value as a `Base64` encoded string.
@@ -214,13 +202,11 @@ fast: true
214
202
215
203
**Parameters for bytes field**
216
204
217
-
```markdown
218
205
| Variable | Description | Default value |
219
206
| ------------- | ------------- | ------------- |
220
207
| `stored` | Whether value is stored in the document store | `true` |
221
208
| `indexed` | Whether value is indexed | `true` |
222
209
| `fast` | Whether value is stored in a fast field | `false` |
223
-
```
224
210
225
211
### Composite types
226
212
@@ -261,7 +247,6 @@ Fields with `null` or missing fields in your JSON document will be silently igno
261
247
262
248
This section describes indexing settings for a given index.
263
249
264
-
```markdown
265
250
| Variable | Description | Default value |
266
251
| ------------- | ------------- | ------------- |
267
252
| `timestamp_field` | Timestamp field used for sharding documents in splits (1). | None |
@@ -271,7 +256,6 @@ This section describes indexing settings for a given index.
271
256
| `merge_policy.max_merge_factor` | Maximum number of splits to merge. | 12 |
272
257
| `resources.num_threads` | Number of threads per source. | 1 |
273
258
| `resources.heap_size` | Indexer heap size per source per index. | 2_000_000_000 |
274
-
```
275
259
276
260
(1) [Learn more on time sharding]( ../design/architecture.md)
277
261
@@ -285,11 +269,9 @@ Indexer works with a default heap of 2 GiB of memory. This does not directly ref
285
269
286
270
This section describes search settings for a given index.
287
271
288
-
```markdown
289
272
| Variable | Description | Default value |
290
273
| ------------- | ------------- | ------------- |
291
274
| `search_default_fields` | Default list of fields that will be used for search. | None |
Copy file name to clipboardExpand all lines: docs/reference/metastore-config.md
+9-9Lines changed: 9 additions & 9 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -12,7 +12,7 @@ For instance:
12
12
- The different sources checkpoints.
13
13
- Some extra information such as the index creation time.
14
14
15
-
The metastore is entirely defined by a single URI. One can set it by editing the `metastore_uri` parameter of the [Quickwit configuration file](https://www.notion.so/Quickwit-configuration-MERGED-3fab5a181a9a43cba83db2fb25b46729) (often named `quickwit.yaml`).
15
+
The metastore is entirely defined by a single URI. One can set it by editing the `metastore_uri` parameter of the [Quickwit configuration file](./quickwit-config.md) (often named `quickwit.yaml`).
16
16
17
17
Currently, Quickwit offers two implementations:
18
18
@@ -48,7 +48,7 @@ Likewise, if you upgrade Quickwit to a version that includes some changes in the
48
48
49
49
For convenience, Quickwit also makes it possible to store its metadata in files using a file-backed metastore. In that case, Quickwit will write one file per index.
50
50
51
-
The metastore is then configured by passing a [Storage URI](https://www.notion.so/Storage-URI-APPROVED-176d8befb8d144fb820bcd0df077a728) that will serve as the root of the metastore storage.
51
+
The metastore is then configured by passing a [Storage URI](./storage-uri.md) that will serve as the root of the metastore storage.
52
52
53
53
The metadata file associated with a given index will then be stored under
54
54
@@ -57,7 +57,7 @@ The metadata file associated with a given index will then be stored under
57
57
For the moment, Quickwit supports two types of storage types:
58
58
59
59
- a local file system URI (e.g., `file:///opt/toto`). It is also valid to pass a file path directly (without file://). `/var/quickwit`. Relative paths will be resolved with respect to the current working directory.
60
-
- S3-compatible storage URI (e.g. `s3://my-bucket/some-path`] ). See the [Storage URI](https://www.notion.so/Storage-URI-APPROVED-176d8befb8d144fb820bcd0df077a728) documentation to configure S3 or S3-compatible storage.
60
+
- S3-compatible storage URI (e.g. `s3://my-bucket/some-path`] ). See the [Storage URI](./storage-uri.md) documentation to configure S3 or S3-compatible storage.
61
61
62
62
### Polling configuration
63
63
@@ -67,10 +67,10 @@ You can also configure it to poll the File-Backed Metastore periodically to keep
67
67
68
68
To configure the polling interval (in seconds only), add a URI fragment to the storage URI like this: `s3://quickwit/my-indexes#polling_interval=30s`
69
69
70
-
<aside>
71
-
👌 Amazon S3 charges $0.0004 per 1000 GET requests. Polling a metastore every 30 seconds will induce a cost of $0.04 per month and per index.
70
+
:::tip
71
+
Amazon S3 charges $0.0004 per 1000 GET requests. Polling a metastore every 30 seconds will induce a cost of $0.04 per month and per index.
0 commit comments