You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Making the index reference doc up to date. (#1068)
* Making the index reference doc up to date.
We recently integrated index-id and index-uri in
the index-reference.
In addition this PR makes more minor change like
adding a source in the example configuration.
* Update docs/reference/index-config.md
Co-authored-by: Evance Soumaoro <[email protected]>
Co-authored-by: Evance Soumaoro <[email protected]>
Copy file name to clipboardExpand all lines: docs/reference/index-config.md
+34-2Lines changed: 34 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -5,8 +5,9 @@ position: 4
5
5
6
6
This page describes how to configure an index.
7
7
8
-
The index configuration lets you define four items:
8
+
In addition to the `index_id`, the index configuration lets you define five items:
9
9
10
+
- The **index-uri**: it defines where the index files should be stored.
10
11
- The **doc mapping**: it defines how a document and the fields it contains are stored and indexed for a given index.
11
12
- The **indexing settings**: it defines the timestamp field used for sharding, and some more advanced parameters like the merge policy.
12
13
- The **search settings**: it defines the default search fields `default_search_fields`, a list of fields that Quickwit will search into if the user query does not explicitly target a field.
@@ -19,8 +20,13 @@ Configuration is set at index creation and cannot be modified except for the sou
19
20
The index configuration format is YAML. When a key is absent from the configuration file, the default value is used.
20
21
Here is a complete example suited for the HDFS logs dataset:
21
22
22
-
```
23
+
```yaml
23
24
version: 0# File format version.
25
+
26
+
index_id: "hdfs"
27
+
28
+
index_uri: "s3://my-bucket/hdfs"
29
+
24
30
doc_mapping:
25
31
field_mappings:
26
32
- name: timestamp
@@ -40,13 +46,39 @@ doc_mapping:
40
46
type: text
41
47
tokenizer: raw
42
48
tag_fields: ["resource.service"]
49
+
43
50
indexing_settings:
44
51
timestamp_field: timestamp
52
+
45
53
search_settings:
46
54
default_search_fields: [severity_text, body]
47
55
56
+
sources:
57
+
- hdfs: hdfs-log-kafka
58
+
source_type: kafka
59
+
params:
60
+
topic: hdfs-logs
61
+
client_params:
62
+
bootstrap.servers: localhost:9092
63
+
group.id: quickwit-consumer-group
64
+
security.protocol: SSL
48
65
```
49
66
67
+
## Index uri
68
+
69
+
The index-uri defines where the index files (also called splits) should be stored.
70
+
This parameter expects a [storage uri](storage-uri).
71
+
72
+
73
+
The `index-uri` parameter is optional.
74
+
By default, the `index-uri` will be computed by concatenating the `index-id` with the
75
+
`default_index_root_uri`defined in the [Quickwit's config](quickwit-config).
76
+
77
+
<aside>
78
+
⚠️ The file storage will not work when running quickwit in distributed mode.
79
+
Today, only the s3 storage is available when running several searcher nodes.
80
+
</aside>
81
+
50
82
## Doc mapping
51
83
52
84
The doc mapping defines how a document and the fields it contains are stored and indexed for a given index. A document is a collection of named fields, each having its own data type (text, binary, date, i64, u64, f64).
0 commit comments