Skip to content

Commit 3437d59

Browse files
DebashisBorgohainO2DebashisBorgohainO2
committed
Update docs
1 parent f169e0a commit 3437d59

File tree

3 files changed

+69
-35
lines changed

3 files changed

+69
-35
lines changed

‎docs/api/stream/setting.md

Lines changed: 66 additions & 34 deletions
Original file line numberDiff line numberDiff line change
@@ -16,21 +16,30 @@ Use the `StreamSettings` schema. All fields are optional.
1616

1717
```json
1818
{
19-
"partition_time_level": "day",
2019
"partition_keys": ["k8s_cluster", "k8s_namespace_name"],
21-
"index_fields": ["k8s_pod_name", "service_name"],
20+
"index_fields": ["k8s_pod_name", "k8s_container_name"],
2221
"full_text_search_keys": ["body"],
23-
"bloom_filter_fields": ["trace_id"],
22+
"bloom_filter_fields": ["k8s_node_name"],
2423
"data_retention": 30,
2524
"flatten_level": 1,
26-
"defined_schema_fields": ["severity", "log_file_path"],
27-
"max_query_range": 168,
25+
"defined_schema_fields": [
26+
"body",
27+
"k8s_cluster",
28+
"k8s_pod_name",
29+
"k8s_app_component",
30+
"log_file_path",
31+
"service_name",
32+
"service_version",
33+
"severity"
34+
],
35+
"max_query_range": 30,
2836
"store_original_data": true,
2937
"approx_partition": false,
3038
"extended_retention_days": [],
3139
"index_original_data": false,
3240
"index_all_values": false
3341
}
42+
3443
```
3544
### Response
3645
```
@@ -51,20 +60,46 @@ PUT /api/{org_id}/streams/{stream_name}/settings
5160

5261
Use the `UpdateStreamSettings` schema. Fields that support `add`, `remove`, or `set` use wrapper syntax.
5362

54-
```
63+
```json
5564
{
56-
"data_retention": 120,
57-
"max_query_range": 30,
58-
"extended_retention_days": {
59-
"add": [
60-
{
61-
"start": "2025-06-16T00:00:00Z",
62-
"end": "2026-02-21T00:00:00Z"
63-
}
65+
"partition_keys": {
66+
"set": ["k8s_cluster", "k8s_namespace_name"]
67+
},
68+
"full_text_search_keys": {
69+
"set": ["body"]
70+
},
71+
"index_fields": {
72+
"set": ["k8s_pod_name", "k8s_container_name"]
73+
},
74+
"bloom_filter_fields": {
75+
"set": ["k8s_node_name"]
76+
},
77+
"data_retention": 30,
78+
"flatten_level": 1,
79+
"defined_schema_fields": {
80+
"set": [
81+
"body",
82+
"k8s_cluster",
83+
"k8s_pod_name",
84+
"k8s_app_component",
85+
"log_file_path",
86+
"service_name",
87+
"service_version",
88+
"severity"
6489
]
65-
}
90+
},
91+
"max_query_range": 120,
92+
"store_original_data": true,
93+
"approx_partition": false,
94+
"extended_retention_days": {
95+
"add": []
96+
},
97+
"index_original_data": false,
98+
"index_all_values": false
6699
}
100+
67101
```
102+
68103
### Response
69104
```
70105
{
@@ -73,23 +108,20 @@ Use the `UpdateStreamSettings` schema. Fields that support `add`, `remove`, or `
73108
```
74109

75110
## Field Description
111+
### StreamSettings Field Reference
76112

77-
| Field name | Description |
78-
|---------------------------|-------------|
79-
| `partition_time_level` | Sets time-based partition granularity: `hour`, `day`, or `month`. Default is `hour` for logs and `day` for metrics. |
80-
| `partition_keys` | Additional fields used to partition data folders. Improves query performance for filtered fields. Use low to medium cardinality fields. |
81-
| `full_text_search_keys` | Fields to tokenize for full-text search. Required for `match_all` or substring queries. Defaults to common log message field names if unset. |
82-
| `index_fields` | Fields to enable exact-match indexing. Improves `field = value` query speed. Use selectively to avoid index bloat. |
83-
| `bloom_filter_fields` | Fields to apply bloom filters. Optimizes rare value lookups. Best for high-cardinality identifiers like `trace_id`. |
84-
| `data_retention` | Number of days to retain stream data. Overrides global retention. Minimum allowed is 3 days. |
85-
| `flatten_level` | Defines how many JSON levels to flatten. `0` flattens all levels. Controls field explosion in nested logs. |
86-
| `defined_schema_fields` | Fields explicitly included in the user-defined schema. Others are dropped or stored raw. Reduces schema and index size. |
87-
| `max_query_range` | Limits the maximum query window in hours. Prevents overly large queries from affecting performance. |
88-
| `store_original_data` | Retains the original log body for dropped fields. Enables viewing but not searching unless `index_original_data` is enabled. |
89-
| `approx_partition` | Enables uniform time slicing for query execution. Improves parallelism for skewed data distributions. |
90-
| `extended_retention_days` | Defines time ranges to retain data beyond standard retention. Must be configured before data expires. |
91-
| `index_original_data` | Indexes the raw log body for full-text search. Allows search across fields not defined in the schema. |
92-
| `index_all_values` | Indexes all fields for exact-match search. Increases ingestion cost. Best for small, stable schemas. |
93-
| `distinct_value_fields` | Maintains summaries of distinct field values. Powers UI filter suggestions and value lookups. |
94-
95-
113+
| Field name | Description |
114+
|---------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
115+
| `partition_keys` | Fields used to create data partitions, shown as `keyValue` or `Hash bucket` in the UI. Improves read performance by skipping unrelated files. Does not make the field searchable. |
116+
| `index_fields` | Fields to create secondary indexes for exact-match filters. Improves query performance on `field = value`. |
117+
| `full_text_search_keys` | Fields tokenized for full-text search. Required for substring or `match_all` queries. Defaults to common log fields if not set. |
118+
| `bloom_filter_fields` | Fields with high-cardinality values to optimize rare value searches. Improves performance by skipping non-matching data blocks. |
119+
| `data_retention` | Number of days to retain data in the stream. Minimum is 3 days. Overrides the global retention setting. |
120+
| `flatten_level` | Maximum depth for flattening nested JSON objects into fields. Helps expose nested keys for querying. |
121+
| `defined_schema_fields` | Fields to retain in the user-defined schema. Others are excluded or stored as raw if `store_original_data` is true. |
122+
| `max_query_range` | Maximum time range in hours for a single query. Prevents resource-heavy long-range queries. |
123+
| `store_original_data` | Stores the full original log body if schema filtering is applied. Allows retrieval of dropped fields. |
124+
| `approx_partition` | Uses evenly divided time ranges for query execution. Helps distribute query load in skewed data. |
125+
| `extended_retention_days` | List of time ranges to retain data beyond `data_retention`. Must be applied before data expires. |
126+
| `index_original_data` | Enables full-text indexing on the raw log body. Allows search across fields not part of the schema. |
127+
| `index_all_values` | Indexes all fields for exact-match lookups. Increases ingestion and index size. Best for fixed schemas. |

‎docs/environment-variables.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -328,12 +328,13 @@ For backward compatibility, we still support etcd but most of you should be able
328328
| AWS_EC2_METADATA_DISABLED | false | No | feature, default enable for `swift`. |
329329
| ZO_S3_FEATURE_HTTP1_ONLY | false | No | feature |
330330
| ZO_S3_FEATURE_HTTP2_ONLY | false | No | feature |
331+
| ZO_S3_FEATURE_BULK_DELETE | false | No | Enables bulk deletion of streams in object stores that support stream deletion. If your object store supports stream delete, you can enable this variable. AWS S3 and Azure ObjectStore are known to support it. When set to **true**, OpenObserve issues a single operation to delete all files under the stream’s storage prefix, reducing deletion time and API usage. |
331332

332333
## Enterprise
333334

334335
Below are the Environment variables only available in the enterprise edition.
335336

336-
## Misc
337+
## Miscellaneous
337338

338339
| Environment Variable | Default Value | Mandatory | Description |
339340
| ------------------------- | ------------- | --------- | --------------------------------------------------------------------------------------- |

‎docs/user-guide/.pages

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,4 +15,5 @@ nav:
1515
- Profile: profile
1616
- Performance: performance
1717
- Best Practices: best-practices
18+
- Migration: migration
1819

0 commit comments

Comments
 (0)