-
Notifications
You must be signed in to change notification settings - Fork 163
Add time series data stream quickstart #2742
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 14 commits
Commits
Show all changes
17 commits
Select commit
Hold shift + click to select a range
69c3a2b
add quickstart
marciw 15fc3fc
fix subs
marciw 56fe8d0
Edits
marciw 7a72f3f
Edits: add stepper, match ES basics, clarity
marciw 40c38dc
Fix circular links
marciw f09de83
Merge branch 'main' into mw-tsds-quickstart
marciw 35d5f30
Merge branch 'main' into mw-tsds-quickstart
marciw 78ad532
just-in-time info
marciw e34bc6b
Merge branch 'main' into mw-tsds-quickstart
marciw 73a6e66
Merge branch 'main' into mw-tsds-quickstart
marciw 0f87cbb
Apply suggestions from reviewers
marciw 79f25cf
Merge branch 'main' into mw-tsds-quickstart
marciw 35dd10e
Merge branch 'main' into mw-tsds-quickstart
marciw 0c76699
Merge branch 'main' into mw-tsds-quickstart
marciw 2a6333d
Apply suggestions from review
marciw c525660
Apply suggestions from review
marciw f985a14
Merge branch 'main' into mw-tsds-quickstart
marciw File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,325 @@ | ||
--- | ||
navigation_title: "Quickstart" | ||
applies_to: | ||
stack: ga | ||
serverless: ga | ||
products: | ||
- id: elasticsearch | ||
--- | ||
|
||
# Quickstart: Time series data stream basics | ||
|
||
Use this quickstart to set up a time series data stream (TSDS), ingest a few documents, and run a basic query. These high-level steps help you see how a TSDS works, so you can decide whether it's right for your data. | ||
|
||
A _time series_ is a sequence of data points collected at regular time intervals. For example, you might track CPU usage or stock price over time. This quickstart uses simplified weather sensor readings to show how a TSDS helps you analyze metrics data over time. | ||
|
||
## Prerequisites | ||
|
||
* Access to [{{dev-tools-app}} Console](/explore-analyze/query-filter/tools/console.md) in {{kib}}, or another way to make {{es}} API requests | ||
|
||
* Cluster and index permissions: | ||
* [Cluster privilege](/deploy-manage/users-roles/cluster-or-deployment-auth/elasticsearch-privileges.md#privileges-list-cluster): `manage_index_templates` | ||
* [Index privileges](/deploy-manage/users-roles/cluster-or-deployment-auth/elasticsearch-privileges.md#privileges-list-indices): `create_doc` and `create_index` | ||
|
||
* Familiarity with [time series data stream concepts](time-series-data-stream-tsds.md) and [{{es}} index and search basics](/solutions/search/get-started.md) | ||
|
||
You can follow this guide using any {{es}} deployment. | ||
To see all deployment options, refer to [](/deploy-manage/deploy.md#choosing-your-deployment-type). | ||
To get started quickly, spin up a cluster [locally in Docker](/deploy-manage/deploy/self-managed/local-development-installation-quickstart.md). | ||
|
||
|
||
## Create and query a TSDS | ||
|
||
:::::{stepper} | ||
::::{step} Create an index template | ||
|
||
To create a data stream, you need an index template to base it on. The template defines the data stream structure and settings. (For this quickstart, you don't need to understand template details.) | ||
|
||
A TSDS uses _dimension_ fields and _metric_ fields. [Dimensions](/manage-data/data-store/data-streams/time-series-data-stream-tsds.md#time-series-dimension) are used to uniquely identify the time series and are typically based on a descriptive property like `location`. [Metrics](/manage-data/data-store/data-streams/time-series-data-stream-tsds.md#time-series-metric) are measurements that change over time. | ||
|
||
Use an [`_index_template` request](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-index-template) to create a template with two identifying dimension fields and two metric fields for weather measurements: | ||
|
||
``` console | ||
PUT _index_template/quickstart-tsds-template | ||
{ | ||
"index_patterns": ["quickstart-*"], | ||
"data_stream": { }, # Indicates this is a data stream, not a regular index. | ||
"priority": 100, | ||
"template": { | ||
"settings": { | ||
"index.mode": "time_series" # The required index mode for TSDS. | ||
}, | ||
"mappings": { | ||
"properties": { | ||
"sensor_id": { | ||
"type": "keyword", | ||
"time_series_dimension": true # Defines a dimension field. | ||
}, | ||
"location": { | ||
"type": "keyword", | ||
"time_series_dimension": true # Another dimension field. | ||
}, | ||
"temperature": { | ||
"type": "half_float", | ||
"time_series_metric": "gauge" # A supported field type for metrics. | ||
}, | ||
"humidity": { | ||
"type": "half_float", | ||
"time_series_metric": "gauge" # A second measurement. | ||
}, | ||
"@timestamp": { | ||
"type": "date" | ||
} | ||
} | ||
} | ||
} | ||
} | ||
|
||
``` | ||
|
||
This example defines a `@timestamp` field for illustration purposes. In most cases, you can use the default `@timestamp` field (which has a default type of `date`) instead of defining a timestamp in the mapping. | ||
|
||
You should get a response of `"acknowledged": true` that confirms the template was created. | ||
|
||
:::: | ||
|
||
::::{step} Add sample documents | ||
|
||
To apply the new index template to a new data stream, use a bulk ingest API request. Adding documents automatically creates a new data stream. | ||
|
||
In step 1, you specified the template index pattern `quickstart-*`. In this step, you set the data stream name to `quickstart-weather`, to match the template pattern. | ||
|
||
Use a [`_bulk` API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-bulk) request, adjusting the timestamps to within a few minutes of the current time: | ||
marciw marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
||
% TODO simplify timestamps | ||
|
||
```console | ||
PUT quickstart-weather/_bulk | ||
{ "create":{ } } | ||
{ "@timestamp": "2025-09-08T21:25:00.000Z", "sensor_id": "STATION-0001", "location": "base", "temperature": 26.7, "humidity": 49.9 } | ||
{ "create":{ } } | ||
{ "@timestamp": "2025-09-08T21:26:00.000Z", "sensor_id": "STATION-0002", "location": "base", "temperature": 27.2, "humidity": 50.1 } | ||
{ "create":{ } } | ||
{ "@timestamp": "2025-09-08T21:35:00.000Z", "sensor_id": "STATION-0003", "location": "base", "temperature": 28.1, "humidity": 48.7 } | ||
{ "create":{ } } | ||
{ "@timestamp": "2025-09-08T21:27:00.000Z", "sensor_id": "STATION-0004", "location": "satellite", "temperature": 32.4, "humidity": 88.9 } | ||
{ "create":{ } } | ||
{ "@timestamp": "2025-09-08T21:36:00.000Z", "sensor_id": "STATION-0005", "location": "satellite", "temperature": 32.3, "humidity": 87.5 } | ||
``` | ||
|
||
The response shows five sample weather data documents. | ||
|
||
:::{dropdown} Example response | ||
|
||
```console-result | ||
{ | ||
"errors": false, | ||
"took": 0, | ||
"items": [ | ||
{ | ||
"create": { | ||
"_index": ".ds-quickstart-weather-2025.09.08-000001", | ||
"_id": "cFJZQJlNh-Xl8V_rAAABmSs3x-A", | ||
"_version": 1, | ||
"result": "created", | ||
"_shards": { | ||
"total": 2, | ||
"successful": 2, | ||
"failed": 0 | ||
}, | ||
"_seq_no": 0, | ||
"_primary_term": 1, | ||
"status": 201 | ||
} | ||
}, | ||
{ | ||
"create": { | ||
"_index": ".ds-quickstart-weather-2025.09.08-000001", | ||
"_id": "c-wsTT0T4CtI3hOuAAABmSs4skA", | ||
"_version": 1, | ||
"result": "created", | ||
"_shards": { | ||
"total": 2, | ||
"successful": 2, | ||
"failed": 0 | ||
}, | ||
"_seq_no": 1, | ||
"_primary_term": 1, | ||
"status": 201 | ||
} | ||
}, | ||
{ | ||
"create": { | ||
"_index": ".ds-quickstart-weather-2025.09.08-000001", | ||
"_id": "Hdee5vMpBvZymWvHAAABmStA76A", | ||
"_version": 1, | ||
"result": "created", | ||
"_shards": { | ||
"total": 2, | ||
"successful": 2, | ||
"failed": 0 | ||
}, | ||
"_seq_no": 2, | ||
"_primary_term": 1, | ||
"status": 201 | ||
} | ||
}, | ||
{ | ||
"create": { | ||
"_index": ".ds-quickstart-weather-2025.09.08-000001", | ||
"_id": "e3Z2UirUQldsjLr2AAABmSs5nKA", | ||
"_version": 1, | ||
"result": "created", | ||
"_shards": { | ||
"total": 2, | ||
"successful": 2, | ||
"failed": 0 | ||
}, | ||
"_seq_no": 3, | ||
"_primary_term": 1, | ||
"status": 201 | ||
} | ||
}, | ||
{ | ||
"create": { | ||
"_index": ".ds-quickstart-weather-2025.09.08-000001", | ||
"_id": "N3-RYtQAp6JEsLRNAAABmStB2gA", | ||
"_version": 1, | ||
"result": "created", | ||
"_shards": { | ||
"total": 2, | ||
"successful": 2, | ||
"failed": 0 | ||
}, | ||
"_seq_no": 4, | ||
"_primary_term": 1, | ||
"status": 201 | ||
} | ||
} | ||
] | ||
} | ||
``` | ||
::: | ||
|
||
:::{tip} | ||
If you get an error about timestamp values, check the error response for the valid timestamp range. For more details, refer to [](/manage-data/data-store/data-streams/time-series-data-stream-tsds.md#tsds-accepted-time-range). | ||
|
||
::: | ||
|
||
:::: | ||
::::{step} Run a query | ||
|
||
Now that your data stream has some documents, you can use the [`_search` endpoint](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-search) to query the data. This sample aggregation shows average temperature for each location, in hourly buckets. (You don't need to understand the details of aggregations to follow this example.) | ||
marciw marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
```console | ||
POST quickstart-weather/_search | ||
{ | ||
"size": 0, | ||
"aggs": { | ||
"by_location": { | ||
"terms": { | ||
"field": "location" # The location dimension defined in the template. | ||
}, | ||
"aggs": { | ||
"avg_temp_per_hour": { | ||
"date_histogram": { | ||
"field": "@timestamp", | ||
"fixed_interval": "1h" | ||
}, | ||
"aggs": { | ||
"avg_temp": { | ||
"avg": { | ||
"field": "temperature" # A metric field defined in the template. | ||
} | ||
} | ||
} | ||
} | ||
} | ||
} | ||
} | ||
} | ||
``` | ||
|
||
:::{dropdown} Example response | ||
|
||
```console-result | ||
{ | ||
"took": 1, | ||
"timed_out": false, | ||
"_shards": { | ||
"total": 1, | ||
"successful": 1, | ||
"skipped": 0, | ||
"failed": 0 | ||
}, | ||
"hits": { | ||
"total": { | ||
"value": 5, | ||
"relation": "eq" | ||
}, | ||
"max_score": null, | ||
"hits": [] | ||
}, | ||
"aggregations": { | ||
"by_location": { | ||
"doc_count_error_upper_bound": 0, | ||
"sum_other_doc_count": 0, | ||
"buckets": [ | ||
{ | ||
"key": "base", | ||
"doc_count": 3, | ||
"avg_temp_per_hour": { | ||
"buckets": [ | ||
{ | ||
"key_as_string": "2025-09-08T21:00:00.000Z", | ||
"key": 1757365200000, | ||
"doc_count": 3, | ||
"avg_temp": { | ||
"value": 27.333333333333332 | ||
} | ||
} | ||
] | ||
} | ||
}, | ||
{ | ||
"key": "satellite", | ||
"doc_count": 2, | ||
"avg_temp_per_hour": { | ||
"buckets": [ | ||
{ | ||
"key_as_string": "2025-09-08T21:00:00.000Z", | ||
"key": 1757365200000, | ||
"doc_count": 2, | ||
"avg_temp": { | ||
"value": 32.359375 | ||
} | ||
} | ||
] | ||
} | ||
} | ||
] | ||
} | ||
} | ||
} | ||
``` | ||
::: | ||
|
||
:::{tip} | ||
You can also try this aggregation in a [data view](/explore-analyze/find-and-organize/data-views.md) in {{kib}}. | ||
::: | ||
|
||
:::: | ||
::::: | ||
|
||
## Next steps | ||
|
||
This quickstart introduced the basics of time series data streams. To learn more, explore these topics: | ||
|
||
* [](/manage-data/data-store/data-streams/time-series-data-stream-tsds.md) | ||
* [](/manage-data/data-store/data-streams/set-up-tsds.md) | ||
|
||
For more information about the APIs used in this quickstart, review the {{es}} API reference documentation: | ||
|
||
* [Bulk API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-bulk) | ||
* [Index template API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-index-template) | ||
* [Search API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-search) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.