Skip to content

Latest commit

 

History

History
122 lines (90 loc) · 5.71 KB

victoriametrics.md

File metadata and controls

122 lines (90 loc) · 5.71 KB

TSBS Supplemental Guide: VictoriaMetrics

VictoriaMetrics is fast, cost-effective and scalable time-series database. It supports InfluxDB, Graphite and OpenTSDB protocols for ingesting data and also can be used as long-term remote storage for Prometheus. This supplemental guide explains how the data generated for TSBS is stored, additional flags available when using the data importer (tsbs_load_victoriametrics), and additional flags available for the query runner (tsbs_run_queries_victoriametrics).

To install all required tools pls do following:

# Install desired binaries. At a minimum this includes tsbs_generate_data,
# tsbs_generate_queries, one tsbs_load_* binary, and one tsbs_run_queries_*
# binary:
$ cd $GOPATH/src/github.com/timescale/tsbs/cmd
$ cd tsbs_generate_data && go install
$ cd ../tsbs_generate_queries && go install
$ cd ../tsbs_load_victoriametrics && go install
$ cd ../tsbs_run_queries_victoriametrics && go install

This should be read after the main README.

Data format

Data generated by tsbs_generate_data for VictoriaMetrics is serialized in the same format as for InfluxDB. This is "pseudo-CSV" format, each reading is composed of a single line where the name of the table is the first item, a comma, followed by several comma-separated items of tags in the format of <label>=<value>, a space, several comma-separated items of fields in the format of <label>=<value>, a space, and finally the timestamp for the reading.

An example for the cpu-only use case:

cpu,hostname=host_0,region=eu-central-1,datacenter=eu-central-1b,rack=21,os=Ubuntu15.10,arch=x86,team=SF,service=6,service_version=0,service_environment=test usage_user=58.1317132304976170,usage_system=2.6224297271376256,usage_idle=24.9969495069947882,usage_nice=61.5854484633778867,usage_iowait=22.9481393231639395,usage_irq=63.6499207106198313,usage_softirq=6.4098777048301052,usage_steal=44.8799140503027445,usage_guest=80.5028770761136201,usage_guest_nice=38.2431182911542820 1451606400000000000

Remember to set -timestamp-start and -timestamp-end flags in range of VictoriaMetrics retention period. All data out of retention period will be automatically deleted after insertion.

One of the ways to generate data for insertion is to use scripts/generate_data.sh:

FORMATS=victoriametrics SCALE=100 TS_START=2021-08-01T00:00:00Z TS_END=2021-08-03T00:00:00Z  ./scripts/generate_data.sh

Important: please ensure that VictoriaMetrics retention setting covers the time range set by TS_START and TS_END params.

tsbs_load_victoriametrics

See recommendations for insertion in InfluxDB protocol.

One of the ways to load data in VictoriaMetrics is to use scripts/load_victoriametrics.sh:

./scripts/load/load_victoriametrics.sh

Assumed that VictoriaMetrics is already installed and ready for insertion on default port 8428. If not - please set DATABASE_PORT variable accordingly. If you're using cluster version of VictoriaMetrics please specify vminsert port (8480 by default) and DATABASE_PATH=insert/0/influx/write, where 0 is tenant ID. See more about URL format here.

Additional Flags

--urls (type: string, default: http://localhost:8428/write)

Comma-separated list of URLs to connect to for inserting data. It can be just a single-version URL or list of VMInsert URLs. Workers will be distributed in a round robin fashion across the URLs. See more about URL format here.

Generating queries

VictoriaMetrics query language MetricsQL has some limitations comparing to SQL. Because of this VictoriaMetrics query generator lacks for implementation of query types for devops use-case:

  • groupby-orderby-limit - results are always ordered by time and can't be limited;
  • lastpoint - can't be queried if datapoint is older than 5 minutes;
  • high-cpu-1, high-cpu-all - can't be queried without grouping by step.

The iot use-case wasn't implemented yet.

One of the ways to generate queries for VictoriaMetrics is to use scripts/generate_queries.sh:

 FORMATS=victoriametrics SCALE=100 TS_START=2021-08-01T00:00:00Z TS_END=2021-08-03T00:00:00Z \
 QUERY_TYPES="cpu-max-all-8 double-groupby-1" ./scripts/generate_queries.sh

Important: generate queries with same params as used for data loading on previous steps.

tsbs_run_queries_victoriametrics

To run generated queries follow examples in documentation:

cat /tmp/bulk_queries/victoriametrics-cpu-max-all-8-queries.gz | gunzip | tsbs_run_queries_victoriametrics

By default, tsbs_run_queries_victoriametrics assumes that VictoriaMetrics is already installed and ready for accepting queries on http://localhost:8428. To change the address, please specify --urls flags. If you're using cluster version of VictoriaMetrics please specify --urls flag as http://localhost:8481/select/0/prometheus, where localhost:8481 is vmselect address and port, and 0 is tenant ID. See more about URL format here.

Additional flags

--urls (type: string, default: http://localhost:8428)

Comma-separated list of URLs to connect to for querying. It can be just a single-version URL or list of VMSelect URLs. Workers will be distributed in a round robin fashion across the URLs. See help for additional info.