Skip to content

Commit 587e6f8

Browse files
stephen-solteszstephen-soltesz
authored
Update README and add example configuration (#32)
* Update readme to use example * Add example configuration and query
1 parent 15b9a45 commit 587e6f8

File tree

4 files changed

+163
-76
lines changed

4 files changed

+163
-76
lines changed

‎README.md‎

Lines changed: 96 additions & 76 deletions
Original file line numberDiff line numberDiff line change
@@ -1,121 +1,141 @@
11
# prometheus-bigquery-exporter
2-
[![Version](https://img.shields.io/github/tag/m-lab/prometheus-bigquery-exporter.svg)](https://github.com/m-lab/prometheus-bigquery-exporter/releases) [![Build Status](https://travis-ci.org/m-lab/prometheus-bigquery-exporter.svg?branch=master)](https://travis-ci.org/m-lab/prometheus-bigquery-exporter) [![Coverage Status](https://coveralls.io/repos/m-lab/prometheus-bigquery-exporter/badge.svg?branch=master)](https://coveralls.io/github/m-lab/prometheus-bigquery-exporter?branch=master) [![GoDoc](https://godoc.org/github.com/m-lab/prometheus-bigquery-exporter?status.svg)](https://godoc.org/github.com/m-lab/prometheus-bigquery-exporter) [![Go Report Card](https://goreportcard.com/badge/github.com/m-lab/prometheus-bigquery-exporter)](https://goreportcard.com/report/github.com/m-lab/prometheus-bigquery-exporter)
32

4-
An exporter for converting BigQuery results into Prometheus metrics.
3+
[![Version](https://img.shields.io/github/tag/m-lab/prometheus-bigquery-exporter.svg)](https://github.com/m-lab/prometheus-bigquery-exporter/releases) [![Build Status](https://travis-ci.org/m-lab/prometheus-bigquery-exporter.svg?branch=master)](https://travis-ci.org/m-lab/prometheus-bigquery-exporter) [![Coverage Status](https://coveralls.io/repos/m-lab/prometheus-bigquery-exporter/badge.svg?branch=master)](https://coveralls.io/github/m-lab/prometheus-bigquery-exporter?branch=master) [![GoDoc](https://godoc.org/github.com/m-lab/prometheus-bigquery-exporter?status.svg)](https://godoc.org/github.com/m-lab/prometheus-bigquery-exporter) [![Go Report Card](https://goreportcard.com/badge/github.com/m-lab/prometheus-bigquery-exporter)](https://goreportcard.com/report/github.com/m-lab/prometheus-bigquery-exporter)
54

6-
# Limitations
5+
An exporter for converting BigQuery results into Prometheus metrics.
76

8-
## No historical values
7+
## Limitations: No historical values
98

109
Prometheus collects the *current* status of a system as reported by an exporter.
1110
Prometheus then associates the values collected with a timestamp of the time of
1211
collection.
1312

1413
*NOTE:* there is no way to associate historical values with timestamps in the
15-
the past!
14+
the past with this exporter!
1615

1716
So, the results of queries run by prometheus-bigquery-exporter should represent
1817
a meaningful value at a fixed point in time relative to the time the query is
1918
made, e.g. total number of tests in a 5 minute window 1 hour ago.
2019

21-
# Query format
20+
## Query Formatting
2221

2322
The prometheus-bigquery-exporter accepts arbitrary BQ queries. However, the
2423
query results must be structured in a predictable way for the exporter to
2524
successfully interpret and convert it into prometheus metrics.
2625

27-
Required columns:
26+
### Metric names and values
2827

29-
* `value` -- every query result must have a "value". Values should be integers
30-
or floats.
28+
Metric names are derived from the query file name and query value columns.
29+
The bigquery-exporter identifies value columns by looking for column names
30+
that match the pattern: `value([.+])`. All characters in the matching group
31+
`([.+])` are appended to the metric prefix taken from the query file name.
32+
For example:
3133

32-
Optional columns:
34+
* Filename: `bq_ndt_test.sql`
35+
* Metric prefix: `bq_ndt_test`
36+
* Column name: `value_count`
37+
* Final metric: `bq_ndt_test_count`
3338

34-
* If there is more than one result row, then the query must also define labels
35-
to distinguish each value. Every column name that is not "value" will create
36-
a label on the resulting metric. For example, results with two columns,
37-
"machine" and "value" would create metrics with labels named "machine" and
38-
values from the results for that row.
39+
Value columns are required (at least one):
3940

40-
Labels should be strings.
41+
* `value([.+])` - every query must define a result "value". Values must
42+
be integers or floats. For a query to return multiple values, prefix each
43+
with "value" and define unique suffixes.
4144

42-
There is no limit on the number of labels, but you should respect the
43-
prometheus best practices by limiting label value cardinality.
45+
Label columns are optional:
4446

45-
## Example query
47+
* If there is more than one result row, then the query must also define labels
48+
to distinguish each value. Every column name that is not "value" will create
49+
a label on the resulting metric. For example, results with two columns,
50+
"machine" and "value" would create metrics with labels named "machine" and
51+
values from the results for that row.
4652

47-
The following query creates a "machine" label and counts the number of tests
53+
Labels must be strings:
4854

49-
```
50-
# TODO: replace with query using views.
51-
# TODO: replace with standard SQL syntax.
52-
SELECT
53-
-- All columns not named "value" are added as metric labels.
54-
CONCAT(REPLACE(REGEXP_EXTRACT(task_filename,
55-
r'gs://.*-(mlab[1-4]-[a-z]{3}[0-9]+)-ndt.*.tgz'), "-", "."),
56-
".measurement-lab.org") AS label_machine,
57-
58-
-- All queries must have a single column named "value"
59-
count(*) as value
60-
61-
FROM
62-
[measurement-lab:public.ndt]
63-
64-
GROUP BY label_machine
65-
ORDER BY value
66-
```
55+
* There is no limit on the number of labels, but you should respect the
56+
prometheus best practices by limiting label value cardinality.
6757

68-
Save the sample query to a file named "ndt_test_cound.sql". The metric name is
69-
derived from the file name. Start the exporter:
58+
Duplicate metrics are an error:
7059

71-
```
72-
bq_exporter --query counter=ndt_test_count.sql
73-
```
60+
* If the query returns multiple rows that are not distinguished by the set of
61+
labels for each row.
7462

75-
Visit http://localhost:9393/metrics and you will find metrics like:
63+
## Example Query
7664

77-
```
78-
ndt_test_count{machine="mlab1.foo01.measurement-lab.org"} 100
79-
ndt_test_count{machine="mlab2.foo01.measurement-lab.org"} 200
80-
...
81-
```
65+
The following query creates a label and groups by each label.
8266

67+
```sql
68+
-- Example data in place of an actual table of values.
69+
WITH example_data as (
70+
SELECT "a" as label, 5 as widgets
71+
UNION ALL
72+
SELECT "b" as label, 2 as widgets
73+
UNION ALL
74+
SELECT "b" as label, 3 as widgets
75+
)
8376

84-
# Testing
77+
SELECT
78+
label, SUM(widgets) as value
79+
FROM
80+
example_data
81+
GROUP BY
82+
label
83+
```
8584

86-
To run the bigquery exporter locally (e.g. with a new query) you can build a
87-
test environment based on the google/cloud-sdk with a golang tools installed.
85+
* Save the sample query to a file named "bq_example.sql".
86+
* Start the exporter:
8887

89-
Use the following steps:
88+
```sh
89+
prometheus-bigquery-exporter -gauge-query bq_example.sql
90+
```
9091

91-
1. Build the testing docker image.
92+
* Visit http://localhost:9348/metrics and you will find metrics like:
9293

93-
```
94-
$ docker build -t bqe.testing -f Dockerfile.testing .
95-
```
94+
```txt
95+
bq_example{label="a"} 5
96+
bq_example{label="b"} 5
97+
...
98+
```
9699

97-
2. Run the testing image, with fowarded ports and shared volume. The
98-
`--volumes-from` option is created automatically by the cloud-sdk base image.
99-
This volume preserves credentials across runs of the docker image.
100+
## Example Configuration
100101

101-
```
102-
$ docker run -p 9050:9050 --rm -ti -v $PWD:/go/src/github.com/m-lab/prometheus-bigquery-exporter --volumes-from gcloud-config bqe.testing
102+
Typical deployments will be in Kubernetes environment, like GKE.
103+
104+
```sh
105+
# Change to the example directory.
106+
cd example
107+
# Deploy the example query as a configmap and example k8s deployment.
108+
./deploy.sh
103109
```
104110

105-
3. Authenticate using your account. Both steps are necessary, the first to run
106-
gcloud commands (which uses user credentials), the second to run the bigquery
107-
exporter (which uses application default credentials).
111+
## Testing
108112

109-
```
110-
# gcloud auth login
111-
# gcloud auth application-default login
112-
```
113+
To run the bigquery exporter locally (e.g. with a new query) you can build
114+
and run locally.
113115

114-
4. Start the bigquery exporter.
116+
Use the following steps:
115117

116-
```
117-
go get -v github.com/m-lab/prometheus-bigquery-exporter/cmd/bigquery_exporter
118-
./go/bin/bigquery_exporter \
119-
--project mlab-sandbox \
120-
--type gauge --query <path-to-some-query-file>/bq_ndt_metrics.sql
121-
```
118+
1. Build the docker image.
119+
120+
```sh
121+
docker build -t bqx-local -f Dockerfile .
122+
```
123+
124+
2. Authenticate using your Google account. Both steps are necessary, the
125+
first to run gcloud commands (which uses user credentials), the second to run
126+
the bigquery exporter (which uses application default credentials).
127+
128+
```sh
129+
gcloud auth login
130+
gcloud auth application-default login
131+
```
132+
133+
3. Run the image, with fowarded ports and access to gcloud credentials.
134+
135+
```sh
136+
docker run -p 9348:9348 --rm \
137+
-v $HOME/.config/gcloud:/root/.config/gcloud \
138+
-v $PWD:/queries -it bqx-local \
139+
-project $GCLOUD_PROJECT \
140+
-guage-query /queries/example/config/bq_example.sql
141+
```

‎example/bigquery.yml‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
apiVersion: apps/v1
2+
kind: Deployment
3+
metadata:
4+
name: bigquery-exporter-5m
5+
spec:
6+
replicas: 1
7+
selector:
8+
matchLabels:
9+
run: bigquery-exporter
10+
template:
11+
metadata:
12+
annotations:
13+
prometheus.io/scrape: 'true'
14+
labels:
15+
run: bigquery-exporter
16+
spec:
17+
containers:
18+
- name: bigquery-exporter
19+
image: measurementlab/prometheus-bigquery-exporter:latest
20+
args: [ "-project=mlab-sandbox",
21+
"-refresh=1m",
22+
"-gauge-query=/queries/bq_example.sql",
23+
]
24+
ports:
25+
- containerPort: 9050
26+
volumeMounts:
27+
- mountPath: /queries
28+
name: bigquery-config
29+
30+
volumes:
31+
- name: bigquery-config
32+
configMap:
33+
name: bigquery-exporter-config

‎example/config/bq_example.sql‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
-- Example query.
2+
WITH example_data as (
3+
SELECT "a" as label, 5 as widgets
4+
UNION ALL
5+
SELECT "b" as label, 2 as widgets
6+
UNION ALL
7+
SELECT "b" as label, 3 as widgets
8+
)
9+
10+
SELECT
11+
label, SUM(widgets) as value
12+
FROM
13+
example_data
14+
GROUP BY
15+
label

‎example/deploy.sh‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
#!/bin/bash
2+
#
3+
# deploy.sh applies an example bigquery exporter to the currently
4+
# selected k8s cluster.
5+
#
6+
# Example:
7+
#
8+
# ./deploy.sh
9+
10+
set -x
11+
set -e
12+
set -u
13+
14+
# Apply the bigquery exporter configurations.
15+
kubectl create configmap bigquery-exporter-config \
16+
--from-file=config \
17+
--dry-run=client -o json | kubectl apply -f -
18+
19+
kubectl apply -f bigquery.yml

0 commit comments

Comments
 (0)