Skip to content

Commit 883d9e3

Browse files
golowanownashif
golowanow
authored and
nashif
committed
ci: elasticsearch: Upload script index map examples
Add several examples for `upload_test_results_es.py` script usage with ElasticSearch index map files for the following use cases: * Twister test results. * Twister test results with recordings. * Memory Footprint data (`twister-footprint.json`). Signed-off-by: Dmitrii Golovanov <[email protected]>
1 parent 6e5f1f1 commit 883d9e3

File tree

5 files changed

+808
-0
lines changed

5 files changed

+808
-0
lines changed

‎scripts/ci/es_upload/README.md

Lines changed: 156 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,156 @@
1+
# ElasticSearch index map examples
2+
3+
This directory contains [ElasticSearch data store](https://github.com/elastic/elasticsearch)
4+
index map examples for use by [upload_test_results_es.py](../upload_test_results_es.py)
5+
script for [Twister](https://docs.zephyrproject.org/latest/develop/test/twister.html)
6+
JSON reports upload.
7+
8+
An index map file allows to create destination index files on ElasticSearch server
9+
with explicit scheme having all required Twister JSON report objects associated
10+
to proper data types, eventually to store the expected document structure.
11+
Besides, it allows to track changes in Twister JSON report scheme
12+
and the corresponding data scheme in the same source code repository.
13+
An alternative way is to have index files created at the ElasticSearch server
14+
by other tools, or rely on default data mapping with potential type misalignments.
15+
16+
The command examples below are simplified with parameters essential for data transformations.
17+
For other command line options and more details run the upload script with `--help`,
18+
or read its source code with inline comments there.
19+
20+
Use these examples as a reference for your setup. Always pay attention to possible changes
21+
in the current Twister report format and in the upload script itself.
22+
Tune resulting data scheme and size depending on your particular needs.
23+
24+
It is recommended to try the upload script with `--dry-run` option first
25+
to check the resulting data without its actual upload.
26+
27+
You should set environment variables `ELASTICSEARCH_SERVER` and `ELASTICSEARCH_KEY`
28+
to connect with your server running the upload script.
29+
30+
31+
## Index creation with a map file
32+
33+
Execute the upload script once for the first time to create your index at the ElasticSearch
34+
server with the appropriate map file, for example:
35+
```bash
36+
python3 ./scripts/ci/upload_test_results_es.py --create-index \
37+
--index YOUR_INDEX_NAME \
38+
--map-file YOUR_INDEX_MAP.json
39+
```
40+
41+
42+
## Upload transformations
43+
44+
The upload script has several command line options to change `twister.json` data:
45+
exclude excess data, extract substrings by regular expressions, change data structure
46+
making it suitable for Kibana default visual components
47+
(see `--help` for more details):
48+
49+
* `--exclude` removes excess testsuite properties not needed to store them
50+
at Elasticsearch index.
51+
52+
* `--transform` applies regexp group parsing rules to string properties extracting
53+
derived object properties.
54+
55+
* `--flatten` changes testsuite data structure in regard of one of its list components:
56+
either `testcases` or `recording`, so each item there becomes an independent data record
57+
inheriting all other testsuite properties, whereas the children object's properties
58+
are renamed with the parent object's name as a prefix:
59+
`testcases_` or `recording_` respectively.
60+
Only one testsuite property object can be flattened this way per index upload.
61+
All other testsuite objects will be treated according to the index structure.
62+
63+
* Other command line options: `--flatten-dict-name`, `--flatten-list-names`,
64+
`--flatten-separator`, `--transpose-separator`, `--escape-separator`.
65+
66+
67+
## Index map examples and use cases
68+
69+
70+
### Twister test results
71+
72+
Create the index:
73+
```bash
74+
python3 ./scripts/ci/upload_test_results_es.py --create-index \
75+
--index zephyr-test-example \
76+
--map-file zephyr_twister_index.json
77+
```
78+
79+
Upload Twister test results 'as-is', without additional transformations:
80+
```bash
81+
python3 ./scripts/ci/upload_test_results_es.py \
82+
--index zephyr-test-example \
83+
./twister-out/**/twister.json
84+
```
85+
86+
87+
### Twister tests with recording
88+
89+
Store test results with `recording` data entries, for example from
90+
[Kernel Timer Behavior](https://github.com/zephyrproject-rtos/zephyr/tree/main/tests/kernel/timer/timer_behavior)
91+
test suite.
92+
93+
Create the index:
94+
```bash
95+
python3 ./scripts/ci/upload_test_results_es.py --create-index \
96+
--index zephyr-test-recording-example-1 \
97+
--map-file zephyr_twister_flat_recording_metrics_index.json
98+
```
99+
100+
Upload data with 'flattened' test suites creating documents for each `recording` data entry.
101+
```bash
102+
python3 ./scripts/ci/upload_test_results_es.py \
103+
--index zephyr-test-recording-example-1 \
104+
--exclude path run_id \
105+
--flatten recording \
106+
./twister-out/**/twister.json
107+
```
108+
109+
110+
### Twister test with recording and extracting more data
111+
112+
Store test results with `recording` data entries which are also scanned by regular expressions
113+
to extract embedded values, for example
114+
[Kernel Latency Benchmarks](https://github.com/zephyrproject-rtos/zephyr/tree/main/tests/benchmarks/latency_measure)
115+
test suite.
116+
117+
Create the index:
118+
```bash
119+
python3 ./scripts/ci/upload_test_results_es.py --create-index \
120+
--index zephyr-test-recording-example-2 \
121+
--map-file zephyr_twister_flat_recording_index.json
122+
```
123+
124+
Upload data with 'flattened' test suites creating documents for each `record` data entry
125+
with 3 additional data properties extracted by regular expressions.
126+
```bash
127+
python3 ./scripts/ci/upload_test_results_es.py \
128+
--index zephyr-test-recording-example-2 \
129+
--exclude path run_id \
130+
--flatten recording \
131+
--transform "{ 'recording_metric': '(?P<recording_metric_object>[^\.]+)\.(?P<recording_metric_action>[^\.]+)\.(?P<recording_metric_details>[^ -]+)' }" \
132+
./twister-out/**/twister.json
133+
```
134+
135+
136+
### Memory Footprint results
137+
138+
To store Memory Footprint data reported in `twister-footprint.json` (see Twister `--footprint-report`).
139+
140+
Create the index:
141+
```bash
142+
python3 ./scripts/ci/upload_test_results_es.py --create-index \
143+
--index zephyr-memory-footprint-example \
144+
--map-file zephyr_twister_flat_footprint_index.json
145+
```
146+
147+
Upload data with 'flattened' footprint entries creating documents for each of them.
148+
```bash
149+
python3 ./scripts/ci/upload_test_results_es.py \
150+
--index zephyr-memory-footprint-example \
151+
--exclude path run_id \
152+
--flatten footprint \
153+
--flatten-list-names "{'children':'name'}" \
154+
--transform "{ 'footprint_name': '^(?P<footprint_area>([^\/]+\/){0,2})(?P<footprint_path>([^\/]*\/)*)(?P<footprint_symbol>[^\/]*)$' }" \
155+
../footprint_data/**/twister_footprint.json
156+
```

‎scripts/ci/es_upload/zephyr_twister_flat_footprint_index.json

Lines changed: 158 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,158 @@
1+
{
2+
"properties": {
3+
"environment": {
4+
"properties": {
5+
"commit_date": {
6+
"type": "date"
7+
},
8+
"os": {
9+
"type": "keyword",
10+
"ignore_above": 256
11+
},
12+
"run_branch": {
13+
"type": "keyword",
14+
"ignore_above": 256
15+
},
16+
"run_workflow": {
17+
"type": "keyword",
18+
"ignore_above": 256
19+
},
20+
"run_id": {
21+
"type": "keyword",
22+
"ignore_above": 256
23+
},
24+
"run_attempt": {
25+
"type": "integer"
26+
},
27+
"run_date": {
28+
"type": "date"
29+
},
30+
"toolchain": {
31+
"type": "keyword",
32+
"ignore_above": 256
33+
},
34+
"zephyr_version": {
35+
"type": "keyword",
36+
"ignore_above": 256
37+
},
38+
"options": {
39+
"type": "object",
40+
"enabled": true,
41+
"dynamic": true,
42+
"subobjects": true
43+
}
44+
}
45+
},
46+
"name": {
47+
"type": "keyword",
48+
"ignore_above": 256
49+
},
50+
"component": {
51+
"type": "keyword",
52+
"ignore_above": 256
53+
},
54+
"sub_component": {
55+
"type": "keyword",
56+
"ignore_above": 256
57+
},
58+
"arch": {
59+
"type": "keyword",
60+
"ignore_above": 256
61+
},
62+
"platform": {
63+
"type": "keyword",
64+
"ignore_above": 256
65+
},
66+
"used_ram": {
67+
"type": "unsigned_long"
68+
},
69+
"used_rom": {
70+
"type": "unsigned_long"
71+
},
72+
"available_ram": {
73+
"type": "unsigned_long"
74+
},
75+
"available_rom": {
76+
"type": "unsigned_long"
77+
},
78+
"retries": {
79+
"type": "integer"
80+
},
81+
"status": {
82+
"type": "keyword",
83+
"ignore_above": 256
84+
},
85+
"build_time": {
86+
"type": "float"
87+
},
88+
"reason": {
89+
"type": "text",
90+
"fields": {
91+
"keyword": {
92+
"type": "keyword",
93+
"ignore_above": 256
94+
}
95+
}
96+
},
97+
"log": {
98+
"type": "text"
99+
},
100+
"footprint_name": {
101+
"type": "text",
102+
"fields": {
103+
"keyword": {
104+
"type": "keyword",
105+
"ignore_above": 256
106+
}
107+
}
108+
},
109+
"footprint_name_depth": {
110+
"type": "integer"
111+
},
112+
"footprint_identifier": {
113+
"type": "text",
114+
"fields": {
115+
"keyword": {
116+
"type": "keyword",
117+
"ignore_above": 256
118+
}
119+
}
120+
},
121+
"footprint_area": {
122+
"type": "text",
123+
"fields": {
124+
"keyword": {
125+
"type": "keyword",
126+
"ignore_above": 256
127+
}
128+
}
129+
},
130+
"footprint_path": {
131+
"type": "text",
132+
"fields": {
133+
"keyword": {
134+
"type": "keyword",
135+
"ignore_above": 256
136+
}
137+
}
138+
},
139+
"footprint_symbol": {
140+
"type": "text",
141+
"fields": {
142+
"keyword": {
143+
"type": "keyword",
144+
"ignore_above": 256
145+
}
146+
}
147+
},
148+
"footprint_address": {
149+
"type": "unsigned_long"
150+
},
151+
"footprint_size": {
152+
"type": "long"
153+
},
154+
"footprint_total_size": {
155+
"type": "unsigned_long"
156+
}
157+
}
158+
}

0 commit comments

Comments
 (0)