Skip to content

Commit 565cafd

Browse files
YANG-DBYANG-DB
authored
Update collection feeds (#1500)
* Add queries & query templates for serviceMap integration add jaeger data provider to the existing data-prepper update integration.schema to reflect mapping fields for specific templates Signed-off-by: YANGDB <[email protected]> * update the integration.schema with the 'feed' concept that would encapsulate the different catalog categories - the collection item now would contain [category name,components list, feeds array] Signed-off-by: YANGDB <[email protected]> * add event.source to the log.mapping default mapping schema Signed-off-by: YANGDB <[email protected]> * add rest API images & diagram for assets dependencies Signed-off-by: YANGDB <[email protected]> * add descriptive information for each log component Signed-off-by: YANGDB <[email protected]> * add fields.json to represent the integration's fields Signed-off-by: YANGDB <[email protected]> --------- Signed-off-by: YANGDB <[email protected]>
1 parent dddecd1 commit 565cafd

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

42 files changed

+2889
-218
lines changed

‎docs/API/API.png‎

166 KB
Loading

‎docs/API/README.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,9 @@
11
# API Mock Tests
22
The [Swagger](swagger.yaml) describes the API used by the Integration repository to support the `catalog`, `registry` and `store` functionalities.
33

4+
Its possible to visualize the REST API using any on the numerous [online editors](https://editor.swagger.io/)
5+
![](API.png)
6+
47
### Setup mock server
58
In order to test the API, we can utilize a swagger based mock library which allows ['contract based testing'](https://github.com/stoplightio/prism).
69

‎docs/API/swagger.yaml‎

Lines changed: 25 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -44,14 +44,14 @@ paths:
4444
schema:
4545
type: array
4646
items:
47-
$ref: '../../schema/system/catalog.schema'
47+
$ref: '../../src/main/resources/schema/system/catalog.schema'
4848
example:
4949
observability:
5050
application/json:
5151
schema:
5252
type: array
5353
items:
54-
$ref: '../../schema/system/catalog.schema'
54+
$ref: '../../src/main/resources/schema/system/catalog.schema'
5555
examples:
5656
catalog: observability
5757
version: '1.0'
@@ -110,14 +110,14 @@ paths:
110110
content:
111111
application/json:
112112
schema:
113-
$ref: '../../schema/system/catalog.schema'
113+
$ref: '../../src/main/resources/schema/system/catalog.schema'
114114
example:
115115
observability:
116116
application/json:
117117
schema:
118118
type: array
119119
items:
120-
$ref: '../../schema/system/catalog.schema'
120+
$ref: '../../src/main/resources/schema/system/catalog.schema'
121121
examples:
122122
catalog: observability
123123
version: '1.0'
@@ -190,14 +190,14 @@ paths:
190190
content:
191191
application/json:
192192
schema:
193-
$ref: '../../schema/system/catalog.schema'
193+
$ref: '../../src/main/resources/schema/system/catalog.schema'
194194
example:
195195
observability:
196196
application/json:
197197
schema:
198198
type: array
199199
items:
200-
$ref: '../../schema/system/catalog.schema'
200+
$ref: '../../src/main/resources/schema/system/catalog.schema'
201201
examples:
202202
catalog: observability
203203
version: '1.0'
@@ -275,7 +275,7 @@ paths:
275275
schema:
276276
type: array
277277
items:
278-
$ref: '../../schema/system/integrations-template-list.schema'
278+
$ref: '../../src/main/resources/schema/system/integrations-template-list.schema'
279279
example:
280280
- template-name: nginx
281281
version: 1.0.0
@@ -313,7 +313,7 @@ paths:
313313
content:
314314
application/zip:
315315
schema:
316-
$ref: '../../schema/system/integration.schema'
316+
$ref: '../../src/main/resources/schema/system/integration.schema'
317317
example:
318318
- BLOB including all images, json files, dashboards and documentations
319319
responses:
@@ -344,7 +344,7 @@ paths:
344344
content:
345345
application/json:
346346
schema:
347-
$ref: '../../schema/system/integration.schema'
347+
$ref: '../../src/main/resources/schema/system/integration.schema'
348348
example:
349349
template-name: nginx
350350
version:
@@ -354,11 +354,12 @@ paths:
354354
description: Nginx HTTP server collector
355355
identification: instrumentationScope.attributes.identification
356356
catalog: observability
357-
components:
358-
- web
359-
- http
360357
collection:
361-
- logs:
358+
- category: logs
359+
components:
360+
- communication
361+
- http
362+
feeds:
362363
- info: access logs
363364
input_type: logfile
364365
dataset: nginx.access
@@ -371,7 +372,9 @@ paths:
371372
- nginx
372373
- error
373374
dataset: nginx.error
374-
- metrics:
375+
- category: metrics
376+
components: [ ]
377+
feeds:
375378
- info: status metrics
376379
input_type: metrics
377380
dataset: nginx.status
@@ -380,7 +383,6 @@ paths:
380383
- status
381384
repository:
382385
url: https://github.com/opensearch-project/observability/tree/main/integrarions/nginx
383-
384386
'400':
385387
description: bad input parameter
386388
/repository/{id}/fields:
@@ -404,7 +406,7 @@ paths:
404406
content:
405407
application/json:
406408
schema:
407-
$ref: '../../schema/system/integration-fields-list.schema'
409+
$ref: '../../src/main/resources/schema/system/integration-fields-list.schema'
408410
example:
409411
template-name: nginx
410412
version: 1.0.0
@@ -818,7 +820,7 @@ paths:
818820
schema:
819821
type: array
820822
items:
821-
$ref: '../../schema/system/integration-instance.schema'
823+
$ref: '../../src/main/resources/schema/system/integration-instance.schema'
822824
example:
823825
template-name: nginx
824826
dataset: prod
@@ -849,7 +851,7 @@ paths:
849851
content:
850852
application/json:
851853
schema:
852-
$ref: '../../schema/system/integration-instance.schema'
854+
$ref: '../../src/main/resources/schema/system/integration-instance.schema'
853855
example:
854856
template-name: nginx
855857
dataset: prod
@@ -893,7 +895,7 @@ paths:
893895
content:
894896
application/json:
895897
schema:
896-
$ref: '../../schema/system/integration-instance.schema'
898+
$ref: '../../src/main/resources/schema/system/integration-instance.schema'
897899
example:
898900
template-name: nginx
899901
dataset: prod
@@ -932,7 +934,7 @@ paths:
932934
content:
933935
application/json:
934936
schema:
935-
$ref: '../../schema/system/integration-instance.schema'
937+
$ref: '../../src/main/resources/schema/system/integration-instance.schema'
936938
example:
937939
template-name: nginx
938940
dataset: prod
@@ -965,7 +967,7 @@ paths:
965967
content:
966968
application/json:
967969
schema:
968-
$ref: '../../schema/system/integration-instance.schema'
970+
$ref: '../../src/main/resources/schema/system/integration-instance.schema'
969971
example:
970972
template-name: nginx
971973
dataset: prod
@@ -994,7 +996,7 @@ paths:
994996
content:
995997
application/json:
996998
schema:
997-
$ref: '../../schema/system/integration-instance.schema'
999+
$ref: '../../src/main/resources/schema/system/integration-instance.schema'
9981000
example:
9991001
template-name: nginx
10001002
dataset: prod
@@ -1034,7 +1036,7 @@ paths:
10341036
content:
10351037
application/json:
10361038
schema:
1037-
$ref: '../../schema/system/integration-instance.schema'
1039+
$ref: '../../src/main/resources/schema/system/integration-instance.schema'
10381040
example:
10391041
template-name: nginx
10401042
dataset: prod

‎docs/Integration-API.md‎

Lines changed: 29 additions & 29 deletions
Original file line numberDiff line numberDiff line change
@@ -23,49 +23,49 @@ Query:
2323
GET _integration/repository?filter=category:logs&component:web
2424
```
2525
**Response**:
26-
```jsoon
26+
```json
2727
{
28-
"name": "nginx",
28+
"template-name": "nginx",
2929
"version": {
30-
"integ": "0.1.0",
30+
"integration": "0.1.0",
3131
"schema": "1.0.0",
3232
"resource": "^1.23.0"
3333
},
3434
"description": "Nginx HTTP server collector",
3535
"Information":"file:///.../schema/logs/info.html",
3636
"identification": "instrumentationScope.attributes.identification",
3737
"catalog": "observability",
38-
"components": [
39-
"web","http"
40-
],
4138
"collection":[
4239
{
43-
"logs": [{
40+
"category": "logs",
41+
"components": [
42+
"communication","http"
43+
],
44+
"feeds": [{
4445
"info": "access logs",
4546
"input_type":"logfile",
4647
"dataset":"nginx.access",
47-
"labels" :["nginx","access"],
48-
"schema": "file:///.../schema/logs/access.json"
48+
"labels" :["nginx","access"]
4949
},
5050
{
5151
"info": "error logs",
5252
"input_type":"logfile",
5353
"labels" :["nginx","error"],
54-
"dataset":"nginx.error",
55-
"schema": "file:///.../schema/logs/error.json"
54+
"dataset":"nginx.error"
5655
}]
5756
},
5857
{
59-
"metrics": [{
58+
"category": "metrics",
59+
"components": [],
60+
"feeds": [{
6061
"info": "status metrics",
6162
"input_type":"metrics",
6263
"dataset":"nginx.status",
63-
"labels" :["nginx","status"],
64-
"schema": "file:///.../schema/logs/status.json"
64+
"labels" :["nginx","status"]
6565
}]
6666
}
6767
],
68-
"repo": {
68+
"repository": {
6969
"github": "https://github.com/opensearch-project/observability/tree/main/integrarions/nginx"
7070
}
7171
}
@@ -139,49 +139,49 @@ POST _integration/store/$instance_name
139139
The $instance_name represents the specific name the integration was instanced with - for example, Nginx Integration can be a template for multiple Nginx instances
140140
each representing different domain / aspect such as geographic.
141141

142-
```jsoon
142+
```json
143143
{
144144
"template-name": "nginx",
145145
"version": {
146-
"integ": "0.1.0",
146+
"integration": "0.1.0",
147147
"schema": "1.0.0",
148148
"resource": "^1.23.0"
149149
},
150150
"description": "Nginx HTTP server collector",
151151
"Information":"file:///.../schema/logs/info.html",
152152
"identification": "instrumentationScope.attributes.identification",
153153
"catalog": "observability",
154-
"components": [
155-
"web","http"
156-
],
157154
"collection":[
158155
{
159-
"logs": [{
156+
"category": "logs",
157+
"components": [
158+
"communication","http"
159+
],
160+
"feeds": [{
160161
"info": "access logs",
161162
"input_type":"logfile",
162163
"dataset":"nginx.access",
163-
"labels" :["nginx","access"],
164-
"schema": "file:///.../schema/logs/access.json"
164+
"labels" :["nginx","access"]
165165
},
166166
{
167167
"info": "error logs",
168168
"input_type":"logfile",
169169
"labels" :["nginx","error"],
170-
"dataset":"nginx.error",
171-
"schema": "file:///.../schema/logs/error.json"
170+
"dataset":"nginx.error"
172171
}]
173172
},
174173
{
175-
"metrics": [{
174+
"category": "metrics",
175+
"components": [],
176+
"feeds": [{
176177
"info": "status metrics",
177178
"input_type":"metrics",
178179
"dataset":"nginx.status",
179-
"labels" :["nginx","status"],
180-
"schema": "file:///.../schema/logs/status.json"
180+
"labels" :["nginx","status"]
181181
}]
182182
}
183183
],
184-
"repo": {
184+
"repository": {
185185
"github": "https://github.com/opensearch-project/observability/tree/main/integrarions/nginx"
186186
}
187187
}

‎docs/Integration-plugin-tasks.md‎

Lines changed: 15 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -93,14 +93,23 @@ _For example the next observability integration:_
9393
...
9494
"collection":[
9595
{
96-
"logs": [{
96+
"category": "logs",
97+
"components": [
98+
"communication","http"
99+
],
100+
"feeds": [{
97101
"info": "access logs",
98-
"input_type":"logfile",
99-
"dataset":"nginx.access", <<- subject to user changes
100-
"namespace": "prod",<<- subject to user changes
101-
"labels" :["nginx","access"],
102-
"schema": "file:///.../schema/logs/access.json"
102+
"input_type":"logfile",
103+
"dataset":"nginx.access",
104+
"labels" :["nginx","access"]
103105
},
106+
{
107+
"info": "error logs",
108+
"input_type":"logfile",
109+
"labels" :["nginx","error"],
110+
"dataset":"nginx.error"
111+
}]
112+
},
104113
...
105114
```
106115
### Loading Step

0 commit comments

Comments
 (0)