Skip to content

Commit 5ebc491

Browse files
auhligauhlig
committed
fix name mapping. use path_keywords.
1 parent 4bdebb8 commit 5ebc491

37 files changed

+2096
-2883
lines changed

‎ChangeLog‎

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,64 @@
11
CHANGES
22
=======
33

4+
* fix name mapping. use path\_keywords
5+
6+
1.0.14
7+
------
8+
9+
* neturon name mapping
10+
11+
1.0.13
12+
------
13+
14+
* fix cinder, nova name mapping
15+
16+
1.0.12
17+
------
18+
19+
* fix manila mapping, regex
20+
21+
1.0.11
22+
------
23+
24+
* add ironic support and tests
25+
26+
1.0.10
27+
------
28+
29+
* manila, nova name lookup fix
30+
* add doc
31+
32+
1.0.9
33+
-----
34+
35+
* add manila mapping and tests
36+
* add pep8 style check
37+
* add build status
38+
* add travis
39+
40+
1.0.8
41+
-----
42+
43+
* glance naming
44+
* fix keystone target\_type\_uri name mapping
45+
* fix keystone test
46+
47+
1.0.7
48+
-----
49+
50+
* enhanced the keystone uri strategy to properly map name lookups to target\_type\_uri
51+
52+
1.0.6
53+
-----
54+
55+
* readme
56+
* Latency and status (#3)
57+
* enable check body of authentication requests
58+
* client\_addr->host\_address
59+
* add /endpoints and test
60+
* [swift] it's not /v1/info but /info
61+
* dry requirements
462
* bump to 1.0.0
563
* fix logging
664
* default target.project\_id to initiator.project\_id if discovery fails

‎README.md‎

Lines changed: 60 additions & 157 deletions
Original file line numberDiff line numberDiff line change
@@ -14,19 +14,21 @@ The OpenStack Watcher is a WSGI middleware capable of analyzing OpenStack traffi
1414

1515
## Principles
1616

17-
The watcher distinguishes between `initiator` and `target` of an action.
18-
`Initiator` describes the resource or the user that starts the request, ` Target` refers to the resource against which the action was performed.
17+
The watcher middleware classifies OpenStack requests based on the Cloud Auditing Data Federation (CADF) specification.
18+
It distinguishes between `initiator` and `target` of an action.
19+
`Initiator` describes the resource or the user who sent the request, `Target` refers to the resource against which the action was performed.
1920

2021
### CADF Specification
2122

22-
Requests are classified according to the CADF specification.
23-
A comprehensive list of of OpenStack requests and their CADF representation can be found here: [Cloud Audit Data Federation - OpenStack Profile (CADF-OpenStack)](https://www.dmtf.org/sites/default/files/standards/documents/DSP2038_1.1.0.pdf).
24-
This middleware follows DMTF specification DSP2038, version 1.1.0 as of 27 April 2015.
23+
The Cloud Audit Data Federation (CADF) specification defines a model for events within the OpenStack platform.
24+
This data model is used by the watcher middleware to classify requests.
25+
More information is provided in the [documentation](./doc/cadf.md).
2526

2627
#### Classification
2728

28-
The watcher classifies requests and records the following attributes.
29-
These are emitted as Prometheus metrics and passed in the GCI environment with the `WATCHER` prefix and in capital letters.
29+
The following attributes are recorded and passed via the WSGI environment through the pipeline.
30+
Moreover, this meta data is emitted as Prometheus metrics.
31+
Note: The attributes in the environment are capitalized.
3032
For example: `WATCHER.ACTION`, `WATCHER.INITIATOR_PROJECT_ID`, `WATCHER.TARGET_PROJECT_ID`, etc. .
3133

3234
- `action`: the CADF action
@@ -48,141 +50,15 @@ For example: `WATCHER.ACTION`, `WATCHER.INITIATOR_PROJECT_ID`, `WATCHER.TARGET_P
4850

4951
- Swift (object-store):
5052
- `target.container_id`: the name/id of the swift container. `None` if not relevant. `Unknown` if it could not be determined.
51-
52-
53-
#### Determine target project
54-
55-
Determining the target of an operation can be hard. The watcher offers 3 mutually exclusive approaches to that:
56-
1. Extract target project id from token.
57-
Assumptions: Initiator is authenticated. Initiator and target are in the same project.
58-
In which case the initiator.project_id as seen in the keystone token will be equal to the target.project_id.
59-
While this may be correct in most cases, if the services policy allows cross-project operations, one of the following approaches is suggested.
60-
61-
2. Extract target project id from request path.
62-
Assumptions: Initiator may be anonymous (not authenticated). The request path contains the project id.
63-
For example swift (object-store) requests might contain the uid of the target project id in the format `../<version>/AUTH_<target_project_id>/..`
64-
This does not allow conclusions on the initiator, but on the target.
65-
66-
3. Extract target project id from service catalog.
67-
Assumption: Initiator is authenticated. The token is scoped to the target project and its service catalog contains endpoint(s) for the relevant service
68-
that include the project id in the format `http(s)://<service>:<port>/<version>/<target_project_id>`.
69-
This requires `include_service_catalog = true` in the keystone.auth_token middleware and does not work when unauthenticated requests are allowed.
70-
See section [keystone auth_token middleware](#keystone-auth_token-middleware).
71-
72-
#### CADF actions
73-
74-
Actions characterize the operation performed by the initiator of a request against a target.
75-
A comprehensive definition of these actions is provided by the CADF specification mentioned above.
76-
The watcher is capable of classifying request actions based on the HTTP method and path as follows:
77-
```
78-
|---------------|-------------------|-------------------|
79-
| HTTP method | Path | Action |
80-
|---------------|-------------------|-------------------|
81-
| GET | | read |
82-
| GET | ../detail | read/list |
83-
| HEAD | | read |
84-
| PUT | | update |
85-
| PATCH | | update |
86-
| POST | | create |
87-
| POST | ../auth/tokens | authenticate |
88-
| DELETE | | delete |
89-
| COPY | | create/copy |
90-
|---------------|-------------------|-------------------|
91-
```
92-
93-
Using this mapping the watcher is capable of classifying request actions correctly in most cases.
94-
However, some cases require a different mapping to alternative actions.
95-
The default classification can be overwritten using a custom_actions mapping
96-
97-
An example for swift (object-store) looks like this:
98-
```yaml
99-
custom_actions:
100-
account:
101-
- method: GET
102-
action_type: read/list
103-
- method: POST
104-
action_type: update
105-
106-
- container:
107-
- method: GET
108-
action_type: read/list
109-
- method: POST
110-
action_type: update
111-
112-
- object:
113-
- method: POST
114-
action_type: update
115-
```
116-
This configuration results in the following mapping:
117-
```
118-
|---------------|-----------------------------------|-------------------|
119-
| HTTP method | Path | Action |
120-
|---------------|-----------------------------------|-------------------|
121-
| GET | ../v1/account | read/list |
122-
| POST | ../v1/account | update |
123-
| GET | ../v1/account/container | read/list |
124-
| POST | ../v1/account/container | update |
125-
| POST | ../v1/account/container/object | update |
126-
| ... | | |
127-
|---------------|-----------------------------------|-------------------|
128-
````
129-
130-
#### Action requests
131-
132-
Some requests may use `POST` or `PUT` with a path including `/action` or `/os-instance-actions` and a json body to perform an action on a resource.
133-
In which case the middleware evaluates the request body to determine the action.
134-
The default mapping will return a CADF action in the following format: `update/<action>`.
135-
A `os-` prefix will be trimmed from the action.
136-
137-
Example: Nova (compute) add security group to an instance and reset state
138-
````
139-
|---------------|-----------------------------------|-------------------------------------------|---------------------------|
140-
| HTTP method | Path | JSON body | CADF action |
141-
|---------------|-----------------------------------|-------------------------------------------|---------------------------|
142-
| POST | /servers/{server_id}/action | { "addSecurityGroup": { "name": "test" }} | update/addSecurityGroup |
143-
| POST | /servers/{server_id}/action | { "os-resetState": { "state": "active" }} | update/resetState |
144-
| ... | ... | | |
145-
|---------------|-----------------------------------|-------------------------------------------|---------------------------|
146-
````
147-
148-
The default behaviour can be overwritten by providing the following configuration:
149-
```
150-
custom_actions:
151-
servers:
152-
server:
153-
action:
154-
- <original_action>: <cadf_action>
155-
- addSecurityGroup: update/addSecurityGroup
156-
- os-resetState: update/os-resetState
157-
```
158-
159-
#### CADF target type URI
160-
161-
The *target type URI* is a CADF specific representation of the request's target URI consisting of
162-
a service-specific prefix and the target URI without version or UUID strings.
163-
164-
In most cases this middleware builds the target type URI by concatenating the `service/<service_type>` prefix and
165-
all parts of the target URI which do not contain a UUID.
166-
In case a UUID is found, it's substituted by the singular of the previous part.
167-
Should this default behaviour not suffice, writing a [custom strategy](./watcher/target_type_uri_strategy.py) might be required.
168-
169-
Examples:
170-
```
171-
|-------------|-------------------------------------------------|-----------------------------------------------|
172-
| Service | Target URI | CADF Target Type URI |
173-
|-------------|-------------------------------------------------|-----------------------------------------------|
174-
| compute | /servers/{server_uuid}/action | service/compute/servers/server/action |
175-
| dns | /v2/zones/{zone_id}/recordsets/{recordset_id} | service/dns/zones/zone/recordsets/recordset |
176-
| ... | ... | ... |
177-
|-------------|-------------------------------------------------|-----------------------------------------------|
178-
```
17953

18054
### Metrics
18155

18256
The openstack-watcher-middleware exposes the following Prometheus metrics via statsD.
18357

184-
`openstack_watcher_api_requests_total` - total count of api requests
185-
`openstack_watcher_api_requests_duration_seconds` - request latency
58+
`openstack_watcher_api_requests_total` - total count of api requests
59+
`openstack_watcher_api_requests_duration_seconds` - request latency in seconds
60+
`openstack_watcher_api_requests_duration_seconds_count` - total number of samples of the request duration metric
61+
`openstack_watcher_api_requests_duration_seconds_sum` - sum of request latency
18662

18763
## Supported Services
18864

@@ -201,6 +77,7 @@ This middleware currently provides CADF-compliant support for the following Open
20177
| Ironic | baremetal |
20278
|-----------------------|-----------------------|
20379
````
80+
20481
Configurations for these services are provided [here](./etc)
20582
Support for additional OpenStack services might require additional action configurations.
20683

@@ -211,29 +88,16 @@ Install via
21188
pip install git+https://github.com/sapcc/openstack-watcher-middleware.git
21289
```
21390

214-
### Keystone auth_token middleware
215-
216-
The watcher determines the initiator of an action via its keystone token,
217-
thus the middleware needs to be added *after* the [keystone.auth_token middleware](https://docs.openstack.org/keystone/queens/admin/identity-auth-token-middleware.html).
218-
Setting `include_service_catalog = true` includes the service catalog on token validation,
219-
which enables the watcher to determine the target project id for a service based on the endpoint(s) found in this scoped catalog.
220-
221-
**Note**:
222-
Setting`delay_auth_decision = true`, configures the auth_token middleware to delegate the authorization decision to downstream WSGI components.
223-
This enables unauthenticated requests, hence the initators project, domain and user uid *cannot* be determined via the keystone token and are `Unknown`.
224-
In which case the initiator can only be characterized by its client address.
225-
Furthermore, the *target.project_id* cannot be extracted from the token nor the service catalog.
226-
If the request path does not include the target.project_id, it will be `Unknown`.
227-
22891
### Pipeline
22992

230-
The watcher should be added after the auth_token middleware.
93+
The watcher should be added after the keystone auth_token middleware to be able to obtain information on the scope (project/domain) of the action.
23194
```
23295
pipeline = .. auth_token watcher ..
23396
```
23497

235-
### Watcher configuration
236-
and configure by adding the following snippet to the paste.ini:
98+
### WSGI configuration
99+
100+
Configuration options in the paste.ini as shown below
237101
```yaml
238102
[filter:watcher]
239103
use = egg:watcher-middleware#watcher
@@ -252,15 +116,54 @@ project_id_from_path = true | false
252116
# determine the project id from the service catalog
253117
project_id_from_service_catalog = true | false
254118

255-
# per default the target.type_uri is prefixed by 'service/<service_type>/'.
119+
# per default the target.type_uri is prefixed by 'service/<service_type>/'
256120
# if the cadf spec. requires a different prefix, it might be given here
257121
# example: swift (object-store)
258-
# service_type = object-store would result in 'service/object-store/', but cadf requires 'service/storage/object/',
259-
# so one needs to set cadf_service_name = service/storage/object
122+
# service_type = object-store
123+
# cadf_service_name = service/storage/object
260124
cadf_service_name = <service_name>
261125

262126
# metrics are emitted via StatsD
263127
statsd_host = 127.0.0.1
264128
statsd_port = 9125
265129
statsd_namespace = openstack_watcher
266130
```
131+
132+
#### Configuration file
133+
134+
Additionally, the watcher might require a configuration file.
135+
For existing services these can be found in the [examples](./etc).
136+
More details are provided [here](./doc/cadf.md)
137+
138+
The following snippet provides an overview of configuration options for a service.
139+
```yaml
140+
# keywords in a request path are followed by the UUID or name of a resource.
141+
# in the target type URI the UUID or name of a resource is replaced by the singular of the keyword.
142+
# a custom value for the singular can also be provided by a mapping <plural>:<singular>
143+
# moreover, if a path ends with a keyword the action will be 'read/list'
144+
path_keywords:
145+
- users
146+
- tokens
147+
- availability-zones: zone
148+
..
149+
150+
# per default every word following a path keyword is replaced.
151+
# however, exclusions can be configured by this list
152+
path_exclusions:
153+
- OS-PKI
154+
- ..
155+
156+
# some request path' are quite hard to map to their target type URI as the replacements can't be derived from the previous part.
157+
# thus, in some cases providing a mapping of <path_regex>: <target_type_URI> might be inevitable
158+
# note: the complete path (including versions, etc. ) needs to be reflected in the regex
159+
regex_path_mapping:
160+
- '\S+/domains/config/[0-9a-zA-Z_]+/default$': 'domains/config/group/default'
161+
162+
# CADF actions are determined by the request method and their path as outlined in the table in the CADF section of this documentation
163+
# however, these can be overwritten using the target type URI and the request method with a custom action_type
164+
custom_actions:
165+
tokens:
166+
- token:
167+
- method: GET
168+
action_type: custom_action
169+
```

0 commit comments

Comments
 (0)