Skip to content

Commit 7a76ba2

Browse files
tommyers-elastictommyers-elastic
authored
add new 'tenant_settings' data stream to o365_metrics integration (#13170)
* add new 'tenant_settings' data stream * add system test * fix failed test by adding "input.type" mapping * tidy up: add changelog, bump version, add sample event * add table entry in README.md * add processor option in data stream maniftest * fix typo in README * remove unused settings (no need for preserve original event - there's no ingest processing happening) * add required permissions to README * add ingest pipeline processor to set latest ECS version * fix ingest pipeline format * run system test for at least 2 collection periods
1 parent 6f2611d commit 7a76ba2

File tree

13 files changed

+364
-6
lines changed

13 files changed

+364
-6
lines changed

‎packages/o365_metrics/_dev/build/docs/README.md‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,8 +6,8 @@ This integration uses the [Microsoft Graph API](https://learn.microsoft.com/en-u
66

77
Following Microsoft 365 Graph Reports can be collected by Microsoft Office 365 Metrics integration.
88

9-
| Report | API | Data-stream Name | Aggregation Level |
10-
|-----------------|-----|-------------|-------------------|
9+
| Report | API | Data-stream Name | Aggregation Level | Required permissions
10+
|-----------------|-----|------------------|-------------------|--------------------|
1111
| [Microsoft 365 Active Users Service User Counts](https://learn.microsoft.com/en-us/microsoft-365/admin/activity-reports/active-users-ww?view=o365-worldwide) | [reportRoot: getOffice365ServicesUserCounts](https://learn.microsoft.com/en-us/graph/api/reportroot-getoffice365servicesusercounts?view=graph-rest-1.0&tabs=http) | Microsoft 365 Active Users metrics | `Period`-based |
1212
| [Microsoft 365 Groups Activity Group Detail](https://learn.microsoft.com/en-us/microsoft-365/admin/activity-reports/office-365-groups-ww?view=o365-worldwide) | [reportRoot: getOffice365GroupsActivityDetail](https://learn.microsoft.com/en-us/graph/api/reportroot-getoffice365groupsactivitydetail?view=graph-rest-1.0&tabs=http) | Microsoft 365 Groups Activity Group Detail | `Day`-based |
1313
| [OneDrive Usage Account Detail](https://learn.microsoft.com/en-us/microsoft-365/admin/activity-reports/onedrive-for-business-usage-ww?view=o365-worldwide) | [reportRoot: getOneDriveUsageAccountDetail](https://learn.microsoft.com/en-us/graph/api/reportroot-getonedriveusageaccountdetail?view=graph-rest-1.0&tabs=http) | Microsoft 365 OneDrive Usage Account Detail | `Day`-based |
@@ -28,6 +28,7 @@ Following Microsoft 365 Graph Reports can be collected by Microsoft Office 365 M
2828
| [Service Health](https://learn.microsoft.com/en-us/graph/service-communications-concept-overview?view=o365-worldwide) | [reportRoot: getServiceHealth](https://learn.microsoft.com/en-us/graph/api/servicehealth-get?view=graph-rest-1.0&tabs=http) | Microsoft 365 Service Health metrics | No aggregation |
2929
| [Subscriptions](https://learn.microsoft.com/en-us/graph/api/resources/subscribedsku?view=graph-rest-1.0?view=o365-worldwide) | [reportRoot: subscribedSkus](https://learn.microsoft.com/en-us/graph/api/subscribedsku-list?view=graph-rest-1.0&tabs=http) | Microsoft 365 Subscriptions metrics | No aggregation |
3030
| [Teamms Call Quality](https://learn.microsoft.com/en-us/graph/api/resources/communications-api-overview?view=graph-rest-1.0?view=o365-worldwide) | [reportRoot: callRecords](https://learn.microsoft.com/en-us/graph/api/callrecords-callrecord-list-sessions?view=graph-rest-1.0&tabs=http) | Microsoft 365 Teams Call Quality metrics | No aggregation |
31+
| Tenant Settings | [organization](https://learn.microsoft.com/en-us/graph/api/resources/organization?view=graph-rest-1.0), [adminReportSettings](https://learn.microsoft.com/en-us/graph/api/resources/adminreportsettings?view=graph-rest-1.0) | Microsoft 365 Tenant Settings | No aggregation | Organization.Read.All, ReportSettings.Read.All
3132

3233
## Setup
3334

‎packages/o365_metrics/_dev/deploy/docker/docker-compose.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,4 +12,4 @@ services:
1212
command:
1313
- http-server
1414
- --addr=:8090
15-
- --config=/files/config.yml
15+
- --config=/files/config.yml

‎packages/o365_metrics/_dev/deploy/docker/files/config.yml‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -283,4 +283,44 @@ rules:
283283
]
284284
}
285285
]
286+
}
287+
- path: /admin/reportSettings
288+
methods: ['GET']
289+
request_headers:
290+
Authorization:
291+
- "Bearer xxxx"
292+
responses:
293+
- status_code: 200
294+
headers:
295+
Content-Type:
296+
- 'application/json'
297+
body: |-
298+
{
299+
"value": {
300+
"@odata.type": "#microsoft.graph.adminReportSettings",
301+
"displayConcealedNames": true
302+
}
303+
}
304+
- path: /organization
305+
methods: ['GET']
306+
query_params:
307+
'$select': 'id,displayName,tenantType'
308+
request_headers:
309+
Authorization:
310+
- "Bearer xxxx"
311+
responses:
312+
- status_code: 200
313+
headers:
314+
Content-Type:
315+
- 'application/json'
316+
body: |-
317+
{
318+
"@odata.context": "https://graph.microsoft.com/v1.0/$metadata#organization(id,displayName,tenantType)",
319+
"value": [
320+
{
321+
"id": "aa40685b-417d-4664-b4ec-8f7640719adb",
322+
"displayName": "azure2",
323+
"tenantType": "AAD"
324+
}
325+
]
286326
}

‎packages/o365_metrics/changelog.yml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,9 @@
11
# newer versions go on top
2+
- version: "0.7.0"
3+
changes:
4+
- description: Add 'tenant settings' data stream.
5+
type: enhancement
6+
link: https://github.com/elastic/integrations/pull/13170
27
- version: "0.6.6"
38
changes:
49
- description: Add support for configuring resource parameters, such as rate_limit, retry, and others.

‎packages/o365_metrics/data_stream/tenant_settings/_dev/test/system/test-default-config.yml‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
input: cel
2+
service: o365_metrics
3+
vars:
4+
url: http://{{Hostname}}:{{Port}}
5+
token_url: http://{{Hostname}}:{{Port}}
6+
azure_tenant_id: "1234"
7+
client_id: "1234"
8+
client_secret: "1234"
9+
data_stream:
10+
vars:
11+
interval: 5s
12+
preserve_original_event: true
13+
assert:
14+
hit_count: 2
15+
fields_present:
16+
- o365.metrics.tenant_settings.display_concealed_names
17+
- o365.metrics.tenant_settings.tenant.display_name
18+
- o365.metrics.tenant_settings.tenant.id
19+
- o365.metrics.tenant_settings.tenant.type

‎packages/o365_metrics/data_stream/tenant_settings/agent/stream/cel.yml.hbs‎

Lines changed: 138 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,138 @@
1+
interval: {{interval}}
2+
{{#if enable_request_tracer}}
3+
resource.tracer.filename: "../../logs/cel/http-request-trace-*.ndjson"
4+
resource.tracer.maxbackups: 5
5+
resource.tracer.maxsize: 5
6+
{{/if}}
7+
{{#if proxy_url}}
8+
resource.proxy_url: {{proxy_url}}
9+
{{/if}}
10+
{{#if resource_ssl}}
11+
resource.ssl:
12+
{{resource_ssl}}
13+
{{/if}}
14+
{{#if resource_timeout}}
15+
resource.timeout: {{resource_timeout}}
16+
{{/if}}
17+
{{#if resource_retry_max_attempts}}
18+
resource.retry.max_attempts: {{resource_retry_max_attempts}}
19+
{{/if}}
20+
{{#if resource_retry_wait_min}}
21+
resource.retry.wait_min: {{resource_retry_wait_min}}
22+
{{/if}}
23+
{{#if resource_retry_wait_max}}
24+
resource.retry.wait_max: {{resource_retry_wait_max}}
25+
{{/if}}
26+
{{#if resource_redirect_forward_headers}}
27+
resource.redirect.forward_headers: {{resource_redirect_forward_headers}}
28+
{{/if}}
29+
{{#if resource_redirect_headers_ban_list}}
30+
resource.redirect.headers_ban_list:
31+
{{#each resource_redirect_headers_ban_list as |item|}}
32+
- {{item}}
33+
{{/each}}
34+
{{/if}}
35+
{{#if resource_redirect_max_redirects}}
36+
resource.redirect.max_redirects: {{resource_redirect_max_redirects}}
37+
{{/if}}
38+
{{#if resource_rate_limit_limit}}
39+
resource.rate_limit.limit: {{resource_rate_limit_limit}}
40+
{{/if}}
41+
{{#if resource_rate_limit_burst}}
42+
resource.rate_limit.burst: {{resource_rate_limit_burst}}
43+
{{/if}}
44+
resource.url: {{url}}
45+
auth.oauth2:
46+
client.id: {{client_id}}
47+
client.secret: {{client_secret}}
48+
provider: azure
49+
scopes:
50+
{{#each token_scopes as |token_scope|}}
51+
- {{token_scope}}
52+
{{/each}}
53+
endpoint_params:
54+
grant_type: client_credentials
55+
{{#if token_url}}
56+
token_url: {{token_url}}/{{azure_tenant_id}}/oauth2/v2.0/token
57+
{{else if azure_tenant_id}}
58+
azure.tenant_id: {{azure_tenant_id}}
59+
{{/if}}
60+
redact:
61+
fields: ~
62+
state:
63+
admin_report_settings_path: '/admin/reportSettings'
64+
orgs_path: '/organization?$select=id,displayName,tenantType'
65+
program: |
66+
state.with(
67+
request("GET", state.url.trim_right("/") + state.admin_report_settings_path).do_request().as(admin_report_settings_resp,
68+
admin_report_settings_resp.StatusCode == 200
69+
?
70+
bytes(admin_report_settings_resp.Body).decode_json().as(admin_report_settings_json,
71+
request("GET", state.url.trim_right("/") + state.orgs_path).do_request().as(orgs_resp,
72+
orgs_resp.StatusCode == 200
73+
?
74+
bytes(orgs_resp.Body).decode_json().as(orgs_json,
75+
{
76+
"events": [{
77+
"o365": {
78+
"metrics": {
79+
"tenant_settings": {
80+
"display_concealed_names": admin_report_settings_json.value.displayConcealedNames,
81+
"tenant": {
82+
"display_name": orgs_json.value[0].displayName,
83+
"id": orgs_json.value[0].id,
84+
"type": orgs_json.value[0].tenantType
85+
}
86+
}
87+
}
88+
}
89+
}]
90+
}
91+
)
92+
:
93+
{
94+
"events": {
95+
"error": {
96+
"code": string(orgs_resp.StatusCode),
97+
"id": string(orgs_resp.Status),
98+
"message": "GET " + state.orgs_path + ": " + (
99+
size(orgs_resp.Body) != 0 ?
100+
string(orgs_resp.Body)
101+
:
102+
string(orgs_resp.Status) + ' (' + string(orgs_resp.StatusCode) + ')'
103+
),
104+
},
105+
},
106+
"want_more": false,
107+
}
108+
)
109+
)
110+
:
111+
{
112+
"events": {
113+
"error": {
114+
"code": string(admin_report_settings_resp.StatusCode),
115+
"id": string(admin_report_settings_resp.Status),
116+
"message": "GET " + state.admin_report_settings_path + ": " + (
117+
size(admin_report_settings_resp.Body) != 0 ?
118+
string(admin_report_settings_resp.Body)
119+
:
120+
string(admin_report_settings_resp.Status) + ' (' + string(admin_report_settings_resp.StatusCode) + ')'
121+
),
122+
},
123+
},
124+
"want_more": false,
125+
}
126+
)
127+
)
128+
tags:
129+
{{#each tags as |tag|}}
130+
- {{tag}}
131+
{{/each}}
132+
{{#contains "forwarded" tags}}
133+
publisher_pipeline.disable_host: true
134+
{{/contains}}
135+
{{#if processors}}
136+
processors:
137+
{{processors}}
138+
{{/if}}

‎packages/o365_metrics/data_stream/tenant_settings/elasticsearch/ingest_pipeline/default.yml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
---
2+
description: Ingest pipeline for o365_metrics 'tenant_settings' data stream.
3+
processors:
4+
- set:
5+
field: ecs.version
6+
value: "8.17.0"

‎packages/o365_metrics/data_stream/tenant_settings/fields/base-fields.yml‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
- name: data_stream.type
2+
type: constant_keyword
3+
description: Data stream type.
4+
- name: data_stream.dataset
5+
type: constant_keyword
6+
description: Data stream dataset.
7+
- name: data_stream.namespace
8+
type: constant_keyword
9+
description: Data stream namespace.
10+
- name: '@timestamp'
11+
type: date
12+
description: Event timestamp.
13+
- name: input.type
14+
type: keyword
15+
description: Input type.

‎packages/o365_metrics/data_stream/tenant_settings/fields/fields.yml‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
- name: o365.metrics.tenant_settings
2+
type: group
3+
fields:
4+
- name: display_concealed_names
5+
type: boolean
6+
description: |
7+
If set to true, all reports conceal user information such as usernames, groups, and sites. If false, all reports show identifiable information. This property represents a setting in the Microsoft 365 admin center.
8+
- name: tenant
9+
type: group
10+
fields:
11+
- name: display_name
12+
type: keyword
13+
description: |
14+
The display name for the tenant.
15+
- name: id
16+
type: keyword
17+
description: |
18+
The tenant ID, a unique identifier representing the organization (or tenant).
19+
- name: type
20+
type: keyword
21+
description: |
22+
Can be one of the following types:
23+
* AAD - An enterprise identity access management (IAM) service that serves business-to-employee and business-to-business (B2B) scenarios.
24+
* AAD B2C An identity access management (IAM) service that serves business-to-consumer (B2C) scenarios.
25+
* CIAM - A customer identity & access management (CIAM) solution that provides an integrated platform to serve consumers, partners, and citizen scenarios.

‎packages/o365_metrics/data_stream/tenant_settings/manifest.yml‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
title: "Tenant settings"
2+
type: metrics
3+
streams:
4+
- input: cel
5+
title: Microsoft 365 tenant settings
6+
enabled: false
7+
description: Collect information about tenant-level settings for Microsoft 365.
8+
template_path: cel.yml.hbs
9+
vars:
10+
- name: interval
11+
type: text
12+
title: Interval
13+
description: The interval at which the API is polled, supported in seconds, minutes, and hours.
14+
show_user: true
15+
required: true
16+
default: 24h
17+
- name: tags
18+
type: text
19+
title: Tags
20+
multi: true
21+
required: false
22+
show_user: false
23+
default:
24+
- o365.metrics.tenant_settings
25+
- name: enable_request_tracer
26+
type: bool
27+
title: Enable request tracing
28+
multi: false
29+
required: false
30+
show_user: false
31+
description: >-
32+
The request tracer logs HTTP requests and responses to the agent's local file-system for debugging configurations. Enabling this request tracing compromises security and should only be used for debugging. See [documentation](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-cel.html#_resource_tracer_filename) for details.
33+
- name: processors
34+
type: yaml
35+
title: Processors
36+
multi: false
37+
required: false
38+
show_user: false
39+
description: >
40+
Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See [Processors](https://www.elastic.co/guide/en/fleet/current/elastic-agent-processor-configuration.html) for details.

0 commit comments

Comments
 (0)