Skip to content

Commit 7ba1791

Browse files
tommyers-elastictommyers-elastic
authored
[o365_metrics] entra id users data stream (#13300)
* add scaffolding for new o365_metrics 'entra_id_users' data stream * add initial /users API call and system test config * add risk information for risky users * change from 'riskUsers' to 'riskDetections' API to reduce licensing constraints (P2 vs P1) * add additional sync error fields * simplify CEL code a little * update README, manifest, changelog. add sample_event * fix issue in CEL program that fails subsequent collections * run system test for at least 2 collection periods * rename 'occurred' -> 'occurred_date_time' * change 'nested' mapping to 'group' for consistency with other datastreams in this package * add section with example event and metrics to docs * change system test to handle new mapping type for `on_premises_provisioning_errors`
1 parent 1bf041f commit 7ba1791

File tree

12 files changed

+676
-5
lines changed

12 files changed

+676
-5
lines changed

‎packages/o365_metrics/_dev/build/docs/README.md‎

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -30,7 +30,7 @@ Following Microsoft 365 Graph Reports can be collected by Microsoft Office 365 M
3030
| [Teamms Call Quality](https://learn.microsoft.com/en-us/graph/api/resources/communications-api-overview?view=graph-rest-1.0?view=o365-worldwide) | [reportRoot: callRecords](https://learn.microsoft.com/en-us/graph/api/callrecords-callrecord-list-sessions?view=graph-rest-1.0&tabs=http) | Microsoft 365 Teams Call Quality metrics | No aggregation | CallRecords.Read.All |
3131
| Tenant Settings | [organization](https://learn.microsoft.com/en-us/graph/api/resources/organization?view=graph-rest-1.0), [adminReportSettings](https://learn.microsoft.com/en-us/graph/api/resources/adminreportsettings?view=graph-rest-1.0) | Microsoft 365 Tenant Settings | No aggregation | Organization.Read.All, ReportSettings.Read.All, Directory.Read.All |
3232
| [App Registrations](https://learn.microsoft.com/en-us/graph/api/resources/application?view=graph-rest-1.0) | [List Applications](https://learn.microsoft.com/en-us/graph/api/application-list?view=graph-rest-1.0&tabs=http) | Microsoft 365 App Registrations | No aggregation | Application.Read.All, User.Read(delegated) |
33-
33+
| Entra ID users | [user](https://learn.microsoft.com/en-us/graph/api/resources/user?view=graph-rest-1.0), [riskDetection](https://learn.microsoft.com/en-us/graph/api/resources/riskdetection?view=graph-rest-1.0) | Microsoft 365 Entra ID user metrics | No aggregation | User.Read.All, IdentityRiskEvent.Read.All
3434

3535
## Setup
3636

@@ -110,6 +110,14 @@ Please refer to the following [document](https://www.elastic.co/guide/en/ecs/cur
110110

111111
{{fields "active_users_services_user_counts"}}
112112

113+
### Entra ID users
114+
115+
Get details about users in Microsoft Entra ID.
116+
117+
{{ event "entra_id_users" }}
118+
119+
{{ fields "entra_id_users" }}
120+
113121
### Mailbox Usage Quota Status
114122

115123
Get details about Mailbox Usage Quota Status from [Microsoft Graph API](https://learn.microsoft.com/en-us/graph/api/reportroot-getmailboxusagequotastatusmailboxcounts?view=graph-rest-1.0&tabs=http).
@@ -351,4 +359,4 @@ Get details about apps registered in Microsoft Entra ID. [Microsoft API](https:/
351359

352360
Please refer to the following [document](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html) for detailed information on ECS fields.
353361

354-
{{fields "app_registrations"}}
362+
{{fields "app_registrations"}}

‎packages/o365_metrics/_dev/deploy/docker/files/config.yml‎

Lines changed: 143 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -418,4 +418,146 @@ rules:
418418
]
419419
}
420420
]
421-
}
421+
}
422+
- path: /users
423+
methods: ['GET']
424+
query_params:
425+
'$select': 'id,userPrincipalName,userType,onPremisesProvisioningErrors,onPremisesSyncEnabled'
426+
request_headers:
427+
Authorization:
428+
- "Bearer xxxx"
429+
responses:
430+
- status_code: 200
431+
headers:
432+
Content-Type:
433+
- 'applciation/json'
434+
body: |-
435+
{
436+
"@odata.context": "https://graph.microsoft.com/v1.0/$metadata#users(id,userPrincipalName,userType)",
437+
"@odata.nextLink": "{{ env "SERVER_ADDRESS" }}/users?$select=id%2cuserPrincipalName%2cuserType%2conPremisesProvisioningErrors%2conPremisesSyncEnabled",
438+
"value": [
439+
{
440+
"id": "6e7b768e-07e2-4810-8459-485f84f8f204",
441+
"userPrincipalName": "[email protected]",
442+
"userType": "Member",
443+
"onPremisesProvisioningErrors": [],
444+
"onPremisesSyncEnabled": null
445+
},
446+
{
447+
"id": "87d349ed-44d7-43e1-9a83-5f2406dee5bd",
448+
"userPrincipalName": "[email protected]",
449+
"userType": "Member",
450+
"onPremisesProvisioningErrors": [],
451+
"onPremisesSyncEnabled": null
452+
}
453+
]
454+
}
455+
- status_code: 200
456+
headers:
457+
Content-Type:
458+
- 'applciation/json'
459+
body: |-
460+
{
461+
"@odata.context": "https://graph.microsoft.com/v1.0/$metadata#users(id,userPrincipalName,userType)",
462+
"@odata.nextLink": "{{ env "SERVER_ADDRESS" }}/users?$select=id%2cuserPrincipalName%2cuserType%2conPremisesProvisioningErrors%2conPremisesSyncEnabled",
463+
"value": [
464+
{
465+
"id": "5bde3e51-d13b-4db1-9948-fe4b109d11a7",
466+
"userPrincipalName": "[email protected]",
467+
"userType": "Member",
468+
"onPremisesProvisioningErrors": [],
469+
"onPremisesSyncEnabled": null
470+
},
471+
{
472+
"id": "4782e723-f4f4-4af3-a76e-25e3bab0d896",
473+
"userPrincipalName": "[email protected]",
474+
"userType": "Member",
475+
"onPremisesProvisioningErrors": [],
476+
"onPremisesSyncEnabled": null
477+
}
478+
]
479+
}
480+
- status_code: 200
481+
headers:
482+
Content-Type:
483+
- 'applciation/json'
484+
body: |-
485+
{
486+
"@odata.context": "https://graph.microsoft.com/v1.0/$metadata#users(id,userPrincipalName,userType)",
487+
"value": [
488+
{
489+
"id": "c03e6eaa-b6ab-46d7-905b-73ec7ea1f755",
490+
"userPrincipalName": "[email protected]",
491+
"userType": "Member",
492+
"onPremisesProvisioningErrors": [{
493+
"category": "PropertyConflict",
494+
"occurredDateTime": "2025-03-25T14:33:19Z",
495+
"propertyCausingError": "UserPrincipalName",
496+
"value": "[email protected]"
497+
}],
498+
"onPremisesSyncEnabled": true
499+
},
500+
{
501+
"id": "013b7b1b-5411-4e6e-bdc9-c4790dae1051",
502+
"userPrincipalName": "[email protected]",
503+
"userType": "Member",
504+
"onPremisesProvisioningErrors": [{
505+
"category": "PropertyConflict",
506+
"occurredDateTime": "2025-03-25T14:33:19Z",
507+
"propertyCausingError": "UserPrincipalName",
508+
"value": "[email protected]"
509+
}],
510+
"onPremisesSyncEnabled": true
511+
}
512+
]
513+
}
514+
- path: /identityProtection/riskDetections
515+
methods: ['GET']
516+
query_params:
517+
'$select': 'riskEventType,riskState,riskLevel,riskDetail'
518+
'$filter': "userId eq '{id:[0-9a-z-]+}'"
519+
request_headers:
520+
Authorization:
521+
- "Bearer xxxx"
522+
responses:
523+
- status_code: 200
524+
headers:
525+
Content-Type:
526+
- 'applciation/json'
527+
body: |-
528+
{
529+
"@odata.context": "https://graph.microsoft.com/v1.0/$metadata#identityProtection/riskDetections(riskEventType,riskState,riskLevel,riskDetail)",
530+
"value": [
531+
{
532+
"riskEventType": "passwordSpray",
533+
"riskState": "remediated",
534+
"riskLevel": "high",
535+
"riskDetail": "userPerformedSecuredPasswordReset"
536+
}
537+
]
538+
}
539+
- status_code: 200
540+
headers:
541+
Content-Type:
542+
- 'applciation/json'
543+
body: |-
544+
{
545+
"@odata.context": "https://graph.microsoft.com/v1.0/$metadata#identityProtection/riskDetections(riskEventType,riskState,riskLevel,riskDetail)",
546+
"value": []
547+
}
548+
- status_code: 401
549+
headers:
550+
Content-Type:
551+
- 'applciation/json'
552+
body: |-
553+
{
554+
"error": {
555+
"code": "InvalidAuthenticationToken",
556+
"message": "Lifetime validation failed, the token is expired.",
557+
"innerError": {
558+
"date": "2025-03-25T14:33:19",
559+
"request-id": "61d323d8-abba-4f69-8ab5-660e86160ec4",
560+
"client-request-id": "61d323d8-abba-4f69-8ab5-660e86160ec4"
561+
}
562+
}
563+
}

‎packages/o365_metrics/changelog.yml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,9 @@
11
# newer versions go on top
2+
- version: "0.9.0"
3+
changes:
4+
- description: Add 'entra ID users' data stream.
5+
type: enhancement
6+
link: https://github.com/elastic/integrations/pull/13300
27
- version: "0.8.2"
38
changes:
49
- description: Fix cel code for `tenant_settings` data stream.

‎packages/o365_metrics/data_stream/entra_id_users/_dev/test/system/test-default-config.yml‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
input: cel
2+
service: o365_metrics
3+
vars:
4+
url: http://{{Hostname}}:{{Port}}
5+
token_url: http://{{Hostname}}:{{Port}}
6+
azure_tenant_id: "1234"
7+
client_id: "1234"
8+
client_secret: "1234"
9+
data_stream:
10+
vars:
11+
interval: 5s
12+
preserve_original_event: true
13+
assert:
14+
hit_count: 12
15+
fields_present:
16+
- o365.metrics.entra_id_users.user.id
17+
- o365.metrics.entra_id_users.user.upn
18+
- o365.metrics.entra_id_users.user.type
19+
- o365.metrics.entra_id_users.risk.error
20+
- o365.metrics.entra_id_users.risk.event_type
21+
- o365.metrics.entra_id_users.risk.level
22+
- o365.metrics.entra_id_users.risk.state
23+
- o365.metrics.entra_id_users.risk.detail
24+
- o365.metrics.entra_id_users.on_premises_provisioning_errors.occurred_date_time
25+
- o365.metrics.entra_id_users.on_premises_provisioning_errors.category
26+
- o365.metrics.entra_id_users.on_premises_provisioning_errors.property_causing_error
27+
- o365.metrics.entra_id_users.on_premises_provisioning_errors.value
28+
- o365.metrics.entra_id_users.on_premises_sync_enabled

‎packages/o365_metrics/data_stream/entra_id_users/agent/stream/cel.yml.hbs‎

Lines changed: 163 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,163 @@
1+
interval: {{interval}}
2+
{{#if enable_request_tracer}}
3+
resource.tracer.filename: "../../logs/cel/http-request-trace-*.ndjson"
4+
resource.tracer.maxbackups: 5
5+
resource.tracer.maxsize: 5
6+
{{/if}}
7+
{{#if proxy_url}}
8+
resource.proxy_url: {{proxy_url}}
9+
{{/if}}
10+
{{#if resource_ssl}}
11+
resource.ssl:
12+
{{resource_ssl}}
13+
{{/if}}
14+
{{#if resource_timeout}}
15+
resource.timeout: {{resource_timeout}}
16+
{{/if}}
17+
{{#if resource_retry_max_attempts}}
18+
resource.retry.max_attempts: {{resource_retry_max_attempts}}
19+
{{/if}}
20+
{{#if resource_retry_wait_min}}
21+
resource.retry.wait_min: {{resource_retry_wait_min}}
22+
{{/if}}
23+
{{#if resource_retry_wait_max}}
24+
resource.retry.wait_max: {{resource_retry_wait_max}}
25+
{{/if}}
26+
{{#if resource_redirect_forward_headers}}
27+
resource.redirect.forward_headers: {{resource_redirect_forward_headers}}
28+
{{/if}}
29+
{{#if resource_redirect_headers_ban_list}}
30+
resource.redirect.headers_ban_list:
31+
{{#each resource_redirect_headers_ban_list as |item|}}
32+
- {{item}}
33+
{{/each}}
34+
{{/if}}
35+
{{#if resource_redirect_max_redirects}}
36+
resource.redirect.max_redirects: {{resource_redirect_max_redirects}}
37+
{{/if}}
38+
{{#if resource_rate_limit_limit}}
39+
resource.rate_limit.limit: {{resource_rate_limit_limit}}
40+
{{/if}}
41+
{{#if resource_rate_limit_burst}}
42+
resource.rate_limit.burst: {{resource_rate_limit_burst}}
43+
{{/if}}
44+
resource.url: {{url}}
45+
auth.oauth2:
46+
client.id: {{client_id}}
47+
client.secret: {{client_secret}}
48+
provider: azure
49+
scopes:
50+
{{#each token_scopes as |token_scope|}}
51+
- {{token_scope}}
52+
{{/each}}
53+
endpoint_params:
54+
grant_type: client_credentials
55+
{{#if token_url}}
56+
token_url: {{token_url}}/{{azure_tenant_id}}/oauth2/v2.0/token
57+
{{else if azure_tenant_id}}
58+
azure.tenant_id: {{azure_tenant_id}}
59+
{{/if}}
60+
redact:
61+
fields: null
62+
state:
63+
users_path: '/users?$select=id,userPrincipalName,userType,onPremisesProvisioningErrors,onPremisesSyncEnabled'
64+
risk_detections_path: '/identityProtection/riskDetections?$select=riskEventType,riskState,riskLevel,riskDetail&$filter=userId%20eq%20'
65+
next_link: null
66+
program: |
67+
state.with(
68+
request(
69+
"GET",
70+
has(state.next_link) && state.next_link != null ? state.next_link : state.url.trim_right("/") + state.users_path
71+
).do_request().as(users_resp,
72+
users_resp.StatusCode == 200
73+
?
74+
bytes(users_resp.Body).decode_json().as(users_json,
75+
{
76+
"events": users_json.value.map(user,
77+
request(
78+
"GET",
79+
state.url.trim_right("/") + state.risk_detections_path + "'" + user.id + "'"
80+
).do_request().as(risk_detections_resp,
81+
risk_detections_resp.StatusCode == 200
82+
?
83+
bytes(risk_detections_resp.Body).decode_json().value.as(risk_detections,
84+
size(risk_detections) == 0 ? {} : risk_detections[0]
85+
)
86+
:
87+
{
88+
"risk_detections_error": [string(risk_detections_resp.StatusCode), string(risk_detections_resp.Body)].join(" ")
89+
}
90+
).as(risk,
91+
{
92+
"o365": {
93+
"metrics": {
94+
"entra_id_users": {
95+
"user": {
96+
"id": user.id,
97+
"upn": user.userPrincipalName,
98+
"type": user.userType,
99+
},
100+
"on_premises_provisioning_errors": user.onPremisesProvisioningErrors.map(err,
101+
{
102+
"category": err.category,
103+
"occurred_date_time": err.occurredDateTime,
104+
"property_causing_error": err.propertyCausingError,
105+
"value": err.value
106+
}
107+
),
108+
"on_premises_sync_enabled": user.onPremisesSyncEnabled,
109+
"risk": has(risk.risk_detections_error)
110+
?
111+
{
112+
"error": risk.risk_detections_error
113+
}
114+
: has(risk.riskLevel)
115+
?
116+
{
117+
"event_type": risk.riskEventType,
118+
"level": risk.riskLevel,
119+
"state": risk.riskState,
120+
"detail": risk.riskDetail
121+
}
122+
:
123+
{}
124+
}
125+
}
126+
}
127+
}
128+
)
129+
),
130+
"want_more": "@odata.nextLink" in users_json,
131+
"next_link": "@odata.nextLink" in users_json ? users_json["@odata.nextLink"] : null
132+
}
133+
)
134+
:
135+
{
136+
"events": {
137+
"error": {
138+
"code": string(users_resp.StatusCode),
139+
"id": string(users_resp.Status),
140+
"message": "GET " + state.users_path + ": " + (
141+
size(users_resp.Body) != 0 ?
142+
string(users_resp.Body)
143+
:
144+
string(users_resp.Status) + ' (' + string(users_resp.StatusCode) + ')'
145+
),
146+
},
147+
},
148+
"want_more": false,
149+
"next_link": null
150+
}
151+
)
152+
)
153+
tags:
154+
{{#each tags as |tag|}}
155+
- {{tag}}
156+
{{/each}}
157+
{{#contains "forwarded" tags}}
158+
publisher_pipeline.disable_host: true
159+
{{/contains}}
160+
{{#if processors}}
161+
processors:
162+
{{processors}}
163+
{{/if}}

‎packages/o365_metrics/data_stream/entra_id_users/elasticsearch/ingest_pipeline/default.yml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
---
2+
description: Ingest pipeline for o365_metrics 'entra_id_users' data stream.
3+
processors:
4+
- set:
5+
field: ecs.version
6+
value: "8.17.0"

0 commit comments

Comments
 (0)