You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/sentinel/datalake/sentinel-lake-overview.md
+32-27Lines changed: 32 additions & 27 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -7,7 +7,7 @@ ms.service: microsoft-sentinel
7
7
ms.subservice: sentinel-graph
8
8
ms.topic: conceptual
9
9
ms.custom: references_regions
10
-
ms.date: 07/16/2025
10
+
ms.date: 08/11/2025
11
11
ms.author: edbaynash
12
12
13
13
ms.collection: ms-security
@@ -16,39 +16,42 @@ ms.collection: ms-security
16
16
17
17
# What is Microsoft Sentinel data lake (preview)?
18
18
19
-
Microsoft Sentinel data lake is a purpose-built, cloud-native security data lake that transforms how organizations manage and analyze security data. Architected as a true data lake, it is designed to ingest, store, and analyze large volumes of diverse security data at scale. By centralizing all your security data into a single, open, and extensible platform, it delivers deep visibility, long-term retention, and advanced analytics.
19
+
Microsoft Sentinel data lake is a purpose-built, cloud-native security data lake that transforms how organizations manage and analyze security data. Designed as a true data lake, it ingests, stores, and analyzes large volumes of diverse security data at scale. By centralizing security data into a single, open-format, extensible platform, it provides deep visibility, long-term retention, and advanced analytics.
20
20
21
-
The data lake makes it cost-effective to bring all your security data into Microsoft Sentinel, eliminating the need to choose between coverage and cost. Retain more data for longer, detect threats with greater context and historical depth, and respond faster, without compromising on security.
21
+
The data lake lets you bring all your security data into Microsoft Sentinel cost-effectively, removing the need to choose between coverage and cost. You can retain more data for longer, detect threats with greater context and historical depth, and respond faster without compromising security.
22
22
23
-
The Microsoft Sentinel data lake is fully managed, without the need to deploy or maintain your data infrastructure. It provides a unified data platform for end-to-end threat analysis and response. It enables you to store one copy of security data across assets, activity logs, and threat intelligence in the lake and leverage multiple analytics tools like KQL and notebooks for deep security analytics.
23
+
The Microsoft Sentinel data lake is fully managed, so you don't need to deploy or maintain data infrastructure. It provides a unified data platform for end-to-end threat analysis and response. It stores a single copy of security data across assets, activity logs, and threat intelligence in the lake and leverages multiple analytics tools like KQL and Jupyter notebooks for deep security analytics.
24
24
25
-
Traditional SIEM solutions struggle with the cost and complexity of storing and querying long-term data. Microsoft Sentinel data lake addresses these challenges in the following ways:
25
+
Traditional SIEM solutions struggle with the cost and complexity of storing and querying long-term security data. Microsoft Sentinel data lake solves these challenges in the following ways:
26
26
27
-
+ Unifying security data across Microsoft Defender XDR, third-party sources and across assets, activity logs, and threat intelligence
28
-
+ Optimizing costs through tiered storage and on-demand data promotion.
29
-
+ Enabling deep security insights with up to 12 years of security data and telemetry that can be queried and deeply analyzed.
27
+
+ Unifying security data across Microsoft Defender XDR, third-party sources and assets, activity logs, and threat intelligence
28
+
+ Optimizing costs with tiered storage, on-demand data promotion, and a single copy of the data
29
+
+ Enabling deep security insights with up to 12 years of security data and telemetry you can query and analyze
30
30
+ Powering AI and automation for faster detection and response.
31
31
32
-
With a single copy of data, Microsoft Sentinel data lake empowers you to run queries in KQL and conduct deeper analysis for forensics, incidence response, and anomaly detection in Jupyter notebooks using sophisticated Python libraries and machine learning tools.
32
+
With a single copy of data, use KQL to run queries and Jupyter notebooks with sophisticated Python libraries and machine learning tools to conduct deeper analysis for forensics, incidence response, and anomaly detection.
33
33
34
34
## Architecture
35
35
36
36
Microsoft Sentinel data lake, built on Azure's scalable infrastructure, facilitates centralized ingestion, analysis, and action across diverse data sources. The Microsoft Sentinel data lake technical architecture includes the following key benefits:
37
37
38
-
+ Single, open-format data copy for efficient and cost-effective storage.
39
-
+ Separation of storage and compute for greater flexibility.
40
-
+ Support for multiple analytics engines to unlock insights from your security data.
41
-
+ Native integration with Microsoft Sentinel SIEM and its security operations workflows.
38
+
+ Open format Parquet data files for interoperability and extensibility
39
+
+ Single copy of data for efficient and cost effective storage
40
+
+ Separation of storage and compute for greater flexibility
41
+
+ Support for multiple analytics engines to unlock insights from your security data
42
+
+ Native integration with Microsoft Sentinel SIEM and its security operations workflows
42
43
43
44
### Storage tiers
44
45
45
46
Microsoft Sentinel is designed with two distinct storage tiers to optimize cost and performance:
46
47
47
-
+ Analytics tier: The existing Microsoft Sentinel data tier enabling querying, visualization, and alerting capabilities to help you proactively identify and resolve issues across your infrastructure and applications.
48
-
+ Data lake tier: A centralized security data lake offering long-term data storage for querying and python-based advanced analytics. The data lake tier is designed for cost-effective storage of large volumes of security data, enabling you to retain data for up to 12 years. For more information on data tiers and retention, see [Manage data tiers and retention in Microsoft Defender portal (preview)](https://aka.ms/manage-data-defender-portal-overview).
48
+
+ Analytics tier: The existing Microsoft Sentinel data tier supporting advanced hunting, alerting, and incident management to help you proactively identify and resolve issues across your infrastructure and applications. This tier is designed for high-performance analytics and real-time data processing.
49
+
+ Data lake tier: Provides centralized long-term storage for querying and Python-based advanced analytics. It's designed for cost effective retention of large volumes of security data for up to 12 years. Data in the analytics tier is mirrored to the lake tier, preserving a single copy of the data.
50
+
51
+
For more information on data tiers and retention, see [Manage data tiers and retention in Microsoft Defender portal (preview)](https://aka.ms/manage-data-defender-portal-overview).
49
52
50
53
51
-
### Supported Data Sources
54
+
### Supported data sources
52
55
53
56
Microsoft Sentinel data lake works with all existing Sentinel data connectors, including:
54
57
+ All Microsoft Defender and Microsoft Sentinel data sources
@@ -64,39 +67,41 @@ Microsoft Sentinel data lake works with all existing Sentinel data connectors, i
64
67
65
68
### Flexible querying with Kusto Query Language
66
69
67
-
Data lake exploration Kusto Query Language (KQL) queries enable you to write and run KQL queries against your data lake resources. You can use the query editor to explore your data, analyze your data lake, and create jobs to promote data from the data lake tier to the analytics tier.
70
+
Data lake exploration Kusto Query Language (KQL) queries let you write and run queries against data lake resources. Use the query editor to explore data, analyze the lake, and create jobs that promote data from the data lake tier to the analytics tier.
68
71
KQL queries offer the following key features:
69
72
70
73
+ KQL query editor: Provides editing and running KQL queries with IntelliSense and autocomplete.
71
74
+ Full support for KQL: Use the full range of KQL capabilities, including machine learning functions and advanced analytics.
72
-
+ Job Creation: Create one-time or scheduled jobs to promote data from the lake to the analytics tier.
75
+
+ Job creation: Create one-time or scheduled jobs to promote data from the lake to the analytics tier.
73
76
74
-
For more information, see [KQL and the Microsoft Sentinel data lake (preview)](kql-overview.md)
77
+
For more information, see [KQL and the Microsoft Sentinel data lake (preview)](kql-overview.md).
75
78
76
-
:::image type="content" source="media/sentinel-lake-overview/data-lake-exploration.png" lightbox="media/sentinel-lake-overview/data-lake-exploration.png" alt-text="A screenshot showing the KQL query editor in the Microsoft Sentinel data lake.":::
79
+
:::image type="content" source="media/sentinel-lake-overview/data-lake-exploration.png" lightbox="media/sentinel-lake-overview/data-lake-exploration.png" alt-text="Screenshot of the KQL query editor in the Microsoft Sentinel data lake.":::
77
80
78
81
### Powerful analytics using Jupyter notebooks
79
82
80
-
Jupyter notebooks in the Microsoft Sentinel data lake provide a powerful environment for data analysis and machine learning. Use Python libraries to build and run machine learning models, conduct advanced analytics, and visualize your data. The notebooks support rich visualizations, enabling you to gain insights from your security data. Schedule notebooks to regularly summarize data, run machine learning models, and promote data from the data lake tier to the analytics tier.
83
+
Jupyter notebooks in the Microsoft Sentinel data lake offer a powerful environment for data analysis and machine learning. Use Python libraries to build and run machine learning models, conduct advanced analytics, and visualize your data. The notebooks support rich visualizations, enabling you to gain insights from your security data. Schedule notebooks to summarize data regularly, run machine learning models, and promote data from the data lake tier to the analytics tier.
81
84
82
85
For more information, see [Jupyter notebooks in the Microsoft Sentinel data lake (preview)](notebooks-overview.md).
83
86
84
-
:::image type="content" source="media/sentinel-lake-overview/notebook.png" lightbox="media/sentinel-lake-overview/notebook.png" alt-text="A screenshot showing a Jupyter notebook.":::
87
+
:::image type="content" source="media/sentinel-lake-overview/notebook.png" lightbox="media/sentinel-lake-overview/notebook.png" alt-text="Screenshot of a Jupyter notebook showing data analysis and visualization.":::
85
88
86
89
### Activity audit
87
-
The Microsoft Sentinel data lake provides audit functionality that tracks activities performed in the data lake. The audit log captures events related to data access, job management, and queries, enabling you to monitor and investigate activities in the data lake.
90
+
The Microsoft Sentinel data lake provides auditing that tracks activities in the lake. The audit log captures data access, job management, and query events, letting you monitor and investigate activity.
88
91
89
92
Some of the activities audited are:
90
-
+ Accessing data in lake via KQL queries
93
+
+ Accessing data in lake with KQL queries
91
94
+ Running notebooks on data lake
92
95
+ Create, edit, run, and delete jobs
93
96
94
-
Auditing is automatically turned on for Microsoft Sentinel data lake. Features that are audited are logged in the audit log automatically.
95
-
For more information on audited data lake activities, see [Audit log for Microsoft Sentinel data lake](./auditing-lake-activities.md)
97
+
Auditing is enabled by default for the Microsoft Sentinel data lake. Audited actions are shown in the audit log.
98
+
99
+
For more information on audited data lake activities, see [Audit log for Microsoft Sentinel data lake](./auditing-lake-activities.md).
96
100
97
101
## Supported regions
98
102
99
-
For a list of supported regions, see [Regions supported for Microsoft Sentinel data lake](../geographical-availability-data-residency.md#regions-supported-for-microsoft-sentinel-data-lake)
103
+
See [Regions supported for Microsoft Sentinel data lake](../geographical-availability-data-residency.md#regions-supported-for-microsoft-sentinel-data-lake) for supported regions.
0 commit comments