Merge pull request #302737 from EdB-MSFT/lake-updates-1507-1

Lake updates 1507 1

Author: v-ccolin

articles/sentinel/graph/kql-jobs.md

@@ -19,7 +19,7 @@ ms.collection: ms-security
 #  Create KQL jobs in the Microsoft Sentinel data lake (preview)
 
 
-A job is a one-time or scheduled task that runs a KQL (Kusto Query Language) query against the data in the lake tier to promote the results to the analytics tier. Once in the analytics tier, use the Advanced hunting KQL editor to query the data. Promoting data to the analytics tier has the following benefits:
+A job is a one-time or scheduled task that runs a KQL (Kusto Query Language) query against the data in the lake tier to promote the results to the analytics tier. Once in the analytics tier, use the advanced hunting KQL editor to query the data. Promoting data to the analytics tier has the following benefits:
 
 + Combine current and historical data in the analytics tier to run advanced analytics and machine learning models on your data.
 
@@ -30,7 +30,7 @@ A job is a one-time or scheduled task that runs a KQL (Kusto Query Language) que
 > [!NOTE] 
 > Storage in the analytics tier incurs higher billing rates than in the data lake tier. To reduce costs, only promote data that you need to analyze further. Use the KQL in your query to project only the columns you need, and filter the data to reduce the amount of data promoted to the analytics tier.  
 
-When promoting data to the analytics tier, make sure that the destination workspace is visible in the Advanced hunting query editor. You can only query connected workspaces in the Advanced hunting query editor. You will not be able to see data promoted to workspaces that aren't connected or to the default workspace in Advance hunting. For more information on connected workspaces, see [Connect a workspace](/defender-xdr/advanced-hunting-microsoft-defender#connect-a-workspace). You can promote data to a new table or append the results to an existing table in the analytics tier. When creating a new table, the table name is suffixed with *_KQL_CL* to indicate that the table was created by a KQL job.  
+When promoting data to the analytics tier, make sure that the destination workspace is visible in the advanced hunting query editor. You can only query connected workspaces in the advanced hunting query editor. You will not be able to see data promoted to workspaces that aren't connected or to the default workspace in advance hunting. For more information on connected workspaces, see [Connect a workspace](/defender-xdr/advanced-hunting-microsoft-defender#connect-a-workspace). You can promote data to a new table or append the results to an existing table in the analytics tier. When creating a new table, the table name is suffixed with *_KQL_CL* to indicate that the table was created by a KQL job.  
 
 
 You can create a job by selecting the **Create job** button a KQL query tab or directly from the **Jobs** management page or by. For more information on the Jobs management page, see [Manage jobs in the Microsoft Sentinel data lake](kql-manage-jobs.md).
@@ -145,7 +145,7 @@ The following standard columns aren't supported for export. These columns are ov
 
 + `TimeGenerated` will be overwritten if it's older that 2 days. To preserve the original event time, we recommend writing the source timestamp to a separate column.
 
-For service limits, see [Microsoft Sentinel data lake (preview) service limits](sentinel-lake-service-limits.md#service-limits-for-kql-jobs).
+For service limits, see [Microsoft Sentinel data lake (preview) service limits](sentinel-lake-service-limits.md#service-parameters-and-limits-for-kql-jobs).
 
 > [!NOTE]
 >  Partial results may be promoted if the job's query exceeds the one hour limit.

articles/sentinel/graph/kql-queries.md

@@ -120,7 +120,7 @@ For sample queries, see [KQL sample queries for the data lake](kql-samples.md).
 + Calling external data via KQL query against the data lake isn't supported. 
 + `Ingestion_time()` function isn't supported on tables in data lake.
 
-For service limits, see [Microsoft Sentinel data lake (preview) service limits](sentinel-lake-service-limits.md#service-limits-for-kql-queries-in-the-lake-tier).
+For service limits, see [Microsoft Sentinel data lake (preview) service limits](sentinel-lake-service-limits.md#service-parameters-and-limits-for-kql-queries-in-the-lake-tier).
 
 For troubleshooting KQL queries, see [Troubleshoot KQL queries in the Microsoft Sentinel data lake](kql-troubleshoot.md).
 

articles/sentinel/graph/notebook-jobs.md

@@ -108,7 +108,7 @@ The page shows a list of jobs and their types. Select a notebook job to view its
 
 ## Service limits and troubleshooting
 
-For a list of service limits for the Microsoft Sentinel data lake, see [Microsoft Sentinel data lake service limits](sentinel-lake-service-limits.md#service-limits-for-vs-code-notebooks).  
+For a list of service limits for the Microsoft Sentinel data lake, see [Microsoft Sentinel data lake service limits](sentinel-lake-service-limits.md#service-parameters-and-limits-for-vs-code-notebooks).  
 
 
 For information on troubleshooting, see [Run notebooks on the Microsoft Sentinel data lake (preview)](notebooks.md#service-limits).

articles/sentinel/graph/notebooks.md

@@ -7,7 +7,7 @@ ms.author: edbaynash
 ms.topic: how-to  
 ms.service: microsoft-sentinel
 ms.subservice: sentinel-graph
-ms.date: 07/09/2025
+ms.date: 07/15/2025
 
 
 # Customer intent: As a security engineer or data scientist, I want to explore and analyze security data in the Microsoft Sentinel data lake using Jupyter notebooks, so that I can gain insights and build advanced analytics solutions.
@@ -173,7 +173,7 @@ You can schedule jobs to run at specific times or intervals using the Microsoft
 
 ## Service limits
 
-For a list of service limits for the Microsoft Sentinel data lake, see [Microsoft Sentinel data lake service limits](sentinel-lake-service-limits.md#service-limits-for-vs-code-notebooks).  
+For a list of service limits for the Microsoft Sentinel data lake, see [Microsoft Sentinel data lake service limits](sentinel-lake-service-limits.md#service-parameters-and-limits-for-vs-code-notebooks).  
 
 ## Troubleshooting 
 
@@ -186,6 +186,7 @@ The following table lists common errors you may encounter when working with note
 | Spark compute | Unable to access Spark Pool – 403 Forbidden. | Output channel – “Window”. | Spark pools aren't displayed. | User doesn't have the required roles to run interactive notebook or schedule job. | Check if you have the required role for interactive notebooks or notebook jobs. |
 | Spark compute | Spark Pool – \<name\> – is being upgraded. | Toast alert. | One of the Spark pools is Not available. | Spark pool is being upgraded to the latest version of Microsoft Sentinel Provider. | Wait for ~20-30 mins for the Pool to be available. |
 | Spark compute | An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe. : org.apache.spark.SparkException: Job aborted due to stage failure: Total size of serialized results (4.0 GB) is bigger than spark.driver.maxResultSize (4.0 GB) | Inline. | Driver memory exceeded or executor failure. | Job ran out of driver memory, or one or more executors failed. | View job run logs or optimize your query. Avoid using toPandas() on large datasets. Consider setting `spark.conf.set("spark.sql.execution.arrow.pyspark.enabled", "true")` if needed. |
+|Spark compute|	Failed to connect to the remote Jupyter Server 'https://api.securityplatform.microsoft.com/spark-notebook/interactive'. Verify the server is running and reachable.|	Toast alert |	User stopped the session, and failed to connect to server. |	User stopped the session.	| Run the cell again to reconnect the session.|
 | VS Code Runtime | Kernel with id – k1 - has been disposed. | Output channel – “Jupyter”. | Kernel not connected. | VS Code lost connection to the compute kernel. | Reselect the Spark pool and execute a cell. |
 | VS Code Runtime | ModuleNotFoundError: No module named 'MicrosoftSentinelProvider'. | Inline. | Module not found. | Missing import for example, Microsoft Sentinel Library library | Run the setup/init cell again. |
 | VS Code Runtime | Cell In[{cell number}], line 1 if: ^ SyntaxError: invalid syntax. | Inline. | Invalid syntax. | Python or PySpark syntax error. | Review code syntax; check for missing colons, parentheses, or quotes. |

articles/sentinel/graph/sentinel-lake-overview.md

@@ -122,3 +122,4 @@ For more information on using the Microsoft Sentinel data lake, see the followin
 + [KQL and the Microsoft Sentinel data lake (preview)](kql-overview.md)
 + [Permissions for the Microsoft Sentinel data lake (preview)](../roles.md#roles-and-permissions-for-the-microsoft-sentinel-data-lake-preview) 
 + [Manage data tiers and retention in Microsoft Defender Portal (preview)](https://aka.ms/manage-data-defender-portal-overview) 
++ [Manage and monitor costs for Microsoft Sentinel](../billing-monitor-costs.md)

articles/sentinel/graph/sentinel-lake-service-limits.md

@@ -13,9 +13,9 @@ ms.author: edbaynash
 ---  
 
 
-# Microsoft Sentinel data lake (preview) service limits
+# Microsoft Sentinel data lake (preview) service parameters and limits
 
-The following service limits apply to the Microsoft Sentinel data lake (preview) service.
+The following service parameters and limits apply to the Microsoft Sentinel data lake (preview) service.
 
 [!INCLUDE [Service limits for VS Code notebooks](../includes/service-limits-notebooks.md)]
 

articles/sentinel/includes/service-limits-kql-jobs.md

@@ -2,18 +2,18 @@
 author: EdB-MSFT
 ms.author: edbayansh
 ms.topic: include
-ms.date: 06/30/2025
+ms.date: 07/15/2025
 ---
 
-## Service limits for KQL jobs
+## Service parameters and limits for KQL jobs
 
-The following table lists the service limits for KQL jobs in the Microsoft Sentinel data lake (Preview).
+The following table lists the service parameters and limits for KQL jobs in the Microsoft Sentinel data lake (Preview).
 
-| Description                        | Limit             
-|-------------------------------------|-----------------|
-| Job query execution timeout         | 1 hour           |
-| Query time range                    | Up to 12 years   |
-| Jobs per tenant (enabled jobs)      | 100 (soft limit) |
-| Concurrent job execution per tenant | 3                |
-| Query scope                         | Single workspace |
-| Number of output tables per job     | 1                |
+| Category                        | Parameter/limit   |
+|-------------------------------------|---------------------|
+| Concurrent job execution per tenant | 3                   |
+| Job query execution timeout         | 1 hour              |
+| Jobs per tenant (enabled jobs)      | 100                 |
+| Number of output tables per job     | 1                   |
+| Query scope                         | Single workspace    |
+| Query time range                    | Up to 12 years      |

articles/sentinel/includes/service-limits-kql-queries.md

@@ -2,18 +2,18 @@
 author: EdB-MSFT
 ms.author: edbayansh
 ms.topic: include
-ms.date: 06/30/2025
+ms.date: 07/15/2025
 ---
 
-## Service limits for KQL queries in the lake tier.
+## Service parameters and limits for KQL queries in the lake tier
 
-The following limitations apply when writing queries in Microsoft Sentinel data lake (Preview).
+The following service parameters limitations apply when writing queries in Microsoft Sentinel data lake (Preview).
 
-|Category | Limit|
-|---|---|
-| Query result rows|  30,000 rows |
-| Query result data | 64 MB. |
-| Query timeout | 8 minutes. |
-|Queryable time range | Up to 12 years, depending on data retention. |
-| Concurrent interactive queries| 45 per minute|
-| Query Scope |	Single workspace |
+| Category                   | Parameter/limit                                         |
+|----------------------------|-----------------------------------------------|
+| Concurrent interactive queries | 45 per minute                             |
+| Query result data          | 64 MB                                         |
+| Query result rows          | 30,000 rows                                   |
+| Query Scope                | Single workspace                              |
+| Query timeout              | 8 minutes                                     |
+| Queryable time range       | Up to 12 years, depending on data retention.  |

articles/sentinel/includes/service-limits-notebooks.md

@@ -2,27 +2,23 @@
 author: EdB-MSFT
 ms.author: edbayansh
 ms.topic: include
-ms.date: 06/30/2025
+ms.date: 07/15/2025
 ---
 
-## Service limits for VS Code Notebooks
+## Service parameters and limits for VS Code Notebooks
 
 
-The following section lists the service limits for Microsoft Sentinel data lake (Preview) when using VS Code Notebooks.
+The following section lists the service parameters and limits for Microsoft Sentinel data lake (Preview) when using VS Code Notebooks.
 
-+ Spark compute session takes about 5-6 minutes to start. You can view the status of the session at the bottom of your VS Code Notebook.
-+ Only [Azure Synapse libraries 3.4](https://github.com/microsoft/synapse-spark-runtime/tree/main#readme) and the Microsoft Sentinel Provider library for abstracted functions are supported for querying lake. Pip installs or custom libraries aren't supported.
-
-
-| Category | Limit |
-|----------|-------|
-| Session start-up time | 5 minutes |
-| Interactive: Session inactivity timeout | 20 minutes |
-| Interactive: Query timeout | 2 hours |
-| Gateway web socket timeout | 2 hours |
-| VS Code UX limit to display records | 100,000 rows |
-| Supported libraries | Azure Synapse libraries, Microsoft Sentinel Provider. Pip install and custom libraries aren't supported |
-| Language | Python |
-| Max concurrent users on interactive querying | 8-10 on Large pool |
-| Max concurrent notebook jobs | 3, subsequent jobs are queued |
-| Custom table in the analytics tier | Custom tables in analytics tier can't be deleted from a notebook; Use Log Analytics to delete these tables. For more information, see [Add or delete tables and columns in Azure Monitor Logs](/azure/azure-monitor/logs/create-custom-table?tabs=azure-portal-1%2Cazure-portal-2%2Cazure-portal-3#delete-a-table)|
+|Category|Parameter/limit|
+|---|---|
+|Custom table in the analytics tier|Custom tables in analytics tier can't be deleted from a notebook; Use Log Analytics to delete these tables. For more information, see [Add or delete tables and columns in Azure Monitor Logs](/azure/azure-monitor/logs/create-custom-table?tabs=azure-portal-1%2Cazure-portal-2%2Cazure-portal-3#delete-a-table)|
+|Gateway web socket timeout|2 hours|
+|Interactive query timeout|2 hours|
+|Interactive session inactivity timeout|20 minutes|
+|Language|Python|
+|Max concurrent notebook jobs|3, subsequent jobs are queued|
+|Max concurrent users on interactive querying|8-10 on Large pool|
+|Session start-up time|Spark compute session takes about 5-6 minutes to start. You can view the status of the session at the bottom of your VS Code Notebook.|
+|Supported libraries|Only [Azure Synapse libraries 3.4](https://github.com/microsoft/synapse-spark-runtime/tree/main#readme) and the Microsoft Sentinel Provider library for abstracted functions are supported for querying the data lake. Pip installs or custom libraries aren't supported.|
+|VS Code UX limit to display records|100,000 rows|

articles/sentinel/includes/service-limits-table-manaement-ingestion.md

@@ -2,22 +2,24 @@
 author: EdB-MSFT
 ms.author: edbayansh
 ms.topic: include
-ms.date: 06/30/2025
+ms.date: 07/15/2025
 ---
 
-## Service limits for tables, data management, and ingestion
+## Service parameters and limits for tables, data management, and ingestion
 
 > [!NOTE] 
-> During public preview Microsoft Sentinel data lake uses a single region. Your primary and other workspaces must be in the same region as your tenant’s home region. Only workspaces in the same region as your tenant’s home region can be attached to the data lake.
+> During preview Microsoft Sentinel data lake uses a single region. Your primary and other workspaces must be in the same region as your tenant’s home region. Only workspaces in the same region as your tenant’s home region can be attached to the data lake.
 
-The following table lists the service limits for the Microsoft Sentinel data lake (Preview) service related to table management, data ingestion, and retention. These limits include, but aren't limited to, Azure Resource Graph data, Microsoft 365 data, and data mirroring.
-
-| Description                                         | Limit                        | 
-|-----------------------------------------------------|------------------------------|
-| Lake Retention (Aux)                                | 12 years                     | 
-| Lake Retention (Asset data)                         | 12 years                     | 
-| Ingestion requests per minute to a data collection endpoint | 15,000               |
-| Data ingestion per minute to a data collection endpoint    | 50 GB                 |
-| Maximum size for field values (Log Analytics)                  | 32 KB (truncated above the limit) | 
-| Default ingestion volume rate threshold in LALog Analytics workspaces   | 6 GB/min uncompressed |
+The following table lists the service parameters and limits for the Microsoft Sentinel data lake (preview) service related to table management, data ingestion, and retention. These limits include, but aren't limited to, Azure Resource Graph data, Microsoft 365 data, and data mirroring.
 
+| Category                                         | Parameter/limit                              |
+|--------------------------------------------------|----------------------------------------------|
+| Data ingestion per minute to a data collection endpoint    | 50 GB                              |
+| Default ingestion volume rate threshold in LALog Analytics workspaces | 6 GB/min uncompressed   |
+| Ingestion requests per minute to a data collection endpoint | 15,000                            |
+| Lake Retention (Asset data)                       | 12 years                                    |
+| Lake Retention (Aux)                              | 12 years                                    |
+| Maximum size for field values (Log Analytics)     | 32 KB (truncated above the limit)           |
+| Table setup latency during onboarding              | 90-120 minutes                             |
+| New table setup latency                            | 90-120 minutes                             |
+| Switching data between tiers latency               | 90-120 minutes                             |