Skip to content

Commit 0295f82

Browse files
authored
Merge pull request #225400 from whhender/analytics-freshness-less-30
Review
2 parents b068652 + 86c87c2 commit 0295f82

25 files changed

+177
-145
lines changed

articles/data-lake-analytics/data-lake-analytics-account-policies.md

Lines changed: 18 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -2,9 +2,9 @@
22
title: Manage Azure Data Lake Analytics Account Policies
33
description: Learn how to use account policies to control usage of a Data Lake Analytics account, such as maximum AUs and maximum jobs.
44
ms.service: data-lake-analytics
5-
ms.reviewer: jasonh
5+
ms.reviewer: whhender
66
ms.topic: how-to
7-
ms.date: 04/30/2018
7+
ms.date: 01/27/2023
88
---
99
# Manage Azure Data Lake Analytics using Account Policies
1010

@@ -18,14 +18,14 @@ These policies apply to all jobs in a Data Lake Analytics account.
1818

1919
### Maximum number of AUs in a Data Lake Analytics account
2020

21-
A policy controls the total number of Analytics Units (AUs) your Data Lake Analytics account can use. By default, the value is set to 250. For example, if this value is set to 250 AUs, you can have one job running with 250 AUs assigned to it, or 10 jobs running with 25 AUs each. Additional jobs that are submitted are queued until the running jobs are finished. When running jobs are finished, AUs are freed up for the queued jobs to run.
21+
A policy controls the total number of Analytics Units (AUs) your Data Lake Analytics account can use. By default, the value is set to 250. For example, if this value is set to 250 AUs, you can have one job running with 250 AUs assigned to it, or 10 jobs running with 25 AUs each. Other jobs that are submitted are queued until the running jobs are finished. When running jobs are finished, AUs are freed up for the queued jobs to run.
2222

2323
To change the number of AUs for your Data Lake Analytics account:
2424

2525
1. In the Azure portal, go to your Data Lake Analytics account.
26-
2. Click **Limits and policies**.
26+
2. Select **Limits and policies**.
2727
3. Under **Maximum AUs**, move the slider to select a value, or enter the value in the text box.
28-
4. Click **Save**.
28+
4. Select **Save**.
2929

3030
> [!NOTE]
3131
> If you need more than the default (250) AUs, in the portal, click **Help+Support** to submit a support request. The number of AUs available in your Data Lake Analytics account can be increased.
@@ -37,23 +37,23 @@ This policy limits how many jobs can run simultaneously. By default, this value
3737
To change the number of jobs that can run simultaneously:
3838

3939
1. In the Azure portal, go to your Data Lake Analytics account.
40-
2. Click **Limits and policies**.
40+
2. Select **Limits and policies**.
4141
3. Under **Maximum Number of Running Jobs**, move the slider to select a value, or enter the value in the text box.
42-
4. Click **Save**.
42+
4. Select **Save**.
4343

4444
> [!NOTE]
4545
> If you need to run more than the default (20) number of jobs, in the portal, click **Help+Support** to submit a support request. The number of jobs that can run simultaneously in your Data Lake Analytics account can be increased.
4646
4747
### How long to keep job metadata and resources
4848

49-
When your users run U-SQL jobs, the Data Lake Analytics service keeps all related files. These files include the U-SQL script, the DLL files referenced in the U-SQL script, compiled resources, and statistics. The files are in the /system/ folder of the default Azure Data Lake Storage account. This policy controls how long these resources are stored before they are automatically deleted (the default is 30 days). You can use these files for debugging, and for performance-tuning of jobs that you'll rerun in the future.
49+
When your users run U-SQL jobs, the Data Lake Analytics service keeps all related files. These files include the U-SQL script, the DLL files referenced in the U-SQL script, compiled resources, and statistics. The files are in the /system/ folder of the default Azure Data Lake Storage account. This policy controls how long these resources are stored before they're automatically deleted (the default is 30 days). You can use these files for debugging, and for performance-tuning of jobs that you'll rerun in the future.
5050

5151
To change how long to keep job metadata and resources:
5252

5353
1. In the Azure portal, go to your Data Lake Analytics account.
54-
2. Click **Limits and policies**.
54+
2. Select **Limits and policies**.
5555
3. Under **Days to Retain Job Queries**, move the slider to select a value, or enter the value in the text box.
56-
4. Click **Save**.
56+
4. Select **Save**.
5757

5858
## Job-level policies
5959

@@ -65,7 +65,7 @@ Data Lake Analytics has two policies that you can set at the job level:
6565

6666
- **Priority**: Users can only submit jobs that have a priority lower than or equal to this value. A higher number indicates a lower priority. By default, this limit is set to 1, which is the highest possible priority.
6767

68-
There is a default policy set on every account. The default policy applies to all users of the account. You can create additional policies for specific users and groups.
68+
There's a default policy set on every account. The default policy applies to all users of the account. You can create more policies for specific users and groups.
6969

7070
> [!NOTE]
7171
> Account-level policies and job-level policies apply simultaneously.
@@ -74,9 +74,9 @@ There is a default policy set on every account. The default policy applies to al
7474

7575
1. In the Azure portal, go to your Data Lake Analytics account.
7676

77-
2. Click **Limits and policies**.
77+
2. Select **Limits and policies**.
7878

79-
3. Under **Job Submission Limits**, click the **Add Policy** button. Then, select or enter the following settings:
79+
3. Under **Job Submission Limits**, select the **Add Policy** button. Then, select or enter the following settings:
8080

8181
1. **Compute Policy Name**: Enter a policy name, to remind you of the purpose of the policy.
8282

@@ -86,19 +86,21 @@ There is a default policy set on every account. The default policy applies to al
8686

8787
4. **Set the Priority Limit**: Set the priority limit that applies to the selected user or group.
8888

89-
4. Click **Ok**.
89+
4. Select **Ok**.
9090

9191
5. The new policy is listed in the **Default** policy table, under **Job Submission Limits**.
9292

9393
## Delete or edit an existing policy
9494

9595
1. In the Azure portal, go to your Data Lake Analytics account.
9696

97-
2. Click **Limits and policies**.
97+
2. Select **Limits and policies**.
9898

9999
3. Under **Job Submission Limits**, find the policy you want to edit.
100100

101-
4. To see the **Delete** and **Edit** options, in the rightmost column of the table, click `...`.## Additional resources for job policies
101+
4. To see the **Delete** and **Edit** options, in the rightmost column of the table, select `...`.
102+
103+
## More resources for job policies
102104

103105
- [Policy overview blog post](/archive/blogs/azuredatalake/managing-your-azure-data-lake-analytics-compute-resources-overview)
104106
- [Account-level policies blog post](/archive/blogs/azuredatalake/managing-your-azure-data-lake-analytics-compute-resources-account-level-policy)

articles/data-lake-analytics/data-lake-analytics-cicd-manage-assemblies.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ title: Manage U-SQL assemblies in a CI/CD pipeline - Azure Data Lake
33
description: 'Learn the best practices for managing U-SQL C# assemblies in a CI/CD pipeline with Azure DevOps.'
44
ms.service: data-lake-analytics
55
ms.topic: how-to
6-
ms.date: 10/30/2018
6+
ms.date: 01/27/2023
77
---
88

99
# Best practices for managing U-SQL assemblies in a CI/CD pipeline
@@ -84,7 +84,7 @@ You can deploy a U-SQL database by using a U-SQL database project or a `.usqldbp
8484

8585
### Deploy a U-SQL database in Azure DevOps
8686

87-
`PackageDeploymentTool.exe` provides the programming and command-line interfaces that help to deploy U-SQL databases. The SDK is included in the [U-SQL SDK Nuget package](https://www.nuget.org/packages/Microsoft.Azure.DataLake.USQL.SDK/), located at `build/runtime/PackageDeploymentTool.exe`.
87+
`PackageDeploymentTool.exe` provides the programming and command-line interfaces that help to deploy U-SQL databases. The SDK is included in the [U-SQL SDK NuGet package](https://www.nuget.org/packages/Microsoft.Azure.DataLake.USQL.SDK/), located at `build/runtime/PackageDeploymentTool.exe`.
8888

8989
In Azure DevOps, you can use a command-line task and this SDK to set up an automation pipeline for the U-SQL database refresh. [Learn more about the SDK and how to set up a CI/CD pipeline for U-SQL database deployment](data-lake-analytics-cicd-overview.md#deploy-u-sql-database-through-azure-pipelines).
9090

articles/data-lake-analytics/data-lake-analytics-data-lake-tools-debug-recurring-job.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,10 @@
11
---
22
title: Debug recurring jobs in Azure Data Lake Analytics
33
description: Learn how to use Azure Data Lake Tools for Visual Studio to debug an abnormal recurring job.
4-
ms.reviewer: jasonh
4+
ms.reviewer: whhender
55
ms.service: data-lake-analytics
66
ms.topic: how-to
7-
ms.date: 05/20/2018
7+
ms.date: 01/27/2023
88
---
99

1010
# Troubleshoot an abnormal recurring job
@@ -55,7 +55,7 @@ You can find all submitted recurring jobs through the job list at the bottom of
5555

5656
![Shortcut menu for comparing jobs](./media/data-lake-analytics-data-lake-tools-debug-recurring-job/compare-job.png)
5757

58-
Pay attention to the big differences between these two jobs. Those differences are probably causing the performance problems. To check further, use the steps in the following diagram:
58+
Pay attention to the differences between these two jobs. Those differences are probably causing the performance problems. To check further, use the steps in the following diagram:
5959

6060
![Process diagram for checking differences between jobs](./media/data-lake-analytics-data-lake-tools-debug-recurring-job/recurring-job-diff-debugging-flow.png)
6161

articles/data-lake-analytics/data-lake-analytics-data-lake-tools-export-database.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,10 @@
11
---
22
title: Export U-SQL database- Azure Data Lake Tools for Visual Studio
33
description: Learn how to use Azure Data Lake Tools for Visual Studio to export a U-SQL database and automatically import it to a local account.
4-
ms.reviewer: jasonh
4+
ms.reviewer: whhender
55
ms.service: data-lake-analytics
66
ms.topic: how-to
7-
ms.date: 11/27/2017
7+
ms.date: 01/27/2023
88
---
99

1010
# Export a U-SQL database
@@ -40,7 +40,7 @@ The export action is completed by running a U-SQL job. Therefore, exporting from
4040

4141
### Step 3: Check the objects list and other configurations
4242

43-
In this step, you can verify the selected objects in the **Export object list** box. If there are any errors, select **Previous** to go back and correctly configure the objects that you want to export.
43+
In this step, you can verify the selected objects in the **Export object list** box. If there are any errors, select **Previous** to go back, and correctly configure the objects that you want to export.
4444

4545
You can also configure other settings for the export target. Configuration descriptions are listed in the following table:
4646

articles/data-lake-analytics/data-lake-analytics-data-lake-tools-local-debug.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,10 @@
11
---
22
title: Debug Azure Data Lake Analytics code locally
33
description: Learn how to use Azure Data Lake Tools for Visual Studio to debug U-SQL jobs on your local workstation.
4-
ms.reviewer: jasonh
4+
ms.reviewer: whhender
55
ms.service: data-lake-analytics
66
ms.topic: how-to
7-
ms.date: 07/03/2018
7+
ms.date: 01/27/2023
88
---
99
# Debug Azure Data Lake Analytics code locally
1010

articles/data-lake-analytics/data-lake-analytics-data-lake-tools-use-vertex-execution-view.md

Lines changed: 9 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -2,27 +2,28 @@
22
title: Vertex Execution View in Data Lake Tools for Visual Studio
33
description: This article describes how to use the Vertex Execution View to exam Data Lake Analytics jobs.
44
ms.service: data-lake-analytics
5-
6-
75
ms.topic: how-to
8-
ms.date: 10/13/2016
6+
ms.date: 01/27/2023
97
---
108
# Use the Vertex Execution View in Data Lake Tools for Visual Studio
9+
1110
Learn how to use the Vertex Execution View to exam Data Lake Analytics jobs.
1211

1312
[!INCLUDE [retirement-flag](includes/retirement-flag.md)]
1413

1514
## Open the Vertex Execution View
16-
Open a U-SQL job in Data Lake Tools for Visual Studio. Click **Vertex Execution View** in the bottom left corner. You may be prompted to load profiles first and it can take some time depending on your network connectivity.
15+
16+
Open a U-SQL job in Data Lake Tools for Visual Studio. Select **Vertex Execution View** in the bottom left corner. You may be prompted to load profiles first and it can take some time depending on your network connectivity.
1717

1818
![Screenshot that shows the Data Lake Analytics Tools Vertex Execution View](./media/data-lake-analytics-data-lake-tools-use-vertex-execution-view/data-lake-tools-open-vertex-execution-view.png)
1919

2020
## Understand Vertex Execution View
21+
2122
The Vertex Execution View has three parts:
2223

2324
![Screenshot that shows the Vertex Execution View with the "Vertex selector" and center-top and center-bottom panes highlighted.](./media/data-lake-analytics-data-lake-tools-use-vertex-execution-view/data-lake-tools-vertex-execution-view.png)
2425

25-
The **Vertex selector** on the left lets you select vertices by features (such as top 10 data read, or choose by stage). One of the most commonly-used filters is to see the **vertices on critical path**. The **Critical path** is the longest chain of vertices of a U-SQL job. Understanding the critical path is useful for optimizing your jobs by checking which vertex takes the longest time.
26+
The **Vertex selector** on the left lets you select vertices by features (such as top 10 data read, or choose by stage). One of the most commonly used filters is to see the **vertices on critical path**. The **Critical path** is the longest chain of vertices of a U-SQL job. Understanding the critical path is useful for optimizing your jobs by checking which vertex takes the longest time.
2627

2728
![Screenshot that shows the Vertex Execution View top-center pane that displays the "running status of all the vertices".](./media/data-lake-analytics-data-lake-tools-use-vertex-execution-view/data-lake-tools-vertex-execution-view-pane2.png)
2829

@@ -31,7 +32,8 @@ The top center pane shows the **running status of all the vertices**.
3132
![Screenshot that shows the Vertex Execution View bottom-center pane that displays information about each vertex.](./media/data-lake-analytics-data-lake-tools-use-vertex-execution-view/data-lake-tools-vertex-execution-view-pane3.png)
3233

3334
The bottom center pane shows information about each vertex:
34-
* Process Name: The name of the vertex instance. It is composed of different parts in StageName|VertexName|VertexRunInstance. For example, the SV7_Split[62].v1 vertex stands for the second running instance (.v1, index starting from 0) of Vertex number 62 in Stage SV7_Split.
35+
36+
* Process Name: The name of the vertex instance. It's composed of different parts in StageName|VertexName|VertexRunInstance. For example, the SV7_Split[62].v1 vertex stands for the second running instance (.v1, index starting from 0) of Vertex number 62 in Stage SV7_Split.
3537
* Total Data Read/Written: The data was read/written by this vertex.
3638
* State/Exit Status: The final status when the vertex is ended.
3739
* Exit Code/Failure Type: The error when the vertex failed.
@@ -43,6 +45,7 @@ The bottom center pane shows information about each vertex:
4345
* Process Create Start Time/Process Queued Time/Process Start Time/Process Complete Time: when the vertex process starts creation; when the vertex process starts to queue; when the certain vertex process starts; when the certain vertex is completed.
4446

4547
## Next steps
48+
4649
* To log diagnostics information, see [Accessing diagnostics logs for Azure Data Lake Analytics](data-lake-analytics-diagnostic-logs.md)
4750
* To see a more complex query, see [Analyze Website logs using Azure Data Lake Analytics](data-lake-analytics-analyze-weblogs.md).
4851
* To view job details, see [Use Job Browser and Job View for Azure Data lake Analytics jobs](data-lake-analytics-data-lake-tools-view-jobs.md)

articles/data-lake-analytics/data-lake-analytics-manage-use-cli.md

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -3,15 +3,15 @@ title: Manage Azure Data Lake Analytics using Azure CLI
33
description: This article describes how to use the Azure CLI to manage Data Lake Analytics jobs, data sources, & users.
44
ms.service: data-lake-analytics
55
ms.topic: how-to
6-
ms.date: 01/29/2018
6+
ms.date: 01/27/2023
77
---
88
# Manage Azure Data Lake Analytics using the Azure CLI
99

1010
[!INCLUDE [manage-selector](../../includes/data-lake-analytics-selector-manage.md)]
1111

1212
[!INCLUDE [retirement-flag](includes/retirement-flag.md)]
1313

14-
Learn how to manage Azure Data Lake Analytics accounts, data sources, users, and jobs using the Azure CLI. To see management topics using other tools, click the tab select above.
14+
Learn how to manage Azure Data Lake Analytics accounts, data sources, users, and jobs using the Azure CLI. To see management topics using other tools, select the tab select above.
1515

1616
## Prerequisites
1717

@@ -39,7 +39,7 @@ Before you begin this tutorial, you must have the following resources:
3939

4040
## Manage accounts
4141

42-
Before running any Data Lake Analytics jobs, you must have a Data Lake Analytics account. Unlike Azure HDInsight, you don't pay for an Analytics account when it is not running a job. You only pay for the time when it is running a job. For more information, see [Azure Data Lake Analytics Overview](data-lake-analytics-overview.md).
42+
Before running any Data Lake Analytics jobs, you must have a Data Lake Analytics account. Unlike Azure HDInsight, you don't pay for an Analytics account when it isn't running a job. You only pay for the time when it's running a job. For more information, see [Azure Data Lake Analytics Overview](data-lake-analytics-overview.md).
4343

4444
### Create accounts
4545

@@ -85,7 +85,7 @@ Data Lake Analytics currently supports the following two data sources:
8585
- [Azure Storage](../storage/common/storage-introduction.md)
8686

8787
When you create an Analytics account, you must designate an Azure Data Lake Storage account to be the default
88-
storage account. The default Data Lake storage account is used to store job metadata and job audit logs. After you have created an Analytics account, you can add additional Data Lake Storage accounts and/or Azure Storage account.
88+
storage account. The default Data Lake storage account is used to store job metadata and job audit logs. After you've created an Analytics account, you can add other Data Lake Storage accounts and/or Azure Storage account.
8989

9090
### Find the default Data Lake Store account
9191

@@ -95,7 +95,7 @@ You can view the default Data Lake Store account used by running the `az dla acc
9595
az dla account show --account "<Data Lake Analytics account name>"
9696
```
9797

98-
### Add additional Blob storage accounts
98+
### Add other Blob storage accounts
9999

100100
```azurecli
101101
az dla account blob-storage add --access-key "<Azure Storage Account Key>" --account "<Data Lake Analytics account name>" --storage-account-name "<Storage account name>"
@@ -105,9 +105,9 @@ You can view the default Data Lake Store account used by running the `az dla acc
105105
> Only Blob storage short names are supported. Don't use FQDN, for example "myblob.blob.core.windows.net".
106106
>
107107
108-
### Add additional Data Lake Store accounts
108+
### Add other Data Lake Store accounts
109109

110-
The following command updates the specified Data Lake Analytics account with an additional Data Lake Store account:
110+
The following command updates the specified Data Lake Analytics account with another Data Lake Store account:
111111

112112
```azurecli
113113
az dla account data-lake-store add --account "<Data Lake Analytics account name>" --data-lake-store-account-name "<Data Lake Store account name>"

0 commit comments

Comments
 (0)