You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/data-lake-analytics/data-lake-analytics-account-policies.md
+18-16Lines changed: 18 additions & 16 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -2,9 +2,9 @@
2
2
title: Manage Azure Data Lake Analytics Account Policies
3
3
description: Learn how to use account policies to control usage of a Data Lake Analytics account, such as maximum AUs and maximum jobs.
4
4
ms.service: data-lake-analytics
5
-
ms.reviewer: jasonh
5
+
ms.reviewer: whhender
6
6
ms.topic: how-to
7
-
ms.date: 04/30/2018
7
+
ms.date: 01/27/2023
8
8
---
9
9
# Manage Azure Data Lake Analytics using Account Policies
10
10
@@ -18,14 +18,14 @@ These policies apply to all jobs in a Data Lake Analytics account.
18
18
19
19
### Maximum number of AUs in a Data Lake Analytics account
20
20
21
-
A policy controls the total number of Analytics Units (AUs) your Data Lake Analytics account can use. By default, the value is set to 250. For example, if this value is set to 250 AUs, you can have one job running with 250 AUs assigned to it, or 10 jobs running with 25 AUs each. Additional jobs that are submitted are queued until the running jobs are finished. When running jobs are finished, AUs are freed up for the queued jobs to run.
21
+
A policy controls the total number of Analytics Units (AUs) your Data Lake Analytics account can use. By default, the value is set to 250. For example, if this value is set to 250 AUs, you can have one job running with 250 AUs assigned to it, or 10 jobs running with 25 AUs each. Other jobs that are submitted are queued until the running jobs are finished. When running jobs are finished, AUs are freed up for the queued jobs to run.
22
22
23
23
To change the number of AUs for your Data Lake Analytics account:
24
24
25
25
1. In the Azure portal, go to your Data Lake Analytics account.
26
-
2.Click**Limits and policies**.
26
+
2.Select**Limits and policies**.
27
27
3. Under **Maximum AUs**, move the slider to select a value, or enter the value in the text box.
28
-
4.Click**Save**.
28
+
4.Select**Save**.
29
29
30
30
> [!NOTE]
31
31
> If you need more than the default (250) AUs, in the portal, click **Help+Support** to submit a support request. The number of AUs available in your Data Lake Analytics account can be increased.
@@ -37,23 +37,23 @@ This policy limits how many jobs can run simultaneously. By default, this value
37
37
To change the number of jobs that can run simultaneously:
38
38
39
39
1. In the Azure portal, go to your Data Lake Analytics account.
40
-
2.Click**Limits and policies**.
40
+
2.Select**Limits and policies**.
41
41
3. Under **Maximum Number of Running Jobs**, move the slider to select a value, or enter the value in the text box.
42
-
4.Click**Save**.
42
+
4.Select**Save**.
43
43
44
44
> [!NOTE]
45
45
> If you need to run more than the default (20) number of jobs, in the portal, click **Help+Support** to submit a support request. The number of jobs that can run simultaneously in your Data Lake Analytics account can be increased.
46
46
47
47
### How long to keep job metadata and resources
48
48
49
-
When your users run U-SQL jobs, the Data Lake Analytics service keeps all related files. These files include the U-SQL script, the DLL files referenced in the U-SQL script, compiled resources, and statistics. The files are in the /system/ folder of the default Azure Data Lake Storage account. This policy controls how long these resources are stored before they are automatically deleted (the default is 30 days). You can use these files for debugging, and for performance-tuning of jobs that you'll rerun in the future.
49
+
When your users run U-SQL jobs, the Data Lake Analytics service keeps all related files. These files include the U-SQL script, the DLL files referenced in the U-SQL script, compiled resources, and statistics. The files are in the /system/ folder of the default Azure Data Lake Storage account. This policy controls how long these resources are stored before they're automatically deleted (the default is 30 days). You can use these files for debugging, and for performance-tuning of jobs that you'll rerun in the future.
50
50
51
51
To change how long to keep job metadata and resources:
52
52
53
53
1. In the Azure portal, go to your Data Lake Analytics account.
54
-
2.Click**Limits and policies**.
54
+
2.Select**Limits and policies**.
55
55
3. Under **Days to Retain Job Queries**, move the slider to select a value, or enter the value in the text box.
56
-
4.Click**Save**.
56
+
4.Select**Save**.
57
57
58
58
## Job-level policies
59
59
@@ -65,7 +65,7 @@ Data Lake Analytics has two policies that you can set at the job level:
65
65
66
66
-**Priority**: Users can only submit jobs that have a priority lower than or equal to this value. A higher number indicates a lower priority. By default, this limit is set to 1, which is the highest possible priority.
67
67
68
-
There is a default policy set on every account. The default policy applies to all users of the account. You can create additional policies for specific users and groups.
68
+
There's a default policy set on every account. The default policy applies to all users of the account. You can create more policies for specific users and groups.
69
69
70
70
> [!NOTE]
71
71
> Account-level policies and job-level policies apply simultaneously.
@@ -74,9 +74,9 @@ There is a default policy set on every account. The default policy applies to al
74
74
75
75
1. In the Azure portal, go to your Data Lake Analytics account.
76
76
77
-
2.Click**Limits and policies**.
77
+
2.Select**Limits and policies**.
78
78
79
-
3. Under **Job Submission Limits**, click the **Add Policy** button. Then, select or enter the following settings:
79
+
3. Under **Job Submission Limits**, select the **Add Policy** button. Then, select or enter the following settings:
80
80
81
81
1.**Compute Policy Name**: Enter a policy name, to remind you of the purpose of the policy.
82
82
@@ -86,19 +86,21 @@ There is a default policy set on every account. The default policy applies to al
86
86
87
87
4.**Set the Priority Limit**: Set the priority limit that applies to the selected user or group.
88
88
89
-
4.Click**Ok**.
89
+
4.Select**Ok**.
90
90
91
91
5. The new policy is listed in the **Default** policy table, under **Job Submission Limits**.
92
92
93
93
## Delete or edit an existing policy
94
94
95
95
1. In the Azure portal, go to your Data Lake Analytics account.
96
96
97
-
2.Click**Limits and policies**.
97
+
2.Select**Limits and policies**.
98
98
99
99
3. Under **Job Submission Limits**, find the policy you want to edit.
100
100
101
-
4. To see the **Delete** and **Edit** options, in the rightmost column of the table, click `...`.## Additional resources for job policies
101
+
4. To see the **Delete** and **Edit** options, in the rightmost column of the table, select `...`.
102
+
103
+
## More resources for job policies
102
104
103
105
-[Policy overview blog post](/archive/blogs/azuredatalake/managing-your-azure-data-lake-analytics-compute-resources-overview)
104
106
-[Account-level policies blog post](/archive/blogs/azuredatalake/managing-your-azure-data-lake-analytics-compute-resources-account-level-policy)
Copy file name to clipboardExpand all lines: articles/data-lake-analytics/data-lake-analytics-cicd-manage-assemblies.md
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -3,7 +3,7 @@ title: Manage U-SQL assemblies in a CI/CD pipeline - Azure Data Lake
3
3
description: 'Learn the best practices for managing U-SQL C# assemblies in a CI/CD pipeline with Azure DevOps.'
4
4
ms.service: data-lake-analytics
5
5
ms.topic: how-to
6
-
ms.date: 10/30/2018
6
+
ms.date: 01/27/2023
7
7
---
8
8
9
9
# Best practices for managing U-SQL assemblies in a CI/CD pipeline
@@ -84,7 +84,7 @@ You can deploy a U-SQL database by using a U-SQL database project or a `.usqldbp
84
84
85
85
### Deploy a U-SQL database in Azure DevOps
86
86
87
-
`PackageDeploymentTool.exe` provides the programming and command-line interfaces that help to deploy U-SQL databases. The SDK is included in the [U-SQL SDK Nuget package](https://www.nuget.org/packages/Microsoft.Azure.DataLake.USQL.SDK/), located at `build/runtime/PackageDeploymentTool.exe`.
87
+
`PackageDeploymentTool.exe` provides the programming and command-line interfaces that help to deploy U-SQL databases. The SDK is included in the [U-SQL SDK NuGet package](https://www.nuget.org/packages/Microsoft.Azure.DataLake.USQL.SDK/), located at `build/runtime/PackageDeploymentTool.exe`.
88
88
89
89
In Azure DevOps, you can use a command-line task and this SDK to set up an automation pipeline for the U-SQL database refresh. [Learn more about the SDK and how to set up a CI/CD pipeline for U-SQL database deployment](data-lake-analytics-cicd-overview.md#deploy-u-sql-database-through-azure-pipelines).
Copy file name to clipboardExpand all lines: articles/data-lake-analytics/data-lake-analytics-data-lake-tools-debug-recurring-job.md
+3-3Lines changed: 3 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,10 +1,10 @@
1
1
---
2
2
title: Debug recurring jobs in Azure Data Lake Analytics
3
3
description: Learn how to use Azure Data Lake Tools for Visual Studio to debug an abnormal recurring job.
4
-
ms.reviewer: jasonh
4
+
ms.reviewer: whhender
5
5
ms.service: data-lake-analytics
6
6
ms.topic: how-to
7
-
ms.date: 05/20/2018
7
+
ms.date: 01/27/2023
8
8
---
9
9
10
10
# Troubleshoot an abnormal recurring job
@@ -55,7 +55,7 @@ You can find all submitted recurring jobs through the job list at the bottom of
55
55
56
56
57
57
58
-
Pay attention to the big differences between these two jobs. Those differences are probably causing the performance problems. To check further, use the steps in the following diagram:
58
+
Pay attention to the differences between these two jobs. Those differences are probably causing the performance problems. To check further, use the steps in the following diagram:
59
59
60
60
Copy file name to clipboardExpand all lines: articles/data-lake-analytics/data-lake-analytics-data-lake-tools-export-database.md
+3-3Lines changed: 3 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,10 +1,10 @@
1
1
---
2
2
title: Export U-SQL database- Azure Data Lake Tools for Visual Studio
3
3
description: Learn how to use Azure Data Lake Tools for Visual Studio to export a U-SQL database and automatically import it to a local account.
4
-
ms.reviewer: jasonh
4
+
ms.reviewer: whhender
5
5
ms.service: data-lake-analytics
6
6
ms.topic: how-to
7
-
ms.date: 11/27/2017
7
+
ms.date: 01/27/2023
8
8
---
9
9
10
10
# Export a U-SQL database
@@ -40,7 +40,7 @@ The export action is completed by running a U-SQL job. Therefore, exporting from
40
40
41
41
### Step 3: Check the objects list and other configurations
42
42
43
-
In this step, you can verify the selected objects in the **Export object list** box. If there are any errors, select **Previous** to go back and correctly configure the objects that you want to export.
43
+
In this step, you can verify the selected objects in the **Export object list** box. If there are any errors, select **Previous** to go back, and correctly configure the objects that you want to export.
44
44
45
45
You can also configure other settings for the export target. Configuration descriptions are listed in the following table:
Open a U-SQL job in Data Lake Tools for Visual Studio. Click **Vertex Execution View** in the bottom left corner. You may be prompted to load profiles first and it can take some time depending on your network connectivity.
15
+
16
+
Open a U-SQL job in Data Lake Tools for Visual Studio. Select **Vertex Execution View** in the bottom left corner. You may be prompted to load profiles first and it can take some time depending on your network connectivity.
17
17
18
18
19
19
20
20
## Understand Vertex Execution View
21
+
21
22
The Vertex Execution View has three parts:
22
23
23
24
24
25
25
-
The **Vertex selector** on the left lets you select vertices by features (such as top 10 data read, or choose by stage). One of the most commonly-used filters is to see the **vertices on critical path**. The **Critical path** is the longest chain of vertices of a U-SQL job. Understanding the critical path is useful for optimizing your jobs by checking which vertex takes the longest time.
26
+
The **Vertex selector** on the left lets you select vertices by features (such as top 10 data read, or choose by stage). One of the most commonlyused filters is to see the **vertices on critical path**. The **Critical path** is the longest chain of vertices of a U-SQL job. Understanding the critical path is useful for optimizing your jobs by checking which vertex takes the longest time.
26
27
27
28
28
29
@@ -31,7 +32,8 @@ The top center pane shows the **running status of all the vertices**.
31
32
32
33
33
34
The bottom center pane shows information about each vertex:
34
-
* Process Name: The name of the vertex instance. It is composed of different parts in StageName|VertexName|VertexRunInstance. For example, the SV7_Split[62].v1 vertex stands for the second running instance (.v1, index starting from 0) of Vertex number 62 in Stage SV7_Split.
35
+
36
+
* Process Name: The name of the vertex instance. It's composed of different parts in StageName|VertexName|VertexRunInstance. For example, the SV7_Split[62].v1 vertex stands for the second running instance (.v1, index starting from 0) of Vertex number 62 in Stage SV7_Split.
35
37
* Total Data Read/Written: The data was read/written by this vertex.
36
38
* State/Exit Status: The final status when the vertex is ended.
37
39
* Exit Code/Failure Type: The error when the vertex failed.
@@ -43,6 +45,7 @@ The bottom center pane shows information about each vertex:
43
45
* Process Create Start Time/Process Queued Time/Process Start Time/Process Complete Time: when the vertex process starts creation; when the vertex process starts to queue; when the certain vertex process starts; when the certain vertex is completed.
44
46
45
47
## Next steps
48
+
46
49
* To log diagnostics information, see [Accessing diagnostics logs for Azure Data Lake Analytics](data-lake-analytics-diagnostic-logs.md)
47
50
* To see a more complex query, see [Analyze Website logs using Azure Data Lake Analytics](data-lake-analytics-analyze-weblogs.md).
48
51
* To view job details, see [Use Job Browser and Job View for Azure Data lake Analytics jobs](data-lake-analytics-data-lake-tools-view-jobs.md)
Learn how to manage Azure Data Lake Analytics accounts, data sources, users, and jobs using the Azure CLI. To see management topics using other tools, click the tab select above.
14
+
Learn how to manage Azure Data Lake Analytics accounts, data sources, users, and jobs using the Azure CLI. To see management topics using other tools, select the tab select above.
15
15
16
16
## Prerequisites
17
17
@@ -39,7 +39,7 @@ Before you begin this tutorial, you must have the following resources:
39
39
40
40
## Manage accounts
41
41
42
-
Before running any Data Lake Analytics jobs, you must have a Data Lake Analytics account. Unlike Azure HDInsight, you don't pay for an Analytics account when it is not running a job. You only pay for the time when it is running a job. For more information, see [Azure Data Lake Analytics Overview](data-lake-analytics-overview.md).
42
+
Before running any Data Lake Analytics jobs, you must have a Data Lake Analytics account. Unlike Azure HDInsight, you don't pay for an Analytics account when it isn't running a job. You only pay for the time when it's running a job. For more information, see [Azure Data Lake Analytics Overview](data-lake-analytics-overview.md).
43
43
44
44
### Create accounts
45
45
@@ -85,7 +85,7 @@ Data Lake Analytics currently supports the following two data sources:
When you create an Analytics account, you must designate an Azure Data Lake Storage account to be the default
88
-
storage account. The default Data Lake storage account is used to store job metadata and job audit logs. After you have created an Analytics account, you can add additional Data Lake Storage accounts and/or Azure Storage account.
88
+
storage account. The default Data Lake storage account is used to store job metadata and job audit logs. After you've created an Analytics account, you can add other Data Lake Storage accounts and/or Azure Storage account.
89
89
90
90
### Find the default Data Lake Store account
91
91
@@ -95,7 +95,7 @@ You can view the default Data Lake Store account used by running the `az dla acc
95
95
az dla account show --account "<Data Lake Analytics account name>"
96
96
```
97
97
98
-
### Add additional Blob storage accounts
98
+
### Add other Blob storage accounts
99
99
100
100
```azurecli
101
101
az dla account blob-storage add --access-key "<Azure Storage Account Key>" --account "<Data Lake Analytics account name>" --storage-account-name "<Storage account name>"
@@ -105,9 +105,9 @@ You can view the default Data Lake Store account used by running the `az dla acc
105
105
> Only Blob storage short names are supported. Don't use FQDN, for example "myblob.blob.core.windows.net".
106
106
>
107
107
108
-
### Add additional Data Lake Store accounts
108
+
### Add other Data Lake Store accounts
109
109
110
-
The following command updates the specified Data Lake Analytics account with an additional Data Lake Store account:
110
+
The following command updates the specified Data Lake Analytics account with another Data Lake Store account:
111
111
112
112
```azurecli
113
113
az dla account data-lake-store add --account "<Data Lake Analytics account name>" --data-lake-store-account-name "<Data Lake Store account name>"
0 commit comments