You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/hdinsight/hadoop/apache-hadoop-using-apache-hive-as-an-etl-tool.md
+15-17Lines changed: 15 additions & 17 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,35 +1,35 @@
1
1
---
2
2
title: Using Apache Hive as an ETL Tool - Azure HDInsight
3
3
description: Use Apache Hive to extract, transform, and load (ETL) data in Azure HDInsight.
4
-
ms.service: hdinsight
5
4
author: ashishthaps
6
5
ms.author: ashishth
7
6
ms.reviewer: jasonh
8
-
ms.custom: hdinsightactive
7
+
ms.service: hdinsight
9
8
ms.topic: conceptual
10
-
ms.date: 11/22/2019
9
+
ms.custom: hdinsightactive,seoapr2020
10
+
ms.date: 04/28/2020
11
11
---
12
12
13
13
# Use Apache Hive as an Extract, Transform, and Load (ETL) tool
14
14
15
-
You typically need to clean and transform incoming data before loading it into a destination suitable for analytics. Extract, Transform, and Load (ETL) operations are used to prepare data and load it into a data destination. Apache Hive on HDInsight can read in unstructured data, process the data as needed, and then load the data into a relational data warehouse for decision support systems. In this approach, data is extracted from the source and stored in scalable storage, such as Azure Storage blobs or Azure Data Lake Storage. The data is then transformed using a sequence of Hive queries and is finally staged within Hive in preparation for bulk loading into the destination data store.
15
+
You typically need to clean and transform incoming data before loading it into a destination suitable for analytics. Extract, Transform, and Load (ETL) operations are used to prepare data and load it into a data destination. Apache Hive on HDInsight can read in unstructured data, process the data as needed, and then load the data into a relational data warehouse for decision support systems. In this approach, data is extracted from the source. Then stored in adaptable storage, such as Azure Storage blobs or Azure Data Lake Storage. The data is then transformed using a sequence of Hive queries. Then staged within Hive in preparation for bulk loading into the destination data store.
16
16
17
17
## Use case and model overview
18
18
19
-
The following figure shows an overview of the use case and model for ETL automation. Input data is transformed to generate the appropriate output. During that transformation, the data can change shape, data type, and even language. ETL processes can convert Imperial to metric, change time zones, and improve precision to properly align with existing data in the destination. ETL processes can also combine new data with existing data to keep reporting up to date, or to provide further insight into existing data. Applications such as reporting tools and services can then consume this data in the desired format.
19
+
The following figure shows an overview of the use case and model for ETL automation. Input data is transformed to generate the appropriate output. During that transformation, the data changes shape, data type, and even language. ETL processes can convert Imperial to metric, change time zones, and improve precision to properly align with existing data in the destination. ETL processes can also combine new data with existing data to keep reporting up to date, or to provide further insight into existing data. Applications such as reporting tools and services can then consume this data in the wanted format.
20
20
21
21
22
22
23
-
Hadoop is typically used in ETL processes that import either a massive number of text files (like CSVs) or a smaller but frequently changing number of text files, or both. Hive is a great tool to use to prepare the data before loading it into the data destination. Hive allows you to create a schema over the CSV and use a SQL-like language to generate MapReduce programs that interact with the data.
23
+
Hadoop is typically used in ETL processes that import either a massive number of text files (like CSVs). Or a smaller but frequently changing number of text files, or both. Hive is a great tool to use to prepare the data before loading it into the data destination. Hive allows you to create a schema over the CSV and use a SQL-like language to generate MapReduce programs that interact with the data.
24
24
25
-
The typical steps to using Hive to perform ETL are as follows:
25
+
The typical steps to using Hive to do ETL are as follows:
26
26
27
27
1. Load data into Azure Data Lake Storage or Azure Blob Storage.
28
28
2. Create a Metadata Store database (using Azure SQL Database) for use by Hive in storing your schemas.
29
29
3. Create an HDInsight cluster and connect the data store.
30
30
4. Define the schema to apply at read-time over data in the data store:
31
31
32
-
```
32
+
```hql
33
33
DROP TABLE IF EXISTS hvac;
34
34
35
35
--create the hvac table on comma-separated sensor data stored in Azure Storage blobs
@@ -61,30 +61,28 @@ Data sources are typically external data that can be matched to existing data in
61
61
62
62
## Output targets
63
63
64
-
You can use Hive to output data to a variety of targets including:
64
+
You can use Hive to output data to different kinds of targets including:
65
65
66
66
* A relational database, such as SQL Server or Azure SQL Database.
67
67
* A data warehouse, such as Azure SQL Data Warehouse.
68
68
* Excel.
69
69
* Azure table and blob storage.
70
70
* Applications or services that require data to be processed into specific formats, or as files that contain specific types of information structure.
71
-
* A JSON Document Store like [Azure Cosmos DB](https://azure.microsoft.com/services/cosmos-db/).
71
+
* A JSON Document Store like Azure Cosmos DB.
72
72
73
73
## Considerations
74
74
75
75
The ETL model is typically used when you want to:
76
76
77
-
* Load stream data or large volumes of semi-structured or unstructured data from external sources into an existing database or information system.
78
-
* Clean, transform, and validate the data before loading it, perhaps by using more than one transformation pass through the cluster.
79
-
* Generate reports and visualizations that are regularly updated. For example, if the report takes too long to generate during the day, you can schedule the report to run at night. To automatically run a Hive query, you can use [Azure Logic Apps](../../logic-apps/logic-apps-overview.md) and PowerShell.
77
+
`*` Load stream data or large volumes of semi-structured or unstructured data from external sources into an existing database or information system.
78
+
`*` Clean, transform, and validate the data before loading it, perhaps by using more than one transformation pass through the cluster.
79
+
`*` Generate reports and visualizations that are regularly updated. For example, if the report takes too long to generate during the day, you can schedule the report to run at night. To automatically run a Hive query, you can use [Azure Logic Apps](../../logic-apps/logic-apps-overview.md) and PowerShell.
80
80
81
81
If the target for the data isn't a database, you can generate a file in the appropriate format within the query, for example a CSV. This file can then be imported into Excel or Power BI.
82
82
83
-
If you need to execute several operations on the data as part of the ETL process, consider how you manage them. If the operations are controlled by an external program, rather than as a workflow within the solution, you need to decide whether some operations can be executed in parallel, and to detect when each job completes. Using a workflow mechanism such as Oozie within Hadoop may be easier than trying to orchestrate a sequence of operations using external scripts or custom programs. For more information about Oozie, see [Workflow and job orchestration](https://msdn.microsoft.com/library/dn749829.aspx).
83
+
If you need to execute several operations on the data as part of the ETL process, consider how you manage them. With operations controlled by an external program, rather than as a workflow within the solution, decide whether some operations can be executed in parallel. And to detect when each job completes. Using a workflow mechanism such as Oozie within Hadoop may be easier than trying to orchestrate a sequence of operations using external scripts or custom programs.
84
84
85
85
## Next steps
86
86
87
87
* [ETL at scale](apache-hadoop-etl-at-scale.md)
88
-
* [Operationalize a data pipeline](../hdinsight-operationalize-data-pipeline.md)
89
-
90
-
<!-- * [ETL Deep Dive](../hdinsight-etl-deep-dive.md) -->
88
+
* [`Operationalize a data pipeline`](../hdinsight-operationalize-data-pipeline.md)
Copy file name to clipboardExpand all lines: articles/mysql/howto-database-threat-protection-portal.md
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,8 +1,8 @@
1
1
---
2
2
title: Advanced Threat Protection - Azure portal - Azure Database for MySQL
3
3
description: Learn how to configure Advanced Threat Protection to detect anomalous database activities indicating potential security threats to the database.
Copy file name to clipboardExpand all lines: articles/postgresql/howto-manage-vnet-using-cli.md
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,8 +1,8 @@
1
1
---
2
2
title: Use virtual network rules - Azure CLI - Azure Database for PostgreSQL - Single Server
3
3
description: This article describes how to create and manage VNet service endpoints and rules for Azure Database for PostgreSQL using Azure CLI command line.
title: Set up Azure multi-factor authentication for Windows Virtual Desktop - Azure
3
-
description: How to set up Azure multi-factor authentication for increased security in Windows Virtual Desktop.
2
+
title: Set up Azure Multi-Factor Authentication for Windows Virtual Desktop - Azure
3
+
description: How to set up Azure Multi-Factor Authentication for increased security in Windows Virtual Desktop.
4
4
services: virtual-desktop
5
5
author: Heidilohr
6
6
7
7
ms.service: virtual-desktop
8
8
ms.topic: conceptual
9
-
ms.date: 04/01/2020
9
+
ms.date: 04/22/2020
10
10
ms.author: helohr
11
11
manager: lizross
12
12
---
13
-
14
-
# Set up Azure Multi-Factor Authentication
13
+
# Enable Azure Multi-Factor Authentication for Windows Virtual Desktop
15
14
16
15
The Windows client for Windows Virtual Desktop is an excellent option for integrating Windows Virtual Desktop with your local machine. However, when you configure your Windows Virtual Desktop account into the Windows Client, there are certain measures you'll need to take to keep yourself and your users safe.
17
16
@@ -23,71 +22,34 @@ While remembering credentials is convenient, it can also make deployments on Ent
23
22
24
23
Here's what you'll need to get started:
25
24
26
-
- Assign all your users one of the following licenses:
27
-
- Microsoft 365 E3 or E5
28
-
- Azure Active Directory Premium P1 or P2
29
-
- Enterprise Mobility + Security E3 or E5
25
+
- Assign users a license that includes Azure Active Directory Premium P1 or P2.
30
26
- An Azure Active Directory group with your users assigned as group members.
31
27
- Enable Azure MFA for all your users. For more information about how to do that, see [How to require two-step verification for a user](../active-directory/authentication/howto-mfa-userstates.md#view-the-status-for-a-user).
32
28
33
-
>[!NOTE]
34
-
>The following setting also applies to the [Windows Virtual Desktop web client](https://rdweb.wvd.microsoft.com/webclient/index.html).
35
-
36
-
## Opt in to the Conditional Access policy
37
-
38
-
1. Open **Azure Active Directory**.
39
-
40
-
2. Go to the **All applications** tab. In the "Application type" drop-down menu, select **Enterprise Applications**, then search for **Windows Virtual Desktop Client**.
41
-
42
-
43
-
44
-
3. Select **Conditional Access**.
45
-
46
-
47
-
48
-
4. Select **+ New policy**.
49
-
50
-
51
-
52
-
5. Enter a **name** for the **rule**, then **select** the *name of the **group** you created in the prerequisites.
53
-
54
-
6. Select **Select**, then select **Done**.
55
-
56
-
7. Next, open **Cloud Apps or actions**.
57
-
58
-
8. On the **Select** panel, select the **Windows Virtual Desktop** Enterprise app.
59
-
60
-
61
-
62
-
>[!NOTE]
63
-
>You should also see the Windows Virtual Desktop Client app selected on the left side of the screen, as shown in the following image. You need both the Windows Virtual Desktop and Windows Virtual Desktop Client Enterprise apps for the policy to work.
64
-
>
65
-
> 
66
-
67
-
9. Select **Select**
68
-
69
-
10. Next, open **Grant**
70
-
71
-
11. Select **Require multi-factor authentication**, then select **Require one of the selected controls**.
72
-
73
-
74
-
75
-
>[!NOTE]
76
-
>If you have MDM-enrolled devices in your organization and don't want them to show the MFA prompt, you can also select **Require device to be marked as compliant**.
29
+
> [!NOTE]
30
+
> The following setting also applies to the [Windows Virtual Desktop web client](https://rdweb.wvd.microsoft.com/webclient/index.html).
77
31
78
-
12. Select **Session**.
32
+
## Create a Conditional Access policy
79
33
80
-
13. Set the **Sign-in frequency**to **Active**, then change its value to **1 Hours**.
34
+
This section will show you how to create a Conditional Access policy that requires multi-factor authentication when connecting to Windows Virtual Desktop.
81
35
82
-
83
-
84
-
>[!NOTE]
85
-
>Active sessions in your Windows Virtual Desktop environment will continue to work as you change the policy. However, if you disconnect or sign off, you'll need to provide your credentials again after 60 minutes. As you change the settings, you can extend the timeout period as much as you want (as long as it aligns with your organization's security policy).
86
-
>
87
-
>The default setting is a rolling window of 90 days, which means the client will ask users to sign in again when they try to access a resource after being inactive on their machine for 90 days or longer.
36
+
1. Sign in to the **Azure portal** as a global administrator, security administrator, or Conditional Access administrator.
37
+
1. Browse to **Azure Active Directory** > **Security** > **Conditional Access**.
38
+
1. Select **New policy**.
39
+
1. Give your policy a name. We recommend that organizations create a meaningful standard for the names of their policies.
40
+
1. Under **Assignments**, select **Users and groups**.
41
+
1. Under **Include**, select **Select users and groups** > **Users and groups** > Choose the group created in the prerequisites stage.
42
+
1. Select **Done**.
43
+
1. Under **Cloud apps or actions** > **Include**, select **Select apps**.
44
+
1. Choose **Windows Virtual Desktop** and **Windows Virtual Desktop Client**, and select **Select** then **Done**.
45
+
46
+
1. Under **Access controls** > **Grant**, select **Grant access**, **Require multi-factor authentication**, and then **Select**.
47
+
1. Under **Access controls** > **Session**, select **Sign-in frequency**, set the value to **1** and the unit to **Hours**, and then **Select**.
48
+
1. Confirm your settings and set **Enable policy** to **On**.
49
+
1. Select **Create** to enable your policy.
88
50
89
-
14. Enable the policy.
51
+
## Next steps
90
52
91
-
15. Select **Create** to confirm the policy.
53
+
-[Learn more about Conditional Access policies](../active-directory/conditional-access/concept-conditional-access-policies.md)
92
54
93
-
You're all done! Feel free to test the policy to make sure your allow list works as intended.
55
+
-[Learn more about user sign in frequency](../active-directory/conditional-access/howto-conditional-access-session-lifetime.md#user-sign-in-frequency)
Copy file name to clipboardExpand all lines: articles/virtual-machines/sizes-gpu.md
+3-1Lines changed: 3 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -31,12 +31,14 @@ GPU optimized VM sizes are specialized virtual machines available with single or
31
31
32
32
## Supported operating systems and drivers
33
33
34
-
To take advantage of the GPU capabilities of Azure N-series VMs, NVIDIA GPU drivers must be installed.
34
+
To take advantage of the GPU capabilities of Azure N-series VMs, NVIDIA or AMD GPU drivers must be installed.
35
35
36
36
The [NVIDIA GPU Driver Extension](/azure/virtual-machines/extensions/hpccompute-gpu-windows) installs appropriate NVIDIA CUDA or GRID drivers on an N-series VM. Install or manage the extension using the Azure portal or tools such as Azure PowerShell or Azure Resource Manager templates. See the [NVIDIA GPU Driver Extension documentation](/azure/virtual-machines/extensions/hpccompute-gpu-windows) for supported operating systems and deployment steps. For general information about VM extensions, see [Azure virtual machine extensions and features](/azure/virtual-machines/extensions/overview).
37
37
38
38
If you choose to install NVIDIA GPU drivers manually, see [N-series GPU driver setup for Windows](/azure/virtual-machines/windows/n-series-driver-setup) or [N-series GPU driver setup for Linux](/azure/virtual-machines/linux/n-series-driver-setup) for supported operating systems, drivers, installation, and verification steps.
39
39
40
+
To manually install the AMD GPU drivers, see [N-series AMD GPU driver setup for Windows](/azure/virtual-machines/windows/n-series-amd-driver-setup) for supported operating systems, drivers, installation, and verification steps.
41
+
40
42
## Deployment considerations
41
43
42
44
- For availability of N-series VMs, see [Products available by region](https://azure.microsoft.com/regions/services/).
0 commit comments