You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/synapse-analytics/concepts-data-flow-overview.md
+5-7Lines changed: 5 additions & 7 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -6,20 +6,18 @@ ms.author: makromer
6
6
ms.reviewer: daperlov
7
7
ms.service: azure-synapse-analytics
8
8
ms.subservice: pipeline
9
-
ms.topic: conceptual
9
+
ms.topic: concept-article
10
10
ms.custom: references_regions
11
11
ms.date: 12/16/2020
12
12
---
13
13
14
-
# Data flows in Azure Synapse Analytics
15
-
16
-
## What are data flows?
14
+
# What are data flows in Azure Synapse Analytics?
17
15
18
16
Data flows are visually designed data transformations in Azure Synapse Analytics. Data flows allow data engineers to develop data transformation logic without writing code. The resulting data flows are executed as activities within Azure Synapse Analytics pipelines that use scaled-out Apache Spark clusters. Data flow activities can be operationalized using existing Azure Synapse Analytics scheduling, control, flow, and monitoring capabilities.
19
17
20
18
Data flows provide an entirely visual experience with no coding required. Your data flows run on Synapse-managed execution clusters for scaled-out data processing. Azure Synapse Analytics handles all the code translation, path optimization, and execution of your data flow jobs.
21
19
22
-
## Getting started
20
+
## Get started
23
21
24
22
Data flows are created from the Develop pane in Synapse studio. To create a data flow, select the plus sign next to **Develop**, and then select **Data Flow**.
25
23
@@ -29,7 +27,7 @@ This action takes you to the data flow canvas, where you can create your transfo
29
27
30
28
## Authoring data flows
31
29
32
-
Data flow has a unique authoring canvas designed to make building transformation logic easy. The data flow canvas is separated into three parts: the top bar, the graph, and the configuration panel.
30
+
Data flow has a unique authoring canvas designed to make building transformation logic easy. The data flow canvas is separated into three parts: the top bar, the graph, and the configuration panel.
33
31
34
32
35
33
@@ -91,7 +89,7 @@ Data flow integrates with existing Azure Synapse Analytics monitoring capabiliti
91
89
92
90
The Azure Synapse Analytics team has created a [performance tuning guide](../data-factory/concepts-data-flow-performance.md?context=/azure/synapse-analytics/context/context) to help you optimize the execution time of your data flows after building your business logic.
93
91
94
-
## Next steps
92
+
## Related content
95
93
96
94
* Learn how to create a [source transformation](../data-factory/data-flow-source.md?context=/azure/synapse-analytics/context/context).
97
95
* Learn how to build your data flows in [debug mode](../data-factory/concepts-data-flow-debug-mode.md?context=/azure/synapse-analytics/context/context).
Copy file name to clipboardExpand all lines: articles/synapse-analytics/data-integration/concepts-data-factory-differences.md
+1-6Lines changed: 1 addition & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -4,7 +4,7 @@ description: Learn how the data integration capabilities of Azure Synapse Analyt
4
4
author: kromerm
5
5
ms.service: azure-synapse-analytics
6
6
ms.subservice: pipeline
7
-
ms.topic: conceptual
7
+
ms.topic: concept-article
8
8
ms.date: 02/15/2022
9
9
ms.author: makromer
10
10
ms.reviewer: whhender
@@ -14,9 +14,6 @@ ms.reviewer: whhender
14
14
15
15
In Azure Synapse Analytics, the data integration capabilities such as Synapse pipelines and data flows are based upon those of Azure Data Factory. For more information, see [what is Azure Data Factory](../../data-factory/introduction.md).
16
16
17
-
18
-
## Available features in ADF & Azure Synapse Analytics
|**Monitoring**| Monitoring of Spark Jobs for Data Flow | ✗ | ✓ *Leverage the Synapse Spark pools*|
31
28
32
-
## Next steps
33
-
34
29
Get started with data integration in your Synapse workspace by learning how to [ingest data into an Azure Data Lake Storage gen2 account](data-integration-data-lake.md).
Copy file name to clipboardExpand all lines: articles/synapse-analytics/database-designer/concepts-database-templates.md
+4-4Lines changed: 4 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,11 +1,11 @@
1
1
---
2
2
title: Azure Synapse database templates concepts
3
-
description: Learn more about the database templates within Azure Synapse
3
+
description: Learn about how you can standardize data in your late with the database templates within Azure Synapse.
4
4
author: gsaurer
5
5
ms.author: gesaur
6
6
ms.service: azure-synapse-analytics
7
7
ms.subservice: database-editor
8
-
ms.topic: conceptual
8
+
ms.topic: concept-article
9
9
ms.date: 11/02/2021
10
10
ms.custom: template-concept
11
11
---
@@ -16,7 +16,7 @@ Azure Synapse Analytics provides industry specific database templates to help st
16
16
17
17
## Enterprise templates
18
18
19
-
Enterprise database templates contain a subset of tables that are most likely to be of interest to an organization within a specific industry. It provides a high-level overview and describes the connectivity between the related business areas. These templates serve as an accelerator for many types of large projects. For example, the retail template has one enterprise template called "Retail".
19
+
Enterprise database templates contain a subset of tables that are most likely to be of interest to an organization within a specific industry. It provides a high-level overview and describes the connectivity between the related business areas. These templates serve as an accelerator for many types of large projects. For example, the retail template has one enterprise template called "Retail".
@@ -48,7 +48,7 @@ Relationships are associations or interactions between any two tables. For examp
48
48
49
49
Lake database allows for the underlying data to be partitioned for a table for better performance. You can set partition configuration in the storage settings of a table in database editor.
50
50
51
-
## Next steps
51
+
## Related content
52
52
53
53
Continue to explore the capabilities of the database designer using the links below.
Copy file name to clipboardExpand all lines: articles/synapse-analytics/database-designer/concepts-lake-database.md
+9-11Lines changed: 9 additions & 11 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -5,31 +5,29 @@ author: matt1883
5
5
ms.author: mahi
6
6
ms.service: azure-synapse-analytics
7
7
ms.subservice: database-editor
8
-
ms.topic: conceptual
8
+
ms.topic: concept-article
9
9
ms.date: 11/02/2021
10
10
ms.custom: template-concept
11
11
---
12
12
13
-
14
13
# Lake database
15
14
16
-
The lake database in Azure Synapse Analytics enables customers to bring together database design, meta information about the data that is stored and a possibility to describe how and where the data should be stored. Lake database addresses the challenge of today's data lakes where it is hard to understand how data is structured.
15
+
The lake database in Azure Synapse Analytics enables customers to bring together database design, meta information about the data that is stored and a possibility to describe how and where the data should be stored. Lake database addresses the challenge of today's data lakes where it's hard to understand how data is structured.
The new database designer in Synapse Studio gives you the possibility to create a data model for your lake database and add additional information to it. Every Entity and Attribute can be described to provide more information about the model, which not only contains Entities but relationships as well. In particular, the inability to model relationships has been a challenge for the interaction on the data lake. This challenge is now addressed with an integrated designer that provides possibilities that have been available in databases but not on the lake. Also the capability to add descriptions and possible demo values to the model allows people who are interacting with it in the future to have information where they need it to get a better understanding about the data.
21
+
The new database designer in Synapse Studio gives you the possibility to create a data model for your lake database and add additional information to it. Every Entity and Attribute can be described to provide more information about the model, which not only contains Entities but relationships as well. In particular, the inability to model relationships has been a challenge for the interaction on the data lake. This challenge is now addressed with an integrated designer that provides possibilities that have been available in databases but not on the lake. Also the capability to add descriptions and possible demo values to the model allows people who are interacting with it in the future to have information where they need it to get a better understanding about the data.
24
22
25
-
> [!NOTE]
23
+
> [!NOTE]
26
24
> The maximum size of metadata in a lake database is 10 GB. Attempting to publish or update a model that exceeds 10 GB in size will fail. To resolve this issue, reduce the model size by removing tables and columns. Consider splitting large models into multiple lake databases to avoid this limit.
27
25
28
-
## Data storage
26
+
## Data storage
29
27
30
-
Lake databases use a data lake on the Azure Storage account to store the data of the database. The data can be stored in Parquet, Delta or CSV format and different settings can be used to optimize the storage. Every lake database uses a linked service to define the location of the root data folder. For every entity, separate folders are created by default within this database folder on the data lake. By default all tables within a lake database use the same format but the formats and location of the data can be changed per entity if that is requested.
28
+
Lake databases use a data lake on the Azure Storage account to store the data of the database. The data can be stored in Parquet, Delta, or CSV format and different settings can be used to optimize the storage. Every lake database uses a linked service to define the location of the root data folder. For every entity, separate folders are created by default within this database folder on the data lake. By default all tables within a lake database use the same format but the formats and location of the data can be changed per entity if that is requested.
31
29
32
-
> [!NOTE]
30
+
> [!NOTE]
33
31
> Publishing a lake database does not create any of the underlying structures or schemas needed to query the data in Spark or SQL. After publishing, load data into your lake database using [pipelines](../data-integration/data-integration-data-lake.md) to begin querying it.
34
32
>
35
33
> Currently, Delta format support for lake databases is not supported in Synapse Studio.
@@ -38,9 +36,9 @@ Lake databases use a data lake on the Azure Storage account to store the data of
38
36
39
37
## Database compute
40
38
41
-
The lake database is exposed in Synapse SQL serverless SQL pool and Apache Spark providing users with the capability to decouple storage from compute. The metadata that is associated with the lake database makes it easy for different compute engines to not only provide an integrated experience but also use additional information (for example, relationships) that was not originally supported on the data lake.
39
+
The lake database is exposed in Synapse SQL serverless SQL pool and Apache Spark providing users with the capability to decouple storage from compute. The metadata that is associated with the lake database makes it easy for different compute engines to not only provide an integrated experience but also use additional information (for example, relationships) that wasn't originally supported on the data lake.
42
40
43
-
## Next steps
41
+
## Related content
44
42
45
43
Continue to explore the capabilities of the database designer using the links below.
46
44
-[Create lake database quick start](quick-start-create-lake-database.md)
Copy file name to clipboardExpand all lines: articles/synapse-analytics/database-designer/quick-start-create-lake-database.md
+12-11Lines changed: 12 additions & 11 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -6,16 +6,17 @@ ms.author: gesaur
6
6
ms.reviewer: wiassaf, jovanpop
7
7
ms.service: azure-synapse-analytics
8
8
ms.subservice: database-editor
9
-
ms.topic: conceptual
9
+
ms.topic: quickstart
10
10
ms.date: 08/16/2022
11
11
ms.custom: template-concept
12
12
---
13
13
14
14
# Quickstart: Create a new lake database leveraging database templates
15
15
16
-
This quick start gives you a complete sample scenario on how you can apply database templates to create a lake database, align data to your new model, and use the integrated experience to analyze the data.
16
+
This quick start gives you a complete sample scenario on how you can apply database templates to create a lake database, align data to your new model, and use the integrated experience to analyze the data.
17
17
18
18
## Prerequisites
19
+
19
20
- At least **Synapse User** role permissions are required for exploring a lake database template from Gallery.
20
21
-**Synapse Administrator**, or **Synapse Contributor** permissions are required on the Azure Synapse workspace for creating a lake database.
21
22
-**Storage Blob Data Contributor** permissions are required on data lake when using create table **From data lake** option.
@@ -26,13 +27,13 @@ Use the new database templates functionality to create a lake database that you
26
27
27
28
For our scenario we will use the `Retail` database template and select the following entities:
28
29
29
-
-**RetailProduct** - A product is anything that can be offered to a market that might satisfy a need by potential customers. That product is the sum of all physical, psychological, symbolic, and service attributes associated with it.
30
-
-**Transaction** - The lowest level of executable work or customer activity.
30
+
-**RetailProduct** - A product is anything that can be offered to a market that might satisfy a need by potential customers. That product is the sum of all physical, psychological, symbolic, and service attributes associated with it.
31
+
-**Transaction** - The lowest level of executable work or customer activity.
31
32
A transaction consists of one or more discrete events.
32
-
-**TransactionLineItem** - The components of a Transaction broken down by Product and Quantity, one per line item.
33
-
-**Party** - A party is an individual, organization, legal entity, social organization, or business unit of interest to the business.
34
-
-**Customer** - A customer is an individual or legal entity that has or has purchased a product or service.
35
-
-**Channel** - A channel is a means by which products or services are sold and/or distributed.
33
+
-**TransactionLineItem** - The components of a Transaction broken down by Product and Quantity, one per line item.
34
+
-**Party** - A party is an individual, organization, legal entity, social organization, or business unit of interest to the business.
35
+
-**Customer** - A customer is an individual or legal entity that has or has purchased a product or service.
36
+
-**Channel** - A channel is a means by which products or services are sold and/or distributed.
36
37
37
38
The easiest way to find entities is by using the search box above the different business areas that contain the tables.
38
39
@@ -83,6 +84,6 @@ You can use lake database to train your machine learning models and score the da
83
84
## Next steps
84
85
85
86
Continue to explore the capabilities of the database designer using the links below.
0 commit comments