You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/purview/tutorial-custom-types.md
+40-35Lines changed: 40 additions & 35 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,6 +1,6 @@
1
1
---
2
-
title: "What are type definitions and how to create custom types in Microsoft Purview"
3
-
description: This tutorial will explain what type definitions are, how to create custom ones and how to initialize assets of those custom types in Microsoft Purview.
2
+
title: Type definitions and how to create custom types in Microsoft Purview
3
+
description: This tutorial will explain what type definitions are, how to create custom type definitions, and how to initialize assets of those custom types in Microsoft Purview.
4
4
author: adinastoll
5
5
ms.author: adnegrau
6
6
ms.service: purview
@@ -9,9 +9,9 @@ ms.date: 02/02/2023
9
9
---
10
10
11
11
12
-
# Tutorial: What are type definitions and how to create custom ones in Microsoft Purview
12
+
# Tutorial: Type definitions and how to create custom types
13
13
14
-
This tutorial will explain what type definitions are, how to create custom ones and how to initialize assets of custom types in Microsoft Purview.
14
+
This tutorial will explain what type definitions are, how to create custom types, and how to initialize assets of custom types in Microsoft Purview.
15
15
16
16
In this tutorial, you'll learn:
17
17
@@ -27,46 +27,51 @@ For this tutorial you'll need:
27
27
28
28
* An Azure account with an active subscription. If you don't have one, you can [create an account for free](https://azure.microsoft.com/free/?WT.mc_id=A261C142F).
29
29
* An active Microsoft Purview (formerly Azure Purview) account. If you don't have one, see the [quickstart for creating a Microsoft Purview account](create-microsoft-purview-portal.md).
30
-
* A bearer token to your Purview account. To establish a bearer token and to call any data plane APIs, see the documentation about how to [call REST APIs for Microsoft Purview data planes](tutorial-using-rest-apis.md).
31
-
* Atlas endpoint of your Purview account. To get your Atlas endpoint, follow the *Atlas endpoint* section from [here](tutorial-atlas-2-2-apis.md#atlas-endpoint).
30
+
* A bearer token to your Microsoft Purview account. To establish a bearer token and to call any data plane APIs, see the documentation about how to [call REST APIs for Microsoft Purview data planes](tutorial-using-rest-apis.md).
31
+
*Apache Atlas endpoint of your Microsoft Purview account. To get your Apache Atlas endpoint, follow the *Apache Atlas endpoint* section from [here](tutorial-atlas-2-2-apis.md#atlas-endpoint).
32
32
33
33
> [!NOTE]
34
-
> Before moving to the hands-on part of the tutorial, the first four sections will explain what System Type is and how it is used in Purview.
34
+
> Before moving to the hands-on part of the tutorial, the first four sections will explain what System Type is and how it is used in Microsoft Purview.
35
35
> All the REST API calls described further will use the **bearer token** and the **endpoint** which are described in the prerequisites.
36
+
>
37
+
> To skip directly to the steps, use these links:
38
+
>
39
+
>*[Create custom type definitions](#create-definitions)
40
+
>*[Initialize assets of custom types](#initialize-assets-of-custom-types)
36
41
37
42
## What is a *Type* in Microsoft Purview
38
43
39
-
Purview relies on the [Type System](https://atlas.apache.org/2.0.0/TypeSystem.html) from Apache Atlas. All metadata objects (assets) managed by Purview (out of the box or through custom types) are modeled using type definitions. Understanding the Type System is fundamental in order to create new custom types in Purview.
44
+
Microsoft Purview relies on the [Type System](https://atlas.apache.org/2.0.0/TypeSystem.html) from Apache Atlas. All metadata objects (assets) managed by Microsoft Purview (out of the box or through custom types) are modeled using type definitions. Understanding the Type System is fundamental to create new custom types in Microsoft Purview.
40
45
41
46
Essentially, a *Type* can be seen as a *Class* from Object Oriented Programming (OOP):
42
47
43
-
*it defines the properties that represent that type
44
-
*each type is uniquely identified by its *name*
45
-
*a*type* can inherit from a *supertType*. This is an equivalent concept as inheritance from OOP. A type that extends a superType will inherit the attributes of the superType.
48
+
*It defines the properties that represent that type
49
+
*Each type is uniquely identified by its *name*
50
+
*A*type* can inherit from a *supertType*. This is an equivalent concept as inheritance from OOP. A type that extends a superType will inherit the attributes of the superType.
46
51
47
-
You can see all type definitions in your Purview account by sending a `GET` request to [All Type Definitions](/rest/api/purview/catalogdataplane/types/get-all-type-definitions) endpoint:
52
+
You can see all type definitions in your Microsoft Purview account by sending a `GET` request to the[All Type Definitions](/rest/api/purview/catalogdataplane/types/get-all-type-definitions) endpoint:
48
53
49
54
```
50
55
GET https://{{ENDPOINT}}/catalog/api/atlas/v2/types/typedefs
51
56
```
52
57
53
58
Alternatively, you can use the [Microsoft Purview Custom Types Tool](https://github.com/microsoft/Purview-Custom-Types-Tool-Solution-Accelerator) to visualize all the type definitions.
54
59
55
-
Atlas has few pre-defined system types that are commonly used as supertypes.
60
+
Apache Atlas has few pre-defined system types that are commonly used as supertypes.
56
61
57
62
For example:
58
63
59
64
***Referenceable**: This type represents all entities that can be searched for using a unique attribute called *qualifiedName*.
60
65
61
-
***Asset**: This type extends from Referenceable and has additional attributes such as: *name*, *description* and *owner*.
66
+
***Asset**: This type extends from Referenceable and has other attributes such as: *name*, *description* and *owner*.
62
67
63
68
***DataSet**: This type extends Referenceable and Asset. Conceptually, it can be used to represent a type that stores data. Types that extend DataSet can be expected to have a Schema. For example, a SQL table.
64
69
65
70
:::image type="content" source="./media/tutorial-custom-types/base-model-diagram.png" alt-text="Diagram showing the relationships between system types." border="false":::
66
71
67
72
## Example of a *Type* definition
68
73
69
-
To better understand the Type system, let us look at an example and see how an **Azure SQL Table** is defined.
74
+
To better understand the Type system, let's look at an example and see how an **Azure SQL Table** is defined.
70
75
71
76
You can get the complete type definition by sending a `GET` request to the Type Definition [endpoint](/rest/api/purview/catalogdataplane/types/get-type-definition-by-name):
72
77
@@ -121,28 +126,28 @@ Below you can see a simplified JSON result:
121
126
}
122
127
```
123
128
124
-
Based on the JSON type definition above, let us look at some properties:
129
+
Based on the JSON type definition, let's look at some properties:
125
130
126
-
***Category** field describes in what category your type is. The list of categories supported by Atlas can be found [here](https://atlas.apache.org/api/v2/json_TypeCategory.html).
131
+
***Category** field describes in what category your type is. The list of categories supported by Apache Atlas can be found [here](https://atlas.apache.org/api/v2/json_TypeCategory.html).
127
132
128
133
***ServiceType** lets you group different type definitions in the same group. In our example, a SQL Table is defined under **Azure SQL Database** service type.
129
134
130
135
> [!NOTE]
131
136
> **Azure SQL Database** is defined with the same *serviceType* as **Azure SQL Table**.
132
137
133
-
This field is particularly useful when browsing assets *by source type* in Purview:
138
+
This field is useful when browsing assets *by source type* in Microsoft Purview:
134
139
135
140
:::image type="content" source="./media/tutorial-custom-types/browse-assets.png" alt-text="Screenshot of the portal showing the path from Data Catalog to Browse to By source type and the asset highlighted.":::
136
141
137
142
***SuperTypes** describes the *"parent"* types you want to "*inherit*" from.
138
143
139
-
***schemaElementsAttributes** from **options** influences what appears in the **Schema** tab of your asset in Purview.
144
+
***schemaElementsAttributes** from **options** influences what appears in the **Schema** tab of your asset in Microsoft Purview.
140
145
141
146
Below you can see an example of how the **Schema** tab looks like for an asset of type Azure SQL Table:
142
147
143
148
:::image type="content" source="./media/tutorial-custom-types/schema-tab.png" alt-text="Screenshot of the schema tab for an Azure SQL Table asset.":::
144
149
145
-
In our JSON above, we can see that **schemaElementsAttributes** points to the relationship attribute called **columns**, which can be seen in the **relationshipAttributeDefs** array:
150
+
In our JSON, we can see that **schemaElementsAttributes** points to the relationship attribute called **columns**, which can be seen in the **relationshipAttributeDefs** array:
146
151
147
152
```json
148
153
...
@@ -158,9 +163,9 @@ In our JSON above, we can see that **schemaElementsAttributes** points to the r
158
163
]
159
164
```
160
165
161
-
This relationship has its own definition. The name of the definition is found in **relationshipTypeName** attribute. In this case it is*azure_sql_table_columns*.
166
+
This relationship has its own definition. The name of the definition is found in **relationshipTypeName** attribute. In this case, it's*azure_sql_table_columns*.
162
167
163
-
* The **cardinality** of this relationship attribute is set to *SET* which suggests that it holds a list of related assets.
168
+
* The **cardinality** of this relationship attribute is set to *SET, which suggests that it holds a list of related assets.
164
169
* The related asset is of type *azure_sql_column*, as visible in the *typeName* attribute.
165
170
166
171
In other words, the *columns* relationship relates the Azure SQL Table to a list of Azure SQL Columns that show up in the Schema tab.
@@ -169,7 +174,7 @@ In other words, the *columns* relationship relates the Azure SQL Table to a list
169
174
170
175
Each relationship consists of two ends, called *endDef1* and *endDef2*.
171
176
172
-
In the example above, *azure_sql_table_columns*is the name of the relationship that characterizes a table (endDef1) and its columns (endDef2).
177
+
In the previous example, *azure_sql_table_columns*was the name of the relationship that characterizes a table (endDef1) and its columns (endDef2).
173
178
174
179
For the full definition, you can do make a `GET` request to the following [endpoint](/rest/api/purview/catalogdataplane/types/get-type-definition-by-name) using *azure_sql_table_columns* as the name:
175
180
@@ -202,7 +207,7 @@ Below you can see a simplified JSON result:
202
207
}
203
208
```
204
209
205
-
***name** is the name of the relationship definition. The value, in this case *azure_sql_table_columns* is used in the *relationshipTypeName* attribute of the entity that has this relationship, as you can see it referenced in the example above.
210
+
***name** is the name of the relationship definition. The value, in this case *azure_sql_table_columns* is used in the *relationshipTypeName* attribute of the entity that has this relationship, as you can see it referenced in the json.
206
211
207
212
***relationshipCategory** is the category of the relationship and it can be either COMPOSITION, AGGREGATION or ASSOCIATION as described [here](https://atlas.apache.org/api/v2/json_RelationshipCategory.html).
208
213
@@ -212,21 +217,21 @@ Below you can see a simplified JSON result:
212
217
213
218
***name** is the attribute that will appear on this entity's relationship attribute
214
219
215
-
***cardinality** is eithr SINGLE, SET or LIST.
220
+
***cardinality** is either SINGLE, SET or LIST.
216
221
217
-
***isContainer** is a boolean and is related to the *cardinality* of this relationship. If the *cardinality* is set to **SET** or **LIST** then *isContainer* should be set to True. This describes whether the relationship is a one-to-one relationship or one-to-many. For example, in a SQL Table, *columns* relationship (of type *azure_sql_table_columns*) is a one-to-many relationship between a table and its columns.
222
+
***isContainer** is a boolean and is related to the *cardinality* of this relationship. If the *cardinality* is set to **SET** or **LIST**, then *isContainer* should be set to True. This describes whether the relationship is a one-to-one relationship or one-to-many. For example, in a SQL Table, *columns* relationship (of type *azure_sql_table_columns*) is a one-to-many relationship between a table and its columns.
218
223
219
224
***endDef2** is the second end of the definition and describes, similarly to endDef1, the properties of the second part of the relationship.
220
225
221
226
## Schema tab
222
227
223
-
As we understood above, the information that shows up in the Schema tab of an Azure SQL Table comes from the Azure SQL Column themselves.
228
+
As we understood before, the information that shows up in the Schema tab of an Azure SQL Table comes from the Azure SQL Column themselves.
224
229
225
230
Selecting one column item, we would see the following:
226
231
227
232
:::image type="content" source="./media/tutorial-custom-types/azure-sql-column.png" alt-text="Screenshot of the addressID column page with the properties tab open and the data type highlighted.":::
228
233
229
-
The question is, how did Purview select the *data_tye* property from the column and showed it in the Schema tab of the table?
234
+
The question is, how did Microsoft Purview select the *data_tye* property from the column and showed it in the Schema tab of the table?
230
235
231
236
:::image type="content" source="./media/tutorial-custom-types/schema-tab-data-type.png" alt-text="Screenshot of the Azure SQL Table page with the schema page open.":::
232
237
@@ -239,7 +244,7 @@ GET https://{{ENDPOINT}}/catalog/api/atlas/v2/types/typedef/name/{name}
239
244
>[!NOTE]
240
245
> {name} in this case is: azure_sql_column
241
246
242
-
A simplified JSON result can be seen below:
247
+
Here's a simplified JSON result:
243
248
244
249
```json
245
250
{
@@ -277,15 +282,15 @@ A simplified JSON result can be seen below:
277
282
278
283
Azure SQL Table used *schemaElementAttribute* to point to a relationship consisting of a list of Azure SQL Columns. The type definition of a column has *schemaAttributes* defined.
279
284
280
-
In this way the Schema tab in the table will display the attribute(s) listed in the *schemaAttributes* of the related assets.
285
+
In this way, the Schema tab in the table will display the attribute(s) listed in the *schemaAttributes* of the related assets.
281
286
282
287
## Create custom type definitions
283
288
284
289
### Why?
285
290
286
291
First, why would someone like to create a custom type definition?
287
292
288
-
There can be cases where there is no built-in type that corresponds to the structure of the metadata you want to import in Microsoft Purview.
293
+
There can be cases where there's no built-in type that corresponds to the structure of the metadata you want to import in Microsoft Purview.
289
294
290
295
In such a case, a new type definition has to be defined.
291
296
@@ -402,7 +407,7 @@ They should be linked together through a 1:n relationship.
402
407
}
403
408
```
404
409
405
-
## Initialize assets of custom type
410
+
## Initialize assets of custom types
406
411
407
412
1. Initialize a new asset of type *custom_type_parent* by making a `POST` request to:
408
413
@@ -429,7 +434,7 @@ They should be linked together through a 1:n relationship.
429
434
}
430
435
```
431
436
432
-
Save the *guid* as you will need it later.
437
+
Save the *guid* as you'll need it later.
433
438
434
439
1. Initialize a new asset of type *custom_type_child* by making a `POST` request to:
435
440
@@ -454,7 +459,7 @@ They should be linked together through a 1:n relationship.
454
459
}
455
460
```
456
461
457
-
Save the *guid* as you will need it later.
462
+
Save the *guid* as you'll need it later.
458
463
459
464
1. Initialize a new relationship of type *custom_parent_child_relationship* by making a `POST` request to:
460
465
@@ -514,7 +519,7 @@ They should be linked together through a 1:n relationship.
514
519
515
520
:::image type="content" source="./media/tutorial-custom-types/relationship.png" alt-text="Screenshot of the Related tab, showing the relationship between the child and parent.":::
516
521
517
-
1. You can create multiple children by inititizing a new child asset and inititialzing a relationship
522
+
1. You can create multiple children by initializing a new child asset and inititialzing a relationship
518
523
519
524
>[!NOTE]
520
525
>The *qualifiedName* is unique per asset, therefore, the second child should be called differently, such as: *custom//custom_type_child:Second_child_object*
0 commit comments