You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/energy-data-services/concepts-csv-parser-ingestion.md
+3-3Lines changed: 3 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -12,9 +12,9 @@ ms.custom: template-concept
12
12
# CSV parser ingestion concepts
13
13
A CSV (comma-separated values) file is a comma delimited text file that is used to save data in a table structured format.
14
14
15
-
A CSV Parser [DAG](https://airflow.apache.org/docs/apache-airflow/1.10.12/concepts.html#dags) allows a customer to load data into Microsoft Azure Data Manager for Energy instance based on a custom schema that is, a schema that doesn't match the [OSDU™](https://osduforum.org) Well Known Schema (WKS). Customers must create and register the custom schema using the Schema service before loading the data.
15
+
A CSV Parser [DAG](https://airflow.apache.org/docs/apache-airflow/1.10.12/concepts.html#dags) allows a customer to load data into Microsoft Azure Data Manager for Energy instance based on a custom schema that is, a schema that doesn't match the [OSDU®](https://osduforum.org) Well Known Schema (WKS). Customers must create and register the custom schema using the Schema service before loading the data.
16
16
17
-
A CSV Parser DAG implements an ELT (Extract Load and Transform) approach to data loading, that is, data is first extracted from the source system in a CSV format, and it's loaded into the Azure Data Manager for Energy instance. It could then be transformed to the [OSDU™](https://osduforum.org) Well Known Schema using a mapping service.
17
+
A CSV Parser DAG implements an ELT (Extract Load and Transform) approach to data loading, that is, data is first extracted from the source system in a CSV format, and it's loaded into the Azure Data Manager for Energy instance. It could then be transformed to the [OSDU®](https://osduforum.org) Well Known Schema using a mapping service.
18
18
19
19
20
20
## What does CSV ingestion do?
@@ -42,7 +42,7 @@ The below workflow diagram illustrates the CSV Parser DAG workflow:
42
42
43
43
To execute the CSV Parser DAG workflow, the user must first create and register the schema using the workflow service. Once the schema is created, the user then uses the File service to upload the CSV file to the Microsoft Azure Data Manager for Energy instances, and also creates the storage record of file generic kind. The file service then provides a file ID to the user, which is used while triggering the CSV Parser workflow using the Workflow service. The Workflow service provides a run ID, which the user could use to track the status of the CSV Parser workflow run.
44
44
45
-
OSDU™ is a trademark of The Open Group.
45
+
OSDU® is a trademark of The Open Group.
46
46
47
47
## Next steps
48
48
Advance to the CSV parser tutorial and learn how to perform a CSV parser ingestion
Copy file name to clipboardExpand all lines: articles/energy-data-services/concepts-ddms.md
+17-17Lines changed: 17 additions & 17 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -11,53 +11,53 @@ ms.custom: template-concept
11
11
12
12
# Domain data management service concepts
13
13
14
-
**Domain Data Management Service (DDMS)** – is a platform component that extends [OSDU™](https://osduforum.org) core data platform with domain specific model and optimizations. DDMS is a mechanism of a platform extension that:
14
+
**Domain Data Management Service (DDMS)** – is a platform component that extends [OSDU®](https://osduforum.org) core data platform with domain specific model and optimizations. DDMS is a mechanism of a platform extension that:
15
15
16
16
* delivers optimized handling of data for each (non-overlapping) "domain."
17
17
* pertains to a single vertical discipline or business area, for example, Petrophysics, Geophysics, Seismic
18
18
* serves a functional aspect of one or more vertical disciplines or business areas, for example, Earth Model
19
-
* delivers high performance capabilities not supported by OSDU™ generic normal APIs.
20
-
* helps achieve the extension of OSDU™ scope to new business areas.
19
+
* delivers high performance capabilities not supported by OSDU® generic normal APIs.
20
+
* helps achieve the extension of OSDU® scope to new business areas.
21
21
* may be developed in a distributed manner with separate resources/sponsors.
22
22
23
-
OSDU™ Technical Standard defines the following types of OSDU™ application types:
23
+
OSDU® Technical Standard defines the following types of OSDU® application types:
| OSDU™™ Embedded Applications | An application developed and managed within the OSDU™ Open-Source community that is built on and deployed as part of the OSDU™ Data Platform distribution. |
28
-
| ISV Extension Applications | An application, developed and managed in the marketplace that is NOT part of THE OSDU™ Data Platform distributions, and when selected is deployed within the OSDU™ Data Platform as add-ons |
29
-
| ISV third Party Applications | An application, developed and managed in the marketplace that integrates with the OSDU™ Data Platform, and runs outside the OSDU™ Data Platform |
27
+
| OSDU®™ Embedded Applications | An application developed and managed within the OSDU® Open-Source community that is built on and deployed as part of the OSDU® Data Platform distribution. |
28
+
| ISV Extension Applications | An application, developed and managed in the marketplace that is NOT part of THE OSDU® Data Platform distributions, and when selected is deployed within the OSDU® Data Platform as add-ons |
29
+
| ISV third Party Applications | An application, developed and managed in the marketplace that integrates with the OSDU® Data Platform, and runs outside the OSDU® Data Platform |
30
30
31
31
32
32
| Characteristics | Embedded | Extension | Third Party |
**IT Developers** build systems to connect data to domain applications (internal and external – for example, Petrel) which enables data managers to deliver projects to geoscientists. The DDMS suite on Azure Data Manager for Energy helps automate these workflows and eliminates time spent managing updates.
45
45
46
-
**Geoscientists** use domain applications for key Exploration and Production workflows such as Seismic interpretation and Well tie analysis. While these users won't directly interact with the DDMS, their expectations for data performance and accessibility will drive requirements for the DDMS in the Foundation Tier. Azure will enable geoscientists to stream cross domain data instantly in OSDU™ compatible applications (for example, Petrel) connected to Azure Data Manager for Energy.
46
+
**Geoscientists** use domain applications for key Exploration and Production workflows such as Seismic interpretation and Well tie analysis. While these users won't directly interact with the DDMS, their expectations for data performance and accessibility will drive requirements for the DDMS in the Foundation Tier. Azure will enable geoscientists to stream cross domain data instantly in OSDU® compatible applications (for example, Petrel) connected to Azure Data Manager for Energy.
47
47
48
48
**Data managers** spend a significant number of time fulfilling requests for data retrieval and delivery. The Seismic, Wellbore, and Petrel Data Services enable them to discover and manage data in one place while tracking version changes as derivatives are created.
49
49
50
50
## Platform landscape
51
51
52
-
Azure Data Manager for Energy is an OSDU™ compatible product, meaning that its landscape and release model are dependent on OSDU™.
52
+
Azure Data Manager for Energy is an OSDU® compatible product, meaning that its landscape and release model are dependent on OSDU®.
53
53
54
-
Currently, OSDU™ certification and release process are not fully defined yet and this topic should be defined as a part of the Azure Data Manager for Energy Foundation Architecture.
54
+
Currently, OSDU® certification and release process are not fully defined yet and this topic should be defined as a part of the Azure Data Manager for Energy Foundation Architecture.
55
55
56
-
OSDU™ R3 M8 is the base for the scope of the Azure Data Manager for Energy Foundation Private – as a latest stable, tested version of the platform.
56
+
OSDU® R3 M8 is the base for the scope of the Azure Data Manager for Energy Foundation Private – as a latest stable, tested version of the platform.
57
57
58
-
## Learn more: OSDU™ DDMS community principles
58
+
## Learn more: OSDU® DDMS community principles
59
59
60
-
[OSDU™ community DDMS Overview](https://community.opengroup.org/osdu/documentation/-/wikis/OSDU™-(C)/Design-and-Implementation/Domain-&-Data-Management-Services#ddms-requirements) provides an extensive overview of DDMS motivation and community requirements from a user, technical, and business perspective. These principles are extended to Azure Data Manager for Energy.
60
+
[OSDU® community DDMS Overview](https://community.opengroup.org/osdu/documentation/-/wikis/OSDU®-(C)/Design-and-Implementation/Domain-&-Data-Management-Services#ddms-requirements) provides an extensive overview of DDMS motivation and community requirements from a user, technical, and business perspective. These principles are extended to Azure Data Manager for Energy.
61
61
62
62
## DDMS requirements
63
63
@@ -85,7 +85,7 @@ A DDMS meets the following requirements, further classified into capability, arc
85
85
| 18 | Workflow composability and customizations || Openness and Extensibility |
86
86
| 19 | Data-Centric Extensibility || Openness and Extensibility |
87
87
88
-
OSDU™ is a trademark of The Open Group.
88
+
OSDU® is a trademark of The Open Group.
89
89
90
90
## Next steps
91
91
Advance to the seismic DDMS sdutil tutorial to learn how to use sdutil to load seismic data into seismic store.
Copy file name to clipboardExpand all lines: articles/energy-data-services/concepts-entitlements.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -91,7 +91,7 @@ For a full list of Entitlement API endpoints, see [OSDU entitlement service](htt
91
91
> [!NOTE]
92
92
> The OSDU documentation refers to v1 endpoints, but the scripts noted in this documentation refer to v2 endpoints, which work and have been successfully validated.
Copy file name to clipboardExpand all lines: articles/energy-data-services/concepts-index-and-search.md
+4-4Lines changed: 4 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -35,22 +35,22 @@ When the *recordChangedMessages* event is received by the `Indexer Service`, it
35
35
36
36
:::image type="content" source="media/concepts-index-and-search/concept-indexer-sequence.png" alt-text="Diagram that shows Indexing sequence flow.":::
37
37
38
-
For more information, see [Indexer service OSDU™ documentation](https://community.opengroup.org/osdu/platform/system/indexer-service/-/blob/release/0.15/docs/tutorial/IndexerService.md) provides information on indexer service
38
+
For more information, see [Indexer service OSDU® documentation](https://community.opengroup.org/osdu/platform/system/indexer-service/-/blob/release/0.15/docs/tutorial/IndexerService.md) provides information on indexer service
39
39
40
40
## Search workflow
41
41
42
42
`Search service` provides a mechanism for discovering indexed metadata documents. The Search API supports full-text search on string fields, range queries on date, numeric, or string field, etc. along with geo-spatial searches.
43
43
44
44
When metadata records are loaded onto the Platform using `Storage service`, we can configure permissions for viewers and owners of the metadata records under the *acl* field. The viewers and owners are assigned via groups as defined in the `Entitlement service`. When performing a search as a user, the matched metadata records will only show up for users who are assigned to the Group.
45
45
46
-
For a detailed tutorial on `Search service`, refer [Search service OSDU™ documentation](https://community.opengroup.org/osdu/platform/system/search-service/-/blob/release/0.15/docs/tutorial/SearchService.md)
46
+
For a detailed tutorial on `Search service`, refer [Search service OSDU® documentation](https://community.opengroup.org/osdu/platform/system/search-service/-/blob/release/0.15/docs/tutorial/SearchService.md)
47
47
48
48
49
49
## Reindex workflow
50
50
Reindex API allows users to reindex a kind without reingesting the records via storage API. For detailed information, refer to
Copy file name to clipboardExpand all lines: articles/energy-data-services/concepts-manifest-ingestion.md
+5-5Lines changed: 5 additions & 5 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -20,7 +20,7 @@ A manifest is a JSON document that has a pre-determined structure for capturing
20
20
21
21
You can find an example manifest json document [here](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/Examples/manifest#manifest-example).
22
22
23
-
The manifest schema has containers for the following OSDU™[Group types](https://community.opengroup.org/osdu/data/data-definitions/-/blob/master/Guides/Chapters/02-GroupType.md#2-group-type):
23
+
The manifest schema has containers for the following OSDU®[Group types](https://community.opengroup.org/osdu/data/data-definitions/-/blob/master/Guides/Chapters/02-GroupType.md#2-group-type):
24
24
25
25
***ReferenceData** (*zero or more*) - A set of permissible values to be used by other (master or transaction) data fields. Examples include *Unit of Measure (feet)*, *Currency*, etc.
26
26
***MasterData** (*zero or more*) - A single source of basic business data used across multiple systems, applications, and/or process. Examples include *Wells* and *Wellbores*
@@ -45,15 +45,15 @@ Azure Data Manager for Energy instance has out-of-the-box support for Manifest-b
The Manifest-based file ingestion workflow consists of the following components:
47
47
***Workflow Service** - A wrapper service running on top of the Airflow workflow engine.
48
-
***Airflow engine** - A workflow orchestration engine that executes workflows registered as DAGs (Directed Acyclic Graphs). Airflow is the chosen workflow engine by the [OSDU™](https://osduforum.org/) community to orchestrate and run ingestion workflows. Airflow isn't directly exposed, instead its features are accessed through the workflow service.
48
+
***Airflow engine** - A workflow orchestration engine that executes workflows registered as DAGs (Directed Acyclic Graphs). Airflow is the chosen workflow engine by the [OSDU®](https://osduforum.org/) community to orchestrate and run ingestion workflows. Airflow isn't directly exposed, instead its features are accessed through the workflow service.
49
49
***Storage Service** - A service that is used to save the manifest metadata records into the data platform.
50
-
***Schema Service** - A service that manages OSDU™ defined schemas in the data platform. Schemas are being referenced during the Manifest-based file ingestion.
50
+
***Schema Service** - A service that manages OSDU® defined schemas in the data platform. Schemas are being referenced during the Manifest-based file ingestion.
51
51
***Entitlements Service** - A service that manages access groups. This service is used during the ingestion for verification of ingestion permissions. This service is also used during the metadata record retrieval for validation of "read" writes.
52
52
***Legal Service** - A service that validates compliance through legal tags.
53
53
***Search Service** is used to perform referential integrity check during the manifest ingestion process.
54
54
55
55
### Pre-requisites
56
-
Before running the Manifest-based file ingestion workflow, customers must ensure that the user accounts running the workflow have access to the core services (Search, Storage, Schema, Entitlement and Legal) and Workflow service (see [Entitlement roles](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/docs/osdu-entitlement-roles.md) for details). As part of Azure Data Manager for Energy instance provisioning, the OSDU™ standard schemas and associated reference data are pre-loaded. Customers must ensure that the user account used for ingesting the manifests is included in appropriate owners and viewers ACLs. Customers must ensure that manifests are configured with correct legal tags, owners and viewers ACLs, reference data, etc.
56
+
Before running the Manifest-based file ingestion workflow, customers must ensure that the user accounts running the workflow have access to the core services (Search, Storage, Schema, Entitlement and Legal) and Workflow service (see [Entitlement roles](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/docs/osdu-entitlement-roles.md) for details). As part of Azure Data Manager for Energy instance provisioning, the OSDU® standard schemas and associated reference data are pre-loaded. Customers must ensure that the user account used for ingesting the manifests is included in appropriate owners and viewers ACLs. Customers must ensure that manifests are configured with correct legal tags, owners and viewers ACLs, reference data, etc.
57
57
58
58
### Workflow sequence
59
59
The following illustration provides the Manifest-based file ingestion workflow:
@@ -65,7 +65,7 @@ The workflow service executes a series of manifest `syntax validation` like mani
65
65
66
66
Once the validations are successful, the system processes the content into storage by writing each valid entity into the data platform using the Storage Service API.
67
67
68
-
OSDU™ is a trademark of The Open Group.
68
+
OSDU® is a trademark of The Open Group.
69
69
70
70
## Next steps
71
71
-[Tutorial: Sample steps to perform a manifest-based file ingestion](tutorial-manifest-ingestion.md)
0 commit comments