You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
ms.custom: template-overview #Required; leave this attribute/value as-is.
10
10
---
11
11
12
-
# Overview of Domain data management services (DDMS)
12
+
# Domain data management services (DDMS)
13
13
14
-
Domain data management services (DDMS) store, access, and retrieve metadata and bulk data from applications connected to the data platform. The energy industry works with data of an extraordinary magnitude, which has significant ramifications for storage and compute requirements. Geoscientists stream terabytes of seismic, well Log, and other data types at full resolution. Immediate responsiveness of data is essential for all stages of petroleum exploration--particularly for geologic interpretation and analysis. Developers therefore use DDMS to deliver seamless and secure consumption of data in the applications they build on Microsoft Energy Data Services Preview. The Microsoft Energy Data Services Preview suite of DDMS adheres to [Open Subsurface Data Universe](https://osduforum.org/) (OSDU™) standards and provides enhancements in performance, geo-availability, and access controls. DDMS is optimized for each data type and can be extended to accommodate new data types. All DDMS preserves raw data and offer multi format support and conversion for consuming applications such as Petrel while tracking lineage. Data within the DDMS is discoverable and governed by entitlement and legal tags.
14
+
The energy industry works with data of an extraordinary magnitude, which has significant ramifications for storage and compute requirements. Geoscientists stream terabytes of seismic, well log, and other data types at full resolution. Immediate responsiveness of data is essential for all stages of petroleum exploration--particularly for geologic interpretation and analysis.
15
+
16
+
## Overview
17
+
Domain data management services (DDMS) store, access, and retrieve metadata and bulk data from applications connected to the data platform. Developers, therefore, use DDMS to deliver seamless and secure consumption of data in the applications they build on Microsoft Energy Data Services Preview. The Microsoft Energy Data Services Preview suite of DDMS adheres to [Open Subsurface Data Universe](https://osduforum.org/) (OSDU™) standards and provides enhancements in performance, geo-availability, and access controls. DDMS service is optimized for each data type and can be extended to accommodate new data types. The DDMS service preserves raw data and offers multi format support and conversion for consuming applications such as Petrel while tracking lineage. Data within the DDMS service is discoverable and governed by entitlement and legal tags.
15
18
16
19
[!INCLUDE [preview features callout](./includes/preview/preview-callout.md)]
17
20
18
-
**OSDU™ definition**
21
+
### OSDU™ definition
19
22
20
23
- Highly optimized storage & access for bulk data, with highly opinionated APIs delivering the data required to enable domain workflows
21
24
- Governed schemas that incorporate domain-specific perspective and type-safe accessors for registered entity types
22
25
23
-
**Aspirational components for any DDMS**
26
+
### Aspirational components for any DDMS
24
27
25
28
- Direct connection to OSDU™ core services: storage, legal, ingestion and entitlements
26
29
- Connection to adjacent or proximal databases (blob storage, Cosmos, external) and client applications
27
30
- Configure infrastructure provisioning to enable optimal performance for data streaming and access
28
31
29
-
**Additional components for most DDMS (may include but not be limited to)**
32
+
### Additional components for most DDMS (may include but not be limited to)
30
33
31
34
- File format converter--for example, for Seismic DDMS: SGY to ZGY, etc.
32
35
- Hierarchy of data organization and chunking - Tenant, project, and data
33
36
34
37
## Use cases and value add
35
38
36
-
### Frictionless Exploration and Production(E&P) Enterprise Data Management(EDM) on Azure
39
+
### Frictionless Exploration and Production(E&P)
37
40
38
-
The Microsoft Energy Data Services Preview DDMS suite enables energy companies to access their data in a manner that is fast, portable, testable and extendible. As a result, they'll achieve unparalleled streaming performance and use the standards and output from OSDU™. In the near term, the Azure DDMS suite will onboard the OSDU™ DDMS and Schlumberger proprietary DMS. At the same time, Microsoft contributes to the OSDU™ community DDMS to ensure compatibility and architectural alignment.
41
+
The Microsoft Energy Data Services Preview DDMS service enables energy companies to access their data in a manner that is fast, portable, testable and extendible. As a result, they'll achieve unparalleled streaming performance and use the standards and output from OSDU™. The Azure DDMS service will onboard the OSDU™ DDMS and Schlumberger proprietary DMS. Microsoft also continues to contribute to the OSDU™ community DDMS to ensure compatibility and architectural alignment.
39
42
40
43
### Seamless connection between applications and data
41
44
42
-
- Deploy and utilize OSDU™ Applications – Customers can deploy applications on top of Microsoft Energy Data Services Preview that are OSDU™ compatible. They're able to connect applications to Core Services and DDMS without spending extensive cycles on deployment.
43
-
- Petrel integration and quick getting started – Customers can easily connect DELFI to Microsoft Energy Data Services Preview, eliminating the cycles associated with Petrel deployments and connection to data management systems.
44
-
- Seamless data experience on OSDU™ compatible apps – By connecting DDMS to apps, Geoscientists can execute integrated E&P workflows with unparalleled performance on Azure and use OSDU™ core services. For example, a geophysicist can pick well ties on a seismic volume in Petrel and stream data from the seismic DMS.
45
+
Customers can deploy applications on top of Microsoft Energy Data Services Preview that have been developed as per the OSDU™ standard. They're able to connect applications to Core Services and DDMS without spending extensive cycles on deployment. Customers can also easily connect DELFI to Microsoft Energy Data Services Preview, eliminating the cycles associated with Petrel deployments and connection to data management systems. By connecting applications to DDMS service, Geoscientists can execute integrated E&P workflows with unparalleled performance on Azure and use OSDU™ core services. For example, a geophysicist can pick well ties on a seismic volume in Petrel and stream data from the seismic DMS.
45
46
46
-
## OSDU™ - Seismic DMS
47
+
## Types of DMS
48
+
Below are the OSDU™ DMS the service supports -
47
49
48
-
### Definition and overview
50
+
### OSDU™ - Seismic DMS
49
51
50
52
Seismic data is a fundamental data type for oil and gas exploration. Seismic data provides a geophysical representation of the subsurface that can be applied for prospect identification and drilling decisions. Typical seismic datasets represent a multi-kilometer survey and are therefore massive in size.
51
53
52
-
Due to this extraordinary data size, geoscientists working on-premises struggle to use seismic in domain applications. They suffer from crashes as the seismic dataset exceeds their workstation's RAM, which leads to significant non-productive time. To achieve performance needed for domain workflows, geoscientists must chunk a seismic dataset and view each chunk in isolation. As a result, users suffer from the time spent wrangling seismic data and the opportunity cost of missing the significant picture view of the subsurface and target reservoirs.
54
+
Due to this extraordinary data size, geoscientists working on-premises struggle to use seismic data in domain applications. They suffer from crashes as the seismic dataset exceeds their workstation's RAM, which leads to significant non-productive time. To achieve performance needed for domain workflows, geoscientists must chunk a seismic dataset and view each chunk in isolation. As a result, users suffer from the time spent wrangling seismic data and the opportunity cost of missing the significant picture view of the subsurface and target reservoirs.
53
55
54
-
The seismic DMS is part of the OSDU™ platform and enables users to connect seismic data to cloud storage to applications. It allows secure access to metadata associated with seismic data to efficiently retrieve and handle large blocks of data for OpenVDS, ZGY, and other seismic data formats. The DMS therefore enables users to stream huge amounts of data in OSDU™ compliant applications in real time. Enabling the seismic DMS on Microsoft Energy Data Services Preview opens a pathway for Azure customers to bring their seismic data to the cloud and take advantage of Azure storage and HPC.
56
+
The seismic DMS is part of the OSDU™ platform and enables users to connect seismic data to cloud storage to applications. It allows secure access to metadata associated with seismic data to efficiently retrieve and handle large blocks of data for OpenVDS, ZGY, and other seismic data formats. The DMS therefore enables users to stream huge amounts of data in OSDU™ compliant applications in real time. Enabling the seismic DMS on Microsoft Energy Data Services Preview opens a pathway for Azure customers to bring their seismic data to the cloud and take advantage of Azure storage and high performance computing.
55
57
56
58
## OSDU™ - Wellbore DMS
57
59
58
-
### Definition and overview
59
-
60
60
Well Logs are measurements taken while drilling, which tells energy companies information about the subsurface. Ultimately, they reveal whether hydrocarbons are present (or if the well is dry). Logs contain many attributes that inform geoscientists about the type of rock, its quality, and whether it contains oil, water, gas, or a mix. Energy companies use these attributes to determine the quality of a reservoir – how much oil or gas is present, its quality, and ultimately, economic viability. Maintaining Well Log data and ensuring easy access to historical logs is critical to energy companies. The Wellbore DMS facilitates access to this data in any OSDU™ compliant application. The Wellbore DMS was contributed by Schlumberger to OSDU™.
61
61
62
62
Well Log data can come in different formats. It's most often indexed by depth or time and the increment of these measurements can vary. Well Logs typically contain multiple attributes for each vertical measurement. Well Logs can therefore be small or for more modern Well Logs that use high frequency data, greater than 1 Gb. Well Log data is smaller than seismic; however, users will want to look at upwards of hundreds of wells at a time. This scenario is common in mature areas that have been heavily drilled such as the Permian Basin in West Texas.
63
63
64
64
Geoscientists therefore want to access numerous well logs in a single session. They often are looking at all historical drilling programs in an area. As a result, they'll look at Well Log data that was collected using a wide variety of instruments and technology. This data will vary widely in format, quality, and sampling. The Wellbore DMS resolves this data through the OSDU™ schemas to deliver the data to the consuming applications.
65
65
66
-
### Services
66
+
Here are the services that the Wellbore DMS offers -
67
67
68
68
-**Objects and Consumption** - The Wellbore DMS can consume Wellbore, log set, log, marker, trajectory, and dip objects. This covers most well related exploration workflows
69
69
-**Lifecycle** – The Wellbore DMS supports the dataset through creation and writing to storage, versioning, lineage, auditing, and deletion
@@ -72,13 +72,9 @@ Geoscientists therefore want to access numerous well logs in a single session. T
72
72
73
73
## OSDU™ - Well Delivery DMS
74
74
75
-
### Definition and overview
76
-
77
75
The Well Delivery DMS stores critical drilling domain information related to the planning and execution of a well. Throughout a drilling program, engineers and domain experts need to access a wide variety of data types including activities, trajectories, risks, subsurface information, equipment used, fluid and cementing, rig utilization, and reports. Integrating this collection of data types together are the cornerstone to drilling insights. At the same time, until now, there was no industry wide standardization or enforced format. The common standards the Well Delivery DMS enables is critical to the Drilling Value Chain as it connects a diverse group of personas including operations, oil companies, service companies, logistics companies, etc.
Microsoft Energy Data Services Preview is a secure, reliable, hyperscale, fully managed cloud-based data platform solution for the energy industry. It is an enterprise-grade data platform that brings together the capabilities of OSDU™ Data Platform, Microsoft's secure and trusted Azure cloud platform, and Schlumberger's extensive domain expertise. It allows customers to free data from silos, provides strong data management, storage, and federation strategy while ensuring compatibility with evolving community standards like OSDU™ and enable value addition through interoperability with both first-party and third-party solutions.
13
+
Microsoft Energy Data Services Preview is a secure, reliable, hyperscale, fully managed cloud-based data platform solution for the energy industry. It is an enterprise-grade data platform that brings together the capabilities of OSDU™ Data Platform, Microsoft's secure and trusted Azure cloud platform, and Schlumberger's extensive domain expertise. It allows customers to free data from silos, provides strong data management, storage, and federation strategy. Microsoft Energy Data Services ensures compatibility with evolving community standards like OSDU™ and enables value addition through interoperability with both first-party and third-party solutions.
14
14
15
15
[!INCLUDE [preview features callout](./includes/preview/preview-callout.md)]
16
16
17
17
## Principles
18
18
19
-
Microsoft Energy Data Services Preview follows certain principles:
19
+
Microsoft Energy Data Services conforms to the following principles:
[!INCLUDE [preview features callout](./includes/preview/preview-callout.md)]
15
15
16
-
Microsoft Energy Data Services receives improvements on an ongoing basis. To stay up to date with the most recent developments, this article provides you with information about:
16
+
Microsoft Energy Data Services is updated on an ongoing basis. To stay up to date with the most recent developments, this article provides you with information about:
17
17
18
18
- The latest releases
19
19
- Known issues
@@ -26,41 +26,34 @@ Microsoft Energy Data Services receives improvements on an ongoing basis. To sta
26
26
27
27
### Key Announcement
28
28
29
-
Microsoft Energy Data Services is now available as a public preview.
29
+
Microsoft Energy Data Services is now available in public preview.
30
30
31
-
Microsoft Energy Data Services conforms with the requirements of the Release-3, [Milestone-12](https://community.opengroup.org/osdu/governance/project-management-committee/-/wikis/M12-Release-Notes) of OSDU™ Technical Standard, Version 1.0.
31
+
Microsoft Energy Data Services is developed in alignment with the emerging requirements of the OSDU™ Technical Standard, Version 1.0. and is currently aligned with Mercury Release(R3), [Milestone-12](https://community.opengroup.org/osdu/governance/project-management-committee/-/wikis/M12-Release-Notes).
32
32
33
33
### Partition & User Management
34
34
35
-
-You can create [new data partitions](how-to-add-more-data-partitions.md)(up to five) in your existing Microsoft Energy Data Services instance. Earlier, you could create a data partition only while provisioning the instance.
36
-
- The domain name for your entitlement groups for [user management](how-to-manage-users.md)is changed to "dataservices.energy".
35
+
-New data partitions can be [created dynamically](how-to-add-more-data-partitions.md)as needed post provising of the platform (up to five). Earlier, data partitions could only be created when provisioning a new instance.
36
+
- The domain name for entitlement groups for [user management](how-to-manage-users.md)has been changed to "dataservices.energy".
37
37
38
38
### Data Ingestion
39
39
40
-
- Microsoft Energy Data Services - Enabled support for user context in ingestion ([ADR: Issue 52](https://community.opengroup.org/osdu/platform/data-flow/ingestion/home/-/issues/52)) (not yet supported in the Azure OSDU™ community edition)
41
-
- In Microsoft Energy Data Services, the User identity is preserved and passed on to all ingestion workflow related services using the newly introduced _x-on-behalf-of_ header.
42
-
- In Azure OSDU™ community edition, by using elevated SPN permissions, users could call the workflow service and trigger an ingestion job even without appropriate service level entitlements on the underlying dependent services. In Microsoft Energy Data Services, a user needs to have appropriate service level entitlements on all dependent services involved in the ingestion workflow
43
-
- In Azure OSDU™ community edition, users who weren't part of the same data entitlement group could accidentally modify and corrupt data (using elevated SPN permissions). In Microsoft Energy Data Services, only users belonging to the same data entitlement group can modify data, and the modified user and date information are captured.
40
+
- Enabled support for user context in ingestion ([ADR: Issue 52](https://community.opengroup.org/osdu/platform/data-flow/ingestion/home/-/issues/52))
41
+
- User identity is preserved and passed on to all ingestion workflow related services using the newly introduced _x-on-behalf-of_ header. A user needs to have appropriate service level entitlements on all dependent services involved in the ingestion workflow and only users with appropriate data level entitlements can modify data.
44
42
- Workflow service payload is restricted to a maximum of 2 MB. If it exceeds, the service will throw an HTTP 413 error. This restriction is placed to prevent workflow requests from overwhelming the server.
45
43
- Microsoft Energy Data Services uses Azure Data Factory (ADF) to run large scale ingestion workloads.
46
44
47
45
### Search
48
46
49
-
- Improved security as Elasticsearch images is now pulled from Microsoft's internal Azure Container Registry instead of public repositories.
47
+
- Improved security as Elasticsearch images are now pulled from Microsoft's internal Azure Container Registry instead of public repositories.
50
48
- Improved security by enabling encryption in transit for Elasticsearch, Registration, and Notification services.
51
49
52
50
### Monitoring
53
51
54
-
- You can create diagnostic settings to export [Airflow](how-to-integrate-airflow-logs-with-azure-monitor.md) and [Elasticsearch](how-to-integrate-elastic-logs-with-azure-monitor.md) logs from your Microsoft Energy Data Services instance to Azure Monitor.
55
-
56
-
### Hardware & Software
57
-
58
-
- Your Microsoft Energy Data Services instance runs on the latest version of Azure Kubernetes Services (AKS) clusters – [version 1.24.](https://kubernetes.io/blog/2022/04/07/upcoming-changes-in-kubernetes-1-24/)
52
+
- Diagnostic settings can be exported from [Airflow](how-to-integrate-airflow-logs-with-azure-monitor.md) and [Elasticsearch](how-to-integrate-elastic-logs-with-azure-monitor.md) to Azure Monitor.
59
53
60
54
### Region Availability
61
55
62
-
- In Preview, Availability Zones will be available by default in select regions. All new Microsoft Energy Data Services instances created in these select zones will have Availability Zones redundancy enabled on them upon creation.
63
-
- The selected regions where Availability Zones will be offered on Microsoft Energy Data Services Preview are - South Central US, East US, West Europe, and North Europe.
56
+
- Currently, Microsoft Energy Data Services is being offered in the following regions - South Central US, East US, West Europe, and North Europe.
0 commit comments