Merge pull request #269041 from bharathim/eds-preview

ttorble · web-flow · commit f6dc5ac7dad6 · 2024-03-15T10:44:39.000Z
Create how to enable eds preview
diff --git a/articles/energy-data-services/faq-energy-data-services.yml b/articles/energy-data-services/faq-energy-data-services.yml
@@ -24,6 +24,24 @@ sections:
           answer: >
             Azure Data Manager for Energy is available to all customers and partners who have a valid Azure subscription. You can create an instance by going to Azure Marketplace and searching for **Azure Data Manager for Energy**.
 
+    - name: External Data Sources
+      questions:
+        - question:  What is the impact of data sharing across tenancy boundaries on security, privacy, and compliance?
+          answer: >
+            Any data transferred across tenancy boundaries by the data provider and data consumer is the responsibility of the data provider and the data consumer. Microsoft will not store or process customer data outside the customer-specified Geo without your authorization. The data provider and consumer can use Customer Managed Encryption Keys (CMEK) and Private Links to secure data at rest and during transit.
+        - question: How are the network bandwidth/data transfer charges handled with EDS usage?
+          answer: >
+            The deployment topology decided by the data provider and data consumer will determine the network bandwidth/data transfer costs.
+        - question: Would data integrity be maintained when data provider deletes source data?
+          answer: > 
+            Currently, EDS does not support reconciling deleted data therefore deleting data at the data provider’s level may result in inconsistent records at the data consumer’s level.
+        - question: What is the impact of secret key rotation at the data consumer’s end?
+          answer: > 
+            Key rotation is fully supported therefore it will not have any impact on EDS functionality.
+        - question: When will EDS be generally available on Azure Data Manager for Energy?
+          answer: >
+            EDS is a preview feature in Azure Data Manager for Energy. We wish to learn from customer implementation of the EDS preview feature to ensure the functionality, performance, and scalability needs are met, before we make the feature generally available.
+
 
     - name: Data residency
       questions:
diff --git a/articles/energy-data-services/how-to-enable-external-data-sources.md b/articles/energy-data-services/how-to-enable-external-data-sources.md
@@ -0,0 +1,75 @@
+---
+title: "Microsoft Azure Data Manager for Energy - How to enable External Data Sources (EDS)"
+description: "This article describes how to enable external data sources in Azure Data Manager for Energy."
+author: bharathim
+ms.author: bselvaraj
+ms.service: energy-data-services
+ms.topic: how-to #Don't change
+ms.date: 03/14/2024
+
+#customer intent: As a Data Manager in Operating company, I want to enable external data sources so that I pull metadata at scheduled intervals into Azure Data Manager for Energy and retrieve bulk data on demand.
+
+---
+
+# How to enable External Data Sources (EDS) Preview?
+
+External Data Sources (EDS) is a capability in [OSDU&trade;](https://osduforum.org/) that allows data from an [OSDU&trade;](https://osduforum.org/) compliant external data source to be shared with an Azure Data Manager for Energy resource. EDS is designed to pull specified data (metadata) from OSDU-compliant data sources via scheduled jobs while leaving associated dataset files (LAS, SEG-Y, etc.) stored at the external source for retrieval on demand.
+
+For more information about External Data Sources (EDS), see [The OSDU Forum 2Q 2022 Newsletter - EDS](https://osduforum.org/wp-content/uploads/2022/06/The-OSDU-Forum-2Q-2022-Newsletter.pdf).  
+
+> [!NOTE]
+> OSDU community shipped EDS as a preview feature in M18 Release, and it is available as a preview feature on Azure Data Manager for Energy in Developer tier only.
+
+> [!IMPORTANT]
+> Limit your Identity Provider (IdP) token to read operations only.
+>
+> Protect your search and delivery wrapper APIs from unauthorized access. As a best practice, ensure only Azure Data Manager for Energy can access data through your wrapper APIs. This helps with data discovery and delivery for EDS Preview.
+
+## Prerequisites
+
+1. Create a new or use an existing key vault to store secrets managed by [OSDU&trade;](https://osduforum.org/) secret service. To learn how to create a key vault with the Azure portal, see  [Quickstart: Create a key vault using the Azure portal](../key-vault/general/quick-create-portal.md).
+  
+    > [!IMPORTANT]
+    > Your key vault must exist in the same tenant and subscription as your Azure Data Manager for Energy resource. 
+
+    > [!TIP]
+    > When you create the key vault, select [Enable purge protection (enforce a mandatory retention period for deleted vaults and vault objects)](../key-vault/general/key-vault-recovery.md?tabs=azure-portal#what-are-soft-delete-and-purge-protection).
+  
+1. In the **Access configuration** tab, under **Permission model** select **Vault access policy**.
+    [![Screenshot of create a key vault.](media/how-to-enable-external-data-sources/create-a-key-vault.jpg)](media/how-to-enable-external-data-sources/create-a-key-vault.jpg#lightbox)
+1. Give permission to Azure Data Manager for Energy Service Principal (SPN) on key vault (existing or new). How to give access to Azure Data Manager for Energy SPN - [Assign an Azure Key Vault access policy (CLI) | Microsoft Learn](../key-vault/general/assign-access-policy.md?tabs=azure-portal).
+    1. Under **Access Policies**, select **Create** to create an access policy: 
+        1. In the Permissions tab, select the permissions.
+        [![Screenshot of select permissions.](media/how-to-enable-external-data-sources/select-permissions.jpg)](media/how-to-enable-external-data-sources/select-permissions.jpg#lightbox) 
+        1. Under the Principal selection pane, enter `AzureEnergyRpaaSAppProd`.
+         [![Screenshot of create an access policy.](media/how-to-enable-external-data-sources/create-an-access-policy.jpg)](media/how-to-enable-external-data-sources/create-an-access-policy.jpg#lightbox) 
+        1. Review + Create the access policy.
+
+## To enable EDS Preview, create an Azure support request
+To enable External Data Sources Preview on your Azure Data Manager for Energy, create an Azure support ticket with the following information: 
+- Subscription ID 
+- Azure Data Manager for Energy developer tier resource name
+- Data partition name (the data partition in which EDS needs to be enabled)
+- Key Vault name (created in [Prerequisites](#prerequisites)) 
+
+> [!NOTE]
+> EDS does not have [multi data partition support](https://community.opengroup.org/osdu/platform/data-flow/ingestion/external-data-sources/core-external-data-workflow/-/issues/51)
+
+We notify you once EDS preview is enabled in your Azure Data Manager for Energy resource.
+
+## Known issues
+- Below issues are specific to [OSDU&trade;](https://osduforum.org/) M18 release: 
+    - EDS ingest DAG results in failures when the data supplier’s wrapper Search service is unavailable. 
+    - EDS Dataset service response provides an empty response when data supplier’s Dataset wrapper service is unavailable. 
+    - Secret service responds with 5xx HTTP response code instead of 4xx in some cases. For example, 
+        - When an application tries to recover a deleted secret, which isn't deleted. 
+        - When an application tries to get an invalid deleted secret. 
+
+## Limitations
+Some EDS capabilities like **Naturalization, Reverse Naturalization, Reference data mapping** are unavailable in the M18 [OSDU&trade;](https://osduforum.org/) release (available in later releases), and hence unavailable in Azure Data Manager for Energy M18 release. These features are available once we upgrade to subsequent [OSDU&trade;](https://osduforum.org/) milestone release.
+
+## FAQ
+See [External data sources FAQ.](faq-energy-data-services.yml#external-data-sources)
+
+> [!div class="nextstepaction"]
+> [How to register an external data source with Azure Data Manager for Energy?](how-to-register-external-data-sources.md) 
diff --git a/articles/energy-data-services/how-to-register-external-data-sources.md b/articles/energy-data-services/how-to-register-external-data-sources.md
@@ -0,0 +1,59 @@
+---
+title: "Microsoft Azure Data Manager for Energy - How to register external data sources"
+description: "This article describes how to register external data sources with Azure Data Manager for Energy."
+author: bharathim
+ms.author: bselvaraj
+ms.service: energy-data-services
+ms.topic: how-to #Don't change
+ms.date: 03/14/2024
+
+#customer intent: As a Data Manager in Operating company, I want to register external data sources with Azure Data Manager for Energy so that I could pull metadata at scheduled intervals and retrieve bulk data on demand.
+
+---
+# How to register an External Data Sources (EDS) with Azure Data Manager for Energy?
+This article explains how to register an External Data Sources (EDS) with Azure Data Manager for energy. EDS allow you to fetch and ingest data (metadata) from external data sources. It also allows you to retrieve bulk data on demand.
+
+## Prerequisites 
+- Download and import API [collection](https://community.opengroup.org/osdu/platform/pre-shipping/-/blob/main/R3-M20/QA_Artifacts_M20/eds_testing_doc/EDS_Ingest_M20_Pre-Shipping_Setup_and_Testing.postman_collection.json?ref_type=heads) and [environment](https://community.opengroup.org/osdu/platform/pre-shipping/-/blob/main/R3-M20/QA_Artifacts_M20/envFilesAndCollections/envFiles/OSDU%20R3%20M20%20RI%20Pre-ship.postman_environment.json?ref_type=heads) files into API test client (like Postman). Make appropriate modifications in environment based on your data source. 
+- Refer **Section 2.2** in [osdu-eds-data-supplier-enablement-guide](https://gitlab.opengroup.org/osdu/r3-program-activities/docs/-/raw/master/R3%20Document%20Snapshot/23-osdu-eds-data-supplier-enablement-guide.pdf) for details on Data source Registration. 
+- Review **Connected Source Registry Entry (CSRE)** and **Connection Source Data Job (CSDJ)** sections in [EDS_Documentation-1.0.docx](https://gitlab.opengroup.org/osdu/subcommittees/ea/projects/extern-data/docs/-/blob/master/Design%20Documents/Training/EDS_Documentation-1.0.docx) to understand the various parameters used in data source registration. 
+- To run EDS, the user must be a member of `service.eds.user` entitlements group. Additionally, to access Secret service, the user should be a member of the following entitlements: `service.secret.viewer`, `service.secret.editor`, `service.secret.admin`.  
+
+## EDS Fetch and Ingest workflow 
+Execute the APIs in the following collections to register your external data source that runs EDS Fetch and Ingest workflow on a schedule: 
+
+1. `001: Pre-req: Validate Schema Registration`
+1. `002: Pre-req: Validate Reference Data` 
+1. `003: Secret Service` 
+1. `004: Pre-req: Add Source Registry` 
+
+After successful data registration, data is regularly fetched from external sources and added to your Azure Data Manager for Energy. 
+
+You can use the Search service to search for your ingested data. 
+
+## Troubleshooting
+
+You could run the below Kusto queries in your Log analytics workspace to identify any issues with Data Source registration. 
+
+
+```kusto
+OEPAirFlowTask 
+| where DagName == "eds_ingest"        
+| where LogLevel == "ERROR" // ERROR/DEBUG/INFO/WARNING
+```
+
+```kusto
+OEPAirFlowTask 
+| where DagName == "eds_scheduler"        
+| where LogLevel == "ERROR" // ERROR/DEBUG/INFO/WARNING
+```
+## Retrieve bulk data on demand
+Use **getRetrievalInstructions** API in `005: Dataset Service collection` to retrieve bulk data from external data sources on demand. 
+
+## References
+* [External data sources FAQ](faq-energy-data-services.yml#external-data-sources)
+* [EDS documentation 1.0](https://gitlab.opengroup.org/osdu/subcommittees/ea/projects/extern-data/docs/-/blob/master/Design%20Documents/Training/EDS_Documentation-1.0.docx)
+* [EDS M18 release notes](https://community.opengroup.org/osdu/governance/project-management-committee/-/wikis/M18-Release-Notes#external-data-sources-eds)
+* [EDS Postman collection](https://community.opengroup.org/osdu/platform/pre-shipping/-/blob/main/R3-M20/QA_Artifacts_M20/eds_testing_doc/EDS_Ingest_M20_Pre-Shipping_Setup_and_Testing.postman_collection.json?ref_type=heads)
+* [EDS supplier enablement guide](https://gitlab.opengroup.org/osdu/r3-program-activities/docs/-/raw/master/R3%20Document%20Snapshot/23-osdu-eds-data-supplier-enablement-guide.pdf)
+* [EDS issues](https://community.opengroup.org/osdu/platform/data-flow/ingestion/external-data-sources/core-external-data-workflow/-/issues)
diff --git a/articles/energy-data-services/index.yml b/articles/energy-data-services/index.yml
@@ -76,6 +76,10 @@ landingContent:
             url: how-to-convert-segy-to-zgy.md
           - text: How to Upload large files using file service
             url: how-to-upload-large-files-using-file-service.md
+          - text: How to enable External Data Sources (EDS)
+            url: how-to-enable-external-data-sources.md
+          - text: How to register External Data Sources (EDS)
+            url: how-to-register-external-data-sources.md
   - title: Administration
     linkLists:
       - linkListType: how-to-guide
diff --git a/articles/energy-data-services/media/how-to-enable-external-data-sources/create-a-key-vault.jpg b/articles/energy-data-services/media/how-to-enable-external-data-sources/create-a-key-vault.jpg
diff --git a/articles/energy-data-services/media/how-to-enable-external-data-sources/create-an-access-policy.jpg b/articles/energy-data-services/media/how-to-enable-external-data-sources/create-an-access-policy.jpg
diff --git a/articles/energy-data-services/media/how-to-enable-external-data-sources/select-permissions.jpg b/articles/energy-data-services/media/how-to-enable-external-data-sources/select-permissions.jpg
diff --git a/articles/energy-data-services/release-notes.md b/articles/energy-data-services/release-notes.md
@@ -28,6 +28,11 @@ This page is updated with the details about the upcoming release approximately a
 ### Azure Data Manager for Energy in Australia East Region
 Azure Data Manager for Energy is now available in the Australia East Region. This new region is enabled for both the Standard and Developer tiers of Azure Data Manager for Energy. You can now select Australia East as your preferred region when creating Azure Data Manage for Energy resource, using the [Azure portal](https://ms.portal.azure.com/#create/Microsoft.AzureDataManagerforEnergy).
 
+### External Data Sources (Preview)
+External Data Sources (EDS) allows data from an [OSDU&trade;](https://osduforum.org/) compliant external data sources to be shared with an Azure Data Manager for Energy resource. EDS is designed to pull specified data (metadata) from OSDU-compliant data sources via scheduled jobs while leaving associated dataset files (LAS, SEG-Y, etc.) stored at the external source for retrieval on demand.
+
+For details, see [How to enable External Data Services (EDS) Preview?](how-to-enable-external-data-sources.md)
+
 ## November 2023
 
 ### Compliant with M18 OSDU&trade; release
@@ -53,7 +58,7 @@ Starting September 2023, the General Availability pricing changes for Azure Data
 Starting July 2023, Azure Data Manager for Energy offers an uptime SLA for its Standard tier offering. You can find the details of our SLA in the document 'Service Level Agreements for Microsoft Online Services (WW)'  published from July 2023 onwards at [Microsoft Licensing Documents & Resource website](https://www.microsoft.com/licensing/docs/view/Service-Level-Agreements-SLA-for-Online-Services?lang=1).
 
 ### Developer tier for accelerating innovation with OSDU&trade;
-Azure Data Manager for Energy is now available in two tiers; Developer and Standard.  All active resources of Azure Data Manager for Energy prior to this release are considered Standard, and now a new Tier option is available called, the 'Developer' tier.  Customers can now select their desired tier when creating their Azure Data Manage for Energy resource, using the [Azure portal](https://aka.ms/adme-create). [Learn more](./quickstart-create-microsoft-energy-data-services-instance.md)
+Azure Data Manager for Energy is now available in two tiers; Developer and Standard.  All active resources of Azure Data Manager for Energy prior to this release are considered Standard, and now a new Tier option is available called the 'Developer' tier.  Customers can now select their desired tier when creating their Azure Data Manage for Energy resource using the [Azure portal](https://aka.ms/adme-create). [Learn more](./quickstart-create-microsoft-energy-data-services-instance.md)
 
 ### Compliant with M16 OSDU&trade; release
 Azure Data Manager for Energy is now compliant with the M16 OSDU&trade; milestone release. With this release, you can take advantage of the latest features and capabilities available in the [OSDU&trade; M16](https://community.opengroup.org/osdu/governance/project-management-committee/-/wikis/M16-Release-Notes).
diff --git a/articles/energy-data-services/toc.yml b/articles/energy-data-services/toc.yml
@@ -68,6 +68,10 @@
           href: how-to-add-more-data-partitions.md
         - name: Deploy OSDU Admin UI
           href: how-to-deploy-osdu-admin-ui.md
+        - name: How to enable External Data Sources (EDS)
+          href: how-to-enable-external-data-sources.md
+        - name: How to register External Data Sources (EDS)
+          href: how-to-register-external-data-sources.md
     - name: Monitoring
       items:
         - name: Airflow task logs integration with Azure Monitor
