Merge pull request #268919 from rcdun/aoi_data_product_factory

Document the Data Product Factory (preview)

Author: v-dirichards

articles/operator-insights/TOC.yml

@@ -33,6 +33,8 @@
     href: ingestion-agent-overview.md
   - name: Managed identity
     href: managed-identity.md
+  - name: Data product factory
+    href: data-product-factory.md
 - name: How-to guides
   expanded: false
   items:

articles/operator-insights/business-continuity-disaster-recovery.md

@@ -40,6 +40,6 @@ Azure Operator Insights has no innate region redundancy. Regional outages affect
 
 #### User-managed redundancy
 
-For maximal redundancy, you can deploy Data Products in an active-active mode. Deploy a second Data Product in a backup Azure region of your choice, and configure your ingestion agents to fork data to both Data Products simultaneously. The backup data product is unaffected by the failure of the primary region. During a regional outage, look at dashboards that use the backup Data Product as the data source. This architecture doubles the cost of the solution.
+For maximal redundancy, you can deploy Data Products in an active-active mode. Deploy a second Data Product in a backup Azure region of your choice, and configure your ingestion agents to fork data to both Data Products simultaneously. The backup Data Product is unaffected by the failure of the primary region. During a regional outage, look at dashboards that use the backup Data Product as the data source. This architecture doubles the cost of the solution.
 
 Alternatively, you could use an active-passive mode. Deploy a second Data Product in a backup Azure region, and configure your ingestion agents to send to the primary Data Product. During a regional outage, reconfigure your ingestion agents to send data to the backup Data Product during a region outage. This architecture gives full access to data created during the outage (starting from the time where you reconfigure the ingestion agents), but during the outage you don't have access to data ingested before that time. This architecture requires a small infrastructure charge for the second Data Product, but no additional data processing charges.

articles/operator-insights/change-ingestion-agent-configuration.md

@@ -8,7 +8,7 @@ ms.service: operator-insights
 ms.topic: how-to
 ms.date: 02/29/2024
 
-#CustomerIntent: As a someone managing an agent that has already been set up, I want to update its configuration so that data products in Azure Operator Insights receive the correct data.
+#CustomerIntent: As a someone managing an agent that has already been set up, I want to update its configuration so that Data Products in Azure Operator Insights receive the correct data.
 ---
 
 # Change configuration for Azure Operator Insights ingestion agents
@@ -19,7 +19,7 @@ In this article, you'll change your ingestion agent configuration and roll back
 
 ## Prerequisites
 
-- Using the documentation for your data product, check for required or recommended configuration for the ingestion agent.
+- Using the documentation for your Data Product, check for required or recommended configuration for the ingestion agent.
 - See [Configuration reference for Azure Operator Insights ingestion agent](ingestion-agent-configuration-reference.md) for full details of the configuration options.
 
 ## Update agent configuration

articles/operator-insights/concept-data-quality-monitoring.md

@@ -34,8 +34,8 @@ Data quality dimensions are the various aspects or characteristics that define t
 
 All data quality dimensions are covered by quality metrics produced by Azure Operator Insights platform. There are two types of the quality metrics:
 
-- Basic - Standard set of checks across all data products.
-- Custom - Custom set of checks, allowing all data products to implement checks that are specific to their product.
+- Basic - Standard set of checks across all Data Products.
+- Custom - Custom set of checks, allowing all Data Products to implement checks that are specific to their product.
 
 The basic quality metrics produced by the platform are available in the following table.
 
@@ -54,7 +54,7 @@ The basic quality metrics produced by the platform are available in the followin
 | Percentiles for lag between data processed and available for querying            | Timeliness    | Processed       |
 | Ages for materialized views                                                      | Timeliness    | Processed       |
 
-The custom data quality metrics are implemented on per data product basis. These metrics cover the accuracy and consistency dimensions. Data product documentation contains description for the custom quality metrics available.
+The custom data quality metrics are implemented on per Data Product basis. These metrics cover the accuracy and consistency dimensions. Data Product documentation contains description for the custom quality metrics available.
 
 ## Monitoring
 

articles/operator-insights/data-product-create.md

@@ -228,7 +228,7 @@ You create the Azure Operator Insights Data Product resource.
     1. Carefully paste the Key Identifier URI that was created when you set up Azure Key Vault as a prerequisite.
 
 1. To add owner(s) for the Data Product, which will also appear in Microsoft Purview, select **Add owner**, enter the email address, and select **Add owners**.
-1. In the Tags tab of the **Create a Data Product** page, select or enter the name/value pair used to categorize your data product resource.
+1. In the Tags tab of the **Create a Data Product** page, select or enter the name/value pair used to categorize your Data Product resource.
 1. Select **Review + create**.
 1. Select **Create**. Your Data Product instance is created in about 20-25 minutes. During this time, all the underlying components are provisioned. After this process completes, you can work with your data ingestion, explore sample dashboards and queries, and so on.
 
@@ -268,7 +268,7 @@ Once your Data Product instance is created, you can deploy a sample insights das
 > [!NOTE] 
 > The reader role is required for you to have access to the insights consumption URL.
 
-3. Download the sample JSON template file for your data product's dashboard:
+3. Download the sample JSON template file for your Data Product's dashboard:
     * Quality of Experience - Affirmed MCC GIGW: [https://go.microsoft.com/fwlink/p/?linkid=2254536](https://go.microsoft.com/fwlink/p/?linkid=2254536)
     * Monitoring - Affirmed MCC: [https://go.microsoft.com/fwlink/p/?linkid=2254551](https://go.microsoft.com/fwlink/?linkid=2254551)
 1. Copy the consumption URL from the Data Product overview screen into the clipboard.
@@ -321,7 +321,7 @@ The consumption URL also allows you to write your own Kusto query to get insight
 
 ## Optionally, delete Azure resources
 
-If you're using this data product to explore Azure Operator Insights, you should delete the resources you've created to avoid unnecessary Azure costs.
+If you're using this Data Product to explore Azure Operator Insights, you should delete the resources you've created to avoid unnecessary Azure costs.
 
 # [Portal](#tab/azure-portal)
 
@@ -339,7 +339,7 @@ az group delete --name "ResourceGroup"
 
 ## Next step
 
-Upload data to your data product. If you're planning to do this with the Azure Operator Insights ingestion agent:
+Upload data to your Data Product. If you're planning to do this with the Azure Operator Insights ingestion agent:
 
-1. Read the documentation for your data product to determine the requirements.
+1. Read the documentation for your Data Product to determine the requirements.
 1. [Install the Azure Operator Insights ingestion agent and configure it to upload data](set-up-ingestion-agent.md).

articles/operator-insights/data-product-factory.md

@@ -0,0 +1,52 @@
+---
+title: What is the data product factory (preview) for Azure Operator Insights?
+description: Learn about the data product factory (preview) for Azure Operator Insights, and how it can help you design and create new Data Products.
+author: rcdun
+ms.author: rdunstan
+ms.reviewer: sergeyche
+ms.service: operator-insights
+ms.topic: overview
+
+#CustomerIntent: As a partner developing a Data Product, I want to understand what the data product factory is so that I can use it.
+---
+
+# What is the Azure Operator Insights data product factory (preview)?
+
+Azure Operator Insights Data Products process data from operator networks, enrich it, and make it available for analysis. They can include prebuilt dashboards, and allow operators to view their data in other analysis tools. For more information, see [What is Azure Operator Insights?](overview.md).
+
+The Azure Operator Insights data product factory (preview) allows partners to easily design and create new Data Products for the Azure Operator Insights platform. Partners can develop pipelines to analyze network data and offer insights, while allowing the Azure Operator Insights platform to process operator-scale data.
+
+:::image type="complex" source="media/data-product-factory/data-product-factory.png" alt-text="Diagram indicating the position of the data product factory between the Azure Operator Insights platform and the Azure Marketplace.":::
+The data product factory is built on the Azure Operator Insights platform, which provides low-latency, transformation and analysis. You can publish Data Products from the data product factory to the Azure Marketplace for monetization.
+:::image-end:::
+
+## Features of the data product factory (preview)
+
+The data product factory (preview) offers:
+
+- Integration with Azure Marketplace for discoverability and monetization.
+- Acceleration of time to business value with "no code" / "low code" techniques that allow rapid onboarding of new data sources from operator networks, more quickly than IT-optimized toolkits.
+- Standardization of key areas, including:
+  - Design of data pipelines for ingesting data, transforming it and generating insights.
+  - Configuration of Microsoft Purview catalogs for data governance.
+  - Data quality metrics.
+- Simpler migration of on-premises analytics pipelines to Azure.
+- Easy integration with partners' own value-add solutions through open and consistent interfaces, such as:
+  - Integration into workflow and ticketing systems empowering automation based on AI-generated insights.
+  - Integration into network-wide management solutions such as OSS/NMS platforms.
+
+## Using the data product factory (preview)
+
+The data product factory (preview) is a self-service environment for partners to design, create, and test new Data Products.
+
+Each Data Product is defined by a data product definition: a set of files defining the transformation, aggregation, summarization, and visualization of the data.
+
+The data product factory is delivered as a GitHub-based SDK containing:
+- A development environment and sandbox for local design and testing. The environment and sandbox provide a tight feedback loop to accelerate the development cycle for ingestion, enrichment, and insights.
+- Documentation including step-by-step tutorials for designing, testing, publishing, and monetizing Data Products.
+- Sample data product definitions to kickstart design and creation.
+- Tools to automatically generate and validate data product definitions.
+
+## Next step
+
+Apply for access to the data product factory SDK by filling in the [application form](https://forms.office.com/r/vMP9bjQr6n).

articles/operator-insights/index.yml

@@ -89,4 +89,12 @@ landingContent:
           - text: Data types
             url: concept-data-types.md
           - text: Data visualization
-            url: concept-data-visualization.md 
+            url: concept-data-visualization.md
+
+  # Card
+  - title: Develop Data Products
+    linkLists:
+      - linkListType: concept
+        links:
+          - text: Azure Operator Insights data product factory (preview)
+            url: data-product-factory.md

articles/operator-insights/ingestion-agent-overview.md

@@ -13,7 +13,7 @@ ms.date: 12/8/2023
 
 # Ingestion agent overview
 
-An _ingestion agent_ uploads data to an Azure Operator Insights data product. We provide an ingestion agent called the Azure Operator Insights ingestion agent that you can install on a Linux virtual machine to upload data from your network. This ingestion agent supports uploading:
+An _ingestion agent_ uploads data to an Azure Operator Insights Data Product. We provide an ingestion agent called the Azure Operator Insights ingestion agent that you can install on a Linux virtual machine to upload data from your network. This ingestion agent supports uploading:
 
 - Affirmed Mobile Content Cloud (MCC) Event Data Record (EDR) data streams.
 - Files stored on an SFTP server.

articles/operator-insights/media/data-product-factory/data-product-factory.png

@@ -54,7 +54,7 @@ To start monitoring a Data Product with Azure Monitor Logs and Log Analytics:
 1. In the **Diagnostic setting** view of your Data Product, create a diagnostic setting that routes the logs that you want to collect to the Log Analytics workspace. To use the example query in this procedure, include **Database Query** (in addition to any other category of logs that you want to collect).
     - For instructions, see [Create diagnostic setting to collect platform logs and metrics in Azure](/azure/azure-monitor/platform/diagnostic-settings). You can use the Azure portal, CLI, or PowerShell.
     - The categories of logs for Azure Operator Insights are listed in [Azure Operator Insights monitoring data reference](monitor-operator-insights-data-reference.md#resource-logs).
-1. To use the example query in this procedure, run a query on the data in your Data Product by following [Query data in the Data Product](data-query.md). This step ensures that Azure Monitor Logs has some data for your data product.
+1. To use the example query in this procedure, run a query on the data in your Data Product by following [Query data in the Data Product](data-query.md). This step ensures that Azure Monitor Logs has some data for your Data Product.
 1. Return to your Data Product resource and select **Logs** from the Azure Operator Insights menu to access Log Analytics.
 1. Run the following query to view the log for the query that you ran on your Data Product, replacing _[email protected]_ with the email address you used when you ran the query. You can also adapt the sample queries in [Sample Kusto queries](#sample-kusto-queries).
     ```kusto