Merge pull request #264913 from rcdun/parjai/datatype_update

Adding the details of the new Data types supported by MCC DP

Author: JamesJBarnett

articles/operator-insights/TOC.yml

@@ -68,6 +68,8 @@
   items:
     - name: Monitoring data
       href: monitor-operator-insights-data-reference.md
+    - name: Device reference schema for the Quality of Experience Affirmed MCC Data Product
+      href: device-reference-schema.md
     - name: Azure CLI for Operator Insights
       href: /cli/azure/service-page/azure%20operator%20insights%20-%20network%20analytics
     - name: PowerShell for Operator Insights

articles/operator-insights/concept-data-types.md

@@ -1,6 +1,6 @@
 ---
 title: Data types - Azure Operator Insights
-description: This article provides an overview of the data types used by Azure Operator Insights Data Products
+description: This article provides an overview of the data types used by Azure Operator Insights Data Products.
 author: rcdun
 ms.author: rdunstan
 ms.reviewer: rathishr
@@ -15,17 +15,28 @@ ms.date: 10/25/2023
 
 A Data Product ingests data from one or more sources, digests and enriches this data, and presents this data to provide domain-specific insights and to support further data analysis.
 
-A data type is used to refer to an individual data source.  Data types can be from outside the Data Product, such as from a network element. Data types can also be created within the Data Product itself by aggregating or enriching information from other data types.
+A data type is used to refer to an individual data source. Data types can be from outside the Data Product, such as from a network element. Data types can also be created within the Data Product itself by aggregating or enriching information from other data types.
 
 Data Product operators can choose the data retention period for each data type.
 
 ## Data type contents
 
 Each data type contains data from a specific source. The primary source for a data type might be a network element within the subject domain. Some data types are derived by aggregating or enriching information from other data types.
 
-- The **Quality of Experience – Affirmed MCC** Data Product includes the *edr* data type that handles Event Data Records from the MCC.  
-- The **Quality of Experience – Affirmed MCC** Data Product also includes a derived *edr-sanitized* data type. This data type contains the same information as  *edr* but with personal data suppressed to support operators' compliance with privacy legislation.
-- The **Monitoring – Affirmed MCC** Data Product includes the *pmstats* data type that contains performance management statistics from the MCC EMS.
+- The **Quality of Experience – Affirmed MCC** Data Product includes the following data types.
+  - `edr`: This data type handles Event Data Records (EDRs) from the MCC.
+  - `edr-sanitized`: This data type contains the same information as `edr` but with personal data suppressed to support operators' compliance with privacy legislation.
+  - `edr-validation`: This data type contains a subset of performance management statistics and provides you with the ability to optionally ingest a minimum number of PMstats tables for a data quality check.
+  - `device`: This optional data type contains device data (for example, device model, make and capabilities) that the Data Product can use to enrich the MCC Event Data Records. To use this data type, you must upload the device reference data in a CSV file. The CSV must conform to the [Device reference schema for the Quality of Experience Affirmed MCC Data Product](device-reference-schema.md).
+  - `enrichment`: This data type holds the enriched Event Data Records and covers multiple sub data types for precomputed aggregations targeted to accelerate specific dashboards, granularities, and queries. These multiple sub data types include:
+      - `agg-enrichment-5m`: contains enriched Event Data Records aggregated over five-minute intervals.
+      - `agg-enrichment-1h`: contains enriched Event Data Records aggregated over one-hour intervals.
+      - `enriched-flow-dcount`: contains precomputed counts used to report the unique IMSIs, MCCs, and Applications over time.
+  - `location`: This optional data type contains data enriched with location information, if you have a source of location data. This covers the following sub data types.
+      - `agg-location-1h`: contains enriched location data aggregated over one-hour intervals.
+      - `enriched-loc-dcount`: contains precomputed counts used to report location data over time.
+ 
+- The **Monitoring – Affirmed MCC** Data Product includes the `pmstats` datatype. This data type contains performance management statistics from the MCC EMS.
 
 ## Data type settings
 

articles/operator-insights/concept-mcc-data-product.md

@@ -1,6 +1,6 @@
 ---
 title: Quality of Experience - Affirmed MCC Data Products - Azure Operator Insights
-description: This article gives an overview of the Azure Operator Insights Data Products provided to monitor the Quality of Experience for the Affirmed Mobile Content Cloud (MCC) 
+description: This article gives an overview of the Azure Operator Insights Data Products provided to monitor the Quality of Experience for the Affirmed Mobile Content Cloud (MCC).
 author: rcdun
 ms.author: rdunstan
 ms.reviewer: rathishr
@@ -13,7 +13,7 @@ ms.date: 10/25/2023
 
 # Quality of Experience - Affirmed MCC Data Product overview
 
-The *Quality of Experience - Affirmed MCC* Data Products support data analysis and insight for operators of the Affirmed Networks Mobile Content Cloud (MCC). They ingest Event Data Records (EDRs) from MCC network elements, and then digest and enrich this data to provide a range of visualizations for the operator.  Operator data scientists have access to the underlying enriched data to support further data analysis.
+The *Quality of Experience - Affirmed MCC* Data Products support data analysis and insight for operators of the Affirmed Networks Mobile Content Cloud (MCC). They ingest Event Data Records (EDRs) from MCC network elements, and then digest and enrich this data to provide a range of visualizations for the operator. Operator data scientists have access to the underlying enriched data to support further data analysis.
 
 ## Background
 
@@ -26,7 +26,7 @@ The Affirmed Networks Mobile Content Cloud (MCC) is a virtualized Evolved Packet
 - Serving GPRS support node and MME (SGSN/MME) is responsible for the delivery of data packets to and from the mobile stations within its geographical service area.
 - Control and User Plane Separation (CUPS), an LTE enhancement that separates control and user plane function to allow independent scaling of functions.
 
-The data produced by the MCC varies according to the functionality.  This variation affects the enrichments and visualizations that are relevant.  Azure Operator Insights provides the following Quality of Experience Data Products to support specific MCC functions.
+The data produced by the MCC varies according to the functionality. This variation affects the enrichments and visualizations that are relevant. Azure Operator Insights provides the following Quality of Experience Data Products to support specific MCC functions.
 
 - **Quality of Experience - Affirmed MCC GIGW**
 - **Quality of Experience - Affirmed MCC PGW/GGSN**
@@ -35,8 +35,17 @@ The data produced by the MCC varies according to the functionality.  This variat
 
 The following data types are provided for all Quality of Experience - Affirmed MCC Data Products.
 
-- *edr* contains data from the Event Data Records (EDRs) written by the MCC network elements.  EDRs record each significant event arising during calls or sessions handled by the MCC. They provide a comprehensive record of what happened, allowing operators to explore both individual problems and more general patterns.
-- *edr-sanitized* contains data from the *edr* data type but with personal data suppressed. Sanitized data types can be used to support data analysis while also enforcing subscriber privacy.
+- `edr`: This data type handles EDRs from the MCC.
+- `edr-sanitized`: This data type contains the same information as `edr` but with personal data suppressed to support operators' compliance with privacy legislation.
+- `edr-validation`: This data type contains a subset of performance management statistics and provides you with the ability to optionally ingest a minimum number of PMstats tables for a data quality check.
+- `device`: This optional data type contains device data (for example, device model, make and capabilities) that the Data Product can use to enrich the MCC Event Data Records. To use this data type, you must upload the device reference data in a CSV file. The CSV must conform to the [Device reference schema for the Quality of Experience Affirmed MCC Data Product](device-reference-schema.md).
+- `enrichment`: This data type holds the enriched Event Data Records and covers multiple sub data types for precomputed aggregations targeted to accelerate specific dashboards, granularities, and queries. These multiple sub data types include:
+    - `agg-enrichment-5m`: contains enriched Event Data Records aggregated over five-minute intervals.
+    - `agg-enrichment-1h`: contains enriched Event Data Records aggregated over one-hour intervals.
+    - `enriched-flow-dcount`: contains precomputed counts used to report the unique IMSIs, MCCs, and Applications over time.
+- `location`: This optional data type contains data enriched with location information, if you have a source of location data. This covers the following sub data types.
+    - `agg-location-1h`: contains enriched location data aggregated over one-hour intervals.
+    - `enriched-loc-dcount`: contains precomputed counts used to report location data over time.
 
 ## Setup
 
@@ -52,5 +61,5 @@ To use the Quality of Experience - Affirmed MCC Data Product:
 - [Monitoring - Affirmed MCC Data Product](concept-monitoring-mcc-data-product.md)
 - [Affirmed Networks MCC documentation](https://manuals.metaswitch.com/MCC) 
 
-    > [!NOTE]
-    > Affirmed Networks login credentials are required to access the MCC product documentation.
+> [!NOTE]
+> Affirmed Networks login credentials are required to access the MCC product documentation.

articles/operator-insights/device-reference-schema.md

@@ -0,0 +1,31 @@
+---
+title: Device schema for the Azure Operator Insights QoE MCC Data Product
+description: Learn about the schema needed to implement the Device data type in the Quality of Experience – Affirmed MCC Data Product for Azure Operator Insights.
+author: parjai
+ms.author: parjai
+ms.reviewer: rdunstan
+ms.topic: reference
+ms.service: operator-insights
+ms.date: 01/31/2024
+---
+
+<!-- #CustomerIntent: As a Data Product user, I want to add the ability to add device reference data to further enrich the MCC Event Data Records-->
+
+# Device reference schema for the Quality of Experience Affirmed MCC Data Product
+
+You can enrich Event Data Record (EDR) data in the Quality of Experience Affirmed MCC Data Product with information about the devices involved in the session. You must provide this device data in the `device` data type. Prepare CSV files that conform to the following schema and upload the files into the input Azure Data Lake Storage. For more information about data types, including the `device` data type, see [Data types](concept-data-types.md).
+
+## Schema for device reference information
+
+| Source field name | Type | Description |
+| --- | --- | --- |
+| `TAC` | String | Type Allocation Code (TAC): a unique identifier assigned to mobile devices. Typically first eight digits of IMEI number. Can have leading zeros if TAC is six or seven digits. Matched against the IMEI in the session EDRs |
+| `Make` | String | The manufacturer or brand of the mobile device. |
+| `Model` | String | The specific model or version of the mobile device. |
+| `DeviceType` | String | Categorizes the mobile device based on its primary function (for example, handheld or feature phone). |
+| `PlatformType` | String | Identifies the underlying operating system or platform running on the mobile device. |
+| `IsOwnedDevice` | String | Indicates if the device model was ranged by the operator. A value of `Y`/`1` signifies it is, while `N`/`0` indicates it isn't. |
+| `Is3G` | String | Indicates whether the mobile device supports 3G. A value of `Y`/`E`/`1` signifies 3G capability, while `N`/`0` indicates its absence. |
+| `IsLTE` | String | Indicates whether the mobile device supports Long-Term Evolution (LTE) technology. A value of `Y`/`E`/`1` signifies LTE capability, while `N`/`0` indicates its absence |
+| `IsVoLTE` | String | Indicates whether the mobile device supports Voice over LTE. A value of `Y`/`E`/`1` signifies VoLTE capability, while `N`/`0` indicates its absence. |
+| `Is5G` | String | Indicates whether the mobile device supports 5G. A value of `Y`/`E`/`1` signifies 5G capability, while `N`/`0` indicates its absence. |