Merge pull request #103681 from KingdomOfEnds/tsi-refresh

v-shils · web-flow · commit ca6e2f0f44f9 · 2020-02-10T12:31:11.000-08:00
TSI ingress and storage improvements
diff --git a/articles/time-series-insights/time-series-insights-update-storage-ingress.md b/articles/time-series-insights/time-series-insights-update-storage-ingress.md
@@ -8,114 +8,153 @@ ms.workload: big-data
 ms.service: time-series-insights
 services: time-series-insights
 ms.topic: conceptual
-ms.date: 12/31/2019
+ms.date: 02/10/2020
 ms.custom: seodec18
 ---
 
 # Data storage and ingress in Azure Time Series Insights Preview
 
-This article describes updates to data storage and ingress for Azure Time Series Insights Preview. It covers the underlying storage structure, file format, and Time Series ID property. It also discusses the underlying ingress process, best practices, and current preview limitations.
+This article describes updates to data storage and ingress for Azure Time Series Insights Preview. It describes the underlying storage structure, file format, and Time Series ID property. The underlying ingress process, best practices, and current preview limitations are also described.
 
 ## Data ingress
 
-Your Azure Time Series Insights environment contains an Ingestion Engine to collect, process, and store time-series data. When planning your environment, there are some considerations to take into account in order to ensure that all incoming data is processed, and to achieve high ingress scale and minimize ingestion latency (the time taken by TSI to read and process data from the event source). 
+Your Azure Time Series Insights environment contains an *ingestion engine* to collect, process, and store time-series data. 
 
-In Time Series Insights Preview, data ingress policies determine where data can be sourced from and what format the data should have.
+There are some considerations to be mindful of to ensure all incoming data is processed, to achieve high ingress scale, and minimize *ingestion latency* (the time taken by Time Series Insights to read and process data from the event source) when [planning your environment](time-series-insights-update-plan.md).
+
+Time Series Insights Preview data ingress policies determine where data can be sourced from and what format the data should have.
 
 ### Ingress policies
 
+*Data ingress* involves how data is sent to an Azure Time Series Insights Preview environment. 
+
+Key configuration, formatting, and best practices are summarized below.
+
 #### Event Sources
 
-Time Series Insights Preview supports the following event sources:
+Azure Time Series Insights Preview supports the following event sources:
 
 - [Azure IoT Hub](../iot-hub/about-iot-hub.md)
 - [Azure Event Hubs](../event-hubs/event-hubs-about.md)
 
-Time Series Insights Preview supports a maximum of two event sources per instance.
+Azure Time Series Insights Preview supports a maximum of two event sources per instance.
 
-> [!WARNING] 
+> [!IMPORTANT] 
 > * You may experience high initial latency when attaching an event source to your Preview environment. 
 > Event source latency depends on the number of events currently in your IoT Hub or Event Hub.
-> * High latency will subside after event source data is first ingested. Contact us by submitting a support ticket through the Azure portal if you experience continued high latency.
+> * High latency will subside after event source data is first ingested. Submit a support ticket through the Azure portal if you experience ongoing high latency.
 
 #### Supported data format and types
 
-Azure Time Series Insights supports UTF8 encoded JSON submitted through Azure IoT Hub or Azure Event Hubs. 
+Azure Time Series Insights supports UTF-8 encoded JSON sent from Azure IoT Hub or Azure Event Hubs. 
 
-Below is the list of supported data types.
+The supported data types are:
 
 | Data type | Description |
-|-----------|------------------|-------------|
-| bool      |   A data type having one of two states: true or false.       |
-| dateTime    |   Represents an instant in time, typically expressed as a date and time of day. DateTimes should be in ISO 8601 format.      |
-| double    |   A double-precision 64-bit IEEE 754 floating point
-| string    |   Text values, comprised of Unicode characters.          |
+|---|---|
+| **bool** | A data type having one of two states: `true` or `false`. |
+| **dateTime** | Represents an instant in time, typically expressed as a date and time of day. Expressed in [ISO 8601](https://www.iso.org/iso-8601-date-and-time-format.html) format. |
+| **double** | A double-precision 64-bit [IEEE 754](https://ieeexplore.ieee.org/document/8766229) floating point. |
+| **string** | Text values, comprised of Unicode characters.          |
 
 #### Objects and arrays
 
-You can send complex types such as objects and arrays as part of your event payload, but your data will undergo a flattening process when stored. For more information on how to shape your JSON events as well as details on complex type and nested object flattening, see the page on [how to shape JSON for ingress and query](./time-series-insights-update-how-to-shape-events.md).
+You may send complex types such as objects and arrays as part of your event payload, but your data will undergo a flattening process when stored. 
 
+Detailed information describing how to shape your JSON events, send complex type, and nested object flattening is available in [How to shape JSON for ingress and query](./time-series-insights-update-how-to-shape-events.md) to assist with planning and optimization.
 
 ### Ingress best practices
 
 We recommend that you employ the following best practices:
 
-* Configure Time Series Insights and your IoT Hub or Event Hub in the same region in order to reduce network incurred ingestion latency.
-* Plan for your scale needs by calculating your anticipated ingestion rate and verifying that it falls within the supported rate listed below
+* Configure Azure Time Series Insights and any IoT Hub or Event Hub in the same region to reduce potential latency.
+
+* [Plan for your scale needs](time-series-insights-update-plan.md) by calculating your anticipated ingestion rate and verifying that it falls within the supported rate listed below.
+
 * Understand how to optimize and shape your JSON data, as well as the current limitations in preview, by reading [how to shape JSON for ingress and query](./time-series-insights-update-how-to-shape-events.md).
 
-### Ingress scale and limitations in preview
+### Ingress scale and Preview limitations 
+
+Azure Time Series Insights Preview ingress limitations are described below.
+
+> [!TIP]
+> Read [Plan your Preview environment](https://docs.microsoft.com/azure/time-series-insights/time-series-insights-update-plan#review-preview-limits) for a comprehensive list of all Preview limits.
 
 #### Per environment limitations
 
 In general, ingress rates are viewed as the factor of the number of devices that are in your organization, event emission frequency, and the size of each event:
 
 *  **Number of devices** × **Event emission frequency** × **Size of each event**.
 
-By default, Time Series Insights preview can ingest incoming data at a rate of up to 1 megabyte per second (MBps) **per TSI environment**. Contact us if this does not meet your requirements, we can support up to 16 MBps for an environment by submitting a support ticket in the Azure portal.
- 
-Example 1: Contoso Shipping has 100,000 devices that emit an event three times per minute. The size of an event is 200 bytes. They’re using an Event Hub with 4 partitions as the TSI event source.
-The ingestion rate for their TSI environment would be: 100,000 devices * 200 bytes/event * (3/60 event/sec) = 1 MBps.
-The ingestion rate per partition would be 0.25 MBps.
-Contoso Shipping’s ingestion rate would be within the preview scale limitation.
+By default, Time Series Insights preview can ingest incoming data at a rate of **up to 1 megabyte per second (MBps) per Time Series Insights environment**.
+
+> [!TIP] 
+> * Environment support for ingesting speeds up to 16 MBps can be provided by request.
+> * Contact us if you require higher throughput by submitting a support ticket through Azure portal.
  
-Example 2: Contoso Fleet Analytics has 60,000 devices that emit an event every second. They are using an IoT Hub 24 partition count of 4 as the TSI event source. The size of an event is 200 bytes.
-The environment ingestion rate would be: 20,000 devices * 200 bytes/event * 1 event/sec = 4 MBps.
-The per partition rate would be 1 MBps.
-Contoso Fleet Analytics would need to submit a request to TSI via the Azure portal for a dedicated environment to achieve this scale.
+* **Example 1:**
+
+    Contoso Shipping has 100,000 devices that emit an event three times per minute. The size of an event is 200 bytes. They’re using an Event Hub with four partitions as the Time Series Insights event source.
+
+    * The ingestion rate for their Time Series Insights environment would be: **100,000 devices * 200 bytes/event * (3/60 event/sec) = 1 MBps**.
+    * The ingestion rate per partition would be 0.25 MBps.
+    * Contoso Shipping’s ingestion rate would be within the preview scale limitation.
+
+* **Example 2:**
+
+    Contoso Fleet Analytics has 60,000 devices that emit an event every second. They are using an IoT Hub 24 partition count of 4 as the Time Series Insights event source. The size of an event is 200 bytes.
 
-#### Hub Partitions and Per Partition Limits
+    * The environment ingestion rate would be: **20,000 devices * 200 bytes/event * 1 event/sec = 4 MBps**.
+    * The per partition rate would be 1 MBps.
+    * Contoso Fleet Analytics can submit a request to Time Series Insights through Azure portal to increase the ingestion rate for their environment.
 
-When planning your TSI environment, it's important to consider the configuration of the event source(s) that you'll be connecting to TSI. Both Azure IoT Hub and Event Hubs utilize partitions to enable horizontal scale for event processing.  A partition is an ordered sequence of events that is held in a hub. The partition count is set during the IoT or Event Hubs’ creation phase, and is not changeable. For more information on determining the partition count, see the Event Hubs' FAQ How many partitions do I need? For TSI environments using IoT Hub, generally most IoT Hubs only need 4 partitions. Whether or not you're creating a new hub for your TSI environment, or using an existing one, you'll need to calculate your per partition ingestion rate to determine if it is within the preview limits. TSI preview currently has a **per partition** limit of 0.5 MB/s. Use the examples below as a reference, and please note the following IoT Hub-specific consideration if you're an IoT Hub user.
+#### Hub partitions and per partition limits
+
+When planning your Time Series Insights environment, it's important to consider the configuration of the event source(s) that you'll be connecting to Time Series Insights. Both Azure IoT Hub and Event Hubs utilize partitions to enable horizontal scale for event processing. 
+
+A *partition* is an ordered sequence of events held in a hub. The partition count is set during the hub creation phase and cannot be changed. 
+
+For Event Hubs partitioning best practices, review [How many partitions do I need?](https://docs.microsoft.com/azure/event-hubs/event-hubs-faq#how-many-partitions-do-i-need)
+
+> [!NOTE]
+> Most IoT Hubs used with Azure Time Series Insights only need four partitions.
+
+Whether you're creating a new hub for your Time Series Insights environment or using an existing one, you'll need to calculate your per partition ingestion rate to determine if it's within the preview limits. 
+
+Azure Time Series Insights Preview currently has a general **per partition limit of 0.5 MBps**.
 
 #### IoT Hub-specific considerations
 
-When a device is created in IoT Hub it is assigned to a partition, and the partition assignment will not change. By doing so, IoT Hub is able to guarantee event ordering. However, this has implications for TSI as a downstream reader in certain scenarios. When messages from multiple devices are forwarded to the hub using the same gateway device ID they will arrive in the same partition, thus potentially exceeding the per partition scale limitation. 
+When a device is created in IoT Hub, it's permanently assigned to a partition. In doing so, IoT Hub is able to guarantee event ordering (since the assignment never changes).
+
+A fixed partition assignment also impacts Time Series Insights instances that are ingesting data sent from IoT Hub downstream. When messages from multiple devices are forwarded to the hub using the same gateway device ID, they may arrive in the same partition at the same time potentially exceeding the per partition scale limits. 
 
 **Impact**:
-If a single partition experiences a sustained rate of ingestion over the preview limitation there is the potential that the TSI reader will not ever catch up before the IoT Hub data retention period has been exceeded. This would cause a loss of data.
 
-We recommend the following: 
+* If a single partition experiences a sustained rate of ingestion over the Preview limit, it's possible that Time Series Insights will not sync all device telemetry before the IoT Hub data retention period has been exceeded. As a result, sent data can be lost if the ingestion limits are consistently exceeded.
 
-* Calculate your per environment and per partition ingestion rate before deploying your solution
-* Ensure that your IoT Hub devices (and thus partitions) are load-balanced to the furthest extend possible
+To mitigate that circumstance, we recommend the following best practices:
 
-> [!WARNING]
+* Calculate your per environment and per partition ingestion rates before deploying your solution.
+* Ensure that your IoT Hub devices are load-balanced to the furthest extent possible.
+
+> [!IMPORTANT]
 > For environments using IoT Hub as an event source, calculate the ingestion rate using the number of hub devices in use to be sure that the rate falls below the 0.5 MBps per partition limitation in preview.
+> * Even if several events arrive simultaneously, the Preview limit will not be exceeded.
 
   ![IoT Hub Partition Diagram](media/concepts-ingress-overview/iot-hub-partiton-diagram.png)
 
-Refer to the following links for more information on throughput units and partitions:
+Refer to the following resources to learn more about optimizing hub throughput and partitions:
 
 * [IoT Hub Scale](https://docs.microsoft.com/azure/iot-hub/iot-hub-scaling)
 * [Event Hub Scale](https://docs.microsoft.com/azure/event-hubs/event-hubs-scalability#throughput-units)
 * [Event Hub Partitions](https://docs.microsoft.com/azure/event-hubs/event-hubs-features#partitions)
 
 ### Data storage
 
-When you create a Time Series Insights Preview pay-as-you-go SKU environment, you create two Azure resources:
+When you create a Time Series Insights Preview *pay-as-you-go* (PAYG) SKU environment, you create two Azure resources:
 
-* A Time Series Insights Preview environment that can optionally include warm store capabilities.
+* An Azure Time Series Insights Preview environment that can be configured for warm storage.
 * An Azure Storage general-purpose V1 blob account for cold data storage.
 
 Data in your warm store is available only via [Time Series Query](./time-series-insights-update-tsq.md) and the [Azure Time Series Insights Preview explorer](./time-series-insights-update-explorer.md). 
@@ -127,7 +166,7 @@ Time Series Insights Preview saves your cold store data to Azure Blob storage in
 
 ### Data availability
 
-Time Series Insights Preview partitions and indexes data for optimum query performance. Data becomes available to query after it’s indexed. The amount of data that's being ingested can affect this availability.
+Azure Time Series Insights Preview partitions and indexes data for optimum query performance. Data becomes available to query after it’s indexed. The amount of data that's being ingested can affect this availability.
 
 > [!IMPORTANT]
 > During the preview, you might experience a period of up to 60 seconds before data becomes available. If you experience significant latency beyond 60 seconds, please submit a support ticket through the Azure portal.
@@ -140,13 +179,16 @@ For a thorough description of Azure Blob storage, read the [Storage blobs introd
 
 ### Your storage account
 
-When you create a Time Series Insights Preview pay-as-you-go environment, an Azure Storage general-purpose V1 blob account is created as your long-term cold store.  
+When you create an Azure Time Series Insights Preview PAYG environment, an Azure Storage general-purpose V1 blob account is created as your long-term cold store.  
+
+Azure Time Series Insights Preview publishes up to two copies of each event in your Azure Storage account. The initial copy has events ordered by ingestion time. That event order is **always preserved** so other services can access your events without sequencing issues. 
 
-Time Series Insights Preview publishes up to two copies of each event in your Azure Storage account. The initial copy has events ordered by ingestion time and is always preserved, so you can use other services to access it. You can use Spark, Hadoop, and other familiar tools to process the raw Parquet files. 
+> [!NOTE]
+> You can also use Spark, Hadoop, and other familiar tools to process the raw Parquet files. 
 
-Time Series Insights Preview repartitions the Parquet files to optimize for the Time Series Insights query. This repartitioned copy of the data is also saved.
+Time Series Insights Preview also repartitions the Parquet files to optimize for the Time Series Insights query. This repartitioned copy of the data is also saved. 
 
-During public preview, data is stored indefinitely in your Azure Storage account.
+During public Preview, data is stored indefinitely in your Azure Storage account.
 
 #### Writing and editing Time Series Insights blobs
 
diff --git a/includes/time-series-insights-preview-limits.md b/includes/time-series-insights-preview-limits.md
@@ -4,7 +4,7 @@ description: include file
 services: digital-twins
 ms.service: digital-twins
 ms.topic: include
-ms.date: 02/03/2020
+ms.date: 02/07/2020
 author: deepakpalled
 ms.author: dpalled
 manager: cshankar
@@ -45,7 +45,7 @@ A maximum of two event sources per instance is supported.
 * Learn how to [Add an event hub source](https://docs.microsoft.com/azure/time-series-insights/time-series-insights-how-to-add-an-event-source-eventhub).
 * Configure [an IoT hub source](https://docs.microsoft.com/azure/time-series-insights/time-series-insights-how-to-add-an-event-source-iothub).
 
-By default, [Preview environments support ingress rates](https://docs.microsoft.com/azure/time-series-insights/time-series-insights-update-storage-ingress) up to **1 megabyte per second (MB/s) per environment**. Customers may scale their Preview environments up to **16 MB/s** throughput if necessary. There is also a per-partition limit of **0.5 MB/s**. 
+By default, [Preview environments support ingress rates](https://docs.microsoft.com/azure/time-series-insights/time-series-insights-update-storage-ingress#ingress-scale-and-preview-limitations) up to **1 megabyte per second (MB/s) per environment**. Customers may scale their Preview environments up to **16 MB/s** throughput if necessary. There is also a per-partition limit of **0.5 MB/s**. 
 
 ### API limits
 
