Merge pull request #5734 from influxdata/jstirnaman/DAR-463

docs(partitioning): enhance best practices and time part templates do…

Author: jstirnaman

content/influxdb/cloud-dedicated/admin/custom-partitions/_index.md

@@ -8,49 +8,9 @@ menu:
     name: Best practices
     parent: Manage data partitioning
 weight: 202
+source: /shared/v3-distributed-admin-custom-partitions/best-practices.md
 ---
 
-Use the following best practices when defining custom partitioning strategies
-for your data stored in {{< product-name >}}.
-
-- [Partition by tags that you commonly query for a specific value](#partition-by-tags-that-you-commonly-query-for-a-specific-value)
-- [Only partition by tags that _always_ have a value](#only-partition-by-tags-that-always-have-a-value)
-- [Avoid over-partitioning](#avoid-over-partitioning)
-
-## Partition by tags that you commonly query for a specific value
-
-Custom partitioning primarily benefits queries that look for a specific tag
-value in the `WHERE` clause. For example, if you often query data related to a
-specific ID, partitioning by the tag that stores the ID helps the InfluxDB
-query engine to more quickly identify what partitions contain the relevant data.
-
-{{% note %}}
-
-#### Use tag buckets for high-cardinality tags
-
-Partitioning using distinct values of tags with many (10K+) unique values can
-actually hurt query performance as partitions are created for each unique tag value.
-Instead, use [tag buckets](/influxdb/cloud-dedicated/admin/custom-partitions/partition-templates/#tag-bucket-part-templates)
-to partition by high-cardinality tags.
-This method of partitioning groups tag values into "buckets" and partitions by bucket.
-{{% /note %}}
-
-## Only partition by tags that _always_ have a value
-
-You should only partition by tags that _always_ have a value.
-If points don't have a value for the tag, InfluxDB can't store them in the correct partitions and, at query time, must read all the partitions.
-
-## Avoid over-partitioning
-
-As you plan your partitioning strategy, keep in mind that data can be
-"over-partitioned"--meaning partitions are so granular that queries end up
-having to retrieve and read many partitions from the object store, which
-hurts query performance.
-
-- Balance the partition time interval with the actual amount of data written
-  during each interval. If a single interval doesn't contain a lot of data,
-  it is better to partition by larger time intervals.
-- Don't partition by tags that you typically don't use in your query workload.
-- Don't partition by distinct values of high-cardinality tags.
-  Instead, [use tag buckets](#use-tag-buckets-for-high-cardinality-tags) to
-  partition by these tags.
+<!-- 
+The content of this page is at /content/shared/v3-distributed-admin-custom-partitions/best-practices.md
+-->

content/influxdb/cloud-dedicated/admin/custom-partitions/best-practices.md

@@ -10,161 +10,9 @@ weight: 202
 related:
   - /influxdb/cloud-dedicated/reference/cli/influxctl/database/create/
   - /influxdb/cloud-dedicated/reference/cli/influxctl/table/create/
+source: /shared/v3-distributed-admin-custom-partitions/define-custom-partitions.md
 ---
 
-Use the [`influxctl` CLI](/influxdb/cloud-dedicated/reference/cli/influxctl/)
-to define custom partition strategies when creating a database or table.
-By default, {{< product-name >}} partitions data by day.
-
-The partitioning strategy of a database or table is determined by a
-[partition template](/influxdb/cloud-dedicated/admin/custom-partitions/#partition-templates)
-which defines the naming pattern for [partition keys](/influxdb/cloud-dedicated/admin/custom-partitions/#partition-keys).
-Partition keys uniquely identify each partition.
-When a partition template is applied to a database, it becomes the default template
-for all tables in that database, but can be overridden when creating a
-table.
-
-- [Create a database with a custom partition template](#create-a-database-with-a-custom-partition-template)
-- [Create a table with a custom partition template](#create-a-table-with-a-custom-partition-template)
-- [Example partition templates](#example-partition-templates)
-
-{{% warn %}}
-
-#### Partition templates can only be applied on create
-
-You can only apply a partition template when creating a database or table.
-You can't update a partition template on an existing resource.
-{{% /warn %}}
-
-Use the following command flags to identify
-[partition template parts](/influxdb/cloud-dedicated/admin/custom-partitions/partition-templates/#tag-part-templates):
-
-- `--template-tag`: An [InfluxDB tag](/influxdb/cloud-dedicated/reference/glossary/#tag)
-  to use in the partition template.
-- `--template-tag-bucket`: An [InfluxDB tag](/influxdb/cloud-dedicated/reference/glossary/#tag)
-  and number of "buckets" to group tag values into.
-  Provide the tag key and the number of buckets to bucket tag values into
-  separated by a comma: `tagKey,N`.
-- `--template-timeformat`: A [Rust strftime date and time](/influxdb/cloud-dedicated/admin/custom-partitions/partition-templates/#time-part-templates)
-  string that specifies the time format in the partition template and determines
-  the time interval to partition by.
-
-{{% note %}}
-A partition template can include up to 7 total tag and tag bucket parts
-and only 1 time part.
-{{% /note %}}
-
-_View [partition template part restrictions](/influxdb/cloud-dedicated/admin/custom-partitions/partition-templates/#restrictions)._
-
-{{% note %}}
-#### Always provide a time format when using custom partitioning
-
-When defining a custom partition template for your database or table using any
-of the `influxctl` `--template-*` flags, always include the `--template-timeformat`
-flag with a time format to use in your partition template.
-Otherwise, InfluxDB omits time from the partition template and won't compact partitions.
-{{% /note %}}
-
-## Create a database with a custom partition template
-
-The following example creates a new `example-db` database and applies a partition
-template that partitions by distinct values of two tags (`room` and `sensor-type`),
-bucketed values of the `customerID` tag, and by day using the time format `%Y-%m-%d`:
-
-<!--Skip database create and delete tests: namespaces aren't reusable-->
-<!--pytest.mark.skip-->
-
-```sh
-influxctl database create \
-  --template-tag room \
-  --template-tag sensor-type \
-  --template-tag-bucket customerID,500 \
-  --template-timeformat '%Y-%m-%d' \
-  example-db
-```
-
-## Create a table with a custom partition template
-
-The following example creates a new `example-table` table in the specified
-database and applies a partition template that partitions by distinct values of
-two tags (`room` and `sensor-type`), bucketed values of the `customerID` tag,
-and by month using the time format `%Y-%m`:
-
-<!--Skip database create and delete tests: namespaces aren't reusable-->
-<!--pytest.mark.skip-->
-
-{{% code-placeholders "DATABASE_NAME" %}}
-
-```sh
-influxctl table create \
-  --template-tag room \
-  --template-tag sensor-type \
-  --template-tag-bucket customerID,500 \
-  --template-timeformat '%Y-%m' \
-  DATABASE_NAME \
-  example-table
-```
-
-{{% /code-placeholders %}}
-
-Replace the following in your command:
-
-- {{% code-placeholder-key %}}`DATABASE_NAME`{{% /code-placeholder-key %}}: your {{% product-name %}} [database](/influxdb/cloud-dedicated/admin/databases/)
-
-<!--actual test
-
-```sh
-
-# Test the preceding command outside of the code block.
-# influxctl authentication requires TTY interaction--
-# output the auth URL to a file that the host can open.
-
-TABLE_NAME=table_TEST_RUN
-script -c "influxctl table create \
-  --template-tag room \
-  --template-tag sensor-type \
-  --template-tag-bucket customerID,500 \
-  --template-timeformat '%Y-%m' \
-  DATABASE_NAME \
-  $TABLE_NAME" \
- /dev/null > /shared/urls.txt
-
-script -c "influxctl query \
- --database DATABASE_NAME \
- --token DATABASE_TOKEN \
- 'SHOW TABLES'" > /shared/temp_tables.txt
-grep -q $TABLE_NAME /shared/temp_tables.txt
-rm /shared/temp_tables.txt
-```
-
+<!-- 
+The content of this page is at /content/shared/v3-distributed-admin-custom-partitions/_define-custom-partitions.md
 -->
-
-## Example partition templates
-
-Given the following [line protocol](/influxdb/cloud-dedicated/reference/syntax/line-protocol/)
-with a `2024-01-01T00:00:00Z` timestamp:
-
-```text
-prod,line=A,station=weld1 temp=81.9,qty=36i 1704067200000000000
-```
-
-##### Partitioning by distinct tag values
-
-| Description             | Tag parts         | Time part  | Resulting partition key  |
-| :---------------------- | :---------------- | :--------- | :----------------------- |
-| By day (default)        |                   | `%Y-%m-%d` | 2024-01-01               |
-| By month                |                   | `%Y-%m`    | 2024-01                  |
-| By year                 |                   | `%Y`       | 2024                     |
-| Single tag, by day      | `line`            | `%Y-%m-%d` | A \| 2024-01-01          |
-| Single tag, by month    | `line`            | `%Y-%m`    | A \| 2024-01             |
-| Single tag, by year     | `line`            | `%Y`       | A \| 2024                |
-| Multiple tags, by day   | `line`, `station` | `%Y-%m-%d` | A \| weld1 \| 2024-01-01 |
-| Multiple tags, by month | `line`, `station` | `%Y-%m`    | A \| weld1 \| 2024-01    |
-| Multiple tags, by year  | `line`, `station` | `%Y`       | A \| weld1 \| 2024       |
-
-##### Partition by tag buckets
-
-| Description                         | Tag part | Tag bucket part | Time part  | Resulting partition key |
-| :---------------------------------- | :------- | :-------------- | :--------- | :---------------------- |
-| Distinct tag, tag buckets, by day   | `line`   | `station,100`   | `%Y-%m-%d` | A \| 3 \| 2024-01-01    |
-| Distinct tag, tag buckets, by month | `line`   | `station,500`   | `%Y-%m`    | A \| 303 \| 2024-01     |

content/influxdb/cloud-dedicated/admin/custom-partitions/define-custom-partitions.md

@@ -8,124 +8,9 @@ menu:
   influxdb_cloud_dedicated:
     parent: Manage data partitioning
 weight: 202
+source: /shared/v3-distributed-admin-custom-partitions/partition-templates.md
 ---
 
-Use partition templates to define the patterns used to generate partition keys.
-A partition key uniquely identifies a partition and is used to name the partition
-Parquet file in the [Object store](/influxdb/cloud-dedicated/reference/internals/storage-engine/#object-store).
-
-A partition template consists of 1-8 _template parts_---dimensions to partition data by.
-Three types of template parts exist:
-
-- **tag**: An [InfluxDB tag](/influxdb/cloud-dedicated/reference/glossary/#tag)
-  to partition by.
-- **tag bucket**: An [InfluxDB tag](/influxdb/cloud-dedicated/reference/glossary/#tag)
-  and number of "buckets" to group tag values into. Data is partitioned by the
-  tag bucket rather than each distinct tag value.
-- {{< req type="key" >}} **time**: A Rust strftime date and time string that specifies the time interval
-  to partition data by. The smallest unit of time included in the time part
-  template is the interval used to partition data.
-
-{{% note %}}
-A partition template must include 1 [time part](#time-part-templates)
-and can include up to 7 total [tag](#tag-part-templates) and [tag bucket](#tag-bucket-part-templates) parts.
-{{% /note %}}
-
-<!-- TOC -->
-- [Restrictions](#restrictions)
-  - [Template part size limit](#template-part-size-limit)
-  - [Reserved keywords](#reserved-keywords)
-  - [Reserved Characters](#reserved-characters)
-- [Tag part templates](#tag-part-templates)
-- [Tag bucket part templates](#tag-bucket-part-templates)
-- [Time part templates](#time-part-templates)
-<!-- /TOC -->
-
-## Restrictions
-
-### Template part size limit
-
-Each template part is limited to 200 bytes in length.
-Anything longer will be truncated at 200 bytes and appended with `#`.
-
-### Partition key size limit
-
-With the truncation of template parts, the maximum length of a partition key is
-1,607 bytes (1.57 KiB).
-
-### Reserved keywords
-
-The following reserved keywords cannot be used in partition templates:
-
-- `time`
-
-### Reserved Characters
-
-If used in template parts, non-ASCII characters and the following reserved
-characters must be [percent encoded](https://developer.mozilla.org/en-US/docs/Glossary/Percent-encoding):
-
-- `|`: Partition key part delimiter
-- `!`: Null or missing partition key part
-- `^`: Empty string partition key part
-- `#`: Key part truncation marker
-- `%`: Required for unambiguous reversal of percent encoding
-
-## Tag part templates
-
-Tag part templates consist of a _tag key_ to partition by.
-Generated partition keys include the unique _tag value_ specific to each partition.
-
-A partition template may include a given tag key only once in template parts 
-that operate on tags (tag value and tag bucket)--for example:
-
-If a template partitions on unique values of `tag_A`, then
-you can't use `tag_A` as a tag bucket part.
-
-## Tag bucket part templates
-
-Tag bucket part templates consist of a _tag key_ to partition by and the
-_number of "buckets" to partition tag values into_--for example:
-
-```
-customerID,500
-```
-
-Values of the `customerID` tag are bucketed into 500 distinct "buckets." 
-Each bucket is identified by the remainder of the tag value hashed into a 32bit
-integer divided by the specified number of buckets:
-
-```rust
-hash(tagValue) % N
-```
-
-Generated partition keys include the unique _tag bucket identifier_ specific to
-each partition.
-
-**Supported number of tag buckets**: 1-1,000
-
-{{% note %}}
-Tag buckets should be used to partition by high cardinality tags or tags with an
-unknown number of distinct values.
-{{% /note %}}
-
-A partition template may include a given tag key only once in template parts 
-that operate on tags (tag value and tag bucket)--for example:
-
-If a template partitions on unique values of `tag_A`, then
-you can't use `tag_A` as a tag bucket part.
-
-## Time part templates
-
-Time part templates use a limited subset of the
-[Rust strftime date and time formatting syntax](https://docs.rs/chrono/latest/chrono/format/strftime/index.html)
-to specify time format in partition keys.
-InfluxDB uses the smallest unit of time included in the time part template as
-the partition interval.
-
-### Date specifiers
-
-| Variable | Example      | Description                                                                                                                                                                         |
-| :------: | :----------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-|   `%Y`   | `2001`       | The full proleptic Gregorian year, zero-padded to 4 digits. chrono supports years from -262144 to 262143. Note: years before 1 BCE or after 9999 CE, require an initial sign (+/-). |
-|   `%m`   | `07`         | Month number (01--12), zero-padded to 2 digits.                                                                                                                                     |
-|   `%d`   | `08`         | Day number (01--31), zero-padded to 2 digits.                                                                                                                                       |
+<!-- 
+The content of this page is at /content/shared/v3-distributed-admin-custom-partitions/_partition-templates.md
+-->