Skip to content

fix(monolith): add configurable db, table, and column limits to enterprise #6242

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 7 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 48 additions & 4 deletions content/influxdb3/enterprise/reference/config-options.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,9 @@ influxdb3 serve
- [tls-minimum-versions](#tls-minimum-version)
- [without-auth](#without-auth)
- [disable-authz](#disable-authz)
- [num-database-limit](#num-database-limit)
- [num-table-limit](#num-table-limit)
- [num-total-columns-per-table-limit](#num-total-columns-per-table-limit)
- [AWS](#aws)
- [aws-access-key-id](#aws-access-key-id)
- [aws-secret-access-key](#aws-secret-access-key)
Expand Down Expand Up @@ -302,8 +305,8 @@ The minimum version for TLS.
Valid values are `tls-1.2` or `tls-1.3`.
Default is `tls-1.2`.

| influxdb3 serve option | Environment variable |
| :---------------------- | :----------------------- |
| influxdb3 serve option | Environment variable |
| :---------------------- | :------------------------------ |
| `--tls-minimum-version` | `INFLUXDB3_TLS_MINIMUM_VERSION` |

---
Expand All @@ -313,13 +316,54 @@ Default is `tls-1.2`.
Disables authentication for all server actions (CLI commands and API requests).
The server processes all requests without requiring tokens or authentication.

| influxdb3 serve option | Environment variable |
| :--------------------- | :----------------------------- |
| `--without-auth` | `INFLUXDB3_START_WITHOUT_AUTH` |

---

#### disable-authz

Optionally disable authz by passing in a comma separated list of resources.
Valid values are `health`, `ping`, and `metrics`.

| influxdb3 serve option | Environment variable |
| :--------------------- | :------------------------ |
| `--disable-authz` | `INFLUXDB3_DISABLE_AUTHZ` |

---

#### num-database-limit

Limits the total number of active databases.
Default is {{% influxdb3/limit "database" %}}.

| influxdb3 serve option | Environment variable |
| :--------------------- | :---------------------------------------- |
| `--num-database-limit` | `INFLUXDB3_ENTERPRISE_NUM_DATABASE_LIMIT` |

---

#### num-table-limit

Limits the total number of active tables across all databases.
Default is {{% influxdb3/limit "table" %}}.

| influxdb3 serve option | Environment variable |
| :--------------------- | :------------------------------------- |
| `--num-table-limit` | `INFLUXDB3_ENTERPRISE_NUM_TABLE_LIMIT` |

---

#### num-total-columns-per-table-limit

Limits the total number of columns per table.
Default is {{% influxdb3/limit "column" %}}.

| influxdb3 serve option | Environment variable |
| :------------------------------------ | :------------------------------------------------------- |
| `--num-total-columns-per-table-limit` | `INFLUXDB3_ENTERPRISE_NUM_TOTAL_COLUMNS_PER_TABLE_LIMIT` |

---

### AWS
Expand Down Expand Up @@ -824,7 +868,7 @@ Sets the thread priority for tokio DataFusion runtime workers.

#### datafusion-max-parquet-fanout

When multiple parquet files are required in a sorted way
When multiple Parquet files are required in a sorted way
(deduplication for example), specifies the maximum fanout.

**Default:** `1000`
Expand All @@ -837,7 +881,7 @@ When multiple parquet files are required in a sorted way

#### datafusion-use-cached-parquet-loader

Uses a cached parquet loader when reading parquet files from the object store.
Uses a cached Parquet loader when reading Parquet files from the object store.

| influxdb3 serve option | Environment variable |
| :--------------------------------------- | :----------------------------------------------- |
Expand Down
29 changes: 19 additions & 10 deletions content/shared/influxdb3-admin/databases/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ stored. Each database can contain multiple tables.
> **If coming from InfluxDB v2, InfluxDB Cloud (TSM), or InfluxDB Cloud Serverless**,
> _database_ and _bucket_ are synonymous.

<!--
{{% show-in "enterprise" %}}
## Retention periods

A database **retention period** is the maximum age of data stored in the database.
Expand All @@ -22,10 +22,9 @@ When a point's timestamp is beyond the retention period (relative to now), the
point is marked for deletion and is removed from the database the next time the
retention enforcement service runs.

The _minimum_ retention period for an InfluxDB database is 1 hour.
The _maximum_ retention period is infinite meaning data does not expire and will
never be removed by the retention enforcement service.
-->
The _maximum_ retention period is infinite (`none`) meaning data does not expire
and will never be removed by the retention enforcement service.
{{% /show-in %}}

## Database, table, and column limits

Expand All @@ -40,9 +39,11 @@ never be removed by the retention enforcement service.
**Maximum number of tables across all databases**: {{% influxdb3/limit "table" %}}

{{< product-name >}} limits the number of tables you can have across _all_
databases to {{% influxdb3/limit "table" %}}. There is no specific limit on how
many tables you can have in an individual database, as long as the total across
all databases is below the limit.
databases to {{% influxdb3/limit "table" %}}{{% show-in "enterprise" %}} by default{{% /show-in %}}.
{{% show-in "enterprise" %}}You can configure the table limit using the
[`--num-table-limit` configuration option](/influxdb3/enterprise/reference/config-options/#num-table-limit).{{% /show-in %}}
InfluxDB doesn't limit how many tables you can have in an individual database,
as long as the total across all databases is below the limit.

Having more tables affects your {{% product-name %}} installation in the
following ways:
Expand All @@ -64,14 +65,16 @@ persists data to Parquet files. Each `PUT` request incurs a monetary cost and
increases the operating cost of {{< product-name >}}.

{{% /expand %}}
{{% expand "**More work for the compactor** _(Enterprise only)_ <em style='opacity:.5;font-weight:normal;'>View more info</em>" %}}
{{% show-in "enterprise" %}}
{{% expand "**More work for the compactor** <em style='opacity:.5;font-weight:normal;'>View more info</em>" %}}

To optimize storage over time, InfluxDB 3 Enterprise has a compactor that
routinely compacts Parquet files.
With more tables and Parquet files to compact, the compactor may need to be scaled
to keep up with demand, adding to the operating cost of InfluxDB 3 Enterprise.

{{% /expand %}}
{{% /show-in %}}
{{< /expand-wrapper >}}

### Column limit
Expand All @@ -80,11 +83,17 @@ to keep up with demand, adding to the operating cost of InfluxDB 3 Enterprise.

Each row must include a time column, with the remaining columns representing
tags and fields.
As a result, a table can have one time column and up to {{% influxdb3/limit "column" -1 %}}
As a result,{{% show-in "enterprise" %}} by default,{{% /show-in %}} a table can
have one time column and up to {{% influxdb3/limit "column" -1 %}}
_combined_ field and tag columns.
If you attempt to write to a table and exceed the column limit, the write
request fails and InfluxDB returns an error.

{{% show-in "enterprise" %}}
You can configure the maximum number of columns per
table using the [`num-total-columns-per-table-limit` configuration option](/influxdb3/enterprise/reference/config-options/#num-total-columns-per-table-limit).
{{% /show-in %}}

Higher numbers of columns has the following side-effects:

{{< expand-wrapper >}}
Expand Down
5 changes: 5 additions & 0 deletions content/shared/influxdb3-admin/databases/create.md
Original file line number Diff line number Diff line change
Expand Up @@ -130,7 +130,12 @@ database_name/retention_policy_name

## Database limit

{{% show-in "enterprise" %}}
**Default maximum number of databases**: {{% influxdb3/limit "database" %}}
{{% /show-in %}}
{{% show-in "core" %}}
**Maximum number of databases**: {{% influxdb3/limit "database" %}}
{{% /show-in %}}

_For more information about {{< product-name >}} database, table, and column limits,
see [Database, table, and column limits](/influxdb3/version/admin/databases/#database-table-and-column-limits)._