Skip to content

Unhide parquet runtime configs #7032

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Sep 21, 2025
Merged

Unhide parquet runtime configs #7032

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 28 additions & 0 deletions docs/configuration/config-file-reference.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4126,6 +4126,22 @@ The `limits_config` configures default and per-tenant limits imposed by Cortex s
# zones are not available.
[query_partial_data: <boolean> | default = false]

# The maximum number of rows that can be fetched when querying parquet storage.
# Each row maps to a series in a parquet file. This limit applies before
# materializing chunks. 0 to disable.
# CLI flag: -querier.parquet-queryable.max-fetched-row-count
[parquet_max_fetched_row_count: <int> | default = 0]

# The maximum number of bytes that can be used to fetch chunk column pages when
# querying parquet storage. 0 to disable.
# CLI flag: -querier.parquet-queryable.max-fetched-chunk-bytes
[parquet_max_fetched_chunk_bytes: <int> | default = 0]

# The maximum number of bytes that can be used to fetch all column pages when
# querying parquet storage. 0 to disable.
# CLI flag: -querier.parquet-queryable.max-fetched-data-bytes
[parquet_max_fetched_data_bytes: <int> | default = 0]

# Maximum number of outstanding requests per tenant per request queue (either
# query frontend or query scheduler); requests beyond this error with HTTP 429.
# CLI flag: -frontend.max-outstanding-requests-per-tenant
Expand Down Expand Up @@ -4224,6 +4240,18 @@ query_rejection:
# CLI flag: -compactor.partition-series-count
[compactor_partition_series_count: <int> | default = 0]

# If set, enables the Parquet converter to create the parquet files.
# CLI flag: -parquet-converter.enabled
[parquet_converter_enabled: <boolean> | default = false]

# The default tenant's shard size when the shuffle-sharding strategy is used by
# the parquet converter. When this setting is specified in the per-tenant
# overrides, a value of 0 disables shuffle sharding for the tenant. If the value
# is < 1 and > 0 the shard size will be a percentage of the total parquet
# converters.
# CLI flag: -parquet-converter.tenant-shard-size
[parquet_converter_tenant_shard_size: <float> | default = 0]

# S3 server-side encryption type. Required to enable server-side encryption
# overrides for a specific tenant. If not set, the default S3 client settings
# are used.
Expand Down
10 changes: 5 additions & 5 deletions pkg/util/validation/limits.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -189,9 +189,9 @@ type Limits struct {
QueryPartialData bool `yaml:"query_partial_data" json:"query_partial_data" doc:"nocli|description=Enable to allow queries to be evaluated with data from a single zone, if other zones are not available.|default=false"`

// Parquet Queryable enforced limits.
ParquetMaxFetchedRowCount int `yaml:"parquet_max_fetched_row_count" json:"parquet_max_fetched_row_count" doc:"hidden"`
ParquetMaxFetchedChunkBytes int `yaml:"parquet_max_fetched_chunk_bytes" json:"parquet_max_fetched_chunk_bytes" doc:"hidden"`
ParquetMaxFetchedDataBytes int `yaml:"parquet_max_fetched_data_bytes" json:"parquet_max_fetched_data_bytes" doc:"hidden"`
ParquetMaxFetchedRowCount int `yaml:"parquet_max_fetched_row_count" json:"parquet_max_fetched_row_count"`
ParquetMaxFetchedChunkBytes int `yaml:"parquet_max_fetched_chunk_bytes" json:"parquet_max_fetched_chunk_bytes"`
ParquetMaxFetchedDataBytes int `yaml:"parquet_max_fetched_data_bytes" json:"parquet_max_fetched_data_bytes"`

// Query Frontend / Scheduler enforced limits.
MaxOutstandingPerTenant int `yaml:"max_outstanding_requests_per_tenant" json:"max_outstanding_requests_per_tenant"`
Expand Down Expand Up @@ -220,8 +220,8 @@ type Limits struct {
CompactorPartitionSeriesCount int64 `yaml:"compactor_partition_series_count" json:"compactor_partition_series_count"`

// Parquet converter
ParquetConverterEnabled bool `yaml:"parquet_converter_enabled" json:"parquet_converter_enabled" doc:"hidden"`
ParquetConverterTenantShardSize float64 `yaml:"parquet_converter_tenant_shard_size" json:"parquet_converter_tenant_shard_size" doc:"hidden"`
ParquetConverterEnabled bool `yaml:"parquet_converter_enabled" json:"parquet_converter_enabled"`
ParquetConverterTenantShardSize float64 `yaml:"parquet_converter_tenant_shard_size" json:"parquet_converter_tenant_shard_size"`

// This config doesn't have a CLI flag registered here because they're registered in
// their own original config struct.
Expand Down
Loading