Skip to content

Tune async insert buffer #284

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Sep 3, 2025
Merged

Tune async insert buffer #284

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 7 additions & 7 deletions cmd/root.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -80,13 +80,13 @@ func init() {
rootCmd.PersistentFlags().String("storage-orchestrator-clickhouse-host", "", "Clickhouse host for orchestrator storage")
rootCmd.PersistentFlags().String("storage-orchestrator-clickhouse-username", "", "Clickhouse username for orchestrator storage")
rootCmd.PersistentFlags().String("storage-orchestrator-clickhouse-password", "", "Clickhouse password for orchestrator storage")
rootCmd.PersistentFlags().Bool("storage-orchestrator-clickhouse-asyncInsert", false, "Clickhouse async insert for orchestrator storage")
rootCmd.PersistentFlags().Bool("storage-orchestrator-clickhouse-asyncInsert", true, "Clickhouse async insert for orchestrator storage")
rootCmd.PersistentFlags().Int("storage-orchestrator-clickhouse-maxRowsPerInsert", 100000, "Clickhouse max rows per insert for orchestrator storage")
rootCmd.PersistentFlags().Int("storage-orchestrator-clickhouse-maxOpenConns", 30, "Clickhouse max open connections for orchestrator storage")
rootCmd.PersistentFlags().Int("storage-orchestrator-clickhouse-maxIdleConns", 30, "Clickhouse max idle connections for orchestrator storage")
rootCmd.PersistentFlags().Bool("storage-orchestrator-clickhouse-disableTLS", false, "Clickhouse disableTLS for orchestrator storage")
rootCmd.PersistentFlags().Bool("storage-orchestrator-clickhouse-enableParallelViewProcessing", false, "Clickhouse enableParallelViewProcessing for orchestrator storage")
rootCmd.PersistentFlags().Bool("storage-orchestrator-clickhouse-enableCompression", false, "Clickhouse enableCompression for orchestrator storage")
rootCmd.PersistentFlags().Bool("storage-orchestrator-clickhouse-enableCompression", true, "Clickhouse enableCompression for orchestrator storage")
rootCmd.PersistentFlags().Int("storage-orchestrator-clickhouse-maxQueryTime", 60, "Clickhouse max query time for orchestrator storage")
rootCmd.PersistentFlags().Int("storage-orchestrator-clickhouse-maxMemoryUsage", 1000000000, "Clickhouse max memory usage in bytes for orchestrator storage")
rootCmd.PersistentFlags().String("storage-orchestrator-postgres-host", "", "PostgreSQL host for orchestrator storage")
Expand All @@ -103,24 +103,24 @@ func init() {
rootCmd.PersistentFlags().String("storage-main-clickhouse-host", "", "Clickhouse host for main storage")
rootCmd.PersistentFlags().String("storage-main-clickhouse-username", "", "Clickhouse username for main storage")
rootCmd.PersistentFlags().String("storage-main-clickhouse-password", "", "Clickhouse password for main storage")
rootCmd.PersistentFlags().Bool("storage-main-clickhouse-asyncInsert", false, "Clickhouse async insert for main storage")
rootCmd.PersistentFlags().Bool("storage-main-clickhouse-asyncInsert", true, "Clickhouse async insert for main storage")
rootCmd.PersistentFlags().Int("storage-main-clickhouse-maxRowsPerInsert", 100000, "Clickhouse max rows per insert for main storage")
rootCmd.PersistentFlags().Int("storage-main-clickhouse-maxOpenConns", 30, "Clickhouse max open connections for main storage")
rootCmd.PersistentFlags().Int("storage-main-clickhouse-maxIdleConns", 30, "Clickhouse max idle connections for main storage")
rootCmd.PersistentFlags().Bool("storage-main-clickhouse-disableTLS", false, "Clickhouse disableTLS for main storage")
rootCmd.PersistentFlags().Bool("storage-main-clickhouse-enableParallelViewProcessing", false, "Clickhouse enableParallelViewProcessing for main storage")
rootCmd.PersistentFlags().Bool("storage-main-clickhouse-enableCompression", false, "Clickhouse enableCompression for main storage")
rootCmd.PersistentFlags().Bool("storage-main-clickhouse-enableCompression", true, "Clickhouse enableCompression for main storage")
rootCmd.PersistentFlags().Int("storage-main-clickhouse-maxQueryTime", 60, "Clickhouse max query time for main storage")
rootCmd.PersistentFlags().Int("storage-main-clickhouse-maxMemoryUsage", 1000000000, "Clickhouse max memory usage in bytes for main storage")
rootCmd.PersistentFlags().String("storage-staging-clickhouse-username", "", "Clickhouse username for staging storage")
rootCmd.PersistentFlags().String("storage-staging-clickhouse-password", "", "Clickhouse password for staging storage")
rootCmd.PersistentFlags().Bool("storage-staging-clickhouse-asyncInsert", false, "Clickhouse async insert for staging storage")
rootCmd.PersistentFlags().Bool("storage-staging-clickhouse-asyncInsert", true, "Clickhouse async insert for staging storage")
rootCmd.PersistentFlags().Int("storage-staging-clickhouse-maxRowsPerInsert", 100000, "Clickhouse max rows per insert for staging storage")
rootCmd.PersistentFlags().Int("storage-staging-clickhouse-maxOpenConns", 30, "Clickhouse max open connections for staging storage")
rootCmd.PersistentFlags().Int("storage-staging-clickhouse-maxIdleConns", 30, "Clickhouse max idle connections for staging storage")
rootCmd.PersistentFlags().Bool("storage-staging-clickhouse-disableTLS", false, "Clickhouse disableTLS for staging storage")
rootCmd.PersistentFlags().Bool("storage-staging-clickhouse-enableParallelViewProcessing", false, "Clickhouse enableParallelViewProcessing for staging storage")
rootCmd.PersistentFlags().Bool("storage-staging-clickhouse-enableCompression", false, "Clickhouse enableCompression for staging storage")
rootCmd.PersistentFlags().Bool("storage-staging-clickhouse-enableCompression", true, "Clickhouse enableCompression for staging storage")
rootCmd.PersistentFlags().Int("storage-staging-clickhouse-maxQueryTime", 60, "Clickhouse max query time for staging storage")
rootCmd.PersistentFlags().Int("storage-staging-clickhouse-maxMemoryUsage", 1000000000, "Clickhouse max memory usage in bytes for staging storage")
rootCmd.PersistentFlags().String("storage-staging-postgres-host", "", "PostgreSQL host for staging storage")
Expand Down Expand Up @@ -198,7 +198,7 @@ func init() {
rootCmd.PersistentFlags().String("migrator-destination-clickhouse-password", "", "Clickhouse password for migrator destination")
rootCmd.PersistentFlags().String("migrator-destination-clickhouse-database", "", "Clickhouse database for migrator destination")
rootCmd.PersistentFlags().Bool("migrator-destination-clickhouse-disableTLS", false, "Clickhouse disableTLS for migrator destination")
rootCmd.PersistentFlags().Bool("migrator-destination-clickhouse-asyncInsert", false, "Clickhouse async insert for migrator destination")
rootCmd.PersistentFlags().Bool("migrator-destination-clickhouse-asyncInsert", true, "Clickhouse async insert for migrator destination")
rootCmd.PersistentFlags().Int("migrator-destination-clickhouse-maxRowsPerInsert", 100000, "Clickhouse max rows per insert for migrator destination")
rootCmd.PersistentFlags().Int("migrator-destination-clickhouse-maxOpenConns", 30, "Clickhouse max open connections for migrator destination")
rootCmd.PersistentFlags().Int("migrator-destination-clickhouse-maxIdleConns", 30, "Clickhouse max idle connections for migrator destination")
Expand Down
3 changes: 3 additions & 0 deletions internal/storage/clickhouse.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -127,6 +127,9 @@ func connectDB(cfg *config.ClickhouseConfig) (clickhouse.Conn, error) {
if cfg.AsyncInsert {
settings["async_insert"] = "1"
settings["wait_for_async_insert"] = "1"
settings["async_insert_busy_timeout_ms"] = "3000" // 3 seconds
settings["async_insert_max_data_size"] = "104857600" // 100MB
settings["async_insert_max_query_number"] = "100000" // 100k
}
Comment on lines +130 to 133
Copy link

@coderabbitai coderabbitai bot Sep 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Verification agent

🧩 Analysis chain

Double-check interaction with Native batch inserts

This connector predominantly uses PrepareBatch (Native protocol). Please verify your ClickHouse version and driver behavior to ensure async_insert settings apply as intended to your insert path; if not, these settings won’t influence batch.Send() semantics. If needed, document the expected behavior or gate settings by mode.

You can confirm server-side support and effective values with:

-- Run on the target cluster as the same user this app uses
SELECT name, value, changed, origin
FROM system.settings
WHERE name IN (
  'async_insert','wait_for_async_insert',
  'async_insert_busy_timeout_max_ms','async_insert_max_data_size','async_insert_max_query_number'
);

Also check that your ClickHouse version documents these settings:

🌐 Web query:

Verify whether ClickHouse async_insert settings (busy_timeout_max_ms, max_data_size, max_query_number) apply to Native protocol inserts, and whether busy_timeout_ms is aliased to busy_timeout_max_ms.

💡 Result:

Short answers:

  • The async_insert settings (async_insert_busy_timeout_max_ms, async_insert_max_data_size, async_insert_max_query_number) control ClickHouse's asynchronous-insert buffering/flush behavior — they apply to async INSERT processing, not to every insert mode. [1][2]

  • Native/binary-protocol inserts do not universally use the async_insert path: many ClickHouse clients/SDKs only support async_insert when sending SQL/text INSERTs (the native Insert API in some clients bypasses async buffering). See the clickhouse-cpp notes that async inserts work for SQL/text but the native Block Insert path does not use it. [2][4]

  • async_insert_busy_timeout_ms is an alias for async_insert_busy_timeout_max_ms. [3]

Sources:

  1. ClickHouse settings docs (async_insert, async_insert_max_data_size, async_insert_max_query_number, busy_timeout descriptions). [clickhouse.com docs].
  2. clickhouse-cpp GitHub README/issues explaining async_insert requires SQL/text inserts and native Insert() (binary Block) doesn’t use async insert. [github.com/ClickHouse/clickhouse-cpp].
  3. ClickHouse source / settings history showing async_insert_busy_timeout_ms aliased to async_insert_busy_timeout_max_ms. [fossies / ClickHouse source].

Document or gate async_insert settings for native protocol

Current async_insert settings (async_insert_busy_timeout_ms/async_insert_busy_timeout_max_ms, async_insert_max_data_size, async_insert_max_query_number) apply only to ClickHouse’s async INSERT (SQL/text) path and are ignored by the native binary protocol used by PrepareBatch/batch.Send(). Remove or conditionally apply these settings for the native batch insert path, or clearly document that they have no effect in native mode.

🤖 Prompt for AI Agents 
In internal/storage/clickhouse.go around lines 130 to 133, the async_insert_*
settings are being applied unconditionally but only affect ClickHouse’s SQL/text
async INSERT path and are ignored by the native binary protocol used by
PrepareBatch/batch.Send(); either stop applying these settings when using the
native client or add a clear comment documenting they have no effect in native
mode. Change the logic to detect the client/protocol (native vs SQL/text) and
only set
async_insert_busy_timeout_ms/async_insert_max_data_size/async_insert_max_query_number
for the SQL/text path, otherwise omit them; if detection is not practical, add
an inline comment and documentation near this block stating these settings do
not apply to native batch inserts so maintainers won’t expect them to affect
PrepareBatch/batch.Send().

return settings
}(),
Expand Down