diff --git a/modules/ROOT/nav.adoc b/modules/ROOT/nav.adoc index d257a17984..95bc056c4f 100644 --- a/modules/ROOT/nav.adoc +++ b/modules/ROOT/nav.adoc @@ -180,7 +180,7 @@ *** xref:manage:topic-recovery.adoc[Topic Recovery] *** xref:manage:whole-cluster-restore.adoc[Whole Cluster Restore] ** xref:manage:iceberg/index.adoc[Iceberg] -*** xref:manage:iceberg/topic-iceberg-integration.adoc[About Iceberg Topics] +*** xref:manage:iceberg/about-iceberg-topics.adoc[About Iceberg Topics] *** xref:manage:iceberg/use-iceberg-catalogs.adoc[Use Iceberg Catalogs] *** xref:manage:iceberg/query-iceberg-topics.adoc[Query Iceberg Topics] *** xref:manage:iceberg/redpanda-topics-iceberg-snowflake-catalog.adoc[Query Iceberg Topics with Snowflake] diff --git a/modules/console/pages/ui/data-transforms.adoc b/modules/console/pages/ui/data-transforms.adoc index 56e5c33c5e..ce160a3d1c 100644 --- a/modules/console/pages/ui/data-transforms.adoc +++ b/modules/console/pages/ui/data-transforms.adoc @@ -1,5 +1,6 @@ -= Manage Data Transforms in Redpanda Console -:description: You can use Redpanda Console to monitor the status and performance metrics of your transform functions. You can also view detailed logs and delete transform functions when they are no longer needed. += Manage Data Transforms in {ui} +:description: Use {ui} to monitor the status and performance metrics of your transform functions. You can also view detailed logs and delete transform functions when they are no longer needed. +// tag::single-source[] {description} @@ -7,18 +8,20 @@ Before you begin, ensure that you have the following: +ifndef::env-cloud[] - Redpanda Console must be xref:console:config/connect-to-redpanda.adoc[connected to a Redpanda cluster]. - Redpanda Console must be xref:console:config/connect-to-redpanda.adoc#admin[configured to connect to the Redpanda Admin API]. +endif::[] - xref:develop:data-transforms/configure.adoc#enable-transforms[Data transforms enabled] in your Redpanda cluster. - At least one transform function deployed to your Redpanda cluster. [[monitor]] == Monitor transform functions -To monitor transform functions in Redpanda Console: +To monitor transform functions: . Navigate to the *Transforms* menu. -. Click on the name of a transform function to view detailed information: +. Click the name of a transform function to view detailed information: - The partitions that the function is running on - The broker (node) ID - Any lag (the amount of pending records on the input topic that have yet to be processed by the transform) @@ -26,18 +29,18 @@ To monitor transform functions in Redpanda Console: [[logs]] == View logs -To view logs for a transform function in Redpanda Console: +To view logs for a transform function: . Navigate to the *Transforms* menu. . Click on the name of a transform function. . Click the *Logs* tab to see the logs. -Redpanda Console displays a limited number of logs for transform functions. To view the full history of logs, use the xref:develop:data-transforms/monitor.adoc#logs[`rpk` command-line tool]. +{ui} displays a limited number of logs for transform functions. To view the full history of logs, use the xref:develop:data-transforms/monitor.adoc#logs[`rpk` command-line tool]. [[delete]] == Delete transform functions -To delete a transform function in Redpanda Console: +To delete a transform function: 1. Navigate to the *Transforms* menu. 2. Find the transform function you want to delete from the list. @@ -50,4 +53,6 @@ Deleting a transform function will remove it from the cluster and stop any furth - xref:develop:data-transforms/how-transforms-work.adoc[] - xref:develop:data-transforms/deploy.adoc[] -- xref:develop:data-transforms/monitor.adoc[] \ No newline at end of file +- xref:develop:data-transforms/monitor.adoc[] + +// end::single-source[] \ No newline at end of file diff --git a/modules/console/pages/ui/edit-topic-configuration.adoc b/modules/console/pages/ui/edit-topic-configuration.adoc index 8216c7bc11..695fe3337d 100644 --- a/modules/console/pages/ui/edit-topic-configuration.adoc +++ b/modules/console/pages/ui/edit-topic-configuration.adoc @@ -1,7 +1,7 @@ -= Edit Topic Configuration in the {ui} += Edit Topic Configuration in {ui} :page-aliases: manage:console/edit-topic-configuration.adoc -// tag::single-source[] :description: Use {ui} to edit the configuration of existing topics in a cluster. +// tag::single-source[] {description} diff --git a/modules/console/pages/ui/programmable-push-filters.adoc b/modules/console/pages/ui/programmable-push-filters.adoc index 7f366dffc6..00381edf3e 100644 --- a/modules/console/pages/ui/programmable-push-filters.adoc +++ b/modules/console/pages/ui/programmable-push-filters.adoc @@ -1,8 +1,8 @@ = Filter Messages with JavaScript in {ui} :page-aliases: console:features/programmable-push-filters.adoc, reference:console/programmable-push-filters.adoc // Do not put page aliases in the single-sourced content -// tag::single-source[] :description: Learn how to filter Kafka records using custom JavaScript code within {ui}. +// tag::single-source[] You can use push-down filters in {ui} to search through large Kafka topics that may contain millions of records. Filters are JavaScript functions executed on the backend, evaluating each record individually. Your function must return a boolean: diff --git a/modules/console/pages/ui/record-deserialization.adoc b/modules/console/pages/ui/record-deserialization.adoc index 57ded2e83f..592cf2f139 100644 --- a/modules/console/pages/ui/record-deserialization.adoc +++ b/modules/console/pages/ui/record-deserialization.adoc @@ -1,7 +1,7 @@ = View Deserialized Messages in {ui} :page-aliases: console:features/record-deserialization.adoc, manage:console/protobuf.adoc, reference:console/record-deserialization.adoc -// tag::single-source[] :description: Learn how {ui} deserializes messages. +// tag::single-source[] In Redpanda, the messages exchanged between producers and consumers contain raw bytes. Schemas work as an agreed-upon format, like a contract, for producers and consumers to serialize and deserialize those messages. If a producer breaks this contract, consumers can fail. diff --git a/modules/console/pages/ui/schema-reg.adoc b/modules/console/pages/ui/schema-reg.adoc index 6ad661f6e0..8a5668efdd 100644 --- a/modules/console/pages/ui/schema-reg.adoc +++ b/modules/console/pages/ui/schema-reg.adoc @@ -1,14 +1,14 @@ = Use Schema Registry in {ui} :page-aliases: manage:schema-reg/schema-reg-ui.adoc :page-categories: Management, Schema Registry -// tag::single-source[] :description: Perform common Schema Registry management operations in the {ui}. +// tag::single-source[] In {ui}, the *Schema Registry* menu lists registered and verified schemas, including their serialization format and versions. Select an individual schema to see which topics it applies to. [NOTE] ==== -The Schema Registry is built into Redpanda, and you can use it with the Schema Registry API or with {ui}. This section describes Schema Registry operations available in {ui}. +The Schema Registry is built into Redpanda, and you can use it with the Schema Registry API or with the UI. This section describes Schema Registry operations available in the UI. ==== ifndef::env-cloud[] diff --git a/modules/develop/pages/data-transforms/build.adoc b/modules/develop/pages/data-transforms/build.adoc index c95b468850..53a5311965 100644 --- a/modules/develop/pages/data-transforms/build.adoc +++ b/modules/develop/pages/data-transforms/build.adoc @@ -1,6 +1,7 @@ = Develop Data Transforms :description: Learn how to initialize a data transforms project and write transform functions in your chosen language. :page-categories: Development, Stream Processing, Data Transforms +// tag::single-source[] {description} @@ -8,7 +9,12 @@ You must have the following development tools installed on your host machine: +ifdef::env-cloud[] +* The xref:manage:rpk/rpk-install.adoc[`rpk` command-line client] installed. +endif::[] +ifndef::env-cloud[] * The xref:get-started:rpk-install.adoc[`rpk` command-line client] installed on your host machine and configured to connect to your Redpanda cluster. +endif::[] * For Golang projects, you must have at least version 1.20 of https://go.dev/doc/install[Go^]. * For Rust projects, you must have the latest stable version of https://rustup.rs/[Rust^]. * For JavaScript and TypeScript projects, you must have the https://nodejs.org/en/download/package-manager[latest long-term-support release of Node.js^]. @@ -46,9 +52,7 @@ For example, if you choose `tinygo-no-goroutines`, the following project files a The `transform.go` file contains a boilerplate transform function. The `transform.yaml` file specifies the configuration settings for the transform function. -See also: - -- xref:develop:data-transforms/configure.adoc[] +See also: xref:develop:data-transforms/configure.adoc[] == Build transform functions @@ -284,7 +288,7 @@ See also: - xref:develop:data-transforms/monitor#logs[View logs for transform functions] - xref:develop:data-transforms/monitor.adoc[Monitor data transforms] - xref:develop:data-transforms/configure.adoc#log[Configure transform logging] -- xref:reference:rpk/rpk-transform/rpk-transform-logs.adoc[] +- xref:reference:rpk/rpk-transform/rpk-transform-logs.adoc[`rpk transform logs` reference] === Avoid state management @@ -458,3 +462,5 @@ xref:develop:data-transforms/configure.adoc[] - xref:develop:data-transforms/how-transforms-work.adoc[] - xref:reference:data-transforms/sdks.adoc[] - xref:reference:rpk/rpk-transform/rpk-transform.adoc[`rpk transform` commands] + +// end::single-source[] \ No newline at end of file diff --git a/modules/develop/pages/data-transforms/configure.adoc b/modules/develop/pages/data-transforms/configure.adoc index 063b7f6edc..984f193412 100644 --- a/modules/develop/pages/data-transforms/configure.adoc +++ b/modules/develop/pages/data-transforms/configure.adoc @@ -1,6 +1,7 @@ = Configure Data Transforms :description: pass:q[Learn how to configure data transforms in Redpanda, including editing the `transform.yaml` file, environment variables, and memory settings. This topic covers both the configuration of transform functions and the WebAssembly (Wasm) engine's environment.] :page-categories: Development, Stream Processing, Data Transforms +// tag::single-source[] {description} @@ -41,7 +42,7 @@ env: You can set the name of the transform function, environment variables, and input and output topics on the command-line when you deploy the transform. These command-line settings take precedence over those specified in the `transform.yaml` file. -See xref:develop:data-transforms/deploy.adoc[]. +See xref:develop:data-transforms/deploy.adoc[] [[built-in]] === Built-In environment variables @@ -62,19 +63,20 @@ This section covers how to configure the Wasm engine environment using Redpanda To use data transforms, you must enable it for a Redpanda cluster using the xref:reference:properties/cluster-properties.adoc#data_transforms_enabled[`data_transforms_enabled`] property. -[[resources]] +ifndef::env-cloud[] === Configure memory resources for data transforms Redpanda reserves memory for each transform function within the broker. You need enough memory for your input record and output record to be in memory at the same time. -Set the following properties based on the number of functions you have and the amount of memory you anticipate needing. +Set the following based on the number of functions you have and the amount of memory you anticipate needing. - xref:reference:properties/cluster-properties.adoc#data_transforms_per_core_memory_reservation[`data_transforms_per_core_memory_reservation`]: Increase this setting if you plan to deploy a large number of data transforms or if your transforms are memory-intensive. Reducing it may limit the number of concurrent transforms. - - xref:reference:properties/cluster-properties.adoc#data_transforms_per_function_memory_limit[`data_transforms_per_function_memory_limit`]: Adjust this setting if individual transform functions require more memory to process records efficiently. Reducing it may cause memory errors in complex transforms. The maximum number of functions that can be deployed to a cluster is equal to `data_transforms_per_core_memory_reservation` / `data_transforms_per_function_memory_limit`. When that limit is hit, Redpanda cannot allocate memory for the VM and the transforms stay in `errored` states. +endif::[] +ifndef::env-cloud[] [[binary-size]] === Configure maximum binary size @@ -88,25 +90,31 @@ Increase this setting if your Wasm binaries are larger than the default limit. S You can set the interval at which data transforms commit their progress using the xref:reference:properties/cluster-properties.adoc#data_transforms_commit_interval_ms[`data_transforms_commit_interval_ms`] property. Adjust this setting to control how frequently the transform function's progress is committed. Shorter intervals may provide more frequent progress updates but can increase load. Longer intervals reduce load but may delay progress updates. +endif::[] [[log]] === Configure transform logging +The following properties configure logging for data transforms: -Redpanda provides several properties to configure logging for data transforms: - +ifndef::env-cloud[] - xref:reference:properties/cluster-properties.adoc#data_transforms_logging_buffer_capacity_bytes[`data_transforms_logging_buffer_capacity_bytes`]: Increase this value if your transform logs are large or if you need to buffer more log data before flushing. Reducing this value may cause more frequent log flushing. - xref:reference:properties/cluster-properties.adoc#data_transforms_logging_flush_interval_ms[`data_transforms_logging_flush_interval_ms`]: Adjust this value to control how frequently logs are flushed to the `transform_logs` topic. Shorter intervals provide more frequent log updates but can increase load. Longer intervals reduce load but may delay log updates. +endif::[] - xref:reference:properties/cluster-properties.adoc#data_transforms_logging_line_max_bytes[`data_transforms_logging_line_max_bytes`]: Increase this value if your log messages are frequently truncated. Setting this value too low may truncate important log information. +ifndef::env-cloud[] [[runtime-limit]] === Configure runtime limits You can set the maximum runtime for starting up a data transform and the time it takes for a single record to be transformed using the xref:reference:properties/cluster-properties.adoc#data_transforms_runtime_limit_ms[`data_transforms_runtime_limit_ms`] property. Adjust this value only if your transform functions need more time to process each record or to start up. +endif::[] == Next steps -xref:develop:data-transforms/deploy.adoc[]. +xref:develop:data-transforms/deploy.adoc[] + +// end::single-source[] \ No newline at end of file diff --git a/modules/develop/pages/data-transforms/deploy.adoc b/modules/develop/pages/data-transforms/deploy.adoc index 5fc358cc10..82e5332a20 100644 --- a/modules/develop/pages/data-transforms/deploy.adoc +++ b/modules/develop/pages/data-transforms/deploy.adoc @@ -1,6 +1,7 @@ = Deploy Data Transforms :description: Learn how to build, deploy, share, and troubleshoot data transforms in Redpanda. :page-categories: Development, Stream Processing, Data Transforms +// tag::single-source[] {description} @@ -10,7 +11,12 @@ Before you begin, ensure that you have the following: - xref:develop:data-transforms/configure.adoc#enable-transforms[Data transforms enabled] in your Redpanda cluster. +ifndef::env-cloud[] - The xref:get-started:rpk-install.adoc[`rpk` command-line client] installed on your host machine and configured to connect to your Redpanda cluster. +endif::[] +ifdef::env-cloud[] +- The xref:manage:rpk/rpk-install.adoc[`rpk` command-line client]. +endif::[] - A xref:develop:data-transforms/build.adoc[data transform] project. [[build]] @@ -120,7 +126,14 @@ rpk transform delete For more details about this command, see xref:reference:rpk/rpk-transform/rpk-transform-delete.adoc[]. +ifndef::env-cloud[] TIP: You can also xref:console:ui/data-transforms.adoc#delete[delete transform functions in Redpanda Console]. +endif::[] + +ifdef::env-cloud[] +TIP: You can also delete transform functions in Redpanda Cloud. +endif::[] + == Troubleshoot @@ -144,5 +157,6 @@ Invalid WebAssembly - the binary is missing required transform functions. Check All transform functions must register a callback with the `OnRecordWritten()` method. For more details, see xref:develop:data-transforms/build.adoc[]. == Next steps +xref:develop:data-transforms/monitor.adoc[Set up monitoring] for data transforms. -xref:develop:data-transforms/monitor.adoc[Set up monitoring] for data transforms. \ No newline at end of file +// end::single-source[] \ No newline at end of file diff --git a/modules/develop/pages/data-transforms/how-transforms-work.adoc b/modules/develop/pages/data-transforms/how-transforms-work.adoc index 7f35a46096..0bb1f79f21 100644 --- a/modules/develop/pages/data-transforms/how-transforms-work.adoc +++ b/modules/develop/pages/data-transforms/how-transforms-work.adoc @@ -1,7 +1,7 @@ = How Data Transforms Work :page-categories: Development, Stream Processing, Data Transforms -include::develop:partial$data-transforms-ga-notice.adoc[] :description: Learn how Redpanda data transforms work. +// tag::single-source[] Redpanda provides the framework to build and deploy inline transformations (data transforms) on data written to Redpanda topics, delivering processed and validated data to consumers in the format they expect. Redpanda does this directly inside the broker, eliminating the need to manage a separate stream processing environment or use third-party tools. @@ -22,7 +22,9 @@ To execute a transform function, Redpanda uses just-in-time (JIT) compilation to When you deploy a data transform to a Redpanda broker, it stores the Wasm bytecode and associated metadata, such as input and output topics and environment variables. The broker then replicates this data across the cluster using internal Kafka topics. When the data is distributed, each shard runs its own instance of the transform function. This process includes several resource management features: - Each shard can run only one instance of the transform function at a time to ensure efficient resource utilization and prevent overload. +ifndef::env-cloud[] - Memory for each function is reserved within the broker with the `data_transforms_per_core_memory_reservation` and `data_transforms_per_function_memory_limit` properties. See xref:develop:data-transforms/configure.adoc#resources[Configure memory for data transforms]. +endif::[] - CPU time is dynamically allocated to the Wasm runtime to ensure that the code does not run forever and cannot block the broker from handling traffic or doing other work, such as Tiered Storage uploads. == Flow of data transforms @@ -74,6 +76,8 @@ This section outlines the limitations of data transforms. These constraints are == Suggested reading -- xref:reference:data-transform-golang-sdk.adoc[] -- xref:reference:data-transform-rust-sdk.adoc[] +- xref:reference:data-transforms/golang-sdk.adoc[] +- xref:reference:data-transforms/rust-sdk.adoc[] - xref:reference:rpk/rpk-transform/rpk-transform.adoc[`rpk transform` commands] + +// end::single-source[] \ No newline at end of file diff --git a/modules/develop/pages/data-transforms/monitor.adoc b/modules/develop/pages/data-transforms/monitor.adoc index b1bd07f4ed..afd30a6b03 100644 --- a/modules/develop/pages/data-transforms/monitor.adoc +++ b/modules/develop/pages/data-transforms/monitor.adoc @@ -1,12 +1,19 @@ = Monitor Data Transforms :description: This topic provides guidelines on how to monitor the health of your data transforms and view logs. :page-categories: Development, Stream Processing, Data Transforms +// tag::single-source[] {description} == Prerequisites -xref:manage:monitoring.adoc[Set up monitoring] for your Redpanda cluster. +ifndef::env-cloud[] +xref:manage:monitoring.adoc[Set up monitoring] for your cluster. +endif::[] + +ifdef::env-cloud[] +xref:manage:monitor-cloud.adoc[Set up monitoring] for your cluster. +endif::[] == Performance @@ -39,7 +46,9 @@ If memory usage is consistently high or exceeds the maximum allocated memory: - Review and optimize your transform functions to reduce memory consumption. This step can involve optimizing data structures, reducing memory allocations, and ensuring efficient handling of records. +ifndef::env-cloud[] - Consider increasing the allocated memory for the Wasm engine. Adjust the xref:develop:data-transforms/configure.adoc#resources[`data_transforms_per_core_memory_reservation`] and xref:develop:data-transforms/configure.adoc#resources[`data_transforms_per_function_memory_limit settings`] to provide more memory to each function and the overall Wasm engine. +endif::[] == Throughput @@ -62,11 +71,18 @@ rpk transform logs Replace `` with the xref:develop:data-transforms/configure.adoc[configured name] of the transform function. -TIP: You can also xref:console:ui/data-transforms.adoc#logs[view logs in Redpanda Console]. +ifndef::env-cloud[] +TIP: You can also xref:console:ui/data-transforms.adoc#logs[view logs in {ui}]. +endif::[] + +ifdef::env-cloud[] +TIP: You can also view logs in the UI. +endif::[] By default, Redpanda provides several settings to manage logging for data transforms, such as buffer capacity, flush interval, and maximum log line length. These settings ensure that logging operates efficiently without overwhelming the system. However, you may need to adjust these settings based on your specific requirements and workloads. For information on how to configure logging, see the xref:develop:data-transforms/configure.adoc#log[Configure transform logging] section of the configuration guide. == Suggested reading - xref:reference:public-metrics-reference.adoc#data_transform_metrics[Data transforms metrics] -- xref:console:ui/data-transforms.adoc[] + +// end::single-source[] diff --git a/modules/develop/pages/data-transforms/run-transforms-index.adoc b/modules/develop/pages/data-transforms/run-transforms-index.adoc index e19100c752..19c8f12578 100644 --- a/modules/develop/pages/data-transforms/run-transforms-index.adoc +++ b/modules/develop/pages/data-transforms/run-transforms-index.adoc @@ -2,4 +2,7 @@ :description: Choose your deployment environment to get started with building and deploying your first transform function in Redpanda. :page-aliases: reference:rpk/rpk-wasm/rpk-wasm.adoc, reference:rpk/rpk-wasm.adoc, reference:rpk/rpk-wasm/rpk-wasm-deploy.adoc, reference:rpk/rpk-wasm/rpk-wasm-generate.adoc, reference:rpk/rpk-wasm/rpk-wasm-remove.adoc, data-management:data-transform.adoc, labs:data-transform/index.adoc :page-layout: index -:page-categories: Development, Stream Processing, Data Transforms \ No newline at end of file +:page-categories: Development, Stream Processing, Data Transforms +// tag::single-source[] + +// end::single-source[] \ No newline at end of file diff --git a/modules/develop/pages/data-transforms/run-transforms.adoc b/modules/develop/pages/data-transforms/run-transforms.adoc index 7909456592..d9f3ca346a 100644 --- a/modules/develop/pages/data-transforms/run-transforms.adoc +++ b/modules/develop/pages/data-transforms/run-transforms.adoc @@ -2,7 +2,8 @@ :description: Learn how to build and deploy your first transform function in Linux deployments. :page-context-links: [{"name": "Linux", "to": "develop:data-transforms/run-transforms.adoc" },{"name": "Kubernetes", "to": "develop:data-transforms/k-run-transforms.adoc" } ] :page-categories: Development, Stream Processing, Data Transforms +// tag::single-source[] -include::develop:partial$data-transforms-ga-notice.adoc[] +include::develop:partial$run-transforms.adoc[] -include::develop:partial$run-transforms.adoc[] \ No newline at end of file +// end::single-source[] \ No newline at end of file diff --git a/modules/develop/pages/data-transforms/test.adoc b/modules/develop/pages/data-transforms/test.adoc index 8be044a3c9..6bad5185b6 100644 --- a/modules/develop/pages/data-transforms/test.adoc +++ b/modules/develop/pages/data-transforms/test.adoc @@ -1,5 +1,6 @@ = Write Integration Tests for Transform Functions :description: pass:q[Learn how to write integration tests for data transform functions in Redpanda, including setting up unit tests and using testcontainers for integration tests.] +// tag::single-source[] {description} @@ -80,4 +81,6 @@ This will execute all tests in the current directory. Integration tests verify that your transform functions work correctly in a real Redpanda environment. You can use https://github.com/testcontainers/testcontainers-go/tree/main[testcontainers] to set up and manage a Redpanda instance for testing. -For more detailed examples and helper code for setting up integration tests, refer to the SDK integration tests on https://github.com/redpanda-data/redpanda/tree/dev/src/transform-sdk/tests[GitHub]. \ No newline at end of file +For more detailed examples and helper code for setting up integration tests, refer to the SDK integration tests on https://github.com/redpanda-data/redpanda/tree/dev/src/transform-sdk/tests[GitHub]. + +// end::single-source[] \ No newline at end of file diff --git a/modules/develop/pages/produce-data/configure-producers.adoc b/modules/develop/pages/produce-data/configure-producers.adoc index a0d138f2bb..6c954b6564 100644 --- a/modules/develop/pages/produce-data/configure-producers.adoc +++ b/modules/develop/pages/produce-data/configure-producers.adoc @@ -21,7 +21,9 @@ The `acks` property sets the number of acknowledgments the producer requires the Redpanda guarantees data safety with fsync, which means flushing to disk. * With `acks=all`, every write is fsynced by default. +ifndef::env-cloud[] * With other `acks` settings, or with `write_caching_default=true` at the cluster level, Redpanda fsyncs to disk according to `raft_replica_max_pending_flush_bytes` and `raft_replica_max_flush_delay_ms`, whichever is reached first. +endif::[] * With `write.caching` enabled at the topic level, Redpanda fsyncs to disk according to `flush.ms` and `flush.bytes`, whichever is reached first. === `acks=0` @@ -66,6 +68,7 @@ to the majority of the brokers responsible for the partition in the cluster. As soon as the fsync call is complete, the message is considered acknowledged and is made visible to readers. +ifndef::env-cloud[] NOTE: This property has an important distinction compared to Kafka's behavior. In Kafka, a message is considered acknowledged without the requirement that it has been fsynced. Messages that have not been fsynced to disk may be lost in the @@ -73,6 +76,18 @@ event of a broker crash. So when using `acks=all`, the Redpanda default configuration is more resilient than Kafka's. You can also consider using xref:develop:config-topics.adoc#configure-write-caching[write caching], which is a relaxed mode of `acks=all` that acknowledges a message as soon as it is received and acknowledged on a majority of brokers, without waiting for it to fsync to disk. This provides lower latency while still ensuring that a majority of brokers acknowledge the write. +endif::[] + +ifdef::env-cloud[] +NOTE: This property has an important distinction compared to Kafka's behavior. In +Kafka, a message is considered acknowledged without the requirement that it has +been fsynced. Messages that have not been fsynced to disk may be lost in the +event of a broker crash. So when using `acks=all`, the Redpanda default +configuration is more resilient than Kafka's. You can also consider +using write caching, which is a relaxed mode of `acks=all` that acknowledges a message as soon as it is received and acknowledged on a majority of brokers, without waiting for it to fsync to disk. This provides lower latency while still ensuring that a majority of brokers acknowledge the write. + +endif::[] + === `retries` This property controls the number of times a message is re-sent to the broker diff --git a/modules/develop/partials/data-transforms-ga-notice.adoc b/modules/develop/partials/data-transforms-ga-notice.adoc deleted file mode 100644 index a5b0ce3c05..0000000000 --- a/modules/develop/partials/data-transforms-ga-notice.adoc +++ /dev/null @@ -1 +0,0 @@ -TIP: Data transforms is generally available for all Redpanda Community and Redpanda Enterprise Edition users. To unlock this feature in Redpanda Cloud, contact https://support.redpanda.com/hc/en-us/requests/new[Redpanda support^]. \ No newline at end of file diff --git a/modules/develop/partials/run-transforms.adoc b/modules/develop/partials/run-transforms.adoc index f0b594155d..15f52d82d0 100644 --- a/modules/develop/partials/run-transforms.adoc +++ b/modules/develop/partials/run-transforms.adoc @@ -1,28 +1,38 @@ +// tag::single-source[] + Data transforms let you run common data streaming tasks, like filtering, scrubbing, and transcoding, within Redpanda. For example, you may have consumers that require you to redact credit card numbers or convert JSON to Avro. Data transforms can also interact with the Redpanda Schema Registry to work with encoded data types. Data transforms use a WebAssembly (Wasm) engine inside a Redpanda broker. A Wasm function acts on a single record in an input topic. You can develop and manage data transforms with xref:reference:rpk/rpk-transform/rpk-transform.adoc[`rpk transform`] commands. NOTE: You should build and deploy transforms from a separate, non-production machine (host machine). Using a separate host machine avoids potential resource conflicts and stability issues on the nodes that run your brokers. -See also: xref:develop:data-transforms/how-transforms-work.adoc[]. +See also: xref:develop:data-transforms/how-transforms-work.adoc[] == Prerequisites You must have the following: +ifndef::env-cloud[] - xref:deploy:deployment-option/self-hosted/index.adoc[A Redpanda cluster] running at least version {page-component-version}. - External access to the Kafka API and the Admin API. +endif::[] ifdef::env-kubernetes[] + Ensure that your Redpanda cluster has xref:manage:kubernetes/networking/external/index.adoc[external access] enabled and is accessible from your host machine using the advertised addresses. + TIP: For a tutorial on setting up a Redpanda cluster with external access, see xref:deploy:deployment-option/self-hosted/kubernetes/get-started-dev.adoc[]. endif::[] + - Development tools installed on your host machine: -** For Golang, you must have at least version 1.20 of https://go.dev/doc/install[Go^]. -** For Rust, you must have the latest stable version of https://rustup.rs/[Rust]. + * For Golang, you must have at least version 1.20 of https://go.dev/doc/install[Go^]. + * For Rust, you must have the latest stable version of https://rustup.rs/[Rust]. +ifndef::env-cloud[] - The xref:get-started:rpk-install.adoc[`rpk` command-line client] installed on your host machine and configured to connect to your Redpanda cluster. -** For JavaScript and TypeScript projects, you must have the https://nodejs.org/en/download/package-manager[latest long-term-support release of Node.js]. +endif::[] +ifdef::env-cloud[] +- The xref:manage:rpk/rpk-install.adoc[`rpk` command-line client] installed on your host machine and configured to connect to your Redpanda cluster. +endif::[] + * For JavaScript and TypeScript projects, you must have the https://nodejs.org/en/download/package-manager[latest long-term-support release of Node.js]. ifdef::env-kubernetes[] + You can use a xref:manage:kubernetes/networking/k-connect-to-redpanda.adoc#rpk-profile[pre-configured `rpk` profile]: @@ -313,8 +323,12 @@ rpk transform deploy --input-topic=input-topic --output-topic=output-topic ```bash echo "hello\nworld" | rpk topic produce input-topic ``` - +ifdef::env-cloud[] +. In Redpanda Cloud, check the records in both the input topic and the output topic. They should be the same. +endif::[] +ifndef::env-cloud[] . http://localhost:8080/topics[Open Redpanda Console] and check the records in both the input topic and the output topic. They should be the same. +endif::[] + You can also verify the content of the output topic in the command-line: + @@ -523,8 +537,12 @@ rpk transform deploy --input-topic=input-topic --output-topic=output-topic ```bash echo "apples,10\npears,11\noranges,5" | rpk topic produce input-topic -k market-stock ``` - +ifdef::env-cloud[] +. In Redpanda Cloud, check the records in both the input topic and the output topic. You should see the following values: +endif::[] +ifndef::env-cloud[] . http://localhost:8080/topics[Open Redpanda Console] and check the records in both the input topic and the output topic. You should see the following values: +endif::[] + [source,json,role="no-copy"] ---- @@ -682,3 +700,5 @@ rpk transform delete data-transforms-tutorial --no-confirm - xref:reference:data-transforms/golang-sdk.adoc[] - xref:reference:data-transforms/rust-sdk.adoc[] - xref:reference:rpk/rpk-transform/rpk-transform.adoc[`rpk transform` commands] + +// end::single-source[] \ No newline at end of file diff --git a/modules/get-started/pages/release-notes/redpanda.adoc b/modules/get-started/pages/release-notes/redpanda.adoc index f604172960..99613fafde 100644 --- a/modules/get-started/pages/release-notes/redpanda.adoc +++ b/modules/get-started/pages/release-notes/redpanda.adoc @@ -62,7 +62,7 @@ The admin panel has been removed from the Redpanda Console UI. To manage users, == Iceberg improvements -xref:manage:iceberg/topic-iceberg-integration.adoc[Iceberg-enabled topics] now support custom partitioning for improved query performance, snapshot expiry, and a dead-letter queue for invalid records. Schema evolution is also supported with schema mutations implemented according to the Iceberg specification. +xref:manage:iceberg/about-iceberg-topics.adoc[Iceberg-enabled topics] now support custom partitioning for improved query performance, snapshot expiry, and a dead-letter queue for invalid records. Schema evolution is also supported with schema mutations implemented according to the Iceberg specification. == Protobuf normalization in Schema Registry diff --git a/modules/manage/pages/audit-logging.adoc b/modules/manage/pages/audit-logging.adoc index e157b90bc4..0b98f49fbb 100644 --- a/modules/manage/pages/audit-logging.adoc +++ b/modules/manage/pages/audit-logging.adoc @@ -3,5 +3,8 @@ :page-context-links: [{"name": "Linux", "to": "manage:audit-logging.adoc" },{"name": "Kubernetes", "to": "manage:kubernetes/security/k-audit-logging.adoc" } ] :page-categories: Management, Security :env-linux: true +// tag::single-source[] -include::manage:partial$audit-logging.adoc[] \ No newline at end of file +include::manage:partial$audit-logging.adoc[] + +// end::single-source[] \ No newline at end of file diff --git a/modules/manage/pages/audit-logging/audit-log-samples.adoc b/modules/manage/pages/audit-logging/audit-log-samples.adoc index d1032a1270..7ba297e6b8 100644 --- a/modules/manage/pages/audit-logging/audit-log-samples.adoc +++ b/modules/manage/pages/audit-logging/audit-log-samples.adoc @@ -1,11 +1,14 @@ = Sample Audit Log Messages :description: Sample Redpanda audit log messages. :page-categories: Management, Security +// tag::single-source[] +ifndef::env-cloud[] [NOTE] ==== include::shared:partial$enterprise-license.adoc[] ==== +endif::[] Redpanda's audit logs comply with version 1.0.0 of the https://github.com/ocsf[Open Cybersecurity Schema Framework (OCSF)]. This provides a predictable and extensible solution that works seamlessly with industry standard tools. This page aggregates several sample log files covering a range of scenarios. @@ -130,7 +133,7 @@ This scenario illustrates a common failure where a user entered the wrong creden The Redpanda Kafka API offers a wide array of options for interacting with your Redpanda clusters. Following are examples of messages from common interactions with the API. -.Create ACL Entry +.Create ACL entry [%collapsible] ==== This example illustrates an ACL update that also requires a superuser authentication. It lists the edited ACL and the updated permissions. This is a management type event. @@ -234,7 +237,7 @@ This example illustrates an ACL update that also requires a superuser authentica ---- ==== -.Metadata Request (with counts) +.Metadata request (with counts) [%collapsible] ==== This shows a message for a scenario where a user requests a set of metadata using rpk. It provides detailed information on the type of request and the information sent to the user. This is a describe type event. @@ -328,6 +331,7 @@ This shows a message for a scenario where a user requests a set of metadata usin ---- ==== +ifndef::env-cloud[] == Admin API events The following examples show audit messages related to use of the Redpanda Admin API. @@ -534,4 +538,7 @@ Similar to the previous example, this example illustrates a user requesting clus "unmapped": {} } ---- -==== \ No newline at end of file +==== +endif::[] + +// end::single-source[] \ No newline at end of file diff --git a/modules/manage/pages/iceberg/topic-iceberg-integration.adoc b/modules/manage/pages/iceberg/about-iceberg-topics.adoc similarity index 85% rename from modules/manage/pages/iceberg/topic-iceberg-integration.adoc rename to modules/manage/pages/iceberg/about-iceberg-topics.adoc index dd43de60fe..65f92a50e4 100644 --- a/modules/manage/pages/iceberg/topic-iceberg-integration.adoc +++ b/modules/manage/pages/iceberg/about-iceberg-topics.adoc @@ -1,7 +1,7 @@ = About Iceberg Topics :description: Learn how Redpanda can integrate topics with Apache Iceberg. :page-categories: Iceberg, Tiered Storage, Management, High Availability, Data Replication, Integration -:page-aliases: manage:topic-iceberg-integration.adoc +:page-aliases: manage:topic-iceberg-integration.adoc, manage:iceberg/topic-iceberg-integration.adoc [NOTE] ==== diff --git a/modules/manage/pages/iceberg/query-iceberg-topics.adoc b/modules/manage/pages/iceberg/query-iceberg-topics.adoc index 3f64830431..7941b28fd3 100644 --- a/modules/manage/pages/iceberg/query-iceberg-topics.adoc +++ b/modules/manage/pages/iceberg/query-iceberg-topics.adoc @@ -7,7 +7,7 @@ include::shared:partial$enterprise-license.adoc[] ==== -When you access Iceberg topics from a data lakehouse or other Iceberg-compatible tools, how you consume the data depends on the topic xref:manage:iceberg/topic-iceberg-integration.adoc#enable-iceberg-integration[Iceberg mode] and whether you've registered a schema for the topic in the xref:manage:schema-reg/schema-reg-overview.adoc[Redpanda Schema Registry]. In either mode, you do not need to rely on complex ETL jobs or pipelines to access real-time data from Redpanda. +When you access Iceberg topics from a data lakehouse or other Iceberg-compatible tools, how you consume the data depends on the topic xref:manage:iceberg/about-iceberg-topics.adoc#enable-iceberg-integration[Iceberg mode] and whether you've registered a schema for the topic in the xref:manage:schema-reg/schema-reg-overview.adoc[Redpanda Schema Registry]. In either mode, you do not need to rely on complex ETL jobs or pipelines to access real-time data from Redpanda. include::manage:partial$iceberg/query-iceberg-topics.adoc[] diff --git a/modules/manage/pages/iceberg/redpanda-topics-iceberg-snowflake-catalog.adoc b/modules/manage/pages/iceberg/redpanda-topics-iceberg-snowflake-catalog.adoc index f78e751996..2160d63d89 100644 --- a/modules/manage/pages/iceberg/redpanda-topics-iceberg-snowflake-catalog.adoc +++ b/modules/manage/pages/iceberg/redpanda-topics-iceberg-snowflake-catalog.adoc @@ -93,7 +93,7 @@ Successfully updated configuration. New configuration version is 2. . You must restart your cluster so that the configuration changes take effect. -. Enable the integration for a topic by configuring the topic property `redpanda.iceberg.mode`. This mode creates an Iceberg table for the topic consisting of two columns, one for the record metadata including the key, and another binary column for the record's value. See xref:manage:iceberg/topic-iceberg-integration.adoc#enable-iceberg-integration[Enable Iceberg integration] for more details on Iceberg modes. The following examples show how to use xref:get-started:rpk-install.adoc[`rpk`] to either create a new topic, or alter the configuration for an existing topic, to set the Iceberg mode to `key_value`. +. Enable the integration for a topic by configuring the topic property `redpanda.iceberg.mode`. This mode creates an Iceberg table for the topic consisting of two columns, one for the record metadata including the key, and another binary column for the record's value. See xref:manage:iceberg/about-iceberg-topics.adoc#enable-iceberg-integration[Enable Iceberg integration] for more details on Iceberg modes. The following examples show how to use xref:get-started:rpk-install.adoc[`rpk`] to either create a new topic, or alter the configuration for an existing topic, to set the Iceberg mode to `key_value`. + .Create a new topic and set `redpanda.iceberg.mode`: [,bash] diff --git a/modules/manage/pages/kubernetes/k-manage-topics.adoc b/modules/manage/pages/kubernetes/k-manage-topics.adoc index e204360cf7..b89d859bce 100644 --- a/modules/manage/pages/kubernetes/k-manage-topics.adoc +++ b/modules/manage/pages/kubernetes/k-manage-topics.adoc @@ -139,7 +139,7 @@ include::shared:partial$enterprise-license.adoc[] In addition to the general prerequisites for managing topics, you must have the following: -- xref:manage:topic-iceberg-integration.adoc[Iceberg support] must be enabled on your Redpanda cluster. +- xref:manage:iceberg/about-iceberg-topics.adoc[Iceberg support] must be enabled on your Redpanda cluster. - xref:manage:kubernetes/tiered-storage/k-tiered-storage.adoc[Tiered Storage] must be enabled on your Redpanda cluster. To create an Iceberg topic, set the `redpanda.iceberg.mode` configuration in the `additionalConfig` property of the `Topic` resource. diff --git a/modules/manage/pages/schema-reg/schema-id-validation.adoc b/modules/manage/pages/schema-reg/schema-id-validation.adoc index 5ecc6c52bd..017988730d 100644 --- a/modules/manage/pages/schema-reg/schema-id-validation.adoc +++ b/modules/manage/pages/schema-reg/schema-id-validation.adoc @@ -2,6 +2,7 @@ :page-categories: Management, Schema Registry, rpk :page-aliases: manage:schema-id-validation.adoc :description: Learn about server-side schema ID validation for clients using SerDes that produce to Redpanda brokers, and learn how to configure Redpanda to inspect and reject records with invalid schema IDs. +// tag::single-source[] You can use server-side schema ID validation for clients using Confluent's SerDes format that produce to Redpanda brokers. You can also configure Redpanda to inspect and reject records with schema IDs that aren't valid according to the configured Subject Name strategy and registered with the Schema Registry. @@ -34,6 +35,7 @@ To use schema ID validation: === Enable schema ID validation +ifndef::env-cloud[] By default, server-side schema ID validation is disabled in Redpanda. To enable schema ID validation, change the xref:reference:cluster-properties.adoc#enable_schema_id_validation[`enable_schema_id_validation`] cluster property from its default value of `none` to either `redpanda` or `compat`: * `none`: Schema validation is disabled (no schema ID checks are done). Associated topic properties cannot be modified. @@ -46,6 +48,17 @@ For example, use `rpk` to set the value of `enable_schema_id_validation` to `red ---- rpk cluster config set enable_schema_id_validation redpanda --api-urls=:9644 ---- +endif::[] + +ifdef::env-cloud[] +To enable schema ID validation, set the `enable_schema_id_validation` cluster property to either `redpanda` or `compat`: + +* `none`: Schema validation is disabled (no schema ID checks are done). Associated topic properties cannot be modified. +* `redpanda`: Schema validation is enabled. Only Redpanda topic properties are accepted. +* `compat`: Schema validation is enabled. Both Redpanda and compatible topic properties are accepted. + +See xref:manage:cluster-maintenance/config-cluster.adoc[] +endif::[] === Set subject name strategy per topic @@ -124,4 +137,6 @@ rpk topic alter-config topic_foo \ --set redpanda.value.schema.id.validation=true \ --set redpanda.value.subject.name.strategy=RecordNameStrategy \ -X brokers=:9092 ----- \ No newline at end of file +---- + +// end::single-source[] \ No newline at end of file diff --git a/modules/manage/pages/schema-reg/schema-reg-api.adoc b/modules/manage/pages/schema-reg/schema-reg-api.adoc index 1d3c9e195a..89a78c3c5f 100644 --- a/modules/manage/pages/schema-reg/schema-reg-api.adoc +++ b/modules/manage/pages/schema-reg/schema-reg-api.adoc @@ -7,7 +7,7 @@ Schemas provide human-readable documentation for an API. They verify that data c [NOTE] ==== -The Schema Registry is built into Redpanda, and you can use it with the API or {ui}. This section describes operations available in the xref:api:ROOT:pandaproxy-schema-registry.adoc[Schema Registry API]. +The Schema Registry is built into Redpanda, and you can use it with the API or the UI. This section describes operations available in the xref:api:ROOT:pandaproxy-schema-registry.adoc[Schema Registry API]. ==== The Redpanda Schema Registry has API endpoints that allow you to perform the following tasks: diff --git a/modules/manage/partials/audit-logging.adoc b/modules/manage/partials/audit-logging.adoc index fe33785e07..40338fbaba 100644 --- a/modules/manage/partials/audit-logging.adoc +++ b/modules/manage/partials/audit-logging.adoc @@ -1,19 +1,26 @@ +ifndef::env-cloud[] + [NOTE] ==== include::shared:partial$enterprise-license.adoc[] ==== +endif::[] + Many scenarios for streaming data include the need for fine-grained auditing of user activity related to the system. This is especially true for regulated industries such as finance, healthcare, and the public sector. Complying with https://pcidssguide.com/whats-new-in-pci-dss-v4-0/[PCI DSS v4] standards, for example, requires verbose and detailed activity auditing, alerting, and analysis capabilities. Redpanda's auditing capabilities support recording both administrative and operational interactions with topics and with users. Redpanda complies with the Open Cybersecurity Schema Framework (OCSF), providing a predictable and extensible solution that works seamlessly with industry standard tools. With audit logging enabled, there should be no noticeable changes in performance other than slightly elevated CPU usage. +ifndef::env-cloud[] NOTE: Audit logging is configured at the cluster level. Redpanda supports excluding specific topics or principals from auditing to help reduce noise in the log. Audit logging is disabled by default. +endif::[] + == Audit log flow -The Redpanda audit log mechanism functions similar to the Kafka flow you may be familiar with. When a user interacts with another user or with a topics, Redpanda writes an event to a specialized audit topic. The audit topic is immutable. Only Redpanda can write to it. Users are prevented from writing to the audit topic directly and the Kafka API cannot create or delete it. +The Redpanda audit log mechanism functions similar to the Kafka flow. When a user interacts with another user or with a topic, Redpanda writes an event to a specialized audit topic. The audit topic is immutable. Only Redpanda can write to it. Users are prevented from writing to the audit topic directly and the Kafka API cannot create or delete it. image:shared:audit-logging-flow.png[Audit log flow] @@ -23,41 +30,54 @@ Messages recorded to the audit log topic comply with the https://schema.ocsf.io/ == Audit log configuration options -Redpanda's audit logging mechanism supports several options to control the volume and availability of audit records. Configuration is applied at the cluster level using the standard xref:manage:cluster-maintenance/cluster-property-configuration.adoc[cluster configuration mechanism]. +Redpanda's audit logging mechanism supports several options to control the volume and availability of audit records. Configuration is applied at the cluster level. + +ifdef::env-cloud[] +To configure audit logging, see xref:manage:cluster-maintenance/config-cluster.adoc[]. -ifdef::env-kubernetes[You can configure these options directly in either the Helm values or the Redpanda resource.] +* xref:reference:properties/cluster-properties.adoc#audit_enabled[`audit_enabled`]: Boolean value to enable audit logging. When you set this to `true`, Redpanda checks for an existing topic named `_redpanda.audit_log`. If none is found, Redpanda automatically creates one for you. Default: `true`. +* xref:reference:properties/cluster-properties.adoc#audit_enabled_event_types[`audit_enabled_event_types`]: List of strings in JSON style identifying the event types to include in the audit log. This may include any of the following: `management, produce, consume, describe, heartbeat, authenticate, schema_registry, admin`. Default: `'["management","authenticate","admin"]'`. +* xref:reference:properties/cluster-properties.adoc#audit_excluded_principals[`audit_excluded_principals`]: List of strings in JSON style identifying the principals the audit logging system should ignore. Principals can be listed as `User:name` or `name`, both are accepted. Default: `null`. + +endif::[] ifdef::env-kubernetes[] +You can configure these options directly in either the Helm values or the Redpanda resource. + * `auditLogging.enabled`: Sets the value of the xref:reference:cluster-properties.adoc#audit_enabled[`audit_enabled`] cluster property to enable audit logging. When you set this to `true`, Redpanda checks for an existing topic named `_redpanda.audit_log`. If none is found, Redpanda automatically creates one for you. Default: `false`. * `auditLogging.partitions`: Sets the value of the xref:reference:cluster-properties.adoc#audit_log_num_partitions[`audit_log_num_partitions`] cluster property to define the number of partitions used by a newly created audit topic. This configuration applies only to the audit log topic and may be different from the cluster or other topic configurations. This cannot be altered for an existing audit log topic. Default: `12`. -* `auditLogging.replicationFactor`: Sets the value of the xref:reference:cluster-properties.adoc#audit_log_replication_factor[`audit_log_replication_factor`] cluster property to define the replication factor for a newly created audit log topic. This configuration applies only to the audit log topic and may be different from the cluster or other topic configurations. This cannot be altered for existing audit log topics. If a value is not provided, Redpanda will use the `internal_topic_replication_factor` cluster config value. Default: `null`. -* `auditLogging.enabledEventTypes`: Sets the value of the xref:reference:cluster-properties.adoc#audit_enabled_event_types[`audit_enabled_event_types`] cluster property. This option is a list of JSON strings identifying the <> to include in the audit log. Valid values include any of the following - `management`, `produce`, `consume`, `describe`, `heartbeat`, `authenticate`, `schema_registry`, `admin`. Default: `'["management","authenticate","admin"]'`. -* `auditLogging.excludedTopics`: Sets the value of the xref:reference:cluster-properties.adoc#audit_excluded_topics[`audit_excluded_topics`] cluster property. This option is a list of JSON strings identifying the topics the audit logging system should ignore. This list cannot include the `_redpanda.audit_log` topic. Redpanda will reject the command if you do attempt to include that topic. Default: `null`. +* `auditLogging.replicationFactor`: Sets the value of the xref:reference:cluster-properties.adoc#audit_log_replication_factor[`audit_log_replication_factor`] cluster property to define the replication factor for a newly created audit log topic. This configuration applies only to the audit log topic and may be different from the cluster or other topic configurations. This cannot be altered for existing audit log topics. If a value is not provided, Redpanda uses the `internal_topic_replication_factor` cluster property value. Default: `null`. +* `auditLogging.enabledEventTypes`: Sets the value of the xref:reference:cluster-properties.adoc#audit_enabled_event_types[`audit_enabled_event_types`] cluster property. This option is a list of JSON strings identifying the <> to include in the audit log. Valid values include any of the following: `management`, `produce`, `consume`, `describe`, `heartbeat`, `authenticate`, `schema_registry`, `admin`. Default: `'["management","authenticate","admin"]'`. +* `auditLogging.excludedTopics`: Sets the value of the xref:reference:cluster-properties.adoc#audit_excluded_topics[`audit_excluded_topics`] cluster property. This option is a list of JSON strings identifying the topics the audit logging system should ignore. This list cannot include the `_redpanda.audit_log` topic. Redpanda rejects the command if you do attempt to include that topic. Default: `null`. * `auditLogging.excludedPrincipals`: Sets the value of the xref:reference:cluster-properties.adoc#audit_excluded_principals[`audit_excluded_principals`] cluster property. This option is a list of JSON strings identifying the principals the audit logging system should ignore. Principals can be listed as `User:name` or `name`, both are accepted. Default: `null`. * `auditLogging.clientMaxBufferSize`: Sets the value of the xref:reference:cluster-properties.adoc#audit_client_max_buffer_size[`audit_client_max_buffer_size`] cluster property to define the number of bytes allocated by the internal audit client for audit messages. When changing this, you must disable audit logging and then re-enable it for the change to take effect. Consider increasing this if your system generates a very large number of audit records in a short amount of time. Default: `16777216`. * `auditLogging.queueDrainIntervalMs`: Sets the value of the xref:reference:cluster-properties.adoc#audit_queue_drain_interval_ms[`audit_queue_drain_interval_ms`] cluster property. Internally, Redpanda batches audit log messages in memory and periodically writes them to the audit log topic. This option defines the period in milliseconds between draining this queue to the audit log topic. Longer intervals may help prevent duplicate messages, especially in high throughput scenarios, but they also increase the risk of data loss during hard shutdowns where the queue is lost. Default: `500`. -* `auditLogging.queueMaxBufferSizePerShard`: Sets the value of the xref:reference:cluster-properties.adoc#audit_queue_max_buffer_size_per_shard[`audit_queue_max_buffer_size_per_shard`] cluster property to define the maximum amount of memory in bytes used by the audit buffer in each shard. Once this size is reached, requests to log additional audit messages will return a non-retryable error. Default: `1048576`. +* `auditLogging.queueMaxBufferSizePerShard`: Sets the value of the xref:reference:cluster-properties.adoc#audit_queue_max_buffer_size_per_shard[`audit_queue_max_buffer_size_per_shard`] cluster property to define the maximum amount of memory in bytes used by the audit buffer in each shard. When this size is reached, requests to log additional audit messages return a non-retryable error. Default: `1048576`. Even though audited event messages are stored to a specialized immutable topic, standard topic settings still apply. For example, you can apply the same Tiered Storage, retention time, and replication settings available to normal topics. These particular options are important for controlling the amount of disk space utilized by your audit topics. IMPORTANT: You cannot change the values of `auditLogging.partitions` and `auditLogging.replicationFactor` after enabling audit logging because these settings impact the creation of the `_redpanda.audit_log` topic. The Kafka API allows you to add partitions or alter the replication factor after enabling audit logging, but Redpanda prevents you from altering these two configuration values directly. + endif::[] -ifndef::env-kubernetes[] + +ifndef::env-cloud,env-kubernetes[] * xref:reference:cluster-properties.adoc#audit_enabled[`audit_enabled`]: Boolean value to enable audit logging. When you set this to `true`, Redpanda checks for an existing topic named `_redpanda.audit_log`. If none is found, Redpanda automatically creates one for you. Default: `false`. * xref:reference:cluster-properties.adoc#audit_log_num_partitions[`audit_log_num_partitions`]: Integer value defining the number of partitions used by a newly created audit topic. This configuration applies only to the audit log topic and may be different from the cluster or other topic configurations. This cannot be altered for an existing audit log topic. Default: `12`. -* xref:reference:cluster-properties.adoc#audit_log_replication_factor[`audit_log_replication_factor`]: Optional Integer value defining the replication factor for a newly created audit log topic. This configuration applies only to the audit log topic and may be different from the cluster or other topic configurations. This cannot be altered for existing audit log topics. If a value is not provided, Redpanda will use the `internal_topic_replication_factor` cluster config value. Default: `null`. +* xref:reference:cluster-properties.adoc#audit_log_replication_factor[`audit_log_replication_factor`]: Optional Integer value defining the replication factor for a newly created audit log topic. This configuration applies only to the audit log topic and may be different from the cluster or other topic configurations. This cannot be altered for existing audit log topics. If a value is not provided, Redpanda uses the `internal_topic_replication_factor` cluster property value. Default: `null`. * xref:reference:cluster-properties.adoc#audit_client_max_buffer_size[`audit_client_max_buffer_size`]: Integer value defining the number of bytes allocated by the internal audit client for audit messages. When changing this, you must disable audit logging and then re-enable it for the change to take effect. Consider increasing this if your system generates a very large number of audit records in a short amount of time. Default: `16777216`. -* xref:reference:cluster-properties.adoc#audit_queue_max_buffer_size_per_shard[`audit_queue_max_buffer_size_per_shard`]: Integer value defining the maximum amount of memory in bytes used by the audit buffer in each shard. Once this size is reached, requests to log additional audit messages will return a non-retryable error. You must restart the cluster when changing this value. Default: `1048576`. -* xref:reference:cluster-properties.adoc#audit_enabled_event_types[`audit_enabled_event_types`]: List of strings in JSON style identifying the event types to include in the audit log. This may include any of the following - `management, produce, consume, describe, heartbeat, authenticate, schema_registry, admin`. Default: `'["management","authenticate","admin"]'`. -* xref:reference:cluster-properties.adoc#audit_excluded_topics[`audit_excluded_topics`]: List of strings in JSON style identifying the topics the audit logging system should ignore. This list cannot include the `_redpanda.audit_log` topic. Redpanda will reject the command if you do attempt to include that topic. Default: `null`. +* xref:reference:cluster-properties.adoc#audit_queue_max_buffer_size_per_shard[`audit_queue_max_buffer_size_per_shard`]: Integer value defining the maximum amount of memory in bytes used by the audit buffer in each shard. When this size is reached, requests to log additional audit messages return a non-retryable error. You must restart the cluster when changing this value. Default: `1048576`. +* xref:reference:cluster-properties.adoc#audit_enabled_event_types[`audit_enabled_event_types`]: List of strings in JSON style identifying the event types to include in the audit log. This may include any of the following: `management, produce, consume, describe, heartbeat, authenticate, schema_registry, admin`. Default: `'["management","authenticate","admin"]'`. +* xref:reference:cluster-properties.adoc#audit_excluded_topics[`audit_excluded_topics`]: List of strings in JSON style identifying the topics the audit logging system should ignore. This list cannot include the `_redpanda.audit_log` topic. Redpanda rejects the command if you do attempt to include that topic. Default: `null`. * xref:reference:cluster-properties.adoc#audit_queue_drain_interval_ms[`audit_queue_drain_interval_ms`]: Internally, Redpanda batches audit log messages in memory and periodically writes them to the audit log topic. This defines the period in milliseconds between draining this queue to the audit log topic. Longer intervals may help prevent duplicate messages, especially in high throughput scenarios, but they also increase the risk of data loss during hard shutdowns where the queue is lost. Default: `500`. * xref:reference:cluster-properties.adoc#audit_excluded_principals[`audit_excluded_principals`]: List of strings in JSON style identifying the principals the audit logging system should ignore. Principals can be listed as `User:name` or `name`, both are accepted. Default: `null`. Even though audited event messages are stored to a specialized immutable topic, standard topic settings still apply. For example, you can apply the same Tiered Storage, retention time, and replication settings available to normal topics. These particular options are important for controlling the amount of disk space utilized by your audit topics. IMPORTANT: You must configure certain audit logging properties before enabling audit logging because these settings impact the creation of the `_redpanda.audit_log` topic itself. These properties include: `audit_log_num_partitions` and `audit_log_replication_factor`. The Kafka API allows you to add partitions or alter the replication factor after enabling audit logging, but Redpanda prevents you from altering these two configuration values directly. + endif::[] +ifndef::env-cloud[] == Audit logging event types Redpanda's auditable events fall into one of eight different event types. The APIs associated with each event type are as follows. @@ -121,13 +141,22 @@ a|* All Schema Registry API calls a|* All Admin API calls |=== +endif::[] == Enable audit logging +ifdef::env-cloud[] +Audit logging is enabled by default. Cluster administrators can configure the audited topics and principals. However, only the Redpanda team can configure the type of audited events. For more information or support, contact your Redpanda account team. + +endif::[] + +ifndef::env-cloud[] All audit log settings are applied at the cluster level. -You can configure audit log settings in the Redpanda Helm chart, using Helm values or the Redpanda resource with the Redpanda Operator. +endif::[] ifdef::env-kubernetes[] +You can configure audit log settings in the Redpanda Helm chart, using Helm values or the Redpanda resource with the Redpanda Operator. + [tabs] ====== Operator:: @@ -212,7 +241,7 @@ spec: enabled: true ---- -If you don't want to use the Topic resource, you can enable audit logging and Redpanda will create the audit topic for you: +If you don't want to use the Topic resource, you can enable audit logging and Redpanda creates the audit topic for you: .`redpanda-cluster.yaml` [,yaml,lines=9-22] @@ -287,16 +316,15 @@ For details, see xref:manage:kubernetes/security/authentication/k-authentication - `auditLogging.enabled`: Enable audit logging. When you set this to `true`, Redpanda checks for an existing topic named `_redpanda.audit_log`. If the topic is not found, Redpanda automatically creates one for you. Default: `false`. endif::[] - -ifndef::env-kubernetes[] -Use the `rpk cluster config` to configure audit logs. Some options will require a cluster restart. You can verify this using `rpk cluster config status`. +ifndef::env-cloud,env-kubernetes[] +Use `rpk cluster config` to configure audit logs. Some options require a cluster restart. You can verify this using `rpk cluster config status`. Some key tuning recommendations for your audit logging settings include: -* If you wish to change the number of partitions or the replication factor for your audit log topic, set the `audit_log_num_partitions` and `audit_log_replication_factor` properties respectively. +* To change the number of partitions or the replication factor for your audit log topic, set the `audit_log_num_partitions` and `audit_log_replication_factor` properties, respectively. * Choose the type of events needed by setting `audit_enabled_event_types` to the desired list of event categories. Keep this as restrictive as possible based on your compliance and security needs to avoid excessive noise in your audit logs. * Identify non-sensitive topics so that you can exclude them from auditing. Specify this list of topics in `audit_excluded_topics`. -* Identify non-sensitive principals so that you can exclude them from auditing. Specify this list of principals in `audit_excluded_principals`. This command accepts names in the form of `name` or `User:name`. +* Identify non-sensitive principals so that you can exclude them from auditing. Specify this list of principals in `audit_excluded_principals`. This command accepts names as `name` or `User:name`. * Set `audit_enabled` to `true`. * <>. @@ -309,23 +337,39 @@ The sequence of commands in `rpk` for this audit log configuration is: rpk cluster config set audit_excluded_principals '["User:principal1", "principal2"]' rpk cluster config set audit_enabled true rpk topic alter-config _redpanda.audit_log --set retention.ms=259200000 + +endif::[] + +ifdef::env-cloud[] + +== Configure retention for audit logs + +Assess the retention needs for your audit logs. You may not need to keep the logs for the default seven days. This is controlled by setting the `retention.ms` property for the `_redpanda.audit_log` topic. + +== Next steps + +xref:manage:audit-logging/audit-log-samples.adoc[See samples of audit log messages] + endif::[] +ifndef::env-cloud[] == Optimize costs for audit logging -When enabled, audit logging can quickly generate a very large amount of data, especially if all event types are selected. Proper configuration of audit logging is critical to avoid filling your disk or using excess Tiered Storage. The configuration options available help ensure your audit logs contain only the volume of data necessary to meeting your regulatory or legal requirements. +When enabled, audit logging can quickly generate a very large amount of data, especially if all event types are selected. Proper configuration of audit logging is critical to avoid filling your disk or using excess Tiered Storage. The configuration options available help ensure your audit logs contain only the volume of data necessary to meet your regulatory or legal requirements. With audit logging, the pattern of message generation may be very different from your typical sources of data. These messages reflect usage of your system as opposed to the operational data your topics typically process. As a result, your retention, replication, and Tiered Storage requirements may differ from your other topics. -A typical scenario with audit logging is to route the messages to an analytics platform like Splunk. If your retention period is too long, you will find that you are storing excessive amounts of replicated messages in both Redpanda and in your analytics suite. Identifying the right balance of retention and replication settings minimizes this duplication while retaining your data in a system that provides actionable intelligence. +A typical scenario with audit logging is to route the messages to an analytics platform like Splunk. If your retention period is too long, you may find that you are storing excessive amounts of replicated messages in both Redpanda and in your analytics suite. Identifying the right balance of retention and replication settings minimizes this duplication while retaining your data in a system that provides actionable intelligence. -Assess the retention needs for your audit logs. You may not need to keep the logs around for the default seven days. This is controlled by setting xref:reference:topic-properties.adoc#retentionms[`retention.ms`] for the `_redpanda.audit_log` topic or by setting xref:reference:cluster-properties.adoc#delete_retention_ms[`delete_retention_ms`] at the cluster level. +Assess the retention needs for your audit logs. You may not need to keep the logs for the default seven days. This is controlled by setting xref:reference:topic-properties.adoc#retentionms[`retention.ms`] for the `_redpanda.audit_log` topic or by setting xref:reference:cluster-properties.adoc#delete_retention_ms[`delete_retention_ms`] at the cluster level. == Next steps -xref:manage:audit-logging/audit-log-samples.adoc[See samples of audit log messages]. +xref:manage:audit-logging/audit-log-samples.adoc[See samples of audit log messages] include::shared:partial$suggested-reading.adoc[] - xref:reference:topic-properties.adoc[] - xref:develop:config-topics.adoc[] + +endif::[] diff --git a/modules/manage/partials/iceberg/about-iceberg-topics.adoc b/modules/manage/partials/iceberg/about-iceberg-topics.adoc index ed217b7c05..dab52d2ac6 100644 --- a/modules/manage/partials/iceberg/about-iceberg-topics.adoc +++ b/modules/manage/partials/iceberg/about-iceberg-topics.adoc @@ -62,7 +62,13 @@ endif::[] == Enable Iceberg integration +ifndef::env-cloud[] To create an Iceberg table for a Redpanda topic, you must set the cluster configuration property config_ref:iceberg_enabled,true,properties/cluster-properties[`iceberg_enabled`] to `true`, and also configure the topic property xref:reference:properties/topic-properties.adoc#redpanda-iceberg-mode[`redpanda.iceberg.mode`]. You can choose to provide a schema if you need the Iceberg table to be structured with defined columns. +endif::[] + +ifdef::env-cloud[] +To create an Iceberg table for a Redpanda topic, you must set the cluster configuration property config_ref:iceberg_enabled,true,properties/cluster-properties[`iceberg_enabled`] to `true`, and also configure the topic property `redpanda.iceberg.mode`. You can choose to provide a schema if you need the Iceberg table to be structured with defined columns. +endif::[] . Set the `iceberg_enabled` configuration option on your cluster to `true`. You must restart your cluster if you change this configuration for a running cluster. ifdef::env-cloud[] @@ -115,7 +121,12 @@ TOPIC STATUS OK ---- + +ifndef::env-cloud[] To improve query performance, consider implementing custom https://iceberg.apache.org/docs/nightly/partitioning/[partitioning^] for the Iceberg topic. Use the xref:reference:properties/topic-properties.adoc#redpanda-iceberg-partition-spec[`redpanda.iceberg.partition.spec`] topic property to define the partitioning scheme: +endif::[] +ifdef::env-cloud[] +To improve query performance, consider implementing custom https://iceberg.apache.org/docs/nightly/partitioning/[partitioning^] for the Iceberg topic. Use the `redpanda.iceberg.partition.spec` topic property to define the partitioning scheme: +endif::[] + [,bash,] ---- @@ -183,7 +194,12 @@ The Iceberg table resides in a namespace called `redpanda` and has the same name == About schema support and translation to Iceberg format -The xref:reference:properties/topic-properties.adoc#redpanda-iceberg-mode[`redpanda.iceberg.mode`] property determines how Redpanda maps the topic data to the Iceberg table structure. You can have the generated Iceberg table match the structure of a Avro or Protobuf schema in the Schema Registry, or you can use the `key_value` mode where Redpanda stores the record values as-is in the table. +ifndef::env-cloud[] +The xref:reference:properties/topic-properties.adoc#redpanda-iceberg-mode[`redpanda.iceberg.mode`] topic property determines how Redpanda maps the topic data to the Iceberg table structure. You can have the generated Iceberg table match the structure of a Avro or Protobuf schema in the Schema Registry, or you can use the `key_value` mode where Redpanda stores the record values as-is in the table. +endif::[] +ifdef::env-cloud[] +The `redpanda.iceberg.mode` topic property determines how Redpanda maps the topic data to the Iceberg table structure. You can have the generated Iceberg table match the structure of a Avro or Protobuf schema in the Schema Registry, or you can use the `key_value` mode where Redpanda stores the record values as-is in the table. +endif::[] The JSON Schema format is not supported. If your topic data is in JSON, use the `key_value` mode. @@ -410,7 +426,12 @@ Querying the Iceberg table for `demo-topic` includes the new column `ts`: Errors may occur when translating records in the `value_schema_id_prefix` mode to the Iceberg table format; for example, if you do not use the Schema Registry wire format with the magic byte, if the schema ID in the record is not found in the Schema Registry, or if an Avro or Protobuf data type cannot be translated to an Iceberg type. +ifndef::env-cloud[] If Redpanda encounters an error while writing a record to the Iceberg table, Redpanda writes the record to a separate dead-letter queue (DLQ) Iceberg table named `~dlq`. To disable the default behavior for a topic and drop the record, set the xref:reference:properties/topic-properties.adoc#redpanda-iceberg-invalid-record-action[`redpanda.iceberg.invalid.record.action`] topic property to `drop`. You can also configure the default cluster-wide behavior for invalid records by setting the `iceberg_invalid_record_action` property. +endif::[] +ifdef::env-cloud[] +If Redpanda encounters an error while writing a record to the Iceberg table, Redpanda writes the record to a separate dead-letter queue (DLQ) Iceberg table named `~dlq`. To disable the default behavior for a topic and drop the record, set the `redpanda.iceberg.invalid.record.action` topic property to `drop`. You can also configure the default cluster-wide behavior for invalid records by setting the `iceberg_invalid_record_action` property. +endif::[] The DLQ table itself uses the `key_value` schema, consisting of two columns: the record metadata including the key, and a binary column for the record's value. @@ -484,7 +505,12 @@ You may need to increase the size of your Redpanda cluster to accommodate the ad === Use custom partitioning +ifndef::env-cloud[] To improve query performance, consider implementing custom https://iceberg.apache.org/docs/nightly/partitioning/[partitioning^] for the Iceberg topic. Use the xref:reference:properties/topic-properties.adoc#redpanda-iceberg-partition-spec[`redpanda.iceberg.partition.spec`] topic property to define the partitioning scheme: +endif::[] +ifdef::env-cloud[] +To improve query performance, consider implementing custom https://iceberg.apache.org/docs/nightly/partitioning/[partitioning^] for the Iceberg topic. Use the `redpanda.iceberg.partition.spec` topic property to define the partitioning scheme: +endif::[] [,bash,] ---- diff --git a/modules/reference/pages/data-transforms/golang-sdk.adoc b/modules/reference/pages/data-transforms/golang-sdk.adoc index 6b58636335..fb70e438b6 100644 --- a/modules/reference/pages/data-transforms/golang-sdk.adoc +++ b/modules/reference/pages/data-transforms/golang-sdk.adoc @@ -2,6 +2,7 @@ :description: Work with data transform APIs in Redpanda using Go. :page-aliases: labs:data-transform/data-transform-api.adoc, reference:data-transform-api.adoc, reference:data-transform-golang-sdk.adoc :page-categories: Development, Stream Processing, Data Transforms +// tag::single-source[] The API reference is in the Go package documentation: @@ -9,7 +10,10 @@ The API reference is in the Go package documentation: - https://pkg.go.dev/github.com/redpanda-data/redpanda/src/transform-sdk/go/transform/sr[Schema Registry client library]: This library provides data transforms with access to the Schema Registry built into Redpanda. +// end::single-source[] + == Suggested reading - xref:develop:data-transforms/versioning-compatibility.adoc[] -- xref:develop:data-transforms/upgrade.adoc[] \ No newline at end of file +- xref:develop:data-transforms/upgrade.adoc[] + diff --git a/modules/reference/pages/data-transforms/js/js-sdk-sr.adoc b/modules/reference/pages/data-transforms/js/js-sdk-sr.adoc index 6010652f37..bfed92c695 100644 --- a/modules/reference/pages/data-transforms/js/js-sdk-sr.adoc +++ b/modules/reference/pages/data-transforms/js/js-sdk-sr.adoc @@ -1,5 +1,6 @@ = JavaScript Schema Registry API for Data Transforms :description: Work with Schema Registry in data transforms using JavaScript. +// tag::single-source[] This page contains the API reference for the Schema Registry client library of the data transforms JavaScript SDK. @@ -129,4 +130,6 @@ Client interface for interacting with Redpanda Schema Registry. == Suggested reading -xref:reference:data-transforms/js/js-sdk.adoc[] \ No newline at end of file +xref:reference:data-transforms/js/js-sdk.adoc[] + +// end::single-source[] \ No newline at end of file diff --git a/modules/reference/pages/data-transforms/js/js-sdk.adoc b/modules/reference/pages/data-transforms/js/js-sdk.adoc index 148a48e273..6232d05e5e 100644 --- a/modules/reference/pages/data-transforms/js/js-sdk.adoc +++ b/modules/reference/pages/data-transforms/js/js-sdk.adoc @@ -1,6 +1,7 @@ = JavaScript API for Data Transforms :description: Work with data transforms using JavaScript. :page-aliases: reference:data-transforms/js-sdk.adoc +// tag::single-source[] This page contains the API reference for the data transforms client library of the JavaScript SDK. @@ -134,4 +135,6 @@ Records may have a collection of headers attached to them. Headers are opaque to == Suggested reading -xref:reference:data-transforms/js/js-sdk-sr.adoc[] \ No newline at end of file +xref:reference:data-transforms/js/js-sdk-sr.adoc[] + +// end::single-source[] \ No newline at end of file diff --git a/modules/reference/pages/data-transforms/rust-sdk.adoc b/modules/reference/pages/data-transforms/rust-sdk.adoc index 3dcccfc1de..160e258be6 100644 --- a/modules/reference/pages/data-transforms/rust-sdk.adoc +++ b/modules/reference/pages/data-transforms/rust-sdk.adoc @@ -1,6 +1,7 @@ = Rust SDK for Data Transforms :description: Work with data transforms using Rust. :page-aliases: reference:data-transform-rust-sdk.adoc +// tag::single-source[] The API reference is in the crate documentation: @@ -8,6 +9,8 @@ The API reference is in the crate documentation: - https://docs.rs/redpanda-transform-sdk-sr/latest/redpanda_transform_sdk_sr/[Schema Registry client library]: This crate provides data transforms with access to the Schema Registry built into Redpanda. +// end::single-source[] + == Suggested reading - xref:develop:data-transforms/versioning-compatibility.adoc[] diff --git a/modules/reference/pages/properties/cluster-properties.adoc b/modules/reference/pages/properties/cluster-properties.adoc index 648061414b..63c9919d32 100644 --- a/modules/reference/pages/properties/cluster-properties.adoc +++ b/modules/reference/pages/properties/cluster-properties.adoc @@ -138,9 +138,12 @@ Defines the number of bytes allocated by the internal audit client for audit mes --- +// tag::audit_enabled[] === audit_enabled +ifndef::env-cloud[] include::reference:partial$enterprise-licensed-property.adoc[] +endif::[] Enables or disables audit logging. When you set this to true, Redpanda checks for an existing topic named `_redpanda.audit_log`. If none is found, Redpanda automatically creates one for you. @@ -156,6 +159,9 @@ Enables or disables audit logging. When you set this to true, Redpanda checks fo --- +// end::audit_enabled[] + +// tag::audit_enabled_event_types[] === audit_enabled_event_types List of strings in JSON style identifying the event types to include in the audit log. This may include any of the following: `management, produce, consume, describe, heartbeat, authenticate, schema_registry, admin`. @@ -170,6 +176,9 @@ List of strings in JSON style identifying the event types to include in the audi --- +// end::audit_enabled_event_types[] + +// tag::audit_excluded_principals[] === audit_excluded_principals List of user principals to exclude from auditing. @@ -184,6 +193,10 @@ List of user principals to exclude from auditing. --- +// end::audit_excluded_principals[] + + +// tag::audit_excluded_topics[] === audit_excluded_topics List of topics to exclude from auditing. @@ -198,6 +211,8 @@ List of topics to exclude from auditing. --- +// end::audit_excluded_topics[] + === audit_log_num_partitions Defines the number of partitions used by a newly-created audit topic. This configuration applies only to the audit log topic and may be different from the cluster or other topic configurations. This cannot be altered for existing audit log topics. @@ -264,6 +279,7 @@ Defines the maximum amount of memory in bytes used by the audit buffer in each s --- + === auto_create_topics_enabled Allow automatic topic creation. To prevent excess topics, this property is not supported on Redpanda Cloud BYOC and Dedicated clusters. You should explicitly manage topic creation for these Redpanda Cloud clusters. @@ -280,6 +296,7 @@ If you produce to a topic that doesn't exist, the topic will be created with def --- + === cluster_id Cluster identifier. @@ -640,6 +657,7 @@ Timeout, in milliseconds, to wait for new topic creation. --- +// tag::data_transforms_binary_max_size[] === data_transforms_binary_max_size The maximum size for a deployable WebAssembly binary that the broker can store. @@ -654,6 +672,9 @@ The maximum size for a deployable WebAssembly binary that the broker can store. --- +// end::data_transforms_binary_max_size[] + +// tag::data_transforms_commit_interval_ms[] === data_transforms_commit_interval_ms The commit interval at which data transforms progress. @@ -672,6 +693,9 @@ The commit interval at which data transforms progress. --- +// end::data_transforms_commit_interval_ms[] + +// tag::data_transforms_enabled[] === data_transforms_enabled Enables WebAssembly-powered data transforms directly in the broker. When `data_transforms_enabled` is set to `true`, Redpanda reserves memory for data transforms, even if no transform functions are currently deployed. This memory reservation ensures that adequate resources are available for transform functions when they are needed, but it also means that some memory is allocated regardless of usage. @@ -686,6 +710,9 @@ Enables WebAssembly-powered data transforms directly in the broker. When `data_t --- +// end::data_transforms_enabled[] + + === data_transforms_logging_buffer_capacity_bytes Buffer capacity for transform logs, per shard. Buffer occupancy is calculated as the total size of buffered log messages; that is, logs emitted but not yet produced. @@ -720,6 +747,7 @@ Flush interval for transform logs. When a timer expires, pending logs are collec --- +// tag::data_transforms_logging_line_max_bytes[] === data_transforms_logging_line_max_bytes Transform log lines truncate to this length. Truncation occurs after any character escaping. @@ -736,6 +764,8 @@ Transform log lines truncate to this length. Truncation occurs after any charact --- +// end::data_transforms_logging_line_max_bytes[] + === data_transforms_per_core_memory_reservation The amount of memory to reserve per core for data transform (Wasm) virtual machines. Memory is reserved on boot. The maximum number of functions that can be deployed to a cluster is equal to `data_transforms_per_core_memory_reservation` / `data_transforms_per_function_memory_limit`. @@ -750,6 +780,7 @@ The amount of memory to reserve per core for data transform (Wasm) virtual machi --- +// tag::data_transforms_per_function_memory_limit[] === data_transforms_per_function_memory_limit The amount of memory to give an instance of a data transform (Wasm) virtual machine. The maximum number of functions that can be deployed to a cluster is equal to `data_transforms_per_core_memory_reservation` / `data_transforms_per_function_memory_limit`. @@ -764,6 +795,8 @@ The amount of memory to give an instance of a data transform (Wasm) virtual mach --- +// end::data_transforms_per_function_memory_limit[] + === data_transforms_read_buffer_memory_percentage include::reference:partial$internal-use-property.adoc[] @@ -968,6 +1001,7 @@ Default number of quota tracking windows. --- +// tag::default_topic_partitions[] === default_topic_partitions Default number of partitions per topic. @@ -986,6 +1020,8 @@ Default number of partitions per topic. --- +// end::default_topic_partitions[] + === default_topic_replications Default replication factor for new topics. @@ -1066,6 +1102,7 @@ Disables the cluster recovery loop. --- +// tag::disable_metrics[] === disable_metrics Disable registering the metrics exposed on the internal `/metrics` endpoint. @@ -1080,6 +1117,9 @@ Disable registering the metrics exposed on the internal `/metrics` endpoint. --- +// end::disable_metrics[] + +// tag::disable_public_metrics[] === disable_public_metrics Disable registering the metrics exposed on the `/public_metrics` endpoint. @@ -1094,6 +1134,8 @@ Disable registering the metrics exposed on the `/public_metrics` endpoint. --- +// end::disable_public_metrics[] + === disk_reservation_percent The percentage of total disk capacity that Redpanda will avoid using. This applies both when cloud cache and log data share a disk, as well @@ -1166,6 +1208,7 @@ Limits the write rate for the controller log. --- +// tag::enable_idempotence[] === enable_idempotence Enable idempotent producers. @@ -1179,6 +1222,7 @@ Enable idempotent producers. *Default:* `true` --- +// end::enable_idempotence[] === enable_host_metrics @@ -1288,9 +1332,12 @@ Enable SASL authentication for Kafka connections. Authorization is required to m --- +// tag::enable_schema_id_validation[] === enable_schema_id_validation +ifndef::env-cloud[] include::reference:partial$enterprise-licensed-property.adoc[] +endif::[] Mode to enable server-side schema ID validation. @@ -1314,6 +1361,9 @@ Mode to enable server-side schema ID validation. --- +// end::enable_schema_id_validation[] + +// tag::enable_transactions[] === enable_transactions Enable transactions (atomic writes). @@ -1328,6 +1378,8 @@ Enable transactions (atomic writes). --- +// end::enable_transactions[] + === enable_usage Enables the usage tracking mechanism, storing windowed history of kafka/cloud_storage metrics over time. @@ -1356,6 +1408,7 @@ Whether new feature flags auto-activate after upgrades (true) or must wait for m --- +// tag::fetch_max_bytes[] === fetch_max_bytes Maximum number of bytes returned in a fetch request. @@ -1372,6 +1425,8 @@ Maximum number of bytes returned in a fetch request. --- +// end::fetch_max_bytes[] + === fetch_pid_d_coeff Derivative coefficient for fetch PID controller. @@ -1400,6 +1455,7 @@ Integral coefficient for fetch PID controller. --- +// tag::fetch_pid_max_debounce_ms[] === fetch_pid_max_debounce_ms The maximum debounce time the fetch PID controller will apply, in milliseconds. @@ -1418,6 +1474,8 @@ The maximum debounce time the fetch PID controller will apply, in milliseconds. --- +// end::fetch_pid_max_debounce_ms[] + === fetch_pid_p_coeff Proportional coefficient for fetch PID controller. @@ -1448,6 +1506,7 @@ A fraction, between 0 and 1, for the target reactor utilization of the fetch sch --- +// tag::fetch_read_strategy[] === fetch_read_strategy The strategy used to fulfill fetch requests. @@ -1468,6 +1527,9 @@ The strategy used to fulfill fetch requests. --- +// end::fetch_read_strategy[] + +// tag::fetch_reads_debounce_timeout[] === fetch_reads_debounce_timeout Time to wait for the next read in fetch requests when the requested minimum bytes was not reached. @@ -1486,6 +1548,8 @@ Time to wait for the next read in fetch requests when the requested minimum byte --- +// end::fetch_reads_debounce_timeout[] + === fetch_session_eviction_timeout_ms Time duration after which the inactive fetch session is removed from the fetch session cache. Fetch sessions are used to implement the incremental fetch requests where a consumer does not send all requested partitions to the server but the server tracks them for the consumer. @@ -1504,6 +1568,7 @@ Time duration after which the inactive fetch session is removed from the fetch s --- +// tag::group_initial_rebalance_delay[] === group_initial_rebalance_delay Delay added to the rebalance phase to wait for new members. @@ -1522,6 +1587,9 @@ Delay added to the rebalance phase to wait for new members. --- +// end::group_initial_rebalance_delay[] + +// tag::group_max_session_timeout_ms[] === group_max_session_timeout_ms The maximum allowed session timeout for registered consumers. Longer timeouts give consumers more time to process messages in between heartbeats at the cost of a longer time to detect failures. @@ -1540,6 +1608,9 @@ The maximum allowed session timeout for registered consumers. Longer timeouts gi --- +// end::group_max_session_timeout_ms[] + +// tag::group_min_session_timeout_ms[] === group_min_session_timeout_ms The minimum allowed session timeout for registered consumers. Shorter timeouts result in quicker failure detection at the cost of more frequent consumer heartbeating, which can overwhelm broker resources. @@ -1558,6 +1629,9 @@ The minimum allowed session timeout for registered consumers. Shorter timeouts r --- +// end::group_min_session_timeout_ms[] + +// tag::group_new_member_join_timeout[] === group_new_member_join_timeout Timeout for new member joins. @@ -1576,6 +1650,9 @@ Timeout for new member joins. --- +// end::group_new_member_join_timeout[] + +// tag::group_offset_retention_check_ms[] === group_offset_retention_check_ms Frequency rate at which the system should check for expired group offsets. @@ -1594,6 +1671,9 @@ Frequency rate at which the system should check for expired group offsets. --- +// end::group_offset_retention_check_ms[] + +// tag::group_offset_retention_sec[] === group_offset_retention_sec Consumer group offset retention seconds. To disable offset retention, set this to null. @@ -1612,6 +1692,9 @@ Consumer group offset retention seconds. To disable offset retention, set this t --- +// end::group_offset_retention_sec[] + +// tag::group_topic_partitions[] === group_topic_partitions Number of partitions in the internal group membership topic. @@ -1630,6 +1713,8 @@ Number of partitions in the internal group membership topic. --- +// end::group_topic_partitions[] + === health_manager_tick_interval How often the health manager runs. @@ -1699,10 +1784,7 @@ Proportional coefficient for the Iceberg backlog controller. Number of shares as **Related topics**: -- xref:manage:iceberg/use-iceberg-catalogs.adoc[] -- xref:manage:iceberg/query-iceberg-topics.adoc[] -- xref:manage:iceberg/redpanda-topics-iceberg-snowflake-catalog.adoc[] -- xref:manage:iceberg/topic-iceberg-integration.adoc[] +- xref:manage:iceberg/about-iceberg-topics.adoc[] --- @@ -1721,10 +1803,11 @@ Base path for the object-storage-backed Iceberg catalog. After Iceberg is enable **Related topics**: - xref:manage:iceberg/use-iceberg-catalogs.adoc[] -- xref:manage:iceberg/topic-iceberg-integration.adoc[] +- xref:manage:iceberg/about-iceberg-topics.adoc[] --- +// tag::iceberg_catalog_commit_interval_ms[] === iceberg_catalog_commit_interval_ms The frequency at which the Iceberg coordinator commits topic files to the catalog. This is the interval between commit transactions across all topics monitored by the coordinator, not the interval between individual commits. @@ -1744,14 +1827,15 @@ The frequency at which the Iceberg coordinator commits topic files to the catalo **Related topics**: - xref:manage:iceberg/use-iceberg-catalogs.adoc[] -- xref:manage:iceberg/redpanda-topics-iceberg-snowflake-catalog.adoc[] -- xref:manage:iceberg/topic-iceberg-integration.adoc[] --- +// end::iceberg_catalog_commit_interval_ms[] + +// tag::iceberg_catalog_type[] === iceberg_catalog_type -Iceberg catalog type that Redpanda will use to commit table metadata updates. Supported types: 'rest', 'object_storage'. +Iceberg catalog type that Redpanda will use to commit table metadata updates. Supported types: `rest`, `object_storage`. *Requires restart:* Yes @@ -1764,14 +1848,21 @@ Iceberg catalog type that Redpanda will use to commit table metadata updates. Su **Related topics**: - xref:manage:iceberg/use-iceberg-catalogs.adoc[] -- xref:manage:iceberg/redpanda-topics-iceberg-snowflake-catalog.adoc[] -- xref:manage:iceberg/topic-iceberg-integration.adoc[] --- +// end::iceberg_catalog_type[] + +// tag::iceberg_default_partition_spec[] === iceberg_default_partition_spec +ifndef::env-cloud[] Default value for the xref:reference:properties/topic-properties.adoc#redpanda-iceberg-partition-spec[`redpanda.iceberg.partition.spec`] topic property that determines the partition spec for the Iceberg table corresponding to the topic. +endif::[] + +ifdef::env-cloud[] +Default value for the `redpanda.iceberg.partition.spec` topic property that determines the partition spec for the Iceberg table corresponding to the topic. +endif::[] *Requires restart:* No @@ -1785,10 +1876,13 @@ Partitions the topic by extracting the hour from `redpanda.timestamp`, grouping **Related topics**: -- xref:manage:iceberg/topic-iceberg-integration.adoc#enable-iceberg-integration[Enable Iceberg] +- xref:manage:iceberg/about-iceberg-topics.adoc#enable-iceberg-integration[Enable Iceberg] --- +// end::iceberg_default_partition_spec[] + +// tag::iceberg_delete[] === iceberg_delete Default value for the `redpanda.iceberg.delete` topic property that determines if the corresponding Iceberg table is deleted upon deleting the topic. @@ -1803,10 +1897,12 @@ Default value for the `redpanda.iceberg.delete` topic property that determines i **Related topics**: -- xref:manage:iceberg/topic-iceberg-integration.adoc[] +- xref:manage:iceberg/about-iceberg-topics.adoc[] --- +// end::iceberg_delete[] + === iceberg_disable_automatic_snapshot_expiry Whether to disable automatic Iceberg snapshot expiry. This property may be useful if the Iceberg catalog expects to perform snapshot expiry on its own. @@ -1835,15 +1931,20 @@ Whether to disable tagging of Iceberg snapshots. These tags are used to ensure t **Related topics**: -- xref:manage:iceberg/use-iceberg-catalogs.adoc[] -- xref:manage:iceberg/query-iceberg-topics.adoc[] -- xref:manage:iceberg/topic-iceberg-integration.adoc[] +- xref:manage:iceberg/about-iceberg-topics.adoc[] --- +// tag::iceberg_enabled[] === iceberg_enabled +ifndef::env-cloud[] Enables the translation of topic data into Iceberg tables. Setting `iceberg_enabled` to `true` activates the feature at the cluster level, but each topic must also set the xref:reference:properties/topic-properties.adoc#redpanda-iceberg-enabled[`redpanda.iceberg.enabled`] topic-level property to `true` to use it. If `iceberg_enabled` is set to `false`, then the feature is disabled for all topics in the cluster, overriding any topic-level settings. +endif::[] + +ifdef::env-cloud[] +Enables the translation of topic data into Iceberg tables. Setting `iceberg_enabled` to `true` activates the feature at the cluster level, but each topic must also set the `redpanda.iceberg.enabled` topic-level property to `true` to use it. If `iceberg_enabled` is set to `false`, then the feature is disabled for all topics in the cluster, overriding any topic-level settings. +endif::[] *Requires restart:* Yes @@ -1855,16 +1956,22 @@ Enables the translation of topic data into Iceberg tables. Setting `iceberg_enab **Related topics**: -- xref:manage:iceberg/use-iceberg-catalogs.adoc[] -- xref:manage:iceberg/query-iceberg-topics.adoc[] -- xref:manage:iceberg/redpanda-topics-iceberg-snowflake-catalog.adoc[] -- xref:manage:iceberg/topic-iceberg-integration.adoc[] +- xref:manage:iceberg/about-iceberg-topics.adoc[] --- +// end::iceberg_enabled[] + +// tag::iceberg_invalid_record_action[] === iceberg_invalid_record_action +ifndef::env-cloud[] Default value for the xref:reference:properties/topic-properties.adoc#redpanda-iceberg-invalid-record-action[`redpanda.iceberg.invalid.record.action`] topic property. +endif::[] + +ifdef::env-cloud[] +Default value for the `redpanda.iceberg.invalid.record.action` topic property. +endif::[] *Requires restart:* No @@ -1874,10 +1981,13 @@ Default value for the xref:reference:properties/topic-properties.adoc#redpanda-i **Related topics**: -- xref:manage:iceberg/topic-iceberg-integration.adoc#manage-dead-letter-queue[Manage dead-letter queue] +- xref:manage:iceberg/about-iceberg-topics.adoc#manage-dead-letter-queue[Manage dead-letter queue] --- +// end::iceberg_invalid_record_action[] + +// tag::iceberg_rest_catalog_authentication_mode[] === iceberg_rest_catalog_authentication_mode The authentication mode for client requests made to the Iceberg catalog. Choose from: `none`, `bearer`, and `oauth2`. In `bearer` mode, the token specified in `iceberg_rest_catalog_token` is used unconditionally, and no attempts are made to refresh the token. In `oauth2` mode, the credentials specified in `iceberg_rest_catalog_client_id` and `iceberg_rest_catalog_client_secret` are used to obtain a bearer token from the URI defined by `iceberg_rest_catalog_oauth2_server_uri.`. @@ -1891,11 +2001,12 @@ The authentication mode for client requests made to the Iceberg catalog. Choose **Related topics**: - xref:manage:iceberg/use-iceberg-catalogs.adoc[] -- xref:manage:iceberg/redpanda-topics-iceberg-snowflake-catalog.adoc[] -- xref:manage:iceberg/topic-iceberg-integration.adoc[] --- +// end::iceberg_rest_catalog_authentication_mode[] + +// tag::iceberg_rest_catalog_client_id[] === iceberg_rest_catalog_client_id The client ID used to query the REST catalog API for the OAuth token. Required if catalog type is set to `rest`. @@ -1911,11 +2022,12 @@ The client ID used to query the REST catalog API for the OAuth token. Required i **Related topics**: - xref:manage:iceberg/use-iceberg-catalogs.adoc[] -- xref:manage:iceberg/redpanda-topics-iceberg-snowflake-catalog.adoc[] -- xref:manage:iceberg/topic-iceberg-integration.adoc[] --- +// end::iceberg_rest_catalog_client_id[] + +// tag::iceberg_rest_catalog_client_secret[] === iceberg_rest_catalog_client_secret Secret used with the client ID to query the OAuth token endpoint for Iceberg REST catalog authentication. Required if catalog type is set to `rest` and `iceberg_rest_catalog_authentication_mode` is set to `oauth2`. @@ -1931,11 +2043,12 @@ Secret used with the client ID to query the OAuth token endpoint for Iceberg RES **Related topics**: - xref:manage:iceberg/use-iceberg-catalogs.adoc[] -- xref:manage:iceberg/redpanda-topics-iceberg-snowflake-catalog.adoc[] -- xref:manage:iceberg/topic-iceberg-integration.adoc[] --- +// end::iceberg_rest_catalog_client_secret[] + +// tag::iceberg_rest_catalog_crl[] === iceberg_rest_catalog_crl The contents of a certificate revocation list for `iceberg_rest_catalog_trust`. Takes precedence over `iceberg_rest_catalog_crl_file`. @@ -1950,6 +2063,8 @@ The contents of a certificate revocation list for `iceberg_rest_catalog_trust`. --- +// end::iceberg_rest_catalog_crl[] + === iceberg_rest_catalog_crl_file Path to certificate revocation list for `iceberg_rest_catalog_trust_file`. @@ -1965,11 +2080,10 @@ Path to certificate revocation list for `iceberg_rest_catalog_trust_file`. **Related topics**: - xref:manage:iceberg/use-iceberg-catalogs.adoc[] -- xref:manage:iceberg/redpanda-topics-iceberg-snowflake-catalog.adoc[] -- xref:manage:iceberg/topic-iceberg-integration.adoc[] --- +// tag::iceberg_rest_catalog_endpoint[] === iceberg_rest_catalog_endpoint URL of Iceberg REST catalog endpoint. @@ -1985,11 +2099,12 @@ URL of Iceberg REST catalog endpoint. **Related topics**: - xref:manage:iceberg/use-iceberg-catalogs.adoc[] -- xref:manage:iceberg/redpanda-topics-iceberg-snowflake-catalog.adoc[] -- xref:manage:iceberg/topic-iceberg-integration.adoc[] --- +// end::iceberg_rest_catalog_endpoint[] + +// tag::iceberg_rest_catalog_oauth2_server_uri[] === iceberg_rest_catalog_oauth2_server_uri The OAuth URI used to retrieve access tokens for Iceberg REST catalog authentication. If left undefined, the deprecated Iceberg catalog endpoint `/v1/oauth/tokens` is used instead. @@ -2005,11 +2120,12 @@ The OAuth URI used to retrieve access tokens for Iceberg REST catalog authentica **Related topics**: - xref:manage:iceberg/use-iceberg-catalogs.adoc[] -- xref:manage:iceberg/redpanda-topics-iceberg-snowflake-catalog.adoc[] -- xref:manage:iceberg/topic-iceberg-integration.adoc[] --- +// end::iceberg_rest_catalog_oauth2_server_uri[] + +// tag::iceberg_rest_catalog_prefix[] === iceberg_rest_catalog_prefix Prefix part of the Iceberg REST catalog URL. Prefix is appended to the catalog path, for example `/v1/\{prefix}/namespaces`. @@ -2025,11 +2141,12 @@ Prefix part of the Iceberg REST catalog URL. Prefix is appended to the catalog p **Related topics**: - xref:manage:iceberg/use-iceberg-catalogs.adoc[] -- xref:manage:iceberg/redpanda-topics-iceberg-snowflake-catalog.adoc[] -- xref:manage:iceberg/topic-iceberg-integration.adoc[] --- +// end::iceberg_rest_catalog_prefix[] + +// tag::iceberg_rest_catalog_request_timeout_ms[] === iceberg_rest_catalog_request_timeout_ms Maximum length of time that Redpanda waits for a response from the REST catalog before aborting the request. @@ -2049,11 +2166,12 @@ Maximum length of time that Redpanda waits for a response from the REST catalog **Related topics**: - xref:manage:iceberg/use-iceberg-catalogs.adoc[] -- xref:manage:iceberg/redpanda-topics-iceberg-snowflake-catalog.adoc[] -- xref:manage:iceberg/topic-iceberg-integration.adoc[] --- +// end::iceberg_rest_catalog_request_timeout_ms[] + +// tag::iceberg_rest_catalog_token[] === iceberg_rest_catalog_token Token used to access the REST Iceberg catalog. If the token is present, Redpanda ignores credentials stored in the properties <> and <>. @@ -2071,14 +2189,19 @@ Required if <>. +The contents of a certificate chain to trust for the REST Iceberg catalog. + +ifndef::end-cloud[] +Takes precedence over <>. +endif::[] *Requires restart:* Yes @@ -2090,6 +2213,8 @@ The contents of a certificate chain to trust for the REST Iceberg catalog. Takes --- +// end::iceberg_rest_catalog_trust[] + === iceberg_rest_catalog_trust_file Path to a file containing a certificate chain to trust for the REST Iceberg catalog. @@ -2105,8 +2230,6 @@ Path to a file containing a certificate chain to trust for the REST Iceberg cata **Related topics**: - xref:manage:iceberg/use-iceberg-catalogs.adoc[] -- xref:manage:iceberg/redpanda-topics-iceberg-snowflake-catalog.adoc[] -- xref:manage:iceberg/topic-iceberg-integration.adoc[] --- @@ -2126,14 +2249,23 @@ Average size per partition of the datalake translation backlog that the backlog **Related topics**: -- xref:manage:iceberg/topic-iceberg-integration.adoc[] +- xref:manage:iceberg/about-iceberg-topics.adoc[] --- +// tag::iceberg_target_lag_ms[] === iceberg_target_lag_ms +ifndef::env-cloud[] Default value for the xref:reference:properties/topic-properties.adoc#redpanda-iceberg-target-lag-ms[`redpanda.iceberg.target.lag.ms`] topic property, which controls how often the data in an Iceberg table is refreshed with new data from the corresponding Redpanda topic. Redpanda attempts to commit all data produced to the topic within the lag target, subject to resource availability. +endif::[] + +ifdef::env-cloud[] +Default value for the `redpanda.iceberg.target.lag.ms` topic property, which controls how often the data in an Iceberg table is refreshed with new data from the corresponding Redpanda topic. Redpanda attempts to commit all data produced to the topic within the lag target, subject to resource availability. + +endif::[] + *Unit:* milliseconds *Requires restart:* No @@ -2148,13 +2280,12 @@ Default value for the xref:reference:properties/topic-properties.adoc#redpanda-i **Related topics**: -- xref:manage:iceberg/use-iceberg-catalogs.adoc[] -- xref:manage:iceberg/query-iceberg-topics.adoc[] -- xref:manage:iceberg/redpanda-topics-iceberg-snowflake-catalog.adoc[] -- xref:manage:iceberg/topic-iceberg-integration.adoc[] +- xref:manage:iceberg/about-iceberg-topics.adoc[] --- +// end::iceberg_target_lag_ms[] + === id_allocator_batch_size The ID allocator allocates messages in batches (each batch is a one log record) and then serves requests from memory without touching the log until the batch is exhausted. @@ -2256,6 +2387,7 @@ Time between cluster join retries in milliseconds. --- +// tag::kafka_batch_max_bytes[] === kafka_batch_max_bytes Maximum size of a batch processed by the server. If the batch is compressed, the limit applies to the compressed batch size. @@ -2273,6 +2405,7 @@ Maximum size of a batch processed by the server. If the batch is compressed, the *Default:* `1048576` --- +// end::kafka_batch_max_bytes[] === kafka_connection_rate_limit @@ -2439,6 +2572,7 @@ Kafka group recovery timeout. --- +// tag::kafka_max_bytes_per_fetch[] === kafka_max_bytes_per_fetch Limit fetch responses to this many bytes, even if the total of partition bytes limits is higher. @@ -2453,6 +2587,8 @@ Limit fetch responses to this many bytes, even if the total of partition bytes l --- +// end::kafka_max_bytes_per_fetch[] + === kafka_memory_batch_size_estimate_for_fetch The size of the batch used to estimate memory consumption for fetch requests, in bytes. Smaller sizes allow more concurrent fetch requests per shard. Larger sizes prevent running out of memory because of too many concurrent fetch requests. @@ -2560,6 +2696,7 @@ Update frequency for Kafka queue depth control. --- +// tag::kafka_qdc_enable[] === kafka_qdc_enable Enable Kafka queue depth control. @@ -2574,6 +2711,8 @@ Enable Kafka queue depth control. --- +// end::kafka_qdc_enable[] + === kafka_qdc_idle_depth Queue depth when idleness is detected in Kafka queue depth control. @@ -2680,6 +2819,7 @@ Window size for Kafka queue depth control latency tracking. --- +// tag::kafka_request_max_bytes[] === kafka_request_max_bytes Maximum size of a single request processed using the Kafka API. @@ -2698,6 +2838,8 @@ Maximum size of a single request processed using the Kafka API. --- +// end::kafka_request_max_bytes[] + === kafka_rpc_server_stream_recv_buf Maximum size of the user-space receive buffer. If `null`, this limit is not applied. @@ -2784,6 +2926,7 @@ Per-shard capacity of the cache for validating schema IDs. --- +// tag::kafka_tcp_keepalive_timeout[] === kafka_tcp_keepalive_timeout TCP keepalive idle timeout in seconds for Kafka connections. This describes the timeout between TCP keepalive probes that the remote site successfully acknowledged. Refers to the TCP_KEEPIDLE socket option. When changed, applies to new connections only. @@ -2802,6 +2945,8 @@ TCP keepalive idle timeout in seconds for Kafka connections. This describes the --- +// end::kafka_tcp_keepalive_timeout[] + === kafka_tcp_keepalive_probe_interval_seconds TCP keepalive probe interval in seconds for Kafka connections. This describes the timeout between unacknowledged TCP keepalives. Refers to the TCP_KEEPINTVL socket option. When changed, applies to new connections only. @@ -3099,6 +3244,7 @@ Period at which to log a warning about using unsafe strings containing control c --- +// tag::log_cleanup_policy[] === log_cleanup_policy Default cleanup policy for topic logs. @@ -3115,6 +3261,8 @@ The topic property xref:./topic-properties.adoc#cleanuppolicy[`cleanup.policy`] --- +// end::log_cleanup_policy[] + === log_compaction_adjacent_merge_self_compaction_count The number of self compactions that must occur before an adjacent compaction is attempted in the log. If set to `null`, every segment in the log must be self-compacted before an adjacent compaction is attempted. @@ -3243,6 +3391,7 @@ The topic property xref:./topic-properties.adoc#messagetimestamptype[`message.ti --- +// tag::log_retention_ms[] === log_retention_ms The amount of time to keep a log file before deleting it (in milliseconds). If set to `-1`, no time limit is applied. This is a cluster-wide default when a topic does not set or disable xref:./topic-properties.adoc#retentionms[`retention.ms`]. @@ -3259,6 +3408,9 @@ The amount of time to keep a log file before deleting it (in milliseconds). If s --- +// end::log_retention_ms[] + +// tag::log_segment_ms[] === log_segment_ms Default lifetime of log segments. If `null`, the property is disabled, and no default lifetime is set. Any value under 60 seconds (60000 ms) is rejected. This property can also be set in the Kafka API using the Kafka-compatible alias, `log.roll.ms`. @@ -3284,6 +3436,8 @@ The topic property xref:./topic-properties.adoc#segmentms[`segment.ms`] override --- +// end::log_segment_ms[] + === log_segment_ms_max Upper bound on topic `segment.ms`: higher values will be clamped to this value. @@ -3320,6 +3474,7 @@ Lower bound on topic `segment.ms`: lower values will be clamped to this value. --- +// tag::log_segment_size[] === log_segment_size Default log segment size in bytes for topics which do not set `segment.bytes`. @@ -3336,6 +3491,8 @@ Default log segment size in bytes for topics which do not set `segment.bytes`. --- +// end::log_segment_size[] + === log_segment_size_jitter_percent Random variation to the segment size limit used for each partition. @@ -4166,6 +4323,7 @@ Raft I/O timeout. --- +// tag::raft_learner_recovery_rate[] === raft_learner_recovery_rate Raft learner recovery rate limit. Throttles the rate of data communicated to nodes (learners) that need to catch up to leaders. This rate limit is placed on a node sending data to a recovering node. Each sending node is limited to this rate. The recovering node accepts data as fast as possible according to the combined limits of all healthy nodes in the cluster. For example, if two nodes are sending data to the recovering node, and `raft_learner_recovery_rate` is 100 MB/sec, then the recovering node will recover at a rate of 200 MB/sec. @@ -4180,6 +4338,8 @@ Raft learner recovery rate limit. Throttles the rate of data communicated to nod --- +// end::raft_learner_recovery_rate[] + === raft_max_buffered_follower_append_entries_bytes_per_shard The total size of append entry requests that may be cached per shard, using the Raft-buffered protocol. When an entry is cached, the leader can continue serving requests because the ordering of the cached requests cannot change. When the total size of cached requests reaches the set limit, back pressure is applied to throttle producers. @@ -4527,6 +4687,7 @@ Timeout for append entry requests issued while replicating entries. --- +// tag::retention_bytes[] === retention_bytes Default maximum number of bytes per partition on disk before triggering deletion of the oldest messages. If `null` (the default value), no limit is applied. @@ -4545,6 +4706,8 @@ The topic property xref:./topic-properties.adoc#retentionbytes[`retention.bytes` --- +// end::retention_bytes[] + === retention_local_strict Flag to allow Tiered Storage topics to expand to consumable retention policy limits. When this flag is enabled, non-local retention settings are used, and local retention settings are used to inform data removal policies in low-disk space scenarios. @@ -4573,6 +4736,7 @@ Trim log data when a cloud topic reaches its local retention limit. When this op --- +// tag::retention_local_target_bytes_default[] === retention_local_target_bytes_default Local retention size target for partitions of topics with object storage write enabled. If `null`, the property is disabled. @@ -4597,6 +4761,8 @@ NOTE: Both `retention_local_target_bytes_default` and `retention_local_target_ms --- +// end::retention_local_target_bytes_default[] + === retention_local_target_capacity_bytes The target capacity (in bytes) that log storage will try to use before additional retention rules take over to trim data to meet the target. When no target is specified, storage usage is unbounded. @@ -4635,6 +4801,7 @@ NOTE: Redpanda Data recommends setting only one of <>. @@ -5520,6 +5701,9 @@ Delete segments older than this age. To ensure transaction state is retained as --- +// end::transaction_coordinator_delete_retention_ms[] + +// tag::transaction_coordinator_log_segment_size[] === transaction_coordinator_log_segment_size The size (in bytes) each log segment should be. @@ -5536,6 +5720,9 @@ The size (in bytes) each log segment should be. --- +// end::transaction_coordinator_log_segment_size[] + +// tag::transaction_coordinator_partitions[] === transaction_coordinator_partitions Number of partitions for transactions coordinator. @@ -5554,6 +5741,9 @@ Number of partitions for transactions coordinator. --- +// end::transaction_coordinator_partitions[] + +// tag::transaction_max_timeout_ms[] === transaction_max_timeout_ms The maximum allowed timeout for transactions. If a client-requested transaction timeout exceeds this configuration, the broker returns an error during transactional producer initialization. This guardrail prevents hanging transactions from blocking consumer progress. @@ -5572,6 +5762,9 @@ The maximum allowed timeout for transactions. If a client-requested transaction --- +// end::transaction_max_timeout_ms[] + +// tag::transactional_id_expiration_ms[] === transactional_id_expiration_ms Expiration time of producer IDs. Measured starting from the time of the last write until now for a given ID. @@ -5590,6 +5783,8 @@ Expiration time of producer IDs. Measured starting from the time of the last wri --- +// end::transactional_id_expiration_ms[] + === tx_timeout_delay_ms Delay before scheduling the next check for timed out transactions. @@ -5748,9 +5943,12 @@ Timeout to wait for leadership in metadata cache. --- + === write_caching_default -The default write caching mode to apply to user topics. Write caching acknowledges a message as soon as it is received and acknowledged on a majority of brokers, without waiting for it to be written to disk. With `acks=all`, this provides lower latency while still ensuring that a majority of brokers acknowledge the write. Fsyncs follow <> and <>, whichever is reached first. +The default write caching mode to apply to user topics. Write caching acknowledges a message as soon as it is received and acknowledged on a majority of brokers, without waiting for it to be written to disk. With `acks=all`, this provides lower latency while still ensuring that a majority of brokers acknowledge the write. + +Fsyncs follow <> and <>, whichever is reached first. The `write_caching_default` cluster property can be overridden with the xref:reference:properties/topic-properties.adoc#writecaching[`write.caching`] topic property. @@ -5772,6 +5970,7 @@ The `write_caching_default` cluster property can be overridden with the xref:ref --- +// tag::zstd_decompress_workspace_bytes[] === zstd_decompress_workspace_bytes Size of the zstd decompression workspace. @@ -5788,3 +5987,5 @@ Size of the zstd decompression workspace. --- +// end::zstd_decompress_workspace_bytes[] + diff --git a/modules/reference/pages/properties/topic-properties.adoc b/modules/reference/pages/properties/topic-properties.adoc index 482c865f46..c8fd9b57b3 100644 --- a/modules/reference/pages/properties/topic-properties.adoc +++ b/modules/reference/pages/properties/topic-properties.adoc @@ -6,6 +6,7 @@ A topic-level property sets a Redpanda or Kafka configuration for a particular t Many topic-level properties have corresponding xref:manage:cluster-maintenance/cluster-property-configuration.adoc[cluster properties] that set a default value for all topics of a cluster. To customize the value for a topic, you can set a topic-level property that overrides the value of the corresponding cluster property. + NOTE: All topic properties take effect immediately after being set. |=== @@ -542,7 +543,7 @@ Enable the Iceberg integration for the topic. You can choose one of three modes. **Related topics**: -- xref:manage:topic-iceberg-integration.adoc[] +- xref:manage:iceberg/about-iceberg-topics.adoc[] ==== redpanda.iceberg.delete @@ -552,7 +553,7 @@ Whether the corresponding Iceberg table is deleted upon deleting the topic. **Related topics**: -- xref:manage:topic-iceberg-integration.adoc[] +- xref:manage:iceberg/about-iceberg-topics.adoc[] ==== redpanda.iceberg.partition.spec @@ -562,7 +563,7 @@ The https://iceberg.apache.org/docs/nightly/partitioning/[partitioning^] specifi **Related topics**: -- xref:manage:topic-iceberg-integration.adoc[] +- xref:manage:iceberg/about-iceberg-topics.adoc[] ==== redpanda.iceberg.invalid.record.action @@ -577,7 +578,7 @@ Whether to write invalid records to a dead-letter queue (DLQ). **Related topics**: -- xref:manage:topic-iceberg-integration.adoc[] +- xref:manage:iceberg/about-iceberg-topics.adoc[] === Redpanda topic properties @@ -645,3 +646,4 @@ If both `delete.retention.ms` and the cluster property config_ref:tombstone_rete - xref:develop:produce-data/configure-producers.adoc[Configure Producers] - xref:develop:config-topics.adoc[Manage Topics] +