From 25ceb1a4b525c0df60d8b4945109b3753628604f Mon Sep 17 00:00:00 2001 From: Angela Simms Date: Wed, 5 Feb 2025 17:13:53 +0000 Subject: [PATCH 1/3] add warning for updating cleanup policy --- modules/develop/pages/config-topics.adoc | 2 ++ modules/develop/partials/cleanup-policy-warning.adoc | 6 ++++++ .../pages/cluster-maintenance/compaction-settings.adoc | 2 ++ modules/manage/pages/kubernetes/k-manage-topics.adoc | 2 ++ modules/reference/pages/properties/topic-properties.adoc | 4 ++++ 5 files changed, 16 insertions(+) create mode 100644 modules/develop/partials/cleanup-policy-warning.adoc diff --git a/modules/develop/pages/config-topics.adoc b/modules/develop/pages/config-topics.adoc index c9843152b4..dd6bb5b286 100644 --- a/modules/develop/pages/config-topics.adoc +++ b/modules/develop/pages/config-topics.adoc @@ -99,6 +99,8 @@ The cleanup policy determines how to clean up the partition log files when they Unlike compacted topics, which keep only the most recent message for a given key, topics configured with a `delete` cleanup policy provide a running history of all changes for those topics. +include::develop:partial$cleanup-policy-warning.adoc[] + For example, to change a topic's policy to `compact`, run: [,bash] diff --git a/modules/develop/partials/cleanup-policy-warning.adoc b/modules/develop/partials/cleanup-policy-warning.adoc new file mode 100644 index 0000000000..330841e830 --- /dev/null +++ b/modules/develop/partials/cleanup-policy-warning.adoc @@ -0,0 +1,6 @@ +ifndef::env-cloud[] +WARNING: Changing the default cleanup policy of Redpanda internal topics can cause unexpected errors and Kafka Connect connector failures. +endif::[] +ifdef::env-cloud[] +WARNING: Changing the default cleanup policy of Redpanda internal topics (`_internal_connectors_offsets`, `_internal_connectors_configs`, and `_internal_connectors_status`) can cause unexpected errors and Kafka Connect connector failures. +endif::[] \ No newline at end of file diff --git a/modules/manage/pages/cluster-maintenance/compaction-settings.adoc b/modules/manage/pages/cluster-maintenance/compaction-settings.adoc index 0dfb0bc8ff..b984f52495 100644 --- a/modules/manage/pages/cluster-maintenance/compaction-settings.adoc +++ b/modules/manage/pages/cluster-maintenance/compaction-settings.adoc @@ -26,6 +26,8 @@ Compaction policy may be applied to a cluster or to an individual topic. If both * `compact`: This triggers only cleanup of records with multiple versions. A record that represents the only version for a given key is not deleted. * `compact,delete`: This combines both policies, deleting records exceeding the retention period while compacting multiple versions of records. +include::develop:partial$cleanup-policy-warning.adoc[] + == Tombstone record removal Compaction also enables deletion of existing records through tombstones. For example, as data is deleted from a source system, clients produce a tombstone record to the log. A tombstone contains a key and the value `null`. Tombstones signal to brokers and consumers that records with the same key prior to it in the log should be deleted. diff --git a/modules/manage/pages/kubernetes/k-manage-topics.adoc b/modules/manage/pages/kubernetes/k-manage-topics.adoc index 4804ebea2f..a868e5beb9 100644 --- a/modules/manage/pages/kubernetes/k-manage-topics.adoc +++ b/modules/manage/pages/kubernetes/k-manage-topics.adoc @@ -126,6 +126,8 @@ The cleanup policy determines how to manage the partition log files when they re If the cleanup policy is `delete` or `compact,delete` you can <>. +include::develop:partial$cleanup-policy-warning.adoc[] + === Configure Iceberg topics Iceberg topics allow you to create a seamless integration with Apache Iceberg by writing records in Redpanda topics to object storage in Iceberg format. diff --git a/modules/reference/pages/properties/topic-properties.adoc b/modules/reference/pages/properties/topic-properties.adoc index dadf9abd59..d71de4316c 100644 --- a/modules/reference/pages/properties/topic-properties.adoc +++ b/modules/reference/pages/properties/topic-properties.adoc @@ -171,6 +171,8 @@ The cleanup policy to apply for log segments of a topic. When `cleanup.policy` is set, it overrides the cluster property xref:cluster-properties.adoc#log_cleanup_policy[`log_cleanup_policy`] for the topic. +include::develop:partial$cleanup-policy-warning.adoc[] + **Default**: `[delete]` **Values**: @@ -179,6 +181,8 @@ When `cleanup.policy` is set, it overrides the cluster property xref:cluster-pro - `[compact]` - Deletes data according to a key-based retention policy, discarding all but the latest value for each key. - `[compact,delete]` - The latest values are kept for each key, while the remaining data is deleted according to retention limits. + + **Related topics**: - xref:manage:cluster-maintenance/disk-utilization.adoc#configure-segment-size[Configure segment size] From 22ae640e756665e6db99533a942327aa5b901852 Mon Sep 17 00:00:00 2001 From: Angela Simms Date: Mon, 24 Mar 2025 13:31:21 +0000 Subject: [PATCH 2/3] Redraft --- modules/develop/pages/config-topics.adoc | 2 +- modules/develop/partials/cleanup-policy-warning.adoc | 6 ------ modules/develop/partials/topic-properties-warning.adoc | 1 + .../pages/cluster-maintenance/compaction-settings.adoc | 2 +- modules/manage/pages/kubernetes/k-manage-topics.adoc | 2 +- modules/reference/pages/properties/topic-properties.adoc | 2 +- 6 files changed, 5 insertions(+), 10 deletions(-) delete mode 100644 modules/develop/partials/cleanup-policy-warning.adoc create mode 100644 modules/develop/partials/topic-properties-warning.adoc diff --git a/modules/develop/pages/config-topics.adoc b/modules/develop/pages/config-topics.adoc index dd6bb5b286..347217b5cc 100644 --- a/modules/develop/pages/config-topics.adoc +++ b/modules/develop/pages/config-topics.adoc @@ -99,7 +99,7 @@ The cleanup policy determines how to clean up the partition log files when they Unlike compacted topics, which keep only the most recent message for a given key, topics configured with a `delete` cleanup policy provide a running history of all changes for those topics. -include::develop:partial$cleanup-policy-warning.adoc[] +include::develop:partial$topic-properties-warning.adoc[] For example, to change a topic's policy to `compact`, run: diff --git a/modules/develop/partials/cleanup-policy-warning.adoc b/modules/develop/partials/cleanup-policy-warning.adoc deleted file mode 100644 index 330841e830..0000000000 --- a/modules/develop/partials/cleanup-policy-warning.adoc +++ /dev/null @@ -1,6 +0,0 @@ -ifndef::env-cloud[] -WARNING: Changing the default cleanup policy of Redpanda internal topics can cause unexpected errors and Kafka Connect connector failures. -endif::[] -ifdef::env-cloud[] -WARNING: Changing the default cleanup policy of Redpanda internal topics (`_internal_connectors_offsets`, `_internal_connectors_configs`, and `_internal_connectors_status`) can cause unexpected errors and Kafka Connect connector failures. -endif::[] \ No newline at end of file diff --git a/modules/develop/partials/topic-properties-warning.adoc b/modules/develop/partials/topic-properties-warning.adoc new file mode 100644 index 0000000000..e8ec219912 --- /dev/null +++ b/modules/develop/partials/topic-properties-warning.adoc @@ -0,0 +1 @@ +WARNING: Modifying the properties of topics that are created and managed by Redpanda applications can cause unexpected errors and connector and cluster failures. \ No newline at end of file diff --git a/modules/manage/pages/cluster-maintenance/compaction-settings.adoc b/modules/manage/pages/cluster-maintenance/compaction-settings.adoc index b984f52495..33226b4069 100644 --- a/modules/manage/pages/cluster-maintenance/compaction-settings.adoc +++ b/modules/manage/pages/cluster-maintenance/compaction-settings.adoc @@ -26,7 +26,7 @@ Compaction policy may be applied to a cluster or to an individual topic. If both * `compact`: This triggers only cleanup of records with multiple versions. A record that represents the only version for a given key is not deleted. * `compact,delete`: This combines both policies, deleting records exceeding the retention period while compacting multiple versions of records. -include::develop:partial$cleanup-policy-warning.adoc[] +include::develop:partial$topic-properties-warning.adoc[] == Tombstone record removal diff --git a/modules/manage/pages/kubernetes/k-manage-topics.adoc b/modules/manage/pages/kubernetes/k-manage-topics.adoc index a868e5beb9..91f6d914bb 100644 --- a/modules/manage/pages/kubernetes/k-manage-topics.adoc +++ b/modules/manage/pages/kubernetes/k-manage-topics.adoc @@ -126,7 +126,7 @@ The cleanup policy determines how to manage the partition log files when they re If the cleanup policy is `delete` or `compact,delete` you can <>. -include::develop:partial$cleanup-policy-warning.adoc[] +include::develop:partial$topic-properties-warning.adoc[] === Configure Iceberg topics diff --git a/modules/reference/pages/properties/topic-properties.adoc b/modules/reference/pages/properties/topic-properties.adoc index d71de4316c..cf83eb25da 100644 --- a/modules/reference/pages/properties/topic-properties.adoc +++ b/modules/reference/pages/properties/topic-properties.adoc @@ -171,7 +171,7 @@ The cleanup policy to apply for log segments of a topic. When `cleanup.policy` is set, it overrides the cluster property xref:cluster-properties.adoc#log_cleanup_policy[`log_cleanup_policy`] for the topic. -include::develop:partial$cleanup-policy-warning.adoc[] +include::develop:partial$topic-properties-warning.adoc[] **Default**: `[delete]` From 69323bd6902e533309d3546a85602d442a05e2f6 Mon Sep 17 00:00:00 2001 From: Angela Simms Date: Mon, 24 Mar 2025 13:37:25 +0000 Subject: [PATCH 3/3] hyperlint suggestion --- modules/develop/partials/topic-properties-warning.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/develop/partials/topic-properties-warning.adoc b/modules/develop/partials/topic-properties-warning.adoc index e8ec219912..62fef970c7 100644 --- a/modules/develop/partials/topic-properties-warning.adoc +++ b/modules/develop/partials/topic-properties-warning.adoc @@ -1 +1 @@ -WARNING: Modifying the properties of topics that are created and managed by Redpanda applications can cause unexpected errors and connector and cluster failures. \ No newline at end of file +WARNING: Modifying the properties of topics that are created and managed by Redpanda applications can cause unexpected errors. This may lead to connector and cluster failures. \ No newline at end of file