Tombstone retention (#829)

Co-authored-by: Joyce Fee <[email protected]>

Author: kbatuigas

modules/develop/pages/kafka-clients.adoc

@@ -54,7 +54,6 @@ ifdef::env-cloud[]
 endif::[]
 ifndef::env-cloud[]
 +
-* The `delete.retention.ms` topic configuration in Kafka is not supported. Tombstone markers are not removed for topics with a `compact` xref:develop:config-topics.adoc#change-the-cleanup-policy[cleanup policy]. Redpanda only deletes tombstone markers when topics with a cleanup policy of `compact,delete` have reached their xref:manage:cluster-maintenance/disk-utilization.adoc#configure-message-retention[retention limits].
 * Quotas per user for bandwidth and API request rates. However, xref:manage:cluster-maintenance/manage-throughput.adoc#client-throughput-limits[quotas per client and per client group] using AlterClientQuotas and DescribeClientQuotas APIs are supported.
 endif::[]
 

modules/get-started/pages/whats-new.adoc

@@ -7,6 +7,10 @@ This topic includes new content added in version {page-component-version}. For a
 * xref:redpanda-cloud:get-started:whats-new-cloud.adoc[] 
 * xref:redpanda-cloud:get-started:cloud-overview.adoc#redpanda-cloud-vs-self-managed-feature-compatibility[Redpanda Cloud vs Self-Managed feature compatibility]
 
+== Tombstone removal 
+
+Redpanda now supports the Kafka `delete.retention.ms` topic configuration. You can specify how long Redpanda keeps xref:manage:cluster-maintenance/compaction-settings.adoc#tombstone-record-removal[tombstone records] for compacted topics by setting `delete.retention.ms` at the topic level, or `tombstone_retention_ms` at the cluster level.
+
 == Mountable topics
 
 For topics with Tiered Storage enabled, you can unmount a topic to safely detach it from a cluster and keep the topic data in the cluster's object storage bucket or container. You can mount the detached topic to either the same origin cluster, or a different one. This allows you to hibernate a topic and free up system resources taken up by the topic, or migrate a topic to a different cluster. See xref:manage:mountable-topics.adoc[Mountable topics] for details.
@@ -58,3 +62,4 @@ The following cluster properties are new in this version:
 
 * xref:reference:properties/cluster-properties.adoc#default_leaders_preference[`default_leaders_preference`]
 * xref:reference:properties/cluster-properties.adoc#rpk_path[`rpk_path`]
+* xref:reference:properties/cluster-properties.adoc#tombstone_retention_ms[`tombstone_retention_ms`]

modules/manage/pages/cluster-maintenance/compaction-settings.adoc

@@ -6,25 +6,77 @@ Configure compaction for your cluster to optimize storage utilization.
 
 == Redpanda compaction overview
 
-Compaction is an optional mechanism intended to reduce the storage needs of Redpanda topics. You can enable compaction through configuration of a cluster or topic's cleanup policy. When compaction is enabled as part of the cleanup policy, a background process executes on a pre-set interval to perform compaction operations. When triggered for a partition, the process purges older versions of messages for a given key and only retains the most recent message in that partition. This is done by analyzing closed segments in the partition, copying the most recent messages for each key into a new segment, then deleting the source segments.
+Compaction is an optional mechanism intended to reduce the storage needs of Redpanda topics. You can enable compaction through configuration of a cluster or topic's cleanup policy. When compaction is enabled as part of the cleanup policy, a background process executes on a pre-set interval to perform compaction operations. When triggered for a partition, the process purges older versions of records for a given key and only retains the most recent record in that partition. This is done by analyzing closed segments in the partition, copying the most recent records for each key into a new segment, then deleting the source segments.
 
 image::shared:compaction-example.png[Example of topic compaction]
 
-This diagram provides an illustration of a compacted topic. Imagine a remote sensor network that uses image recognition to track appearances of red pandas in a geographic area. The sensor network employs special devices that send messages to a topic when they detect one. You might enable compaction to reduce topic storage while still maintaining a record in the topic of the last time each device saw a red panda, perhaps to see if they stop frequenting a given area. The left side of the diagram shows all messages sent across the topic. The right side illustrates the results of compaction; older messages for certain keys are deleted from the message log.
+This diagram illustrates a compacted topic. Imagine a remote sensor network that uses image recognition to track appearances of red pandas in a geographic area. The sensor network employs special devices that send records to a topic when they detect one. You might enable compaction to reduce topic storage while still maintaining a record in the topic of the last time each device saw a red panda, perhaps to see if they stop frequenting a given area. The left side of the diagram shows all records sent across the topic. The right side illustrates the results of compaction; older records for certain keys are deleted from the log.
 
-NOTE: If your application requires consuming every message for a given key, consider using the `delete` xref:develop:config-topics#change-the-cleanup-policy.adoc[cleanup policy] instead.
+NOTE: If your application requires consuming every record for a given key, consider using the `delete` xref:develop:config-topics#change-the-cleanup-policy.adoc[cleanup policy] instead.
 
-IMPORTANT:  When using Tiered Storage, compaction functions at the local storage level. As long as a segment remains in local storage, its messages are eligible for compaction. Once a segment is uploaded to tiered storage and removed from local storage it is not retrieved for further compaction operations. A key may therefore appear in multiple segments between Tiered Storage and local storage.
+IMPORTANT:  When using xref:manage:tiered-storage.adoc[Tiered Storage], compaction functions at the local storage level. As long as a segment remains in local storage, its records are eligible for compaction. Once a segment is uploaded to object storage and removed from local storage it is not retrieved for further compaction operations. A key may therefore appear in multiple segments between Tiered Storage and local storage.
 
 While compaction reduces storage needs, Redpanda's compaction (just like Kafka's) does not guarantee perfect de-duplication of a topic. It represents a best effort mechanism to reduce storage needs but duplicates of a key may still exist within a topic. Compaction is not a complete topic operation, either, since it operates on subsets of each partition within the topic.
 
 == Configure cleanup policy
 
 Compaction policy may be applied to a cluster or to an individual topic. If both are set, the topic-level policy overrides the cluster-level policy. The cluster-level xref:reference:cluster-properties.adoc#log_cleanup_policy[`log_cleanup_policy`] and the topic-level xref:reference:topic-properties.adoc#cleanuppolicy[`cleanup.policy`] support the following three options:
 
-* `delete`: Messages are deleted from the topic once the specified retention period (time and/or size allocations) is exceeded. This is the default mechanism and is analogous to disabling compaction.
-* `compact`: This triggers only cleanup of messages with multiple versions. A message that represents the only version for a given key is not deleted.
-* `compact,delete`: This combines both policies, deleting messages exceeding the retention period while compacting multiple versions of messages.
+* `delete`: Records are deleted from the topic once the specified retention period (time and/or size allocations) is exceeded. This is the default mechanism and is analogous to disabling compaction.
+* `compact`: This triggers only cleanup of records with multiple versions. A record that represents the only version for a given key is not deleted.
+* `compact,delete`: This combines both policies, deleting records exceeding the retention period while compacting multiple versions of records.
+
+== Tombstone record removal
+
+Compaction also enables deletion of existing records through tombstones. For example, as data is deleted from a source system, clients produce a tombstone record to the log. A tombstone contains a key and the value `null`. Tombstones signal to brokers and consumers that records with the same key prior to it in the log should be deleted. 
+
+You can specify how long Redpanda keeps these tombstones for compacted topics using both a cluster configuration property config_ref:tombstone_retention_ms,true,properties/cluster-properties[] and a topic configuration property xref:reference:properties/topic-properties.adoc#deleteretentionms[`delete.retention.ms`]. If both are set, the topic-level tombstone retention policy overrides the cluster-level policy.
+
+[NOTE]
+====
+Redpanda does not currently remove tombstone records for compacted topics with Tiered Storage enabled. 
+
+You cannot enable `tombstone_retention_ms` if you have enabled any of the Tiered Storage cluster properties `cloud_storage_enabled`, `cloud_storage_enable_remote_read`, and `cloud_storage_enable_remote_write`.
+
+On the topic level, you cannot enable `delete.retention.ms` at the same time as the Tiered Storage topic configuration properties `redpanda.remote.read` and `redpanda.remote.write`.
+====
+
+To set the cluster-level tombstone retention policy, run the command:
+
+[,bash]
+----
+rpk cluster config set tombstone_retention_ms=100
+----
+
+You can unset the tombstone retention policy for a topic so it inherits the cluster-wide default policy:
+
+[,bash]
+----
+rpk topic alter-config <topic-name> --delete delete.retention.ms
+----
+
+To override the cluster-wide default for a specific topic:
+
+[,bash]
+----
+rpk topic alter-config <topic-name> --set delete.retention.ms=5
+----
+
+To disable tombstone removal for a specific topic:
+
+[,bash]
+----
+rpk topic alter-config <topic-name> --set delete.retention.ms=-1
+----
+
+Redpanda removes tombstones as follows:
+
+* For topics with a `compact` only cleanup policy: Tombstones are removed when the topic exceeds the tombstone retention limit. The `delete.retention.ms` or `tombstone_retention_ms` values therefore also set the time bound that a consumer has in order to see a complete view of the log with tombstones present before they are removed.
+* For topics with a `compact,delete` cleanup policy: Both the tombstone retention policy and standard garbage collection can remove tombstone records.
+
+If obtaining a complete snapshot of the log, including tombstone records, is important to your consumers, set the tombstone retention value such that consumers have enough time for their reads to complete before tombstones are removed. Consumers may not see tombstones if their reads take longer than `delete.retention.ms` and `tombstone_retention_ms`. The trade-offs to ensuring tombstone visibility to consumers are increased disk usage and potentially slower compaction. 
+
+On the other hand, if more frequent cleanup of tombstones is important for optimizing workloads and space management, consider setting a shorter tombstone retention, for example the typical default of 24 hours (86400000 ms).
 
 == Compaction policy settings
 

modules/reference/pages/properties/topic-properties.adoc

@@ -547,6 +547,25 @@ NOTE: Although `replication.factor` isn't returned or displayed by xref:referenc
 
 ---
 
+[[deleteretentionms]]
+==== delete.retention.ms
+
+The retention time for tombstone records in a compacted topic. Redpanda removes tombstone records after the retention limit is exceeded.
+
+If you have enabled Tiered Storage and set <<redpandaremoteread,`redpanda.remote.read`>> or <<redpandaremotewrite,`redpanda.remote.write`>> for the topic, you cannot enable tombstone removal. 
+
+If both `delete.retention.ms` and the cluster property config_ref:tombstone_retention_ms,true,properties/cluster-properties[] are set, `delete.retention.ms` overrides the cluster level tombstone retention for an individual topic.
+
+*Unit:* milliseconds
+
+**Default**: null
+
+**Related topics**:
+
+- xref:manage:cluster-maintenance/compaction-settings.adoc#tombstone-record-removal[Tombstone record removal]
+
+---
+
 == Related topics
 
 - xref:develop:produce-data/configure-producers.adoc[Configure Producers]