Leader pinning (#809)

Co-authored-by: Angela Simms <[email protected]>
Co-authored-by: Joyce Fee <[email protected]>
Co-authored-by: Paulo Borges <[email protected]>

Author: 4 people

modules/ROOT/nav.adoc

@@ -26,6 +26,7 @@
 ** xref:develop:produce-data/index.adoc[Produce Data]
 *** xref:develop:produce-data/configure-producers.adoc[]
 *** xref:develop:produce-data/idempotent-producers.adoc[Idempotent Producers]
+*** xref:develop:produce-data/leader-pinning.adoc[]
 ** xref:develop:consume-data/index.adoc[Consume Data]
 *** xref:develop:consume-data/consumer-offsets.adoc[Consumer Offsets]
 *** xref:develop:consume-data/follower-fetching.adoc[Follower Fetching]

modules/develop/pages/produce-data/leader-pinning.adoc

@@ -0,0 +1,51 @@
+= Leader Pinning
+// tag::single-source[]
+:description: Learn about leader pinning and how to configure a preferred partition leader location based on cloud availability zones or regions.
+
+Produce requests that write data to Redpanda topics go through the topic partition leader, which syncs messages across its follower replicas. For a Redpanda cluster deployed across multiple availability zones (AZs), leader pinning ensures that a topic's partition leaders are geographically closer to clients, which helps decrease networking costs and guarantees lower latency.
+
+If consumers are located in the same preferred region or AZ for leader pinning, and you have not set up xref:develop:consume-data/follower-fetching.adoc[follower fetching], leader pinning can also help reduce networking costs on consume requests.
+
+== Prerequisites
+
+[NOTE]
+====
+include::shared:partial$enterprise-license.adoc[]
+====
+
+Before you can enable leader pinning, you must xref:manage:rack-awareness.adoc#configure-rack-awareness[configure rack awareness] on the cluster. If the config_ref:enable_rack_awareness,true,properties/cluster-properties[] cluster configuration property is set to `false`, leader pinning is disabled across the cluster.
+
+== Configure leader pinning
+
+You can use both a topic configuration property and a cluster configuration property to configure leader pinning.
+
+You can set the topic configuration property for individual topics only, or set the cluster-wide configuration property that will enable leader pinning by default for all topics. You can also use a combination in which a default setting applies across the cluster, and you toggle the setting on or off for specific topics.
+
+This configuration is based on the following scenario: you have Redpanda deployed in a multi-AZ or multi-region cluster, and you have configured each broker so that the config_ref:rack,true,properties/broker-properties[] configuration property contains rack IDs corresponding to the AZ IDs:
+
+* Set the topic configuration property xref:reference:properties/topic-properties.adoc#redpandaleaderspreference[`redpanda.leaders.preference`]. The property accepts the following string values:
++
+--
+** `none`: Opt out the topic from leader pinning.
+** `racks:<rack1>[,<rack2>,...]`: Specify the preferred location (rack) of all topic partition leaders. The list can contain one or more rack IDs, and you can list the IDs in any order. Spaces in the list are ignored, for example: `racks:rack1,rack2` and `racks: rack1,  rack2` are equivalent. You cannot specify empty rack IDs, for example: `racks: rack1,,rack2`. If you specify multiple IDs, Redpanda tries to distribute the partition leader locations equally across brokers in these racks.
+--
++
+This property inherits the default value from the cluster property `default_leaders_preference`.
+
+* Set the cluster configuration property config_ref:default_leaders_preference,true,properties/cluster-properties[], which specifies the default leader pinning configuration for all topics that don’t have `redpanda.leaders.preference` explicitly set. It accepts values in the same format as `redpanda.leaders.preference`. Default: `none` 
++
+This property also affects internal topics, such as `__consumer_offsets` and transaction coordinators. All offset tracking and transaction coordination requests get placed within the preferred regions or AZs for all clients, so you see end-to-end latency and networking cost benefits.
+
+If there is more than one broker in the preferred AZ (or AZs), leader pinning distributes partition leaders uniformly across brokers in the AZ.
+
+== Leader pinning failover across availability zones
+
+If there are three AZs: A, B, and C, and A becomes unavailable, the failover behavior is as follows:
+
+* A topic with "A" as the preferred leader AZ will have its partition leaders uniformly distributed across B and C.
+* A topic with "A,B" as the preferred leader AZs will have its partition leaders in B.
+* A topic with “B” as the preferred leader AZ will have its partition leaders in B as well.
+
+== Suggested reading
+
+* xref:develop:consume-data/follower-fetching.adoc[]

modules/reference/pages/properties/cluster-properties.adoc

@@ -820,14 +820,22 @@ The recursion depth after which debug logging is enabled automatically for the l
 
 === default_leaders_preference
 
-Default settings for preferred location of topic partition leaders. It can be either "none" (no preference), or "racks:<rack1>,<rack2>,..." (prefer brokers with rack id from the list).
+Default settings for preferred location of topic partition leaders. It can be either "none" (no preference), or "racks:<rack1>,<rack2>,..." (prefer brokers with rack ID from the list).
+
+The list can contain one or more rack IDs. If you specify multiple IDs, Redpanda tries to distribute the partition leader locations equally across brokers in these racks.
+
+If config_ref:enable_rack_awareness,true,properties/cluster-properties[] is set to `false`, leader pinning is disabled across the cluster.
 
 *Requires restart:* No
 
 *Visibility:* `user`
 
 *Default:* `none`
 
+*Related topics*:
+
+- xref:develop:produce-data/leader-pinning.adoc[Leader pinning]
+
 ---
 
 === default_num_windows

modules/reference/pages/properties/topic-properties.adoc

@@ -508,6 +508,28 @@ CAUTION: Setting `redpanda.remote.readreplica` together with either `redpanda.re
 
 Configure Redpanda-specific topic properties.
 
+[[redpandaleaderspreference]]
+==== redpanda.leaders.preference
+
+The preferred location (rack) for partition leaders of a topic.
+
+This property inherits the value from the config_ref:default_leaders_preference,true,properties/cluster-properties[] cluster configuration property. You may override the cluster-wide setting by specifying the value for individual topics.
+
+If the cluster configuration property config_ref:enable_rack_awareness,true,properties/cluster-properties[] is set to `false`, leader pinning is disabled across the cluster.
+
+**Default**: `none`
+
+**Values**:
+
+- `none`: Opt out the topic from leader pinning.
+- `racks:<rack1>[,<rack2>,...]`: Specify the preferred location (rack) of all topic partition leaders. The list can contain one or more rack IDs. If you specify multiple IDs, Redpanda tries to distribute the partition leader locations equally across brokers in these racks.
+
+**Related topics**:
+
+- xref:develop:produce-data/leader-pinning.adoc[Leader pinning]
+
+---
+
 ==== replication.factor
 
 The number of replicas of a topic to save in different nodes (brokers) of a cluster.