feat(regionserver): add graceful shutdown configuration

CHANGELOG.md

@@ -26,12 +26,17 @@
 
 ## [24.11.1] - 2025-01-09
 
+### Changed
+
+- Support moving regions to other Pods during graceful shutdown of region servers ([#570]).
+
 ### Fixed
 
 - BREAKING: Use distinct ServiceAccounts for the Stacklets, so that multiple Stacklets can be
   deployed in one namespace. Existing Stacklets will use the newly created ServiceAccounts after
   restart ([#594]).
 
+[#570]: https://github.com/stackabletech/hbase-operator/pull/570
 [#594]: https://github.com/stackabletech/hbase-operator/pull/594
 
 ## [24.11.0] - 2024-11-18

Cargo.toml

@@ -21,6 +21,7 @@ rstest = "0.24"
 serde = { version = "1.0", features = ["derive"] }
 serde_json = "1.0"
 serde_yaml = "0.9"
+shell-escape = "0.1"
 snafu = "0.8"
 stackable-operator = { git = "https://github.com/stackabletech/operator-rs.git", tag = "stackable-operator-0.85.0" }
 product-config = { git = "https://github.com/stackabletech/product-config.git", tag = "0.7.0" }

deploy/helm/hbase-operator/crds/crds.yaml

@@ -688,6 +688,9 @@ spec:
                           description: Time period Pods have to gracefully shut down, e.g. `30m`, `1h` or `2d`. Consult the operator documentation for details.
                           nullable: true
                           type: string
+                        hbaseOpts:
+                          nullable: true
+                          type: string
                         hbaseRootdir:
                           nullable: true
                           type: string
@@ -775,6 +778,34 @@ spec:
                               nullable: true
                               type: boolean
                           type: object
+                        regionMover:
+                          default:
+                            ack: null
+                            maxThreads: null
+                            runBeforeShutdown: null
+                          description: Before terminating a region server pod, the RegionMover tool can be invoked to transfer local regions to other servers. This may cause a lot of network traffic in the Kubernetes cluster if the entire HBase stacklet is being restarted. The operator will compute a timeout period for the region move that will not exceed the graceful shutdown timeout.
+                          properties:
+                            ack:
+                              description: If enabled (default), the region mover will confirm that regions are available on the source as well as the target pods before and after the move.
+                              nullable: true
+                              type: boolean
+                            additionalMoverOptions:
+                              default: []
+                              description: Additional options to pass to the region mover.
+                              items:
+                                type: string
+                              type: array
+                            maxThreads:
+                              description: Maximum number of threads to use for moving regions.
+                              format: uint16
+                              minimum: 0.0
+                              nullable: true
+                              type: integer
+                            runBeforeShutdown:
+                              description: Move local regions to other servers before terminating a region server's pod.
+                              nullable: true
+                              type: boolean
+                          type: object
                         requestedSecretLifetime:
                           description: Request secret (currently only autoTls certificates) lifetime from the secret operator, e.g. `7d`, or `30d`. Please note that this can be shortened by the `maxCertificateLifetime` setting on the SecretClass issuing the TLS certificate.
                           nullable: true
@@ -938,6 +969,9 @@ spec:
                                 description: Time period Pods have to gracefully shut down, e.g. `30m`, `1h` or `2d`. Consult the operator documentation for details.
                                 nullable: true
                                 type: string
+                              hbaseOpts:
+                                nullable: true
+                                type: string
                               hbaseRootdir:
                                 nullable: true
                                 type: string
@@ -1025,6 +1059,34 @@ spec:
                                     nullable: true
                                     type: boolean
                                 type: object
+                              regionMover:
+                                default:
+                                  ack: null
+                                  maxThreads: null
+                                  runBeforeShutdown: null
+                                description: Before terminating a region server pod, the RegionMover tool can be invoked to transfer local regions to other servers. This may cause a lot of network traffic in the Kubernetes cluster if the entire HBase stacklet is being restarted. The operator will compute a timeout period for the region move that will not exceed the graceful shutdown timeout.
+                                properties:
+                                  ack:
+                                    description: If enabled (default), the region mover will confirm that regions are available on the source as well as the target pods before and after the move.
+                                    nullable: true
+                                    type: boolean
+                                  additionalMoverOptions:
+                                    default: []
+                                    description: Additional options to pass to the region mover.
+                                    items:
+                                      type: string
+                                    type: array
+                                  maxThreads:
+                                    description: Maximum number of threads to use for moving regions.
+                                    format: uint16
+                                    minimum: 0.0
+                                    nullable: true
+                                    type: integer
+                                  runBeforeShutdown:
+                                    description: Move local regions to other servers before terminating a region server's pod.
+                                    nullable: true
+                                    type: boolean
+                                type: object
                               requestedSecretLifetime:
                                 description: Request secret (currently only autoTls certificates) lifetime from the secret operator, e.g. `7d`, or `30d`. Please note that this can be shortened by the `maxCertificateLifetime` setting on the SecretClass issuing the TLS certificate.
                                 nullable: true

docs/modules/hbase/pages/usage-guide/operations/graceful-shutdown.adoc

@@ -1,6 +1,6 @@
 = Graceful shutdown
 
-You can configure the graceful shutdown as described in xref:concepts:operations/graceful_shutdown.adoc[].
+You can configure the graceful shutdown grace period as described in xref:concepts:operations/graceful_shutdown.adoc[].
 
 == Masters
 
@@ -15,7 +15,7 @@ However, there is no message in the log acknowledging the graceful shutdown.
 
 == RegionServers
 
-As a default, RegionServers have `60 minutes` to shut down gracefully.
+By default, RegionServers have `60 minutes` to shut down gracefully.
 
 They use the same mechanism described above.
 In contrast to the Master servers, they will, however, acknowledge the graceful shutdown with a message in the logs:
@@ -26,6 +26,61 @@ In contrast to the Master servers, they will, however, acknowledge the graceful
 2023-10-11 12:38:05,060 INFO  [shutdown-hook-0] regionserver.HRegionServer: ***** STOPPING region server 'test-hbase-regionserver-default-0.test-hbase-regionserver-default.kuttl-test-topical-parakeet.svc.cluster.local,16020,1697027870348' *****
 ----
 
+The operator allows for finer control over the shutdown process of region servers.
+For each region server pod, the region mover tool may be invoked before terminating the region server's pod.
+The affected regions are transferred to other pods thus ensuring that the data is still available.
+
+Here is an example:
+
+[source,yaml]
+----
+spec:
+  regionServers:
+    config:
+      regionMover:
+        runBeforeShutdown: true # <1>
+        maxThreads: 5 # <2>
+        ack: false # <3>
+        additionalMoverOptions: ["--designatedFile", "/path/to/designatedFile"] # <4>
+----
+<1>: Run the region mover tool before shutting down the region server. Default is `false`.
+<2>: Maximum number of threads to use for moving regions. Default is 1.
+<3>: Enable or disable region confirmation on the present and target servers. Default is `true`.
+<4>: Extra options to pass to the region mover tool.
+
+For a list of additional options accepted by the region mover use the `--help` option first:
+
+[source]
+----
+$ /stackable/hbase/bin/hbase org.apache.hadoop.hbase.util.RegionMover --help
+usage: hbase org.apache.hadoop.hbase.util.RegionMover <options>
+Options:
+ -r,--regionserverhost <arg>   region server <hostname>|<hostname:port>
+ -o,--operation <arg>          Expected: load/unload/unload_from_rack/isolate_regions
+ -m,--maxthreads <arg>         Define the maximum number of threads to use to unload and reload the regions
+ -i,--isolateRegionIds <arg>   Comma separated list of Region IDs hash to isolate on a RegionServer and put region
+                               server in draining mode. This option should only be used with '-o isolate_regions'. By
+                               putting region server in decommission/draining mode, master can't assign any new region
+                               on this server. If one or more regions are not found OR failed to isolate successfully,
+                               utility will exist without putting RS in draining/decommission mode. Ex.
+                               --isolateRegionIds id1,id2,id3 OR -i id1,id2,id3
+ -x,--excludefile <arg>        File with <hostname:port> per line to exclude as unload targets; default excludes only
+                               target host; useful for rack decommisioning.
+ -d,--designatedfile <arg>     File with <hostname:port> per line as unload targets;default is all online hosts
+ -f,--filename <arg>           File to save regions list into unloading, or read from loading; default
+                               /tmp/<usernamehostname:port>
+ -n,--noack                    Turn on No-Ack mode(default: false) which won't check if region is online on target
+                               RegionServer, hence best effort. This is more performant in unloading and loading but
+                               might lead to region being unavailable for some time till master reassigns it in case the
+                               move failed
+ -t,--timeout <arg>            timeout in seconds after which the tool will exit irrespective of whether it finished or
+                               not;default Integer.MAX_VALUE
+----
+
+NOTE: There is no need to explicitly specify a timeout for the region movement. The operator will compute an appropriate timeout that cannot exceed the `gracefulShutdownTimeout` for region servers.
+
+IMPORTANT: The ZooKeeper connection must be available during the time the region mover is running for the graceful shutdown process to succeed.
+
 == RestServers
 
 As a default, RestServers have `5 minutes` to shut down gracefully.

rust/crd/Cargo.toml

@@ -12,6 +12,7 @@ publish = false
 product-config.workspace = true
 serde.workspace = true
 serde_json.workspace = true
+shell-escape.workspace = true
 snafu.workspace = true
 stackable-operator.workspace = true
 strum.workspace = true