Merge pull request ceph#61441 from ivancich/wip-dynamic-resharding-min-shards

rgw: allow per-bucket minimum number of shards

Reviewed-by: Casey Bodley <[email protected]>

Author: ivancich

doc/radosgw/dynamicresharding.rst

@@ -6,20 +6,20 @@ RGW Dynamic Bucket Index Resharding
 
 .. versionadded:: Luminous
 
-A large bucket index can lead to performance problems, which can
-be addressed by sharding bucket indexes.
-Until Luminous, changing the number of bucket shards (resharding)
-needed to be done offline, with RGW services disabled.
-Since the Luminous release Ceph has supported online bucket resharding.
+A bucket index object with too many entries can lead to performance
+problems. This can be addressed by resharding bucket indexes.  Until
+Luminous, changing the number of bucket shards (resharding) could only
+be done offline, with RGW services disabled.  Since the Luminous
+release Ceph has supported online bucket resharding.
 
 Each bucket index shard can handle its entries efficiently up until
-reaching a certain threshold. If this threshold is
-exceeded the system can suffer from performance issues. The dynamic
-resharding feature detects this situation and automatically increases
-the number of shards used by a bucket's index, resulting in a
-reduction of the number of entries in each shard. This
-process is transparent to the user. Writes to the target bucket
-are blocked (but reads are not) briefly during resharding process.
+reaching a certain threshold number. If this threshold is exceeded the
+system can suffer from performance issues. The dynamic resharding
+feature detects this situation and automatically increases the number
+of shards used by a bucket's index, resulting in a reduction of the
+number of entries in each shard. This process is transparent to the
+user. Writes to the target bucket can be blocked briefly during
+resharding process, but reads are not.
 
 By default dynamic bucket index resharding can only increase the
 number of bucket index shards to 1999, although this upper-bound is a
@@ -29,10 +29,16 @@ spread the number of entries across the bucket index
 shards more evenly.
 
 Detection of resharding opportunities runs as a background process
-that periodically
-scans all buckets. A bucket that requires resharding is added to
-a queue. A thread runs in the background and processes the queueued
-resharding tasks, one at a time and in order.
+that periodically scans all buckets. A bucket that requires resharding
+is added to a queue. A thread runs in the background and processes the
+queueued resharding tasks one at a time.
+
+Starting with Tentacle, dynamic resharding has the ability to reduce
+the number of shards. Once the condition allowing reduction is noted,
+there is a time delay before it will actually be executed, in case the
+number of objects increases in the near future. The goal of the delay
+is to avoid thrashing where resharding keeps getting re-invoked on
+buckets that fluctuate in numbers of objects.
 
 Multisite
 =========
@@ -48,6 +54,8 @@ Configuration
 .. confval:: rgw_dynamic_resharding
 .. confval:: rgw_max_objs_per_shard
 .. confval:: rgw_max_dynamic_shards
+.. confval:: rgw_dynamic_resharding_may_reduce
+.. confval:: rgw_dynamic_resharding_reduction_wait
 .. confval:: rgw_reshard_bucket_lock_duration
 .. confval:: rgw_reshard_thread_interval
 .. confval:: rgw_reshard_num_logs
@@ -136,7 +144,7 @@ For example, the output at each dynamic resharding stage is shown below:
 Cancel pending bucket resharding
 --------------------------------
 
-Note: Bucket resharding operations cannot be cancelled while executing. ::
+Note: Bucket resharding tasks cannot be cancelled once they start executing. ::
 
    # radosgw-admin reshard cancel --bucket <bucket_name>
 
@@ -158,6 +166,20 @@ since the former is prime. A variety of web sites have lists of prime
 numbers; search for "list of prime numbers" with your favorite
 search engine to locate some web sites.
 
+Setting a bucket's minimum number of shards
+-------------------------------------------
+
+::
+
+   # radosgw-admin bucket set-min-shards --bucket <bucket_name> --num-shards <min number of shards>
+
+Since dynamic resharding can now reduce the number of shards,
+administrators may want to prevent the number of shards from becoming
+too low, for example if the expect the number of objects to increase
+in the future. This command allows administrators to set a per-bucket
+minimum. This does not, however, prevent administrators from manually
+resharding to a lower number of shards.
+
 Troubleshooting
 ===============
 

src/rgw/driver/rados/rgw_bucket.cc

@@ -2815,6 +2815,7 @@ void init_default_bucket_layout(CephContext *cct, rgw::BucketLayout& layout,
 
   if (shards) {
     layout.current_index.layout.normal.num_shards = *shards;
+    layout.current_index.layout.normal.min_num_shards = *shards;
   } else if (cct->_conf->rgw_override_bucket_index_max_shards > 0) {
     layout.current_index.layout.normal.num_shards =
       cct->_conf->rgw_override_bucket_index_max_shards;

src/rgw/driver/rados/rgw_rados.cc

@@ -10823,6 +10823,7 @@ int RGWRados::cls_bucket_head_async(const DoutPrefixProvider *dpp, const RGWBuck
 void RGWRados::calculate_preferred_shards(const DoutPrefixProvider* dpp,
 					  const uint64_t num_objs,
 					  const uint32_t num_source_shards,
+					  const uint32_t min_layout_shards,
 					  bool& need_resharding,
 					  uint32_t* suggested_num_shards)
 {
@@ -10834,6 +10835,7 @@ void RGWRados::calculate_preferred_shards(const DoutPrefixProvider* dpp,
 
   RGWBucketReshard::calculate_preferred_shards(dpp,
 					       max_dynamic_shards,
+					       min_layout_shards,
 					       max_objs_per_shard,
 					       is_multisite,
 					       num_objs,
@@ -10867,8 +10869,11 @@ int RGWRados::check_bucket_shards(const RGWBucketInfo& bucket_info,
   uint32_t suggested_num_shards = 0;
   const uint32_t num_source_shards =
     rgw::current_num_shards(bucket_info.layout);
+  const uint32_t min_layout_shards =
+    rgw::current_min_layout_shards(bucket_info.layout);
 
-  calculate_preferred_shards(dpp, num_objs, num_source_shards,
+  calculate_preferred_shards(dpp, num_objs,
+			     num_source_shards, min_layout_shards,
 			     need_resharding, &suggested_num_shards);
   if (! need_resharding) {
     return 0;

src/rgw/driver/rados/rgw_rados.h

@@ -1630,6 +1630,7 @@ int restore_obj_from_cloud(RGWLCCloudTierCtx& tier_ctx,
   void calculate_preferred_shards(const DoutPrefixProvider* dpp,
 				  const uint64_t num_objs,
 				  const uint32_t current_shard_count,
+				  const uint32_t min_layout_shards,
 				  bool& need_resharding,
 				  uint32_t* suggested_num_shard_count = nullptr);
 

src/rgw/driver/rados/rgw_reshard.cc

@@ -69,23 +69,13 @@ const std::initializer_list<uint16_t> RGWBucketReshard::reshard_primes = {
 };
 
 
-uint32_t RGWBucketReshard::get_prime_shard_count(
-  uint32_t shard_count,
-  uint32_t max_dynamic_shards,
-  uint32_t min_dynamic_shards)
-{
+uint32_t RGWBucketReshard::nearest_prime(uint32_t shard_count)  {
   uint32_t prime_shard_count =
     get_prime_shards_greater_or_equal(shard_count);
 
   // if we cannot find a larger prime number, then just use what was
   // passed in
-  if (! prime_shard_count) {
-    prime_shard_count = shard_count;
-  }
-
-  // keep within min/max bounds
-  return std::min(max_dynamic_shards,
-		  std::max(min_dynamic_shards, prime_shard_count));
+  return prime_shard_count ? prime_shard_count : shard_count;
 }
 
 
@@ -96,6 +86,7 @@ uint32_t RGWBucketReshard::get_prime_shard_count(
 void RGWBucketReshard::calculate_preferred_shards(
   const DoutPrefixProvider* dpp,
   const uint32_t max_dynamic_shards,
+  const uint32_t min_layout_shards,
   const uint64_t max_objs_per_shard,
   const bool is_multisite,
   const uint64_t num_objs,
@@ -139,10 +130,13 @@ void RGWBucketReshard::calculate_preferred_shards(
   }
 
   if (prefer_prime) {
-    calculated_num_shards = get_prime_shard_count(
-      calculated_num_shards, max_dynamic_shards, min_dynamic_shards);
+    calculated_num_shards = nearest_prime(calculated_num_shards);
   }
 
+  calculated_num_shards =
+    std::min(max_dynamic_shards,
+	     std::max({ calculated_num_shards, min_dynamic_shards, min_layout_shards }));
+
   ldpp_dout(dpp, 20) << __func__ << ": reshard " << verb <<
     " suggested; current average (objects/shard) is " <<
     float(num_objs) / current_num_shards << ", which is not within " <<
@@ -461,6 +455,7 @@ static int init_target_layout(rgw::sal::RadosStore* store,
   rgw::bucket_index_layout_generation target;
   target.layout.type = rgw::BucketIndexType::Normal;
   target.layout.normal.num_shards = new_num_shards;
+  target.layout.normal.min_num_shards = current.layout.normal.min_num_shards;
   target.gen = current.gen + 1;
 
   if (bucket_info.reshard_status == cls_rgw_reshard_status::IN_PROGRESS) {
@@ -1256,7 +1251,7 @@ int RGWBucketReshard::do_reshard(const rgw::bucket_index_layout_generation& curr
     // block the client op and complete the resharding
     ceph_assert(bucket_info.layout.resharding == rgw::BucketReshardState::InProgress);
     ret = reshard_process(current, max_op_entries, target_shards_mgr, verbose_json_out, out,
-                              formatter, bucket_info.layout.resharding, dpp, y);
+			  formatter, bucket_info.layout.resharding, dpp, y);
     if (ret < 0) {
       ldpp_dout(dpp, 0) << __func__ << ": failed in progress state of reshard ret = " << ret << dendl;
       return ret;
@@ -1637,6 +1632,9 @@ int RGWReshard::process_entry(const cls_rgw_reshard_entry& entry,
     ret = store->getRados()->get_bucket_stats(dpp, bucket_info,
 					      bucket_info.layout.current_index,
 					      -1, nullptr, nullptr, stats, nullptr, nullptr);
+    if (ret < 0) {
+      return clean_up("unable to access buckets current stats");
+    }
 
     // determine current number of bucket entries across shards
     uint64_t num_entries = 0;
@@ -1645,15 +1643,17 @@ int RGWReshard::process_entry(const cls_rgw_reshard_entry& entry,
     }
 
     const uint32_t current_shard_count =
-      rgw::num_shards(bucket_info.get_current_index().layout.normal);
+      rgw::current_num_shards(bucket_info.layout);
+    const uint32_t min_layout_shards =
+      rgw::current_min_layout_shards(bucket_info.layout);
 
     bool needs_resharding { false };
     uint32_t suggested_shard_count { 0 };
     // calling this rados function determines various rados values
     // needed to perform the calculation before calling
     // calculating_preferred_shards() in this class
     store->getRados()->calculate_preferred_shards(
-      dpp, num_entries, current_shard_count,
+      dpp, num_entries, current_shard_count, min_layout_shards,
       needs_resharding, &suggested_shard_count);
 
     // if we no longer need resharding or currently need to expand
@@ -1711,7 +1711,6 @@ int RGWReshard::process_entry(const cls_rgw_reshard_entry& entry,
   }
 
   // all checkes passed; we can reshard...
-
   RGWBucketReshard br(store, bucket_info, bucket_attrs, nullptr);
 
   ReshardFaultInjector f; // no fault injected

src/rgw/driver/rados/rgw_reshard.h

@@ -160,14 +160,12 @@ class RGWBucketReshard {
     }
   }
 
-  // returns a preferred number of shards given a calculated number of
-  // shards based on max_dynamic_shards and the list of prime values
-  static uint32_t get_prime_shard_count(uint32_t suggested_shards,
-					uint32_t max_dynamic_shards,
-					uint32_t min_dynamic_shards);
+  // returns a preferred number of shards as a prime value
+  static uint32_t nearest_prime(uint32_t suggested_shards);
 
   static void calculate_preferred_shards(const DoutPrefixProvider* dpp,
 					 const uint32_t max_dynamic_shards,
+					 const uint32_t min_layout_shards,
 					 const uint64_t max_objs_per_shard,
 					 const bool is_multisite,
 					 const uint64_t num_objs,

src/rgw/radosgw-admin/radosgw-admin.cc

@@ -1,10 +1,9 @@
-
 // -*- mode:C++; tab-width:8; c-basic-offset:2; indent-tabs-mode:t -*-
 // vim: ts=8 sw=2 smarttab ft=cpp
 
 /*
  * Copyright (C) 2025 IBM 
-*/
+ */
 
 #include <cerrno>
 #include <string>
@@ -169,6 +168,7 @@ void usage()
   cout << "  bucket check unlinked            check for object versions that are not visible in a bucket listing \n";
   cout << "  bucket chown                     link bucket to specified user and update its object ACLs\n";
   cout << "  bucket reshard                   reshard bucket\n";
+  cout << "  bucket set-min-shards            set the minimum number of shards that dynamic resharding will consider for a bucket\n";
   cout << "  bucket rewrite                   rewrite all objects in the specified bucket\n";
   cout << "  bucket sync checkpoint           poll a bucket's sync status until it catches up to its remote\n";
   cout << "  bucket sync disable              disable bucket sync\n";
@@ -699,6 +699,7 @@ enum class OPT {
   BUCKET_RM,
   BUCKET_REWRITE,
   BUCKET_RESHARD,
+  BUCKET_SET_MIN_SHARDS,
   BUCKET_CHOWN,
   BUCKET_RADOS_LIST,
   BUCKET_SHARD_OBJECTS,
@@ -937,6 +938,7 @@ static SimpleCmd::Commands all_cmds = {
   { "bucket rm", OPT::BUCKET_RM },
   { "bucket rewrite", OPT::BUCKET_REWRITE },
   { "bucket reshard", OPT::BUCKET_RESHARD },
+  { "bucket set-min-shards", OPT::BUCKET_SET_MIN_SHARDS },
   { "bucket chown", OPT::BUCKET_CHOWN },
   { "bucket radoslist", OPT::BUCKET_RADOS_LIST },
   { "bucket rados list", OPT::BUCKET_RADOS_LIST },
@@ -8803,6 +8805,51 @@ int main(int argc, const char **argv)
     }
   } // OPT_RESHARD_CANCEL
 
+  if (opt_cmd == OPT::BUCKET_SET_MIN_SHARDS) {
+    if (bucket_name.empty()) {
+      cerr << "ERROR: bucket not specified" << std::endl;
+      return -EINVAL;
+    }
+
+    if (!num_shards_specified) {
+      cerr << "ERROR: --num-shards not specified" << std::endl;
+      return -EINVAL;
+    }
+
+    if (num_shards < 1) {
+      cerr << "ERROR: --num-shards must be at least 1" << std::endl;
+      return -EINVAL;
+    }
+
+    int ret = init_bucket(tenant, bucket_name, bucket_id, &bucket);
+    if (ret < 0) {
+      return -ret;
+    }
+    auto& bucket_info = bucket->get_info();
+
+    const rgw::BucketIndexType type =
+      bucket_info.layout.current_index.layout.type;
+    if (type != rgw::BucketIndexType::Normal) {
+      cerr << "ERROR: the bucket's layout is type " << type <<
+	" instead of type " << rgw::BucketIndexType::Normal <<
+	" and therefore does not have a "
+	"minimum number of shards that can be altered" << std::endl;
+      return EINVAL;
+    }
+
+    uint32_t& min_num_shards =
+      bucket_info.layout.current_index.layout.normal.min_num_shards;
+    min_num_shards = num_shards;
+
+    ret = bucket->put_info(dpp(), false, real_time(), null_yield);
+    if (ret < 0) {
+      cerr << "ERROR: failed writing bucket instance info: " << cpp_strerror(-ret) << std::endl;
+      return -ret;
+    }
+
+    return 0;
+  } // SET_MIN_SHARDS
+
   if (opt_cmd == OPT::OBJECT_UNLINK) {
     int ret = init_bucket(tenant, bucket_name, bucket_id, &bucket);
     if (ret < 0) {

src/rgw/rgw_bucket_layout.cc

@@ -81,29 +81,37 @@ void decode_json_obj(BucketHashType& t, JSONObj *obj)
 // bucket_index_normal_layout
 void encode(const bucket_index_normal_layout& l, bufferlist& bl, uint64_t f)
 {
-  ENCODE_START(1, 1, bl);
+  ENCODE_START(2, 1, bl);
   encode(l.num_shards, bl);
   encode(l.hash_type, bl);
+  encode(l.min_num_shards, bl);
   ENCODE_FINISH(bl);
 }
 void decode(bucket_index_normal_layout& l, bufferlist::const_iterator& bl)
 {
-  DECODE_START(1, bl);
+  DECODE_START(2, bl);
   decode(l.num_shards, bl);
   decode(l.hash_type, bl);
+  if (struct_v >= 2) {
+    decode(l.min_num_shards, bl);
+  }
   DECODE_FINISH(bl);
 }
 void encode_json_impl(const char *name, const bucket_index_normal_layout& l, ceph::Formatter *f)
 {
   f->open_object_section(name);
   encode_json("num_shards", l.num_shards, f);
   encode_json("hash_type", l.hash_type, f);
+  encode_json("min_num_shards", l.min_num_shards, f);
   f->close_section();
 }
 void decode_json_obj(bucket_index_normal_layout& l, JSONObj *obj)
 {
   JSONDecoder::decode_json("num_shards", l.num_shards, obj);
   JSONDecoder::decode_json("hash_type", l.hash_type, obj);
+
+  // if not set in json, set to default value of 1
+  JSONDecoder::decode_json("min_num_shards", l.min_num_shards, obj, 1);
 }
 
 // bucket_index_layout