Merge pull request ceph#62179 from benhanokh/s3_full_object_dedup

rgw/dedup: full object dedup

Author: yuvalif

doc/radosgw/s3_objects_dedup.rst

@@ -0,0 +1,79 @@
+======================
+Full RGW Object Dedup:
+======================
+Add a radosgw-admin command to collect and report deduplication stats
+
+.. note:: This utility doesn’t perform dedup and doesn’t make any
+          change to the existing system and will only collect
+          statistics and report them.
+
+----
+
+***************
+Admin commands:
+***************
+- ``radosgw-admin dedup stats``:
+   Collects & displays last dedup statistics
+- ``radosgw-admin dedup pause``:
+   Pauses active dedup session (dedup resources are not released)
+- ``radosgw-admin dedup resume``:
+   Resumes a paused dedup session
+- ``radosgw-admin dedup abort``:
+   Aborts active dedup session and release all resources used by it
+- ``radosgw-admin dedup estimate``
+    Starts a new dedup estimate session (aborting first existing session if exists)
+
+----
+
+****************
+Skipped Objects:
+****************
+Dedup Estimates skips the following objects:
+
+- Objects smaller than 4MB (unless they are multipart)
+- Objects with different placement rules
+- Objects with different pools
+- Objects with different same storage-classes
+
+The Dedup process itself (which will be released later) will also skip
+**compressed** and **user-encrypted** objects, but the estimate
+process will accept them (since we don't have access to that
+information during the estimate process)
+
+----
+
+********************
+Estimate Processing:
+********************
+The Dedup Estimate process collects all the needed information directly from
+the bucket-indices reading one full bucket-index object with 1000's of
+entries at a time.
+
+The Bucket-Indices objects are sharded between the participating
+members so every bucket-index object is read exactly one time.
+The sharding allow processing to scale almost linearly spliting the
+load evenly between the participating members.
+
+The Dedup Estimate process does not access the objects themselves
+(data/metadata) which means its processing time won't be affected by
+the underlined media storing the objects (SSD/HDD) since the bucket-indices are
+virtually always stored on a fast medium (SSD with heavy memory
+caching)
+
+----
+
+*************
+Memory Usage:
+*************
+ +---------------++-----------+
+ | RGW Obj Count |  Memory    |
+ +===============++===========+
+ | | ____1M      | | ___8MB   |
+ | | ____4M      | | __16MB   |
+ | | ___16M      | | __32MB   |
+ | | ___64M      | | __64MB   |
+ | | __256M      | | _128MB   |
+ | | _1024M( 1G) | | _256MB   |
+ | | _4096M( 4G) | | _512MB   |
+ | | 16384M(16G) | | 1024MB   |
+ +---------------+------------+

qa/suites/rgw/dedup/%

@@ -0,0 +1 @@
+../.qa/

qa/suites/rgw/dedup/.qa

@@ -0,0 +1 @@
+.qa/rgw_frontend/beast.yaml

qa/suites/rgw/dedup/beast.yaml

@@ -0,0 +1 @@
+.qa/objectstore/bluestore-bitmap.yaml

qa/suites/rgw/dedup/bluestore-bitmap.yaml

@@ -0,0 +1,13 @@
+roles:
+- [mon.a, mon.c, mgr.y, osd.0, osd.1, osd.2, osd.3, client.0]
+- [mon.b, mgr.x, osd.4, osd.5, osd.6, osd.7, client.1]
+- [client.2]
+openstack:
+- volumes: # attached to each instance
+    count: 4
+    size: 10 # GB
+overrides:
+  ceph:
+    conf:
+      osd:
+        osd shutdown pgref assert: true

qa/suites/rgw/dedup/fixed-3-rgw.yaml

@@ -0,0 +1 @@
+.qa/rgw/ignore-pg-availability.yaml

qa/suites/rgw/dedup/ignore-pg-availability.yaml

@@ -0,0 +1,12 @@
+overrides:
+  ceph:
+    conf:
+      client:
+        setuser: ceph
+        setgroup: ceph
+        debug rgw: 20
+        debug rgw dedup: 20
+  rgw:
+    storage classes:
+      LUKEWARM:
+      FROZEN:

qa/suites/rgw/dedup/overrides.yaml

@@ -0,0 +1 @@
+.qa/distros/supported-random-distro$/