Merge pull request ceph#64762 from mkogan1/wip-ratelim-list-v001

rgw: rate limit for LIST and DELETE ops

Author: mkogan1

PendingReleaseNotes

@@ -603,6 +603,13 @@ CephFS: Disallow delegating preallocated inode ranges to clients. Config
 * RGW: The serialized format of notification and topics has changed, so that 
   new/updated topics will be unreadable by old RGWs. We recommend completing 
   the RGW upgrades before creating or modifying any notification topics.
+* RGW: New rate limiting options have been added to control the number of
+  bucket listing requests (`radosgw-admin ratelimit set ... --max-list-ops=`)
+  and delete operations (`radosgw-admin ratelimit set ... --max-delete-ops=`)
+  per accumulation interval, along with the ability to configure the
+  accumulation interval duration (`rgw_ratelimit_interval`).
+  Their default value is `0`, which keeps backward compatibility to the
+  previous behavior of READ and WRITE operations rate limiting.
 * RBD: Trailing newline in passphrase files (`<passphrase-file>` argument in
   `rbd encryption format` command and `--encryption-passphrase-file` option
   in other commands) is no longer stripped.

doc/radosgw/admin.rst

@@ -622,9 +622,21 @@ Rate Limit Management
 =====================
 
 Quotas can be set for The Ceph Object Gateway on users and buckets. The "rate
-limit" includes the maximum number of read operations (read ops) and write
-operations (write ops) per minute as well as the number of bytes per minute
-that can be written or read per user or per bucket.
+limit" includes the maximum number of read operations and write operations
+per accumulation interval as well as the number of bytes per accumulation interval
+that can be written or read per user or per bucket. It also includes the maximum
+number of list requests and delete operations per accumulation interval.
+The accumulation interval is configured by the :confval:`rgw_ratelimit_interval` option.
+The default value is 60 seconds.
+(Note: S3 Multi-Object Delete operation are currently not supported by rate limiting)
+
+The configured limits should be divided by the number of active object gateways. For example,
+if "user A" is to be be limited to 10 ops per minute and there are two object gateways in the cluster,
+then the limit on "user A" should be 5 (10 ops per minute / 2 RGWs).
+If the requests are not balanced between RGWs, the rate limit might be underutilized.
+For example: if the ops limit is 5 and there are two RGWs,
+but the Load Balancer sends load to only one of those RGWs,
+the effective limit is 5 ops, because this limit is enforced per RGW.
 
 Read Requests and Write Requests
 --------------------------------
@@ -636,8 +648,8 @@ How Metrics Work
 Each object gateway tracks per-user metrics separately from bucket metrics.
 These metrics are not shared with other gateways. The configured limits should
 be divided by the number of active object gateways. For example, if "user A" is
-to be be limited to 10 ops per minute and there are two object gateways in the
-cluster, then the limit on "user A" should be ``5`` (10 ops per minute / 2
+to be be limited to 10 ops per accumulation interval and there are two object gateways in the
+cluster, then the limit on "user A" should be ``5`` (10 ops per accumulation interval / 2
 RGWs). If the requests are **not** balanced between RGWs, the rate limit might
 be underutilized. For example: if the ops limit is ``5`` and there are two
 RGWs, **but** the Load Balancer sends load to only one of those RGWs, the
@@ -654,10 +666,10 @@ rate limit is reached during the execution of the request. The RGW keeps track
 of a "debt" consisting of bytes used in excess of the configured value; users
 or buckets that incur this kind of debt are prevented from sending more
 requests until the "debt" has been repaid. The maximum size of the "debt" is
-twice the max-read/write-bytes per minute. If "user A" is subject to a 1-byte
-read limit per minute and they attempt to ``GET`` an object that is 1 GB in size,
+twice the max-read/write-bytes per accumulation interval. If "user A" is subject to a 1-byte
+read limit per accumulation interval and they attempt to ``GET`` an object that is 1 GB in size,
 then the ``GET`` action will fail. After "user A" has completed this 1 GB
-operation, RGW blocks the user's requests for up to two minutes. After this
+operation, RGW blocks the user's requests for up to two accumulation intervals. After this
 time has elapsed, "user A" will be able to send ``GET`` requests again.
 
 
@@ -668,19 +680,27 @@ time has elapsed, "user A" will be able to send ``GET`` requests again.
   user.
 
 - **Maximum Read Ops:** The ``--max-read-ops`` setting allows you to limit read
-  bytes per minute per RGW instance. A ``0`` value disables throttling. 
+  bytes per accumulation interval per RGW instance. A ``0`` value disables throttling.
 
 - **Maximum Read Bytes:** The ``--max-read-bytes`` setting allows you to limit
-  read bytes per minute per RGW instance. A ``0`` value disables throttling. 
+  read bytes per accumulation interval per RGW instance. A ``0`` value disables throttling.
 
 - **Maximum Write Ops:** The ``--max-write-ops`` setting allows you to specify
-  the maximum number of write ops per minute per RGW instance. A ``0`` value
+  the maximum number of write ops per accumulation interval per RGW instance. A ``0`` value
   disables throttling.
 
 - **Maximum Write Bytes:** The ``--max-write-bytes`` setting allows you to
-  specify the maximum number of write bytes per minute per RGW instance. A
+  specify the maximum number of write bytes per accumulation interval per RGW instance. A
   ``0`` value disables throttling.
- 
+
+- **Maximum List Ops:** The ``--max-list-ops`` setting allows you to
+  specify the maximum number of bucket listing requests per accumulation interval per RGW instance.
+  A ``0`` value disables throttling.
+
+- **Maximum Delete Ops:** The ``--max-delete-ops`` setting allows you to
+  specify the maximum number of delete operations per accumulation interval per RGW instance.
+  A ``0`` value disables throttling.
+
 - **Rate Limit Scope:** The ``--ratelimit-scope`` option sets the scope for the
   rate limit.  The options are ``bucket`` , ``user`` and ``anonymous``. Bucket
   rate limit apply to buckets.  The user rate limit applies to a user.  The
@@ -699,7 +719,8 @@ parameters:
 
    radosgw-admin ratelimit set --ratelimit-scope=user --uid=<uid> \
                                  <[--max-read-ops=<num ops>] [--max-read-bytes=<num bytes>] \
-                                 [--max-write-ops=<num ops>] [--max-write-bytes=<num bytes>]>
+                                 [--max-write-ops=<num ops>] [--max-write-bytes=<num bytes>] \
+                                 [--max-list-ops=<num ops>] [--max-delete-ops=<num ops>]>
 
 An example of using ``radosgw-admin ratelimit set`` to set a rate limit might
 look like this: 
@@ -764,7 +785,8 @@ The following is the general form of commands that set rate limit parameters:
 
    radosgw-admin ratelimit set --ratelimit-scope=bucket --bucket=<bucket> \
                                  <[--max-read-ops=<num ops>] [--max-read-bytes=<num bytes>] \
-                                 [--max-write-ops=<num ops>] [--max-write-bytes=<num bytes>]>
+                                 [--max-write-ops=<num ops>] [--max-write-bytes=<num bytes>] \
+                                 [--max-list-ops=<num ops>] [--max-delete-ops=<num ops>]>
 
 An example of using ``radosgw-admin ratelimit set`` to set a rate limit for a
 bucket might look like this: 

doc/radosgw/adminops.rst

@@ -2072,8 +2072,9 @@ as mentioned in Set Bucket Quota section above.
 Rate Limit
 ==========
 
-The Admin Operations API enables you to set and get ratelimit configurations on users and on bucket and global rate limit configurations. See `Rate Limit Management`_ for additional details. 
-Rate Limit includes the maximum number of operations and/or bytes per minute, separated by read and/or write, to a bucket and/or by a user and the maximum storage size in megabytes.
+The Admin Operations API enables you to set and get ratelimit configurations on users and on bucket and global rate limit configurations. See `Rate Limit Management`_ for additional details.
+Rate Limit includes the maximum number of operations and/or bytes per accumulation interval, separated by read and/or write (Additionally list and get operations),
+to a bucket and/or by a user and the maximum storage size in megabytes.
 
 To view rate limit, the user must have a ``ratelimit=read`` capability. To set,
 modify or disable a ratelimit, the user must have ``ratelimit=write`` capability.
@@ -2087,16 +2088,22 @@ Valid parameters for quotas include:
 - **User:** The ``uid`` option allows you to specify a rate limit for a user.
 
 - **Maximum Read Bytes:** The ``max-read-bytes`` setting allows you to specify
-  the maximum number of read bytes per minute. A 0 value disables this setting.
+  the maximum number of read bytes per accumulation interval. A 0 value disables this setting.
 
 - **Maximum Write Bytes:** The ``max-write-bytes`` setting allows you to specify
-  the maximum number of write bytes per minute. A 0 value disables this setting.
+  the maximum number of write bytes per accumulation interval. A 0 value disables this setting.
 
 - **Maximum Read Ops:** The ``max-read-ops`` setting allows you to specify
-  the maximum number of read ops per minute. A 0 value disables this setting.
+  the maximum number of read ops per accumulation interval. A 0 value disables this setting.
 
 - **Maximum Write Ops:** The ``max-write-ops`` setting allows you to specify
-  the maximum number of write ops per minute. A 0 value disables this setting.
+  the maximum number of write ops per accumulation interval. A 0 value disables this setting.
+
+- **Maximum List Ops:** The ``max-list-ops`` setting allows you to specify
+  the maximum number of bucket listing requests per accumulation interval. A 0 value disables this setting.
+
+- **Maximum Delete Ops:** The ``max-delete-ops`` setting allows you to specify
+  the maximum number of delete operations per accumulation interval. A 0 value disables throttling.
 
 - **Global:** The ``global`` option allows you to specify a global rate limit.
   The value should be either 'True' or 'False'.
@@ -2123,7 +2130,7 @@ Set User Rate Limit
 To set a rate limit, the user must have ``ratelimit`` capability set with ``write``
 permission. ::
 
-	POST /{admin}/ratelimit?ratelimit-scope=user&uid=<uid><[&max-read-bytes=<bytes>][&max-write-bytes=<bytes>][&max-read-ops=<ops>][&max-write-ops=<ops>][enabled=<True|False>]>
+	POST /{admin}/ratelimit?ratelimit-scope=user&uid=<uid><[&max-read-bytes=<bytes>][&max-write-bytes=<bytes>][&max-read-ops=<ops>][&max-write-ops=<ops>][&max-list-ops=<ops>][&max-delete-ops=<ops>][&enabled=<True|False>]>
 
 
 
@@ -2143,7 +2150,7 @@ Set Rate Limit for an Individual Bucket
 To set a rate limit, the user must have ``ratelimit`` capability set with ``write``
 permission. ::
 
-	POST /{admin}/ratelimit?bucket=<bucket-name>&ratelimit-scope=bucket<[&max-read-bytes=<bytes>][&max-write-bytes=<bytes>][&max-read-ops=<ops>][&max-write-ops=<ops>]>
+	POST /{admin}/ratelimit?bucket=<bucket-name>&ratelimit-scope=bucket<[&max-read-bytes=<bytes>][&max-write-bytes=<bytes>][&max-read-ops=<ops>][&max-write-ops=<ops>][&max-list-ops=<ops>][&max-delete-ops=<ops>][&enabled=<True|False>]>
 
 
 
@@ -2163,7 +2170,7 @@ Set Global User Rate Limit
 To set a rate limit, the user must have ``ratelimit`` capability set with ``write``
 permission. ::
 
-	POST /{admin}/ratelimit?ratelimit-scope=user&global=<True|False><[&max-read-bytes=<bytes>][&max-write-bytes=<bytes>][&max-read-ops=<ops>][&max-write-ops=<ops>][enabled=<True|False>]>
+	POST /{admin}/ratelimit?ratelimit-scope=user&global=<True|False><[&max-read-bytes=<bytes>][&max-write-bytes=<bytes>][&max-read-ops=<ops>][&max-write-ops=<ops>][&max-list-ops=<ops>][&max-delete-ops=<ops>][&enabled=<True|False>]>
 
 
 
@@ -2173,7 +2180,7 @@ Set Global Rate Limit Bucket
 To set a rate limit, the user must have ``ratelimit`` capability set with ``write``
 permission. ::
 
-	POST /{admin}/ratelimit?ratelimit-scope=bucket&global=<True|False><[&max-read-bytes=<bytes>][&max-write-bytes=<bytes>][&max-read-ops=<ops>][&max-write-ops=<ops>]>
+	POST /{admin}/ratelimit?ratelimit-scope=bucket&global=<True|False><[&max-read-bytes=<bytes>][&max-write-bytes=<bytes>][&max-read-ops=<ops>][&max-write-ops=<ops>][&max-list-ops=<ops>][&max-delete-ops=<ops>][&enabled=<True|False>]>
 
 
 
@@ -2183,7 +2190,7 @@ Set Global Anonymous User Rate Limit
 To set a rate limit, the user must have ``ratelimit`` capability set with ``write``
 permission. ::
 
-	POST /{admin}/ratelimit?ratelimit-scope=anon&global=<True|False><[&max-read-bytes=<bytes>][&max-write-bytes=<bytes>][&max-read-ops=<ops>][&max-write-ops=<ops>][enabled=<True|False>]>
+	POST /{admin}/ratelimit?ratelimit-scope=anon&global=<True|False><[&max-read-bytes=<bytes>][&max-write-bytes=<bytes>][&max-read-ops=<ops>][&max-write-ops=<ops>][&max-list-ops=<ops>][&max-delete-ops=<ops>][&enabled=<True|False>]>
 
 
 

qa/tasks/radosgw_admin_rest.py

@@ -982,3 +982,74 @@ def task(ctx, config):
     # TESTCASE 'ratelimit' 'global' 'modify' 'anonymous' 'enabled' 'succeeds'
     (ret, out) = rgwadmin_rest(admin_conn, ['ratelimit', 'modify'], {'ratelimit-scope' : 'bucket', 'global': 'true', 'enabled' : 'true'})
     assert ret == 200
+
+    # TESTCASE 'ratelimit' 'user' 'modify' 'max-list-ops = 100' 'succeeds'
+    (ret, out) = rgwadmin_rest(admin_conn, ['ratelimit', 'modify'], {'ratelimit-scope' : 'user', 'uid' : ratelimit_user, 'max-list-ops' : '100'})
+    assert ret == 200
+    (ret, out) = rgwadmin_rest(admin_conn, ['ratelimit', 'info'], {'ratelimit-scope' : 'user', 'uid' : ratelimit_user})
+    assert ret == 200
+    user_ratelimit = out['user_ratelimit']
+    assert user_ratelimit['max_list_ops'] == 100
+
+    # TESTCASE 'ratelimit' 'user' 'modify' 'max-delete-ops = 50' 'succeeds'
+    (ret, out) = rgwadmin_rest(admin_conn, ['ratelimit', 'modify'], {'ratelimit-scope' : 'user', 'uid' : ratelimit_user, 'max-delete-ops' : '50'})
+    assert ret == 200
+    (ret, out) = rgwadmin_rest(admin_conn, ['ratelimit', 'info'], {'ratelimit-scope' : 'user', 'uid' : ratelimit_user})
+    assert ret == 200
+    user_ratelimit = out['user_ratelimit']
+    assert user_ratelimit['max_delete_ops'] == 50
+
+    # TESTCASE 'ratelimit' 'bucket' 'modify' 'max-list-ops = 200' 'succeeds'
+    (ret, out) = rgwadmin_rest(admin_conn, ['ratelimit', 'modify'], {'ratelimit-scope' : 'bucket', 'bucket' : ratelimit_bucket, 'max-list-ops' : '200'})
+    assert ret == 200
+    (ret, out) = rgwadmin_rest(admin_conn, ['ratelimit', 'info'], {'ratelimit-scope' : 'bucket', 'bucket' : ratelimit_bucket})
+    assert ret == 200
+    bucket_ratelimit = out['bucket_ratelimit']
+    assert bucket_ratelimit['max_list_ops'] == 200
+
+    # TESTCASE 'ratelimit' 'bucket' 'modify' 'max-delete-ops = 75' 'succeeds'
+    (ret, out) = rgwadmin_rest(admin_conn, ['ratelimit', 'modify'], {'ratelimit-scope' : 'bucket', 'bucket' : ratelimit_bucket, 'max-delete-ops' : '75'})
+    assert ret == 200
+    (ret, out) = rgwadmin_rest(admin_conn, ['ratelimit', 'info'], {'ratelimit-scope' : 'bucket', 'bucket' : ratelimit_bucket})
+    assert ret == 200
+    bucket_ratelimit = out['bucket_ratelimit']
+    assert bucket_ratelimit['max_delete_ops'] == 75
+
+    # TESTCASE 'ratelimit' 'global' 'modify' 'bucket' 'max-list-ops = 500' 'succeeds'
+    (ret, out) = rgwadmin_rest(admin_conn, ['ratelimit', 'modify'], {'ratelimit-scope' : 'bucket', 'global': 'true', 'max-list-ops' : '500'})
+    assert ret == 200
+    (ret, out) = rgwadmin_rest(admin_conn, ['ratelimit', 'info'], {'global' : 'true'})
+    assert ret == 200
+    # Check that global bucket ratelimit has the list ops set
+    assert 'bucket_ratelimit' in out
+    assert out['bucket_ratelimit']['max_list_ops'] == 500
+
+    # TESTCASE 'ratelimit' 'global' 'modify' 'bucket' 'max-delete-ops = 300' 'succeeds'
+    (ret, out) = rgwadmin_rest(admin_conn, ['ratelimit', 'modify'], {'ratelimit-scope' : 'bucket', 'global': 'true', 'max-delete-ops' : '300'})
+    assert ret == 200
+    (ret, out) = rgwadmin_rest(admin_conn, ['ratelimit', 'info'], {'global' : 'true'})
+    assert ret == 200
+    # Check that global bucket ratelimit has the delete ops set
+    assert 'bucket_ratelimit' in out
+    assert out['bucket_ratelimit']['max_delete_ops'] == 300
+
+    # TESTCASE 'ratelimit' 'user' 'modify' 'multiple ops at once' 'succeeds'
+    (ret, out) = rgwadmin_rest(admin_conn, ['ratelimit', 'modify'], {
+        'ratelimit-scope' : 'user',
+        'uid' : ratelimit_user,
+        'max-read-ops' : '1000',
+        'max-write-ops' : '800',
+        'max-list-ops' : '600',
+        'max-delete-ops' : '400',
+        'enabled' : 'true'
+    })
+    assert ret == 200
+    (ret, out) = rgwadmin_rest(admin_conn, ['ratelimit', 'info'], {'ratelimit-scope' : 'user', 'uid' : ratelimit_user})
+    assert ret == 200
+    user_ratelimit = out['user_ratelimit']
+    assert user_ratelimit['enabled'] == True
+    assert user_ratelimit['max_read_ops'] == 1000
+    assert user_ratelimit['max_write_ops'] == 800
+    assert user_ratelimit['max_list_ops'] == 600
+    assert user_ratelimit['max_delete_ops'] == 400
+

src/common/options/rgw.yaml.in

@@ -4508,6 +4508,20 @@ options:
   default: false
   services:
   - rgw
+- name: rgw_ratelimit_interval
+  type: uint
+  level: advanced
+  desc: Time window for rate limiting in seconds
+  long_desc: This option sets the time window for rate limiting accumulation in seconds.
+    Requests that exceed the configured rate limits within this time window will be rejected.
+    The default is a 60 second token bucket.
+  default: 60
+  min: 1
+  services:
+  - rgw
+  flags:
+  - startup
+  with_legacy: true
 - name: rgw_redis_connection_pool_size
   type: int
   level: basic