Skip to content

Commit 6a0d503

Browse files
kamoltatkamoltat
authored
Merge pull request ceph#56233 from kamoltat/wip-ksirivad-fix-64802
RADOS: Generalize stretch mode pg temp handling to be usable without stretch mode Samuel Just <[email protected]>
2 parents da6c7b0 + 7b41aff commit 6a0d503

File tree

12 files changed

+1640
-0
lines changed

12 files changed

+1640
-0
lines changed

‎doc/rados/operations/pools.rst‎

Lines changed: 111 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -737,6 +737,117 @@ Managing pools that are flagged with ``--bulk``
737737
===============================================
738738
See :ref:`managing_bulk_flagged_pools`.
739739

740+
Setting values for a stretch pool
741+
=================================
742+
To set values for a stretch pool, run a command of the following form:
743+
744+
.. prompt:: bash $
745+
746+
ceph osd pool stretch set {pool-name} {peering_crush_bucket_count} {peering_crush_bucket_target} {peering_crush_bucket_barrier} {crush_rule} {size} {min_size} [--yes-i-really-mean-it]
747+
748+
Here are the break downs of the arguments:
749+
750+
.. describe:: {pool-name}
751+
752+
The name of the pool. It must be an existing pool, this command doesn't create a new pool.
753+
754+
:Type: String
755+
:Required: Yes.
756+
757+
.. describe:: {peering_crush_bucket_count}
758+
759+
The value is used along with peering_crush_bucket_barrier to determined whether the set of
760+
OSDs in the chosen acting set can peer with each other, based on the number of distinct
761+
buckets there are in the acting set.
762+
763+
:Type: Integer
764+
:Required: Yes.
765+
766+
.. describe:: {peering_crush_bucket_target}
767+
768+
This value is used along with peering_crush_bucket_barrier and size to calculate
769+
the value bucket_max which limits the number of OSDs in the same bucket from getting chose to be in the acting set of a PG.
770+
771+
:Type: Integer
772+
:Required: Yes.
773+
774+
.. describe:: {peering_crush_bucket_barrier}
775+
776+
The type of bucket a pool is stretched across, e.g., rack, row, or datacenter.
777+
778+
:Type: String
779+
:Required: Yes.
780+
781+
.. describe:: {crush_rule}
782+
783+
The crush rule to use for the stretch pool. The type of pool must match the type of crush_rule
784+
(replicated or erasure).
785+
786+
:Type: String
787+
:Required: Yes.
788+
789+
.. describe:: {size}
790+
791+
The number of replicas for objects in the stretch pool.
792+
793+
:Type: Integer
794+
:Required: Yes.
795+
796+
.. describe:: {min_size}
797+
798+
The minimum number of replicas required for I/O in the stretch pool.
799+
800+
:Type: Integer
801+
:Required: Yes.
802+
803+
.. describe:: {--yes-i-really-mean-it}
804+
805+
This flag is required to confirm that you really want to by-pass
806+
the safety checks and set the values for a stretch pool, e.g,
807+
when you are trying to set ``peering_crush_bucket_count`` or
808+
``peering_crush_bucket_target`` to be more than the number of buckets in the crush map.
809+
810+
:Type: Flag
811+
:Required: No.
812+
813+
.. _setting_values_for_a_stretch_pool:
814+
815+
Unsetting values for a stretch pool
816+
===================================
817+
To move the pool back to non-stretch, run a command of the following form:
818+
819+
.. prompt:: bash $
820+
821+
ceph osd pool stretch unset {pool-name}
822+
823+
Here are the break downs of the argument:
824+
825+
.. describe:: {pool-name}
826+
827+
The name of the pool. It must be an existing pool that is stretched,
828+
i.e., it has already been set with the command `ceph osd pool stretch set`.
829+
830+
:Type: String
831+
:Required: Yes.
832+
833+
Showing values of a stretch pool
834+
================================
835+
To show values for a stretch pool, run a command of the following form:
836+
837+
.. prompt:: bash $
838+
839+
ceph osd pool stretch show {pool-name}
840+
841+
Here are the break downs of the argument:
842+
843+
.. describe:: {pool-name}
844+
845+
The name of the pool. It must be an existing pool that is stretched,
846+
i.e., it has already been set with the command `ceph osd pool stretch set`.
847+
848+
:Type: String
849+
:Required: Yes.
850+
740851
.. _Pool, PG and CRUSH Config Reference: ../../configuration/pool-pg-config-ref
741852
.. _Bloom Filter: https://en.wikipedia.org/wiki/Bloom_filter
742853
.. _setting the number of placement groups: ../placement-groups#set-the-number-of-placement-groups

‎doc/rados/operations/stretch-mode.rst‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -81,6 +81,18 @@ Data Center B. In a situation of this kind, the loss of Data Center A means
8181
that the data is lost and Ceph will not be able to operate on it. This
8282
situation is surprisingly difficult to avoid using only standard CRUSH rules.
8383

84+
Individual Stretch Pools
85+
========================
86+
Setting individual ``stretch pool`` is an option that allows for the configuration
87+
of specific pools to be distributed across ``two or more data centers``.
88+
This is achieved by executing the ``ceph osd pool stretch set`` command on each desired pool,
89+
as opposed to applying a cluster-wide configuration ``with stretch mode``.
90+
See :ref:`setting_values_for_a_stretch_pool`
91+
92+
Use ``stretch mode`` when you have exactly ``two data centers`` and require a uniform
93+
configuration across the entire cluster. Conversely, opt for a ``stretch pool``
94+
when you need a particular pool to be replicated across ``more than two data centers``,
95+
providing a more granular level of control and a larger cluster size.
8496

8597
Stretch Mode
8698
============

‎qa/suites/rados/singleton/all/3-az-stretch-cluster-netsplit.yaml‎

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
roles:
2+
- - mon.a
3+
- mon.b
4+
- mon.c
5+
- osd.0
6+
- osd.1
7+
- osd.2
8+
- mgr.a
9+
- mgr.b
10+
- - mon.d
11+
- mon.e
12+
- mon.f
13+
- osd.3
14+
- osd.4
15+
- osd.5
16+
- mgr.c
17+
- mgr.d
18+
- - mon.g
19+
- mon.h
20+
- mon.i
21+
- osd.6
22+
- osd.7
23+
- osd.8
24+
- mgr.e
25+
- mgr.f
26+
- - client.0
27+
28+
openstack:
29+
- volumes: # attached to each instance
30+
count: 3
31+
size: 10 # GB
32+
overrides:
33+
ceph:
34+
conf:
35+
global:
36+
mon election default strategy: 3
37+
mon:
38+
client mount timeout: 60
39+
osd pool default size: 6
40+
osd_pool_default_min_size: 3
41+
osd_pool_default_pg_autoscale_mode: off
42+
debug mon: 30
43+
tasks:
44+
- install:
45+
- ceph:
46+
pre-mgr-commands:
47+
- sudo ceph config set mgr mgr_pool false --force
48+
log-ignorelist:
49+
- overall HEALTH_
50+
- \(OSDMAP_FLAGS\)
51+
- \(OSD_
52+
- \(PG_
53+
- \(POOL_
54+
- \(CACHE_POOL_
55+
- \(OBJECT_
56+
- \(SLOW_OPS\)
57+
- \(REQUEST_SLOW\)
58+
- \(TOO_FEW_PGS\)
59+
- slow request
60+
- \(POOL_APP_NOT_ENABLED\)
61+
- overall HEALTH_
62+
- \(MGR_DOWN\)
63+
- \(MON_DOWN\)
64+
- \(PG_AVAILABILITY\)
65+
- \(SLOW_OPS\)
66+
- \[WRN\]
67+
- workunit:
68+
clients:
69+
client.0:
70+
- mon/mon-stretch-pool.sh
71+
- cephfs_test_runner:
72+
modules:
73+
- tasks.test_netsplit_3az_stretch_pool

‎qa/suites/rados/singleton/all/mon-stretch-pool.yaml‎

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
roles:
2+
- - mon.a
3+
- mon.b
4+
- mon.c
5+
- osd.0
6+
- osd.1
7+
- osd.2
8+
- mgr.x
9+
- client.0
10+
- mon.d
11+
- mon.e
12+
- mon.f
13+
- osd.3
14+
- osd.4
15+
- osd.5
16+
- mon.g
17+
- mon.h
18+
- mon.i
19+
- osd.6
20+
- osd.7
21+
- osd.8
22+
23+
openstack:
24+
- volumes: # attached to each instance
25+
count: 3
26+
size: 10 # GB
27+
overrides:
28+
ceph:
29+
conf:
30+
global:
31+
mon election default strategy: 3
32+
mon:
33+
client mount timeout: 60
34+
osd pool default size: 6
35+
osd_pool_default_min_size: 3
36+
osd_pool_default_pg_autoscale_mode: off
37+
debug mon: 30
38+
tasks:
39+
- install:
40+
- ceph:
41+
pre-mgr-commands:
42+
- sudo ceph config set mgr mgr_pool false --force
43+
log-ignorelist:
44+
- overall HEALTH_
45+
- \(OSDMAP_FLAGS\)
46+
- \(OSD_
47+
- \(PG_
48+
- \(POOL_
49+
- \(CACHE_POOL_
50+
- \(OBJECT_
51+
- \(SLOW_OPS\)
52+
- \(REQUEST_SLOW\)
53+
- \(TOO_FEW_PGS\)
54+
- slow request
55+
- \(POOL_APP_NOT_ENABLED\)
56+
- overall HEALTH_
57+
- \(MGR_DOWN\)
58+
- \(MON_DOWN\)
59+
- \(PG_AVAILABILITY\)
60+
- \(SLOW_OPS\)
61+
- workunit:
62+
clients:
63+
client.0:
64+
- mon/mon-stretch-pool.sh
65+
- cephfs_test_runner:
66+
modules:
67+
- tasks.stretch_cluster

‎qa/tasks/ceph_manager.py‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3248,6 +3248,14 @@ def get_mon_quorum(self):
32483248
j = json.loads(out)
32493249
return j['quorum']
32503250

3251+
def get_mon_quorum_names(self):
3252+
"""
3253+
Extract monitor quorum names from the cluster
3254+
"""
3255+
out = self.raw_cluster_cmd('quorum_status')
3256+
j = json.loads(out)
3257+
return j['quorum_names']
3258+
32513259
def wait_for_mon_quorum_size(self, size, timeout=300):
32523260
"""
32533261
Loop until quorum size is reached.

0 commit comments

Comments
 (0)