Skip to content

Commit bc103d8

Browse files
adk3798adk3798
authored
Merge pull request ceph#59086 from phlogistonjohn/jjm-smb-ctdb-clustering
smb: ctdb clustering Reviewed-by: Adam King <[email protected]>
2 parents 3c92a0b + 1c89597 commit bc103d8

File tree

21 files changed

+1154
-94
lines changed

21 files changed

+1154
-94
lines changed

‎doc/cephadm/services/smb.rst‎

Lines changed: 30 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -9,10 +9,10 @@ SMB Service
99
.. warning::
1010

1111
SMB support is under active development and many features may be
12-
missing or immature. Additionally, a Manager module to automate
13-
SMB clusters and SMB shares is in development. Once that feature
14-
is developed it will be the preferred method for managing
15-
SMB on ceph.
12+
missing or immature. A Ceph MGR module, named smb, is available to help
13+
organize and manage SMB related featues. Unless the smb module
14+
has been determined to be unsuitable for your needs we recommend using that
15+
module over directly using the smb service spec.
1616

1717

1818
Deploying Samba Containers
@@ -78,6 +78,7 @@ features
7878
An empty list is valid. Supported terms:
7979

8080
* ``domain``: Enable domain member mode
81+
* ``clustered``: Enable Samba native cluster mode
8182

8283
config_uri
8384
A string containing a (standard or de-facto) URI that identifies a
@@ -109,6 +110,30 @@ include_ceph_users:
109110
The cephx keys for each user in the list will automatically be added to
110111
the keyring in the container.
111112

113+
cluster_meta_uri:
114+
A string containing a URI that identifies where the cluster structure
115+
metadata will be stored. Required if ``clustered`` feature is set. Must be
116+
a RADOS pseudo-URI.
117+
118+
cluster_lock_uri:
119+
A string containing a URI that identifies where Samba/CTDB will store a
120+
cluster lock. Required if ``clustered`` feature is set. Must be a RADOS
121+
pseudo-URI.
122+
123+
124+
.. note::
125+
126+
If one desires clustering between smbd instances (also known as
127+
High-Availability or "transparent state migration") the feature flag
128+
``clustered`` is needed. If this flag is not specified cephadm may deploy
129+
multiple smb servers but they will lack the coordination needed of an actual
130+
Highly-Avaiable cluster. When the ``clustered`` flag is specified cephadm
131+
will deploy additional containers that manage this coordination.
132+
Additionally, the cluster_meta_uri and cluster_lock_uri values must be
133+
specified. The former is used by cephadm to describe the smb cluster layout
134+
to the samba containers. The latter is used by Samba's CTDB component to
135+
manage an internal cluster lock.
136+
112137

113138
Configuring an SMB Service
114139
--------------------------
@@ -134,7 +159,7 @@ it accepts.
134159
When one has composed a configuration it should be stored in a location
135160
that the Samba Container can access. The recommended approach for running
136161
Samba Containers within Ceph orchestration is to store the configuration
137-
in the Ceph cluster. There are two ways to store the configuration
162+
in the Ceph cluster. There are a few ways to store the configuration
138163
in ceph:
139164

140165
RADOS
@@ -202,9 +227,5 @@ A non-exhaustive list of important limitations for the SMB service follows:
202227
configured so that it can resolve the Active Directory (AD) domain or the
203228
``custom_dns`` option may be used. In both cases DNS hosts for the AD domain
204229
must still be reachable from whatever network segment the ceph cluster is on.
205-
* Proper clustering/high-availability/"transparent state migration" is not yet
206-
supported. If a placement causes more than service to be created these
207-
services will act independently and may lead to unexpected behavior if clients
208-
access the same files at once.
209230
* Services must bind to TCP port 445. Running multiple SMB services on the same
210231
node is not yet supported and will trigger a port-in-use conflict.

‎doc/mgr/smb.rst‎

Lines changed: 17 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -55,7 +55,7 @@ Create Cluster
5555

5656
.. code:: bash
5757
58-
$ ceph smb cluster create <cluster_id> {user|active-directory} [--domain-realm=<domain_realm>] [--domain-join-user-pass=<domain_join_user_pass>] [--define-user-pass=<define_user_pass>] [--custom-dns=<custom_dns>] [--placement=<placement>]
58+
$ ceph smb cluster create <cluster_id> {user|active-directory} [--domain-realm=<domain_realm>] [--domain-join-user-pass=<domain_join_user_pass>] [--define-user-pass=<define_user_pass>] [--custom-dns=<custom_dns>] [--placement=<placement>] [--clustering=<clustering>]
5959
6060
Create a new logical cluster, identified by the cluster id value. The cluster
6161
create command must specify the authentication mode the cluster will use. This
@@ -88,6 +88,14 @@ custom_dns
8888
not configured to resolve DNS entries within AD domain(s).
8989
placement
9090
A Ceph orchestration :ref:`placement specifier <orchestrator-cli-placement-spec>`
91+
clustering
92+
Optional. Control if a cluster abstraction actually uses Samba's clustering
93+
mechanism. The value may be one of ``default``, ``always``, or ``never``.
94+
A ``default`` value indicates that clustering should be enabled if the
95+
placement count value is any value other than 1. A value of ``always``
96+
enables clustering regardless of the placement count. A value of ``never``
97+
disables clustering regardless of the placement count. If unspecified,
98+
``default`` is assumed.
9199

92100
Remove Cluster
93101
++++++++++++++
@@ -360,6 +368,14 @@ custom_dns
360368
placement
361369
Optional. A Ceph Orchestration :ref:`placement specifier
362370
<orchestrator-cli-placement-spec>`. Defaults to one host if not provided
371+
clustering
372+
Optional. Control if a cluster abstraction actually uses Samba's clustering
373+
mechanism. The value may be one of ``default``, ``always``, or ``never``.
374+
A ``default`` value indicates that clustering should be enabled if the
375+
placement count value is any value other than 1. A value of ``always``
376+
enables clustering regardless of the placement count. A value of ``never``
377+
disables clustering regardless of the placement count. If unspecified,
378+
``default`` is assumed.
363379
custom_smb_global_options
364380
Optional mapping. Specify key-value pairs that will be directly added to
365381
the global ``smb.conf`` options (or equivalent) of a Samba server. Do

‎qa/suites/orch/cephadm/smb/tasks/deploy_smb_mgr_basic.yaml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,7 @@ tasks:
3939
ceph smb cluster create modusr1 user
4040
--define-user-pass=user1%t3stP4ss1
4141
--define-user-pass=user2%t3stP4ss2
42+
--placement=count:1
4243
- cmd: ceph smb share create modusr1 share1 cephfs / --subvolume=smb/sv1
4344
- cmd: ceph smb share create modusr1 share2 cephfs / --subvolume=smb/sv2
4445
# Wait for the smb service to start

‎qa/suites/orch/cephadm/smb/tasks/deploy_smb_mgr_ctdb_res_basic.yaml‎

Lines changed: 135 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,135 @@
1+
roles:
2+
# Test is for basic smb deployment & functionality. one node cluster is OK
3+
- - host.a
4+
- mon.a
5+
- mgr.x
6+
- osd.0
7+
- osd.1
8+
- client.0
9+
- - host.b
10+
- mon.b
11+
- osd.2
12+
- osd.3
13+
- - host.c
14+
- mon.c
15+
- osd.4
16+
- osd.5
17+
# Reserve a host for acting as an smb client
18+
- - host.d
19+
- cephadm.exclude
20+
overrides:
21+
ceph:
22+
log-only-match:
23+
- CEPHADM_
24+
tasks:
25+
- cephadm.configure_samba_client_container:
26+
role: host.d
27+
- cephadm:
28+
29+
- cephadm.shell:
30+
host.a:
31+
- ceph fs volume create cephfs
32+
- cephadm.wait_for_service:
33+
service: mds.cephfs
34+
35+
- cephadm.shell:
36+
host.a:
37+
# add subvolgroup & subvolumes for test
38+
- cmd: ceph fs subvolumegroup create cephfs smb
39+
- cmd: ceph fs subvolume create cephfs sv1 --group-name=smb --mode=0777
40+
- cmd: ceph fs subvolume create cephfs sv2 --group-name=smb --mode=0777
41+
# set up smb cluster and shares
42+
- cmd: ceph mgr module enable smb
43+
# TODO: replace sleep with poll of mgr state?
44+
- cmd: sleep 30
45+
- cmd: ceph smb apply -i -
46+
stdin: |
47+
# --- Begin Embedded YAML
48+
- resource_type: ceph.smb.cluster
49+
cluster_id: uctdb1
50+
auth_mode: user
51+
user_group_settings:
52+
- {source_type: resource, ref: ug1}
53+
placement:
54+
count: 3
55+
- resource_type: ceph.smb.usersgroups
56+
users_groups_id: ug1
57+
values:
58+
users:
59+
- {name: user1, password: t3stP4ss1}
60+
- {name: user2, password: t3stP4ss2}
61+
groups: []
62+
- resource_type: ceph.smb.share
63+
cluster_id: uctdb1
64+
share_id: share1
65+
cephfs:
66+
volume: cephfs
67+
subvolumegroup: smb
68+
subvolume: sv1
69+
path: /
70+
- resource_type: ceph.smb.share
71+
cluster_id: uctdb1
72+
share_id: share2
73+
cephfs:
74+
volume: cephfs
75+
subvolumegroup: smb
76+
subvolume: sv2
77+
path: /
78+
# --- End Embedded YAML
79+
# Wait for the smb service to start
80+
- cephadm.wait_for_service:
81+
service: smb.uctdb1
82+
# Since this is a true cluster there should be a clustermeta in rados
83+
- cephadm.shell:
84+
host.a:
85+
- cmd: rados --pool=.smb -N uctdb1 get cluster.meta.json /dev/stdout
86+
87+
# Check if shares exist
88+
- cephadm.exec:
89+
host.d:
90+
- sleep 30
91+
- "{{ctx.samba_client_container_cmd|join(' ')}} smbclient -U user1%t3stP4ss1 //{{'host.a'|role_to_remote|attr('ip_address')}}/share1 -c ls"
92+
- "{{ctx.samba_client_container_cmd|join(' ')}} smbclient -U user2%t3stP4ss2 //{{'host.a'|role_to_remote|attr('ip_address')}}/share2 -c ls"
93+
94+
# verify CTDB is healthy, cluster well formed
95+
- cephadm.exec:
96+
host.a:
97+
- "{{ctx.cephadm}} ls --no-detail | {{ctx.cephadm}} shell jq -r 'map(select(.name | startswith(\"smb.uctdb1\")))[-1].name' > /tmp/svcname"
98+
- "{{ctx.cephadm}} enter -n $(cat /tmp/svcname) ctdb status > /tmp/ctdb_status"
99+
- cat /tmp/ctdb_status
100+
- grep 'pnn:0 .*OK' /tmp/ctdb_status
101+
- grep 'pnn:1 .*OK' /tmp/ctdb_status
102+
- grep 'pnn:2 .*OK' /tmp/ctdb_status
103+
- grep 'Number of nodes:3' /tmp/ctdb_status
104+
- rm -rf /tmp/svcname /tmp/ctdb_status
105+
106+
# Test a different host in the cluster
107+
- cephadm.exec:
108+
host.d:
109+
- sleep 30
110+
- "{{ctx.samba_client_container_cmd|join(' ')}} smbclient -U user1%t3stP4ss1 //{{'host.c'|role_to_remote|attr('ip_address')}}/share1 -c ls"
111+
- "{{ctx.samba_client_container_cmd|join(' ')}} smbclient -U user2%t3stP4ss2 //{{'host.c'|role_to_remote|attr('ip_address')}}/share2 -c ls"
112+
113+
- cephadm.shell:
114+
host.a:
115+
- cmd: ceph smb apply -i -
116+
stdin: |
117+
# --- Begin Embedded YAML
118+
- resource_type: ceph.smb.cluster
119+
cluster_id: uctdb1
120+
intent: removed
121+
- resource_type: ceph.smb.usersgroups
122+
users_groups_id: ug1
123+
intent: removed
124+
- resource_type: ceph.smb.share
125+
cluster_id: uctdb1
126+
share_id: share1
127+
intent: removed
128+
- resource_type: ceph.smb.share
129+
cluster_id: uctdb1
130+
share_id: share2
131+
intent: removed
132+
# --- End Embedded YAML
133+
# Wait for the smb service to be removed
134+
- cephadm.wait_for_service_not_present:
135+
service: smb.uctdb1

‎qa/suites/orch/cephadm/smb/tasks/deploy_smb_mgr_ctdb_res_dom.yaml‎

Lines changed: 138 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,138 @@
1+
roles:
2+
# Test is for basic smb deployment & functionality. one node cluster is OK
3+
- - host.a
4+
- mon.a
5+
- mgr.x
6+
- osd.0
7+
- osd.1
8+
- client.0
9+
- - host.b
10+
- mon.b
11+
- osd.2
12+
- osd.3
13+
- - host.c
14+
- mon.c
15+
- osd.4
16+
- osd.5
17+
# Reserve a host for acting as a domain controller and smb client
18+
- - host.d
19+
- cephadm.exclude
20+
overrides:
21+
ceph:
22+
log-only-match:
23+
- CEPHADM_
24+
tasks:
25+
- cephadm.deploy_samba_ad_dc:
26+
role: host.d
27+
- cephadm:
28+
29+
- cephadm.shell:
30+
host.a:
31+
- ceph fs volume create cephfs
32+
- cephadm.wait_for_service:
33+
service: mds.cephfs
34+
35+
- cephadm.shell:
36+
host.a:
37+
# add subvolgroup & subvolumes for test
38+
- cmd: ceph fs subvolumegroup create cephfs smb
39+
- cmd: ceph fs subvolume create cephfs sv1 --group-name=smb --mode=0777
40+
- cmd: ceph fs subvolume create cephfs sv2 --group-name=smb --mode=0777
41+
# set up smb cluster and shares
42+
- cmd: ceph mgr module enable smb
43+
# TODO: replace sleep with poll of mgr state?
44+
- cmd: sleep 30
45+
- cmd: ceph smb apply -i -
46+
stdin: |
47+
# --- Begin Embedded YAML
48+
- resource_type: ceph.smb.cluster
49+
cluster_id: adctdb1
50+
auth_mode: active-directory
51+
domain_settings:
52+
realm: DOMAIN1.SINK.TEST
53+
join_sources:
54+
- source_type: resource
55+
ref: join1-admin
56+
custom_dns:
57+
- "{{ctx.samba_ad_dc_ip}}"
58+
placement:
59+
count: 3
60+
- resource_type: ceph.smb.join.auth
61+
auth_id: join1-admin
62+
auth:
63+
username: Administrator
64+
password: Passw0rd
65+
- resource_type: ceph.smb.share
66+
cluster_id: adctdb1
67+
share_id: share1
68+
cephfs:
69+
volume: cephfs
70+
subvolumegroup: smb
71+
subvolume: sv1
72+
path: /
73+
- resource_type: ceph.smb.share
74+
cluster_id: adctdb1
75+
share_id: share2
76+
cephfs:
77+
volume: cephfs
78+
subvolumegroup: smb
79+
subvolume: sv2
80+
path: /
81+
# --- End Embedded YAML
82+
# Wait for the smb service to start
83+
- cephadm.wait_for_service:
84+
service: smb.adctdb1
85+
# Since this is a true cluster there should be a clustermeta in rados
86+
- cephadm.shell:
87+
host.a:
88+
- cmd: rados --pool=.smb -N adctdb1 get cluster.meta.json /dev/stdout
89+
90+
# Check if shares exist
91+
- cephadm.exec:
92+
host.d:
93+
- sleep 30
94+
- "{{ctx.samba_client_container_cmd|join(' ')}} smbclient -U DOMAIN1\\\\ckent%1115Rose. //{{'host.a'|role_to_remote|attr('ip_address')}}/share1 -c ls"
95+
- "{{ctx.samba_client_container_cmd|join(' ')}} smbclient -U DOMAIN1\\\\ckent%1115Rose. //{{'host.a'|role_to_remote|attr('ip_address')}}/share2 -c ls"
96+
97+
# verify CTDB is healthy, cluster well formed
98+
- cephadm.exec:
99+
host.a:
100+
- "{{ctx.cephadm}} ls --no-detail | {{ctx.cephadm}} shell jq -r 'map(select(.name | startswith(\"smb.adctdb1\")))[-1].name' > /tmp/svcname"
101+
- "{{ctx.cephadm}} enter -n $(cat /tmp/svcname) ctdb status > /tmp/ctdb_status"
102+
- cat /tmp/ctdb_status
103+
- grep 'pnn:0 .*OK' /tmp/ctdb_status
104+
- grep 'pnn:1 .*OK' /tmp/ctdb_status
105+
- grep 'pnn:2 .*OK' /tmp/ctdb_status
106+
- grep 'Number of nodes:3' /tmp/ctdb_status
107+
- rm -rf /tmp/svcname /tmp/ctdb_status
108+
109+
# Test a different host in the cluster
110+
- cephadm.exec:
111+
host.d:
112+
- sleep 30
113+
- "{{ctx.samba_client_container_cmd|join(' ')}} smbclient -U DOMAIN1\\\\ckent%1115Rose. //{{'host.c'|role_to_remote|attr('ip_address')}}/share1 -c ls"
114+
- "{{ctx.samba_client_container_cmd|join(' ')}} smbclient -U DOMAIN1\\\\ckent%1115Rose. //{{'host.c'|role_to_remote|attr('ip_address')}}/share2 -c ls"
115+
116+
- cephadm.shell:
117+
host.a:
118+
- cmd: ceph smb apply -i -
119+
stdin: |
120+
# --- Begin Embedded YAML
121+
- resource_type: ceph.smb.cluster
122+
cluster_id: adctdb1
123+
intent: removed
124+
- resource_type: ceph.smb.join.auth
125+
auth_id: join1-admin
126+
intent: removed
127+
- resource_type: ceph.smb.share
128+
cluster_id: adctdb1
129+
share_id: share1
130+
intent: removed
131+
- resource_type: ceph.smb.share
132+
cluster_id: adctdb1
133+
share_id: share2
134+
intent: removed
135+
# --- End Embedded YAML
136+
# Wait for the smb service to be removed
137+
- cephadm.wait_for_service_not_present:
138+
service: smb.adctdb1

0 commit comments

Comments
 (0)