Skip to content

Commit 8a31988

Browse files
xandrusxandrus
authored
Merge pull request #17 from rocinante-sys/master
Give the option to generate configuration on container startup
2 parents 38a1948 + 94c71aa commit 8a31988

File tree

10 files changed

+225
-9
lines changed

10 files changed

+225
-9
lines changed

‎README.md‎

Lines changed: 140 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -116,6 +116,145 @@ Also on Linux, CGI Server container is available at the IP address: [http://172.
116116

117117
Your MooseFS Docker cluster is persistent. It means all files you created in the `/mnt/moosefs` folder will remain there even after turning containers off.
118118
All data and metadata files are stored in the host `./data` directory.
119+
120+
# Pass config as env variable
121+
122+
There might be situations where you would want to setup a config file on the container start.
123+
For that scenario you can pass the config file as a base64 encoded text. For example lets say
124+
you want to setup your chunk servers to connect to master in k8s cluster where IP's are dynamically assigned
125+
to pods. You have you master yaml definition set up as:
126+
```
127+
apiVersion: apps/v1
128+
kind: Deployment
129+
metadata:
130+
name: moosefs-master
131+
namespace: storage
132+
spec:
133+
replicas: 1
134+
selector:
135+
matchLabels:
136+
app: moosefs-master
137+
template:
138+
metadata:
139+
labels:
140+
app: moosefs-master
141+
spec:
142+
nodeSelector:
143+
"beta.kubernetes.io/os": linux
144+
containers:
145+
- name: moosefs-master
146+
image: rocinantesystems/moosefs-master:latest
147+
resources:
148+
requests:
149+
cpu: 100m
150+
memory: 128Mi
151+
limits:
152+
cpu: 250m
153+
memory: 256Mi
154+
ports:
155+
- containerPort: 9419
156+
- containerPort: 9420
157+
- containerPort: 9421
158+
volumeMounts:
159+
- name: moosefs-master-mfs
160+
mountPath: /var/lib/mfs
161+
volumes:
162+
- name: moosefs-master-mfs
163+
azureDisk:
164+
kind: Managed
165+
diskName: MooseMasterMfs
166+
diskURI: /subscriptions/<subscriptionID>/resourceGroups/<resourceGroup>/providers/Microsoft.Compute/disks/MooseMasterMfs
167+
---
168+
apiVersion: v1
169+
kind: Service
170+
metadata:
171+
name: moosefs-master
172+
namespace: storage
173+
spec:
174+
type: NodePort
175+
ports:
176+
- port: 9419
177+
targetPort: 9419
178+
name: listen-metalogger
179+
- port: 9420
180+
targetPort: 9420
181+
name: listen
182+
- port: 9421
183+
targetPort: 9421
184+
name: listen-client
185+
selector:
186+
app: moosefs-master
187+
```
188+
189+
This will reserve an IP in the cluster where the ports will be reached.
190+
191+
In order for your chunkservers to automatically connect to this IP you would need to have your `mfschunkserver.cfg` defined as:
192+
```
193+
MASTER_HOST = $MOOSEFS_MASTER_SERVICE_HOST
194+
CSSERV_LISTEN_PORT = $MOOSEFS_CHUNKSERVER_SERVICE_PORT
195+
DATA_PATH = /mnt/hdd0/mfs
196+
```
197+
`MOOSEFS_MASTER_SERVICE_HOST` variable is set by k8s cluster and contains IP where the master service is accessible by. `MOOSEFS_CHUNKSERVER_SERVICE_PORT` this is the port on which we will expose our chunk server.
198+
199+
Base64 encoded config data is:
200+
```
201+
TUFTVEVSX0hPU1QgPSAkTU9PU0VGU19NQVNURVJfU0VSVklDRV9IT1NUCkNTU0VSVl9MSVNURU5fUE9SVCA9ICRNT09TRUZTX0NIVU5LU0VSVkVSX1NFUlZJQ0VfUE9SVApEQVRBX1BBVEggPSAvbW50L2hkZDAvbWZzCg==
202+
```
203+
204+
Now we can spin up chunkserver(s) with:
205+
```
206+
apiVersion: apps/v1
207+
kind: Deployment
208+
metadata:
209+
name: moosefs-chunkserver-1
210+
namespace: storage
211+
spec:
212+
replicas: 1
213+
selector:
214+
matchLabels:
215+
app: moosefs-chunkserver-1
216+
template:
217+
metadata:
218+
labels:
219+
app: moosefs-chunkserver-1
220+
spec:
221+
nodeSelector:
222+
"beta.kubernetes.io/os": linux
223+
containers:
224+
- name: moosefs-chunkserver-1
225+
env:
226+
- name: MFS_CHUNKSERVER_CONFIG
227+
value: TUFTVEVSX0hPU1QgPSAkTU9PU0VGU19NQVNURVJfU0VSVklDRV9IT1NUCkNTU0VSVl9MSVNURU5fUE9SVCA9ICRNT09TRUZTX0NIVU5LU0VSVkVSX1NFUlZJQ0VfUE9SVApEQVRBX1BBVEggPSAvbW50L2hkZDAvbWZzCg==
228+
- name: SIZE
229+
value: 16
230+
image: rocinantesystems/moosefs-chunkserver:latest
231+
ports:
232+
- containerPort: 9422
233+
volumeMounts:
234+
- name: moosefs-chunkserver-data-1
235+
mountPath: /mnt/hdd0
236+
volumes:
237+
- name: moosefs-chunkserver-data-1
238+
azureDisk:
239+
kind: Managed
240+
diskName: MfsHangfireData1
241+
diskURI: /subscriptions/<subscriptionID>/resourceGroups/<resourceGroup>/providers/Microsoft.Compute/disks/MfsHangfireData1
242+
---
243+
apiVersion: v1
244+
kind: Service
245+
metadata:
246+
name: moosefs-chunkserver-1
247+
namespace: storage
248+
spec:
249+
type: NodePort
250+
ports:
251+
- port: 9422
252+
targetPort: 9422
253+
selector:
254+
app: moosefs-chunkserver-1
255+
```
256+
Repeat this for other chunk servers modifying your base64 string accordingly. If you leave all the chunkservers on default port `9422` you can use same base64 encoded string `TUFTVEVSX0hPU1QgPSAkTU9PU0VGU19NQVNURVJfU0VSVklDRV9IT1NUCkRBVEFfUEFUSCA9IC9tbnQvaGRkMC9tZnMK` which will only set correct `MASTER_HOST` and `DATA_PATH`
257+
119258
# Docker Hub
120259

121260
| Image name | Image size | Pulls | Stars | Build |
@@ -124,4 +263,4 @@ All data and metadata files are stored in the host `./data` directory.
124263
| [moosefs/chunkserver](https://hub.docker.com/r/moosefs/chunkserver/) | ![Docker Image Size (latest by date)](https://img.shields.io/docker/image-size/moosefs/chunkserver?sort=date) | [![chunkserver](https://img.shields.io/docker/pulls/moosefs/chunkserver)](https://hub.docker.com/r/moosefs/chunkserver/) | ![chunkserver](https://img.shields.io/docker/stars/moosefs/chunkserver) | ![chunkserver](https://img.shields.io/docker/build/moosefs/chunkserver) |
125264
| [moosefs/client](https://hub.docker.com/r/moosefs/client/) | ![Docker Image Size (latest by date)](https://img.shields.io/docker/image-size/moosefs/client?sort=date) | [![client](https://img.shields.io/docker/pulls/moosefs/client)](https://hub.docker.com/r/moosefs/client/) | ![client](https://img.shields.io/docker/stars/moosefs/client) | ![client](https://img.shields.io/docker/build/moosefs/client) |
126265
| [moosefs/metalogger](https://hub.docker.com/r/moosefs/metalogger/) | ![Docker Image Size (latest by date)](https://img.shields.io/docker/image-size/moosefs/metalogger?sort=date) | [![metalogger](https://img.shields.io/docker/pulls/moosefs/metalogger)](https://hub.docker.com/r/moosefs/cgi/) | ![metalogger](https://img.shields.io/docker/stars/moosefs/metalogger) | ![metalogger](https://img.shields.io/docker/build/moosefs/metalogger) |
127-
| [moosefs/cgi](https://hub.docker.com/r/moosefs/cgi/) | ![Docker Image Size (latest by date)](https://img.shields.io/docker/image-size/moosefs/cgi?sort=date) | [![cgi](https://img.shields.io/docker/pulls/moosefs/cgi)](https://hub.docker.com/r/moosefs/cgi/) | ![cgi](https://img.shields.io/docker/stars/moosefs/cgi) | ![cgi](https://img.shields.io/docker/build/moosefs/cgi) |
266+
| [moosefs/cgi](https://hub.docker.com/r/moosefs/cgi/) | ![Docker Image Size (latest by date)](https://img.shields.io/docker/image-size/moosefs/cgi?sort=date) | [![cgi](https://img.shields.io/docker/pulls/moosefs/cgi)](https://hub.docker.com/r/moosefs/cgi/) | ![cgi](https://img.shields.io/docker/stars/moosefs/cgi) | ![cgi](https://img.shields.io/docker/build/moosefs/cgi) |

‎moosefs-cgi/Dockerfile‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,4 +13,8 @@ RUN apt-get update && apt-get install -y moosefs-cgi moosefs-cgiserv moosefs-cli
1313
# Expose ports master ports
1414
EXPOSE 9425
1515

16-
CMD ["mfscgiserv", "-f"]
16+
# Add start script
17+
ADD cgiserver.sh /usr/sbin/cgiserver.sh
18+
RUN chown root:root /usr/sbin/cgiserver.sh ; chmod 700 /usr/sbin/cgiserver.sh
19+
20+
CMD ["cgiserver.sh"]

‎moosefs-cgi/cgiserver.sh‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
#!/usr/bin/env bash
2+
3+
CMD="mfscgiserv -f"
4+
5+
#Add host if set
6+
if [ ! -z ${MASTER_HOST+X} ];
7+
then
8+
CMD="$CMD -H $MASTER_HOST"
9+
fi
10+
11+
#Add host if set
12+
if [ ! -z ${MASTER_PORT+X} ];
13+
then
14+
CMD="$CMD -P $MASTER_PORT"
15+
fi
16+
17+
exec $CMD

‎moosefs-chunkserver/Dockerfile‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
FROM debian:buster
22

33
# Install wget, lsb-release and curl
4-
RUN apt-get update && apt-get install -y wget gnupg2
4+
RUN apt-get update && apt-get install -y wget gnupg2 gettext-base
55

66
# Add key
77
RUN wget -O - http://ppa.moosefs.com/moosefs.key 2>/dev/null | apt-key add - 2>/dev/null

‎moosefs-chunkserver/chunkserver.sh‎

Lines changed: 22 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,29 @@
11
#!/usr/bin/env bash
22

3+
mkdir -p /mnt/hdd0/mfs
34
# Set correct owner
4-
chown -R mfs:mfs /mnt/hdd0 /var/lib/mfs
5+
chown -R mfs:mfs /mnt/hdd0 /mnt/hdd0/mfs /var/lib/mfs
56

6-
#Add size to hdd is defined
7+
# Overwrite mfschunkserver.cfg if passed in
8+
# this will base64 decode MFS_CHUNKSERVER_CONFIG variable text
9+
# substitute any env variables in decoded text
10+
# save text into /etc/mfs/mfschunkserver.cfg
11+
if [ ! -z ${MFS_CHUNKSERVER_CONFIG+X} ];
12+
then
13+
echo $MFS_CHUNKSERVER_CONFIG | base64 -d | envsubst > /etc/mfs/mfschunkserver.cfg
14+
fi
15+
16+
# Overwrite mfshdd.cfg if passed in
17+
# this will base64 decode MFS_HDD_CONFIG variable text
18+
# substitute any env variables in decoded text
19+
# save text into /etc/mfs/mfshdd.cfg
20+
if [ ! -z ${MFS_HDD_CONFIG+X} ];
21+
then
22+
echo $MFS_HDD_CONFIG | base64 -d | envsubst > /etc/mfs/mfshdd.cfg
23+
fi
24+
25+
26+
#Add size to hdd if defined
727
if [ -z ${SIZE+X} ];
828
then
929
echo "/mnt/hdd0" > /etc/mfs/mfshdd.cfg

‎moosefs-client/Dockerfile‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,8 @@ RUN echo "deb http://ppa.moosefs.com/3.0.115/apt/debian/buster buster main" > /e
1010
# Install MooseFS client and cli
1111
RUN apt-get update && apt-get install -y moosefs-client moosefs-cli
1212

13-
# Make a moosefs mountpoint
14-
RUN mkdir -p /mnt/moosefs
13+
# Add start script
14+
ADD mount.sh /usr/sbin/mount.sh
15+
RUN chown root:root /usr/sbin/mount.sh ; chmod 700 /usr/sbin/mount.sh ; mkdir -p /mnt/moosefs
1516

16-
CMD ["mfsmount", "-f", "/mnt/moosefs"]
17+
CMD ["mount.sh"]

‎moosefs-client/mount.sh‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
#!/usr/bin/env bash
2+
3+
CMD="mfsmount /mnt/moosefs -f"
4+
5+
#Add host if set
6+
if [ ! -z ${MASTER_HOST+X} ];
7+
then
8+
CMD="$CMD -H $MASTER_HOST"
9+
fi
10+
11+
#Add host if set
12+
if [ ! -z ${MASTER_PORT+X} ];
13+
then
14+
CMD="$CMD -P $MASTER_PORT"
15+
fi
16+
17+
exec $CMD

‎moosefs-master/master.sh‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,15 @@ MFS_ENV="${MFS_ENV:-PROD}"
66
#Set correct owner
77
chown -R mfs:mfs /var/lib/mfs
88

9+
# Overwrite mfsmaster.cfg if passed in
10+
# this will base64 decode MFS_MASTER_CONFIG variable text
11+
# substitute any env variables in decoded text
12+
# save text into /etc/mfs/mfsmaster.cfg
13+
if [ ! -z ${MFS_MASTER_CONFIG+X} ];
14+
then
15+
echo $MFS_MASTER_CONFIG | base64 -d | envsubst > /etc/mfs/mfsmaster.cfg
16+
fi
17+
918
# We have to be sure that we have metadata files
1019
if [ -f /var/lib/mfs/metadata.mfs ];
1120
then

‎moosefs-metalogger/Dockerfile‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
FROM debian:buster
22

33
# Install wget and gnupg2
4-
RUN apt-get update && apt-get install -y wget gnupg2
4+
RUN apt-get update && apt-get install -y wget gnupg2 gettext-base
55

66
# Add key
77
RUN wget -O - http://ppa.moosefs.com/moosefs.key 2>/dev/null | apt-key add - 2>/dev/null

‎moosefs-metalogger/metalogger.sh‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,5 +6,14 @@ MFS_ENV="${MFS_ENV:-PROD}"
66
# Set a correct owner
77
chown -R mfs:mfs /var/lib/mfs
88

9+
# Overwrite mfsmetalogger.cfg if passed in
10+
# this will base64 decode MFS_METALOGGER_CONFIG variable text
11+
# substitute any env variables in decoded text
12+
# save text into /etc/mfs/mfsmetalogger.cfg
13+
if [ ! -z ${MFS_METALOGGER_CONFIG+X} ];
14+
then
15+
echo $MFS_METALOGGER_CONFIG | base64 -d | envsubst > /etc/mfs/mfsmetalogger.cfg
16+
fi
17+
918
# Run
1019
exec mfsmetalogger -f

0 commit comments

Comments
 (0)