Skip to content

Commit 0d14b72

Browse files
badlopbadlop
committed
New page "Distribution" that covers the erlang distribution topics
1 parent db0c5f0 commit 0d14b72

File tree

7 files changed

+282
-198
lines changed

7 files changed

+282
-198
lines changed

‎content/admin/guide/clustering.md‎

Lines changed: 54 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -16,11 +16,11 @@ clustering is a must have feature.
1616
## How it Works
1717

1818
A XMPP domain is served by one or more ejabberd nodes. These nodes can
19-
be run on different machines that are connected via a network. They all
20-
must have the ability to connect to port 4369 of all another nodes, and
21-
must have the same magic cookie (see Erlang/OTP documentation, in other
22-
words the file `~ejabberd/.erlang.cookie` must be the same on all
23-
nodes). This is needed because all nodes exchange information about
19+
be run on different machines that are connected via a reliable network.
20+
21+
Those ejabberd nodes get connected between them using
22+
[Erlang Distribution](distribution.md),
23+
and then those nodes exchange information about
2424
connected users, s2s connections, registered services, etc…
2525

2626
Each ejabberd node has the following modules:
@@ -60,45 +60,72 @@ domain of the packet’s destination exists. If that is the case, the s2s
6060
manager routes the packet to the process serving this connection,
6161
otherwise a new connection is opened.
6262

63-
## Before you get started
63+
## Managing nodes in a cluster
6464

65-
Before you start implementing clustering, there are a few things you
66-
need to take into account:
65+
Before you setup clustering, there are a few things you need to take into account:
6766

6867
- Cluster should be set up in a single data center: The clustering in
6968
ejabberd Community Server relies on low latency networking. While it may
7069
work across regions, it is recommended that you run an ejabberd
7170
cluster in a single Amazon region.
72-
- Clustering relies on Erlang features and Mnesia shared schemas. Before
71+
72+
- Clustering ejabberd relies on [Erlang Distribution](distribution.md)
73+
and Mnesia shared schemas. Before
7374
getting started, it is best to get familiar with the Erlang environment
7475
as this guide will heavily reference Erlang terms.
7576

76-
## Clustering Setup
77+
### Preparation
7778

78-
def:erlang node
79-
: A node in erlang connected to a cluster.
80-
See [Erlang node name](security.md#erlang-node-name)
79+
Make sure all the nodes that you plan to group in the cluster have the same
80+
[cookie](distribution.md#cookie) file.
81+
You can simply copy the file `.erlang.cookie`
82+
from one of the nodes to all the other nodes.
8183

82-
## Adding a node to a cluster
84+
Then check that each ejabberd node has a different erlang [node name](distribution.md#node-name)
85+
set in the option `ERLANG_NODE` in the file `ejabberdctl.cfg`.
86+
For example, let's assume in a machine named `machine1` you configure:
8387

84-
Suppose you have already configured ejabberd on one node named
85-
`ejabberd01`. Let's create an additional node (`ejabberd02`) and connect them
86-
together.
88+
``` sh
89+
ERLANG_NODE=ejabberd1@machine1
90+
```
8791

88-
1. Copy the `/home/ejabberd/.erlang.cookie` file from `ejabberd01` to
89-
`ejabberd02`.
92+
and in `machine2` you configure:
9093

91-
Alternatively you could pass the `-setcookie <value>`
92-
option to all `erl` commands below.
94+
``` sh
95+
ERLANG_NODE=ejabberd2@machine2
96+
```
9397

94-
2. Make sure your new ejabberd node is properly configured. Usually,
95-
you want to have the same `ejabberd.yml` config file on the new node that on the
96-
other cluster nodes.
98+
Make sure your new ejabberd node is properly configured. Usually,
99+
you want to have the same `ejabberd.yml` config file on the new node that on the
100+
other cluster nodes.
97101

98-
3. Adding a node to the cluster is done by starting a new ejabberd node within the same network, and running [join_cluster](../../developer/ejabberd-api/admin-api.md#join_cluster) from a cluster node. On the `ejabberd02` node for example, as ejabberd is already started, run the following command as the ejabberd daemon user, using the ejabberdctl script:
102+
### Adding a node to a cluster
103+
104+
Let's assume the ejabberd server running in `ejabberd1@machine1`
105+
already contains a database with accounts and other content,
106+
and let's consider that node as the initial cluster.
107+
108+
Now let's add to that cluster a new node `ejabberd2@machine2`
109+
which has no valuable information in its database
110+
(the contents of its Mnesia database will get removed and overwritten).
111+
112+
Go to machine2, check that you are accessing the second node, tell it to
113+
[join the cluster](../../developer/ejabberd-api/admin-api.md#join_cluster)
114+
of the first node, and check the
115+
[list](../../developer/ejabberd-api/admin-api.md#list_cluster)
116+
of nodes in the cluster:
99117

100118
``` sh
101-
ejabberdctl --no-timeout join_cluster 'ejabberd@ejabberd01'
119+
$ ejabberdctl status
120+
The node ejabberd2@machine2 is started with status: started
121+
ejabberd 25.12 is running in that node
122+
123+
$ ejabberdctl --no-timeout join_cluster ejabberd1@machine1
124+
Trying to join that cluster, wait a few seconds and check the list of nodes.
125+
126+
$ ejabberdctl list_cluster
127+
ejabberd1@machine1
128+
ejabberd2@machine2
102129
```
103130

104131
This enables ejabberd's internal replications to be launched across all nodes so new nodes can start receiving messages from other nodes and be registered in the routing tables.
@@ -125,7 +152,7 @@ command must be run as the ejabberd daemon user, from one node of the
125152
cluster:
126153

127154
``` sh
128-
ejabberdctl leave_cluster 'ejabberd@ejabberd02'
155+
ejabberdctl leave_cluster 'ejabberd2@machine2'
129156
```
130157

131158
The removed node must be running while calling leave_cluster to make

‎content/admin/guide/distribution.md‎

Lines changed: 158 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,158 @@
1+
# Erlang Distribution
2+
3+
## Overview
4+
5+
ejabberd uses the
6+
[Distributed Erlang](https://www.erlang.org/doc/system/distributed.html)
7+
feature for two purposes:
8+
9+
- [`ejabberdctl`](managing.md#ejabberdctl) shell script
10+
that connects to your running ejabberd node to allow you executing
11+
[API](../../developer/ejabberd-api/index.md) commands in it,
12+
opening an interactive erlang shell, ...
13+
14+
- [clustering](clustering.md) several ejabberd nodes that
15+
you deploy in separate machines to serve a large XMPP domain
16+
with fault-tolerance and scalability.
17+
18+
There are three topics involved in erlang distribution:
19+
20+
- [Cookie](#cookie), automatically generated, shared by your ejabberd nodes, and secret to the world
21+
- [Node Name](#node-name) that you pick for each ejabberd node to identify them in a cluster
22+
- **Port** number to use, determined using either:
23+
- [ERL_DIST_PORT](#erl_dist_port) environment variable that lets you assign your desired port number
24+
- [epmd](#epmd) program that assigns a random port number and maps names with ports
25+
26+
## Cookie
27+
28+
def:cookie
29+
: Random alphanumeric string assigned to each erlang [](def:node) used to secure connections between erlang nodes.
30+
See [Security in distributed erlang](https://www.erlang.org/doc/system/distributed#security).
31+
32+
An Erlang node
33+
reads the cookie at startup from the command-line parameter
34+
`-setcookie`. If not indicated, the cookie is read from the file
35+
`$HOME/.erlang.cookie`.
36+
37+
If this file does not exist, it is created
38+
immediately with a random cookie in the user `$HOME` path.
39+
This means the user running ejabberd must have a `$HOME`,
40+
and have write access to that path.
41+
So, when you create a new account in your system for running ejabberd,
42+
either allow it to have a `$HOME`,
43+
or set as `$HOME` a path where ejabberd will have write access.
44+
Depending on your setup, examples could be:
45+
46+
``` sh
47+
adduser --home /usr/local/var/lib/ejabberd ejabberd
48+
```
49+
50+
or
51+
52+
``` sh
53+
adduser --home /var/lib/ejabberd ejabberd
54+
```
55+
56+
Two Erlang nodes communicate only if
57+
they have the same cookie. Setting a cookie on the Erlang node allows
58+
you to structure your Erlang network and define which nodes are allowed
59+
to connect to which.
60+
61+
Thanks to Erlang cookies, you can prevent access to the Erlang node by
62+
mistake, for example when there are several Erlang nodes running
63+
different programs in the same machine.
64+
65+
Setting a secret cookie is a simple method to difficult unauthorized
66+
access to your Erlang node. However, the cookie system is not ultimately
67+
effective to prevent unauthorized access or intrusion to an Erlang node.
68+
The communication between Erlang nodes are not encrypted, so the [](def:cookie)
69+
could be read sniffing the traffic on the network. The recommended way
70+
to secure the Erlang node is to block the port 4369.
71+
72+
## Node Name
73+
74+
def:erlang node
75+
: A node in erlang connected to a cluster.
76+
See [Erlang node name](distribution.md#node-name)
77+
78+
| Variable in ejabberdctl.cfg | Corresponding argument in Erlang/OTP |
79+
|---------------------------|--------------------------------------|
80+
| `ERLANG_NODE=ejabberd@localhost` | -[name](https://www.erlang.org/doc/apps/erts/erl_cmd#name) `ejabberd@localhost` <br>-[sname](https://www.erlang.org/doc/apps/erts/erl_cmd#sname) `ejabberd` |
81+
82+
An Erlang node may have a node name. The name can be short (if indicated
83+
with the command-line parameter `-sname`) or long (if indicated with the
84+
parameter `-name`). Starting an Erlang node with `-sname` limits the
85+
communication between Erlang nodes to the LAN.
86+
87+
Using the argument `-sname` instead of `-name` is a simple method to
88+
difficult unauthorized access to your Erlang node. However, it is not
89+
ultimately effective to prevent access to the Erlang node, because it
90+
may be possible to fake the fact that you are on another network using a
91+
modified version of Erlang [](def:epmd). The recommended way to secure the
92+
Erlang node is to block the port 4369.
93+
94+
## `ERL_DIST_PORT`
95+
96+
| Variable in ejabberdctl.cfg | Corresponding argument in Erlang/OTP |
97+
|---------------------------|--------------------------------------|
98+
| `INET_DIST_INTERFACE` | -kernel [inet_dist_use_interface](https://www.erlang.org/doc/apps/kernel/kernel_app.html#inet_dist_use_interface) |
99+
| `ERL_DIST_PORT=5210` | -[erl_epmd_port](https://www.erlang.org/docs/28/apps/erts/erl_cmd#erl_epmd_port) `5210` -[start_epmd](https://www.erlang.org/docs/28/apps/erts/erl_cmd.html#start_epmd) `false` |
100+
101+
The TCP port where your Erlang node listens for Erlang distribution connections
102+
can be set with the environment variable `ERL_DIST_PORT` in the file `ejabberdctl.cfg`.
103+
When attempting to connect to other remote nodes,
104+
it will also try to connect to that port number.
105+
106+
By default this variable is set to `5210`,
107+
but you can set any number you prefer as long as it isn't used in the machine,
108+
and it is the same for all the other ejabberd nodes that will build the cluster.
109+
110+
When building a cluster of several Erlang nodes,
111+
all of them must have the same `ERL_DIST_PORT`,
112+
because this is used for listening and also for connecting.
113+
Consequently, if you want to build a cluster of several nodes in the same machine,
114+
each node must have a different IP address in `INET_DIST_INTERFACE`.
115+
116+
Just as a small note: since Erlang/OTP 27.2, the argument `-erl_epmd_port` is obsoleted by
117+
[-kernel erl_epmd_node_listen_port](https://www.erlang.org/docs/28/apps/kernel/kernel_app#erl_epmd_node_listen_port).
118+
However, ejabberd does not yet use the new argument because ejabberd supports Erlang/OTP 25+.
119+
120+
121+
## epmd
122+
123+
| Variable in ejabberdctl.cfg | Corresponding argument in Erlang/OTP |
124+
|---------------------------|--------------------------------------|
125+
| `INET_DIST_INTERFACE` | -kernel [inet_dist_use_interface](https://www.erlang.org/doc/apps/kernel/kernel_app.html#inet_dist_use_interface) |
126+
| `FIREWALL_WINDOW=XX-YY` | -kernel [inet_dist_listen_min](https://www.erlang.org/doc/apps/kernel/kernel_app.html#inet_dist_listen) `XX` inet_dist_listen_max `YY` |
127+
| `ERL_EPMD_ADDRESS` | [ERL_EPMD_ADDRESS](https://www.erlang.org/doc/apps/erts/epmd_cmd.html#environment-variables) |
128+
129+
def:epmd
130+
: Small name server included in Erlang/OTP and used by Erlang
131+
programs when establishing distributed Erlang communications.
132+
See [`epmd (Erlang Port Mapper Daemon)`](https://www.erlang.org/doc/apps/erts/epmd_cmd.html)
133+
134+
When [ERL_DIST_PORT](#erl_dist_port) is not set in `ejabberdctl.cfg`,
135+
Erlang starts the `epmd` program (which listens in port `4369`),
136+
and connects to that program to ask for a random port number to use.
137+
138+
You should block the port `4369` in the firewall in such a way that
139+
only the programs in your machine can access it,
140+
or configure the environment variable `ERL_EPMD_ADDRESS`
141+
in the file `ejabberdctl.cfg`.
142+
143+
When building a cluster of several ejabberd nodes,
144+
if you don't set [ERL_DIST_PORT](#erl_dist_port)
145+
with the same port number in `ejabberdctl.cfg` in all the nodes,
146+
then you must open the port `4369` for all the machines
147+
involved in the cluster so their epmd programs can be accessed.
148+
Remember to block the port so Internet doesn't have access to it.
149+
150+
Once an Erlang node solved the node name of another Erlang node using
151+
EPMD and port `4369`, the nodes communicate directly. The ports used in
152+
this case by default are random,
153+
and can be restricted with the environment variable `FIREWALL_WINDOW`
154+
in the file `ejabberdctl.cfg`.
155+
156+
The network interface where the Erlang node will listen and accept connections
157+
can be configured with the environment variable `INET_DIST_INTERFACE`
158+
in the file `ejabberdctl.cfg`.

0 commit comments

Comments
 (0)