1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
|
<?xml version="1.0" encoding="utf-8" ?>
<!DOCTYPE chapter SYSTEM "chapter.dtd">
<chapter>
<header>
<copyright>
<year>2003</year><year>2022</year>
<holder>Ericsson AB. All Rights Reserved.</holder>
</copyright>
<legalnotice>
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
</legalnotice>
<title>Distributed Erlang</title>
<prepared></prepared>
<docno></docno>
<date></date>
<rev></rev>
<file>distributed.xml</file>
</header>
<section>
<title>Distributed Erlang System</title>
<p>A <em>distributed Erlang system</em> consists of a number of
Erlang runtime systems communicating with each other. Each such
runtime system is called a <em>node</em>. Message passing between
processes at different nodes, as well as links and monitors, are
transparent when pids are used. Registered names, however, are
local to each node. This means that the node must be specified as well
when sending messages, and so on, using registered names.</p>
<p>The distribution mechanism is implemented using TCP/IP sockets.
How to implement an alternative carrier is described in the
<seeguide marker="erts:alt_dist">ERTS User's Guide</seeguide>.</p>
<warning>
<p>
Starting a distributed node without also specifying
<seecom marker="erts:erl#proto_dist"><c>-proto_dist inet_tls</c></seecom>
will expose the node to attacks that may give the attacker
complete access to the node and in extension the cluster.
When using un-secure distributed nodes, make sure that the
network is configured to keep potential attackers out.
See the <seeguide marker="ssl:ssl_distribution">
Using SSL for Erlang Distribution</seeguide> User's Guide
for details on how to setup a secure distributed node.
</p>
</warning>
</section>
<section>
<title>Nodes</title>
<p>
A <em>node</em> is an executing Erlang runtime system that has
been given a name, using the command-line flag
<seecom marker="erts:erl#name"><c>-name</c></seecom> (long names) or
<seecom marker="erts:erl#sname"><c>-sname</c></seecom> (short names).
</p>
<p>The format of the node name is an atom <c>name@host</c>.
<c>name</c> is the name given by the user. <c>host</c> is
the full host name if long names are used, or the first part of
the host name if short names are used. Function
<seemfa marker="erts:erlang#node/0"><c>node()</c></seemfa>
returns the name of the node.</p>
<p><em>Example:</em></p>
<pre>
% <input>erl -name dilbert</input>
(dilbert@uab.ericsson.se)1> <input>node().</input>
'dilbert@uab.ericsson.se'
% <input>erl -sname dilbert</input>
(dilbert@uab)1> <input>node().</input>
dilbert@uab</pre>
<p>
The node name can also be given in runtime by calling
<seemfa marker="kernel:net_kernel#start/1"><c>net_kernel:start/1</c></seemfa>.
</p>
<p><em>Example:</em></p>
<pre>
% <input>erl</input>
1> <input>node().</input>
nonode@nohost
2> <input>net_kernel:start([dilbert,shortnames]).</input>
{ok,<0.102.0>}
(dilbert@uab)3> <input>node().</input>
dilbert@uab</pre>
<note>
<p>A node with a long node name cannot communicate with a node
with a short node name.</p>
</note>
</section>
<section>
<title>Node Connections</title>
<p>The nodes in a distributed Erlang system are loosely connected.
The first time the name of another node is used, for example, if
<c>spawn(Node,M,F,A)</c> or <c>net_adm:ping(Node)</c> is called,
a connection attempt to that node is made.</p>
<p>Connections are by default transitive. If a node A connects to
node B, and node B has a connection to node C, then node A
also tries to connect to node C. This feature can be turned off by
using the command-line flag <c>-connect_all false</c>, see the
<seecom marker="erts:erl">erl(1)</seecom> manual page in ERTS.</p>
<p>If a node goes down, all connections to that node are removed.
Calling <c>erlang:disconnect_node(Node)</c> forces disconnection
of a node.</p>
<p>The list of (visible) nodes currently connected to is returned by
<c>nodes()</c>.</p>
</section>
<section>
<title>epmd</title>
<p>The Erlang Port Mapper Daemon <em>epmd</em> is automatically
started at every host where an Erlang node is started. It is
responsible for mapping the symbolic node names to machine
addresses. See the
<seecom marker="erts:epmd">epmd(1)</seecom> manual page in ERTS.</p>
</section>
<section>
<title>Hidden Nodes</title>
<p>In a distributed Erlang system, it is sometimes useful to
connect to a node without also connecting to all other nodes.
An example is some kind of O&M functionality used to
inspect the status of a system, without disturbing it. For this
purpose, a <em>hidden node</em> can be used.</p>
<p>A hidden node is a node started with the command-line flag
<c>-hidden</c>. Connections between hidden nodes and other nodes
are not transitive, they must be set up explicitly. Also, hidden
nodes does not show up in the list of nodes returned by
<c>nodes()</c>. Instead, <c>nodes(hidden)</c> or
<c>nodes(connected)</c> must be used. This means, for example,
that the hidden node is not added to the set of nodes that
<c>global</c> is keeping track of.</p>
</section>
<section>
<marker id="dyn_node_name"/>
<title>Dynamic Node Name</title>
<p>
If the node name is set to <em><c>undefined</c></em> the node will
be started in a special mode to be the temporary client
of another node. The node will then request a dynamic node
name from the first node it connects to. In addition these
distribution settings will be set:
</p>
<pre><seecom marker="erts:erl#dist_listen">-dist_listen false</seecom> <seecom marker="erts:erl#hidden">-hidden</seecom> <seeapp marker="kernel:kernel_app#dist_auto_connect">-kernel dist_auto_connect never</seeapp></pre>
<p>
As <c>-dist_auto_connect</c> is set to <c>never</c>,
<seemfa marker="kernel:net_kernel#connect_node/1"><c>net_kernel:connect_node/1</c></seemfa>
must be called in order to setup connections. If the first established
connection is closed (which gave the node its dynamic name), then any
other connections will also be closed and the node will lose its dynamic
node name. A new call to
<seemfa marker="kernel:net_kernel#connect_node/1"><c>net_kernel:connect_node/1</c></seemfa>
can be made to get a new dynamic node name. The node name may
change if the distribution is dropped and then set up again.
</p>
<note>
<p>
The <em>dynamic node name</em> feature is supported from OTP 23.
Both the temporary client node and the first connected peer node
(supplying the dynamic node name) must be at least OTP 23 for it to
work.
</p>
</note>
</section>
<section>
<title>C Nodes</title>
<p>A <em>C node</em> is a C program written to act as a hidden node
in a distributed Erlang system. The library <em>Erl_Interface</em>
contains functions for this purpose. For more information about
C nodes, see the <seeguide marker="erl_interface:ei_users_guide">
Erl_Interface</seeguide> application and
<seeguide marker="system/tutorial:introduction#interoperability tutorial">
Interoperability Tutorial.</seeguide>.</p>
</section>
<section>
<title>Security</title>
<note>
<p>
"Security" here does <em>not</em> mean
cryptographically secure, but rather
security against accidental misuse,
such as preventing a node from connecting
to a cluster with which it is not intended to communicate.
</p>
<p>
Furthermore, the communication between nodes
is per default in clear text.
If you need strong security, please see
<seeguide marker="ssl:ssl_distribution">
Using TLS for Erlang Distribution
</seeguide>
in the SSL application's User's Guide.
</p>
<p>
Also, the default random cookie mentioned
in the following text is not very unpredictable.
A better one can be generated using primitives
in the <c>crypto</c> module, though this still does not make
the initial handshake cryptographically secure.
And inter-node communication is still in clear text.
</p>
</note>
<p>Authentication determines which nodes are allowed to communicate
with each other. In a network of different Erlang nodes, it is
built into the system at the lowest possible level.
All nodes use a <em>magic cookie</em>,
which is an Erlang atom, when connecting another node.</p>
<p>During the connection setup, after node names have been exchanged,
the magic cookies the nodes present to each other are compared.
If they do not match, the connection is rejected.
The cookies themselves are never transferred,
instead they are compared using hashed challenges,
although not in a cryptographically secure manner.</p>
<p>At start-up, a node has a random atom assigned as its default
magic cookie and the cookie of other nodes is assumed to be
<c>nocookie</c>. The first action of the Erlang network
authentication server (<c>auth</c>) is then to search for a file named
<c>.erlang.cookie</c> in the <seeerl marker="erts:init#home">
user's home directory</seeerl> and then in
<seeerl marker="stdlib:filename#user_config">
<c>filename:basedir(user_config, "erlang")</c></seeerl>.
If none of the files exist, a <c>.erlang.cooke</c> file is created
in the user's home directory.
The UNIX permissions mode of the file is set to octal
400 (read-only by user) and its content is a random string. An
atom <c>Cookie</c> is created from the contents of the file and
the cookie of the local node is set to this using
<c>erlang:set_cookie(Cookie)</c>. This sets the default cookie
that the local node will use for all other nodes.</p>
<p>Thus, groups of users with identical cookie files get Erlang
nodes that can communicate freely since they use the same
magic cookie. Users who want to run nodes
where the cookie files are on different file systems
must make certain that their cookie files are identical.</p>
<p>For a node <c>Node1</c> using magic cookie <c>Cookie</c> to be
able to connect to, and to accept a connection from, another node
<c>Node2</c> that uses a different cookie <c>DiffCookie</c>,
the function <c>erlang:set_cookie(Node2, DiffCookie)</c> must
first be called at <c>Node1</c>. Distributed systems with
multiple home directories (differing cookie files)
can be handled in this way.</p>
<note>
<p>With this setup <c>Node1</c> and <c>Node2</c> agree
on which cookie to use: <c>Node1</c> uses its explicitly
configured <c>DiffCookie</c> for <c>Node2</c>,
and <c>Node2</c> uses its default cookie <c>DiffCookie</c>.</p>
<p>You can also use a <c>DiffCookie</c> that neither
<c>Node1</c> nor <c>Node2</c> has as its default cookie,
if you also call <c>erlang:set_cookie(Node1, DiffCookie)</c>
in <c>Node2</c> before establishing connection</p>
<p>Because node names are exchanged during connection setup
before cookies are selected, connection setup works regardless
of which node that initiates it.</p>
<p>Note that to configure <c>Node1</c> to use <c>Node2</c>'s
default cookie when communicating with <c>Node2</c>,
<em>and vice versa</em> results in a broken configuration
(if the cookies are different) because then both nodes
use the other node's (differing) cookie.</p>
</note>
<p>The default when a connection is established between two nodes,
is to immediately connect all other visible nodes as well. This
way, there is always a fully connected network. If there are
nodes with different cookies, this method can be inappropriate
(since it may not be feasible to configure
different cookies for all possible nodes)
and the command-line flag <c>-connect_all false</c> must be set,
see the <seecom marker="erts:erl">erl(1)</seecom>
manual page in ERTS.</p>
<p>The magic cookie of the local node can be retrieved by calling
<c>erlang:get_cookie()</c>.</p>
</section>
<section>
<title>Distribution BIFs</title>
<p>Some useful BIFs for distributed programming
(for more information, see the <seeerl marker="erts:erlang">
erlang(3)</seeerl> manual page in ERTS:</p>
<table>
<row>
<cell align="left" valign="middle"><em>BIF</em></cell>
<cell align="left" valign="middle"><em>Description</em></cell>
</row>
<row>
<cell align="left" valign="middle"><c>erlang:disconnect_node(Node)</c></cell>
<cell align="left" valign="middle">Forces the disconnection of a node.</cell>
</row>
<row>
<cell align="left" valign="middle"><c>erlang:get_cookie()</c></cell>
<cell align="left" valign="middle">Returns the magic cookie of the current node.</cell>
</row>
<row>
<cell align="left" valign="middle"><c>erlang:get_cookie(Node)</c></cell>
<cell align="left" valign="middle">Returns the magic cookie
for node <c>Node</c>.</cell>
</row>
<row>
<cell align="left" valign="middle"><c>is_alive()</c></cell>
<cell align="left" valign="middle">Returns <c>true</c> if the runtime system is a node and can connect to other nodes, <c>false</c> otherwise.</cell>
</row>
<row>
<cell align="left" valign="middle"><c>monitor_node(Node, true|false)</c></cell>
<cell align="left" valign="middle">Monitors the status of
<c>Node</c>. A message<c>{nodedown, Node}</c> is received
if the connection to it is lost.</cell>
</row>
<row>
<cell align="left" valign="middle"><c>node()</c></cell>
<cell align="left" valign="middle">Returns the name of the current node. Allowed in guards.</cell>
</row>
<row>
<cell align="left" valign="middle"><c>node(Arg)</c></cell>
<cell align="left" valign="middle">Returns the node where <c>Arg</c>, a pid, reference, or port, is located.</cell>
</row>
<row>
<cell align="left" valign="middle"><c>nodes()</c></cell>
<cell align="left" valign="middle">Returns a list of all visible nodes this node is connected to.</cell>
</row>
<row>
<cell align="left" valign="middle"><c>nodes(Arg)</c></cell>
<cell align="left" valign="middle">Depending on <c>Arg</c>,
this function can return a list not only of visible nodes,
but also hidden nodes and previously known nodes, and so on.</cell>
</row>
<row>
<cell align="left" valign="middle"><c>erlang:set_cookie(Cookie)</c></cell>
<cell align="left" valign="middle">Sets the magic cookie,
<c>Cookie</c> to use when connecting all nodes
that have no explicit cookie set with
<c>erlang:set_cookie/2</c>.</cell>
</row>
<row>
<cell align="left" valign="middle"><c>erlang:set_cookie(Node, Cookie)</c></cell>
<cell align="left" valign="middle">Sets the magic cookie used
when connecting <c>Node</c>. If <c>Node</c> is the
current node, <c>Cookie</c> is used when connecting
all nodes that have no explicit cookie set with this function.</cell>
</row>
<row>
<cell align="left" valign="middle"><c>spawn[_link|_opt](Node, Fun)</c></cell>
<cell align="left" valign="middle">Creates a process at a remote node.</cell>
</row>
<row>
<cell align="left" valign="middle"><c>spawn[_link|opt](Node, Module, FunctionName, Args)</c></cell>
<cell align="left" valign="middle">Creates a process at a remote node.</cell>
</row>
<tcaption>Distribution BIFs</tcaption>
</table>
</section>
<section>
<title>Distribution Command-Line Flags</title>
<p>Examples of command-line flags used for distributed programming
(for more information, see the <seecom marker="erts:erl">erl(1)
</seecom> manual page in ERTS:</p>
<table>
<row>
<cell align="left" valign="middle"><em>Command-Line Flag</em></cell>
<cell align="left" valign="middle"><em>Description</em></cell>
</row>
<row>
<cell align="left" valign="middle"><c>-connect_all false</c></cell>
<cell align="left" valign="middle">Only explicit connection
set-ups are used.</cell>
</row>
<row>
<cell align="left" valign="middle"><c>-hidden</c></cell>
<cell align="left" valign="middle">Makes a node into a hidden node.</cell>
</row>
<row>
<cell align="left" valign="middle"><c>-name Name</c></cell>
<cell align="left" valign="middle">Makes a runtime system into a node, using long node names.</cell>
</row>
<row>
<cell align="left" valign="middle"><c>-setcookie Cookie</c></cell>
<cell align="left" valign="middle">Same as calling <c>erlang:set_cookie(Cookie)</c>.</cell>
</row>
<row>
<cell align="left" valign="middle"><c>-setcookie Node Cookie</c></cell>
<cell align="left" valign="middle">Same as calling <c>erlang:set_cookie(Node, Cookie)</c>.</cell>
</row>
<row>
<cell align="left" valign="middle"><c>-sname Name</c></cell>
<cell align="left" valign="middle">Makes a runtime system into a node, using short node names.</cell>
</row>
<tcaption>Distribution Command-Line Flags</tcaption>
</table>
</section>
<section>
<title>Distribution Modules</title>
<p>Examples of modules useful for distributed programming:</p>
<p>In the Kernel application:</p>
<table>
<row>
<cell align="left" valign="middle"><em>Module</em></cell>
<cell align="left" valign="middle"><em>Description</em></cell>
</row>
<row>
<cell align="left" valign="middle"><c>global</c></cell>
<cell align="left" valign="middle">A global name registration facility.</cell>
</row>
<row>
<cell align="left" valign="middle"><c>global_group</c></cell>
<cell align="left" valign="middle">Grouping nodes to global name registration groups.</cell>
</row>
<row>
<cell align="left" valign="middle"><c>net_adm</c></cell>
<cell align="left" valign="middle">Various Erlang net administration routines.</cell>
</row>
<row>
<cell align="left" valign="middle"><c>net_kernel</c></cell>
<cell align="left" valign="middle">Erlang networking kernel.</cell>
</row>
<tcaption>Kernel Modules Useful For Distribution.</tcaption>
</table>
<p>In the STDLIB application:</p>
<table>
<row>
<cell align="left" valign="middle"><em>Module</em></cell>
<cell align="left" valign="middle"><em>Description</em></cell>
</row>
<row>
<cell align="left" valign="middle"><c>slave</c></cell>
<cell align="left" valign="middle">Start and control of slave nodes.</cell>
</row>
<tcaption>STDLIB Modules Useful For Distribution.</tcaption>
</table>
</section>
</chapter>
|