Skip to content

Commit ce960b8

Browse files
chaenchaen
committed
docs (tornado): document how to install HTTPs services
1 parent aee762d commit ce960b8

File tree

4 files changed

+210
-23
lines changed

4 files changed

+210
-23
lines changed

‎docs/source/AdministratorGuide/ServerInstallations/HTTPSServices.rst‎

Lines changed: 194 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,194 @@
1+
.. _https_services:
2+
3+
==================================
4+
Installing HTTPs services in DIRAC
5+
==================================
6+
7+
.. contents::
8+
9+
Background
10+
**********
11+
12+
For many years, DIRAC services have been exposed through the historical DISET (``dips://``) protocol.
13+
Few years ago DIRAC developers started exposing DIRAC services using the HTTPs protocol.
14+
This page explains how to migrate existing DIPs protocol to HTTPs. For a developer view, refer to :ref:`httpsTornado`.
15+
16+
For a summary presentation you can check `this <https://indico.cern.ch/event/852597/contributions/4331720/attachments/2241040/3799728/HttpsInDIRAC.pdf>`_.
17+
18+
NB: not all DIPs services can be exposed through HTTPs. For a comprehensve list, please refer to :ref:`scalingLimitations`.
19+
20+
General principle
21+
=================
22+
23+
Contrary to the DISET protocol where each service would have its own process and port opened, HTTPs services are served by a unique process and port, based on Tornado.
24+
25+
There are exceptions to that, due to the use of global variables in some parts of DIRAC. Namely:
26+
27+
* Master CS
28+
29+
All other services can follow the standard procedure described below.
30+
31+
First, the following configuration subsections have to be added to CS::
32+
33+
# "Main" section
34+
DIRAC
35+
{
36+
Setups
37+
{
38+
...
39+
Tornado = Production
40+
}
41+
}
42+
43+
# Add Tornado to Systems section
44+
Systems
45+
{
46+
...
47+
Tornado
48+
{
49+
Production
50+
{
51+
Port = 443
52+
}
53+
}
54+
}
55+
56+
57+
Installation of an HTTPs based service
58+
======================================
59+
60+
This procedure is to be used if you are want to serve a new service with HTTPs.
61+
62+
63+
Case 1: you do NOT run the equivalent DISET service
64+
---------------------------------------------------
65+
66+
This is the most trivial case. Just run ``dirac-install-tornado-service`` with the service you are interested in. This will install an ``runit`` component running ``tornado-start-all``.
67+
68+
Case 2: you run the equivalent DISET service
69+
--------------------------------------------
70+
71+
Because the CS already contains the handler definition for DISET, ``dirac-install-tornado-service`` will not modify it. Thus, you have to update it yourself, before running the command, otherwise ``tornado-start-all`` will not find any service to run, and the installation will be shown as failed.
72+
73+
Procedure:
74+
75+
#. Update by hand the CS of the desired service:
76+
77+
* Remove the port definition
78+
* Modify the handler to point to the Tornado handler
79+
* add ``Protocol=https``
80+
81+
The example the follows is for the "DIRAC File Catalog" (DFC) service. This would normally be in CS as::
82+
83+
Systems
84+
{
85+
...
86+
DataManagement
87+
{
88+
Production
89+
{
90+
URLs
91+
{
92+
...
93+
FileCatalog = dips://my.server.org:9197/DataManagement/FileCatalog
94+
}
95+
Services
96+
{
97+
...
98+
FileCatalog
99+
{
100+
...
101+
Port = 9197
102+
Protocol = dips
103+
...
104+
}
105+
}
106+
...
107+
}
108+
}
109+
}
110+
111+
112+
And you need to change it to::
113+
114+
Systems
115+
{
116+
...
117+
DataManagement
118+
{
119+
Production
120+
{
121+
URLs
122+
{
123+
...
124+
FileCatalog = https://my.server.org:8443/DataManagement/FileCatalog
125+
}
126+
Services
127+
{
128+
...
129+
FileCatalog
130+
{
131+
...
132+
Protocol = https
133+
HandlerPath = DIRAC/DataManagementSystem/Service/TornadoFileCatalogHandler.py
134+
...
135+
}
136+
}
137+
...
138+
}
139+
}
140+
}
141+
142+
143+
#. Run ``dirac-install-tornado-service`` or restart the tornado component if already running.
144+
145+
.. note::
146+
This means that from now on, the DISET service cannot be restarted anymore, as its configuration would be wrong.
147+
So, go ahead and remove (by hand, for now) the `runit`-related directories of the DISET service.
148+
149+
Example of configuration before/after:
150+
151+
.. literalinclude:: /../../src/DIRAC/WorkloadManagementSystem/ConfigTemplate.cfg
152+
:start-after: ##BEGIN JobMonitoring
153+
:end-before: ##END
154+
:dedent: 2
155+
:caption: JobMonitoring configuration for DISET
156+
157+
.. literalinclude:: /../../src/DIRAC/WorkloadManagementSystem/ConfigTemplate.cfg
158+
:start-after: ##BEGIN TornadoJobMonitoring
159+
:end-before: ##END
160+
:dedent: 2
161+
:caption: JobMonitoring configuration for HTTPs
162+
163+
In any case, do not forget to update the URL of the service you just installed, such that other services can reach it.
164+
165+
166+
Adding more tornado instances on a different machine
167+
====================================================
168+
169+
Simply use ``dirac-install-tornado-service`` with no arguments on the new machine.
170+
171+
172+
MasterCS special case
173+
=====================
174+
175+
The master Configuration Server is different and needs to be run in a separate process. In order to do so:
176+
177+
* Do NOT specify ``Protocol=https`` in the service description, otherwise it will be ran with all the other Tornado services.
178+
* If you run on the same machine as other TornadoService, specify a ``Port`` in the service description (you can keep the existing 9135, if already there).
179+
* Modify the content of the Configuration so that URLs are updated to use HTTPs instead of DIPs.
180+
* Add `HandlerPath = DIRAC/ConfigurationSystem/Service/TornadoConfigurationHandler.py` in the etc/dirac.cfg configuration file of the machine where the master CS is running (needed for bootstrap).
181+
182+
Finally, there is no automatic installations script. So just install a CS as you normally would do, and then edit the ``run`` file like that::
183+
184+
#!/bin/bash
185+
rcfile=/opt/dirac/bashrc
186+
[ -e $rcfile ] && source $rcfile
187+
#
188+
export DIRAC_USE_TORNADO_IOLOOP=Yes
189+
exec 2>&1
190+
#
191+
[ "service" = "agent" ] && renice 20 -p $$
192+
#
193+
#
194+
exec tornado-start-CS -ddd

‎docs/source/AdministratorGuide/ServerInstallations/index.rst‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@ This sections constains the documentation for installing new DIRAC Servers or se
1212

1313
InstallingDiracServer
1414
InstallingWebAppDIRAC
15+
HTTPSServices
1516
centralizedLogging
1617
rabbitmq
1718
scalingAndLimitations

‎docs/source/DeveloperGuide/TornadoServices/index.rst‎

Lines changed: 11 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@ This page summarizes the changes between DISET and HTTPS. You can all also see t
1717
- `Presentation of HTTPS migration <https://docs.google.com/presentation/d/1NZ8iKRv3c0OL1_RTXL21hP6YsAUXcKSCqDL2uhkf8Oc/edit?usp=sharing>`_.
1818
- `Summary presentation (latest) <https://indico.cern.ch/event/945474/>`_.
1919

20+
For installing HTTPs services, please refer to :ref:`https_services` administration page.
2021

2122

2223
*******
@@ -147,29 +148,8 @@ Options available are:
147148

148149
This start method can be useful for developing new service or create starting script for a specific service, like the Configuration System (as master).
149150

150-
151-
MasterCS special case
152-
*********************
153-
154-
The master CS is different because it uses the same global variable (``gConfig``) but uses it also to write config. Because of that, it needs to run in a separate process. In order to do so:
155-
156-
* Do NOT specify ``Protocol=https`` in the service description, otherwise it will be ran with all the other Tornado services
157-
* If you run on the same machine as other TornadoService, specify a ``Port`` in the service description
158-
159-
Finally, there is no automatic installations script. So just install a CS as you normally would do, and then edit the ``run`` file like that::
160-
161-
diff --git a/run b/run.new
162-
index d45dce1..f5f3b55 100755
163-
--- a/run
164-
+++ b/run.new
165-
@@ -7,6 +7,6 @@
166-
[ "service" = "agent" ] && renice 20 -p $$
167-
#
168-
#
169-
- exec python $DIRAC/DIRAC/Core/scripts/dirac-service.py Configuration/Server --cfg /opt/dirac/pro/etc/Configuration_Server.cfg < /dev/null
170-
+ export DIRAC_USE_TORNADO_IOLOOP=Yes
171-
+ exec tornado-start-CS -ddd
172-
151+
The master CS is different because it uses the same global variable (``gConfig``) but uses it also to write config. Because of that, it needs to run in a separate process.
152+
It needs to be started with ``tornado-start-CS`` script.
173153

174154

175155
TransferClient
@@ -355,9 +335,17 @@ Two special python packages are needed:
355335
Install a service
356336
*****************
357337

338+
Initial install: first modify one config (with port and so on) before running ``tornado-install``
339+
358340
``dirac-install-tornado-service`` is your friend. This will install a runit component running ``tornado-start-all``.
359341
Nothing is ready yet to install specific tornado service, like the master CS.
360342

343+
Migrate from dips to https
344+
**************************
345+
346+
comment out port, set Protocol = https, change handler
347+
348+
361349
Start the server
362350
****************
363351

‎src/DIRAC/WorkloadManagementSystem/ConfigTemplate.cfg‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@ Services
1717
Default = authenticated
1818
}
1919
}
20+
##BEGIN JobMonitoring
2021
JobMonitoring
2122
{
2223
Port = 9130
@@ -25,6 +26,8 @@ Services
2526
Default = authenticated
2627
}
2728
}
29+
##END
30+
##BEGIN TornadoJobMonitoring
2831
TornadoJobMonitoring
2932
{
3033
Protocol = https
@@ -33,6 +36,7 @@ Services
3336
Default = authenticated
3437
}
3538
}
39+
##END
3640
JobStateUpdate
3741
{
3842
Port = 9136

0 commit comments

Comments
 (0)