Skip to content

Commit 82c4711

Browse files
fstagnifstagni
authored
Merge pull request #5632 from chaen/rel-v7r3_doc_installHTTPs
[v7r3] Proper documentation for HTTPs services
2 parents 1145103 + ce960b8 commit 82c4711

File tree

5 files changed

+238
-47
lines changed

5 files changed

+238
-47
lines changed

‎docs/source/AdministratorGuide/ServerInstallations/HTTPSServices.rst‎

Lines changed: 194 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,194 @@
1+
.. _https_services:
2+
3+
==================================
4+
Installing HTTPs services in DIRAC
5+
==================================
6+
7+
.. contents::
8+
9+
Background
10+
**********
11+
12+
For many years, DIRAC services have been exposed through the historical DISET (``dips://``) protocol.
13+
Few years ago DIRAC developers started exposing DIRAC services using the HTTPs protocol.
14+
This page explains how to migrate existing DIPs protocol to HTTPs. For a developer view, refer to :ref:`httpsTornado`.
15+
16+
For a summary presentation you can check `this <https://indico.cern.ch/event/852597/contributions/4331720/attachments/2241040/3799728/HttpsInDIRAC.pdf>`_.
17+
18+
NB: not all DIPs services can be exposed through HTTPs. For a comprehensve list, please refer to :ref:`scalingLimitations`.
19+
20+
General principle
21+
=================
22+
23+
Contrary to the DISET protocol where each service would have its own process and port opened, HTTPs services are served by a unique process and port, based on Tornado.
24+
25+
There are exceptions to that, due to the use of global variables in some parts of DIRAC. Namely:
26+
27+
* Master CS
28+
29+
All other services can follow the standard procedure described below.
30+
31+
First, the following configuration subsections have to be added to CS::
32+
33+
# "Main" section
34+
DIRAC
35+
{
36+
Setups
37+
{
38+
...
39+
Tornado = Production
40+
}
41+
}
42+
43+
# Add Tornado to Systems section
44+
Systems
45+
{
46+
...
47+
Tornado
48+
{
49+
Production
50+
{
51+
Port = 443
52+
}
53+
}
54+
}
55+
56+
57+
Installation of an HTTPs based service
58+
======================================
59+
60+
This procedure is to be used if you are want to serve a new service with HTTPs.
61+
62+
63+
Case 1: you do NOT run the equivalent DISET service
64+
---------------------------------------------------
65+
66+
This is the most trivial case. Just run ``dirac-install-tornado-service`` with the service you are interested in. This will install an ``runit`` component running ``tornado-start-all``.
67+
68+
Case 2: you run the equivalent DISET service
69+
--------------------------------------------
70+
71+
Because the CS already contains the handler definition for DISET, ``dirac-install-tornado-service`` will not modify it. Thus, you have to update it yourself, before running the command, otherwise ``tornado-start-all`` will not find any service to run, and the installation will be shown as failed.
72+
73+
Procedure:
74+
75+
#. Update by hand the CS of the desired service:
76+
77+
* Remove the port definition
78+
* Modify the handler to point to the Tornado handler
79+
* add ``Protocol=https``
80+
81+
The example the follows is for the "DIRAC File Catalog" (DFC) service. This would normally be in CS as::
82+
83+
Systems
84+
{
85+
...
86+
DataManagement
87+
{
88+
Production
89+
{
90+
URLs
91+
{
92+
...
93+
FileCatalog = dips://my.server.org:9197/DataManagement/FileCatalog
94+
}
95+
Services
96+
{
97+
...
98+
FileCatalog
99+
{
100+
...
101+
Port = 9197
102+
Protocol = dips
103+
...
104+
}
105+
}
106+
...
107+
}
108+
}
109+
}
110+
111+
112+
And you need to change it to::
113+
114+
Systems
115+
{
116+
...
117+
DataManagement
118+
{
119+
Production
120+
{
121+
URLs
122+
{
123+
...
124+
FileCatalog = https://my.server.org:8443/DataManagement/FileCatalog
125+
}
126+
Services
127+
{
128+
...
129+
FileCatalog
130+
{
131+
...
132+
Protocol = https
133+
HandlerPath = DIRAC/DataManagementSystem/Service/TornadoFileCatalogHandler.py
134+
...
135+
}
136+
}
137+
...
138+
}
139+
}
140+
}
141+
142+
143+
#. Run ``dirac-install-tornado-service`` or restart the tornado component if already running.
144+
145+
.. note::
146+
This means that from now on, the DISET service cannot be restarted anymore, as its configuration would be wrong.
147+
So, go ahead and remove (by hand, for now) the `runit`-related directories of the DISET service.
148+
149+
Example of configuration before/after:
150+
151+
.. literalinclude:: /../../src/DIRAC/WorkloadManagementSystem/ConfigTemplate.cfg
152+
:start-after: ##BEGIN JobMonitoring
153+
:end-before: ##END
154+
:dedent: 2
155+
:caption: JobMonitoring configuration for DISET
156+
157+
.. literalinclude:: /../../src/DIRAC/WorkloadManagementSystem/ConfigTemplate.cfg
158+
:start-after: ##BEGIN TornadoJobMonitoring
159+
:end-before: ##END
160+
:dedent: 2
161+
:caption: JobMonitoring configuration for HTTPs
162+
163+
In any case, do not forget to update the URL of the service you just installed, such that other services can reach it.
164+
165+
166+
Adding more tornado instances on a different machine
167+
====================================================
168+
169+
Simply use ``dirac-install-tornado-service`` with no arguments on the new machine.
170+
171+
172+
MasterCS special case
173+
=====================
174+
175+
The master Configuration Server is different and needs to be run in a separate process. In order to do so:
176+
177+
* Do NOT specify ``Protocol=https`` in the service description, otherwise it will be ran with all the other Tornado services.
178+
* If you run on the same machine as other TornadoService, specify a ``Port`` in the service description (you can keep the existing 9135, if already there).
179+
* Modify the content of the Configuration so that URLs are updated to use HTTPs instead of DIPs.
180+
* Add `HandlerPath = DIRAC/ConfigurationSystem/Service/TornadoConfigurationHandler.py` in the etc/dirac.cfg configuration file of the machine where the master CS is running (needed for bootstrap).
181+
182+
Finally, there is no automatic installations script. So just install a CS as you normally would do, and then edit the ``run`` file like that::
183+
184+
#!/bin/bash
185+
rcfile=/opt/dirac/bashrc
186+
[ -e $rcfile ] && source $rcfile
187+
#
188+
export DIRAC_USE_TORNADO_IOLOOP=Yes
189+
exec 2>&1
190+
#
191+
[ "service" = "agent" ] && renice 20 -p $$
192+
#
193+
#
194+
exec tornado-start-CS -ddd

‎docs/source/AdministratorGuide/ServerInstallations/index.rst‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@ This sections constains the documentation for installing new DIRAC Servers or se
1212

1313
InstallingDiracServer
1414
InstallingWebAppDIRAC
15+
HTTPSServices
1516
centralizedLogging
1617
rabbitmq
1718
scalingAndLimitations

‎docs/source/DeveloperGuide/TornadoServices/index.rst‎

Lines changed: 11 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@ This page summarizes the changes between DISET and HTTPS. You can all also see t
1717
- `Presentation of HTTPS migration <https://docs.google.com/presentation/d/1NZ8iKRv3c0OL1_RTXL21hP6YsAUXcKSCqDL2uhkf8Oc/edit?usp=sharing>`_.
1818
- `Summary presentation (latest) <https://indico.cern.ch/event/945474/>`_.
1919

20+
For installing HTTPs services, please refer to :ref:`https_services` administration page.
2021

2122

2223
*******
@@ -147,29 +148,8 @@ Options available are:
147148

148149
This start method can be useful for developing new service or create starting script for a specific service, like the Configuration System (as master).
149150

150-
151-
MasterCS special case
152-
*********************
153-
154-
The master CS is different because it uses the same global variable (``gConfig``) but uses it also to write config. Because of that, it needs to run in a separate process. In order to do so:
155-
156-
* Do NOT specify ``Protocol=https`` in the service description, otherwise it will be ran with all the other Tornado services
157-
* If you run on the same machine as other TornadoService, specify a ``Port`` in the service description
158-
159-
Finally, there is no automatic installations script. So just install a CS as you normally would do, and then edit the ``run`` file like that::
160-
161-
diff --git a/run b/run.new
162-
index d45dce1..f5f3b55 100755
163-
--- a/run
164-
+++ b/run.new
165-
@@ -7,6 +7,6 @@
166-
[ "service" = "agent" ] && renice 20 -p $$
167-
#
168-
#
169-
- exec python $DIRAC/DIRAC/Core/scripts/dirac-service.py Configuration/Server --cfg /opt/dirac/pro/etc/Configuration_Server.cfg < /dev/null
170-
+ export DIRAC_USE_TORNADO_IOLOOP=Yes
171-
+ exec tornado-start-CS -ddd
172-
151+
The master CS is different because it uses the same global variable (``gConfig``) but uses it also to write config. Because of that, it needs to run in a separate process.
152+
It needs to be started with ``tornado-start-CS`` script.
173153

174154

175155
TransferClient
@@ -355,9 +335,17 @@ Two special python packages are needed:
355335
Install a service
356336
*****************
357337

338+
Initial install: first modify one config (with port and so on) before running ``tornado-install``
339+
358340
``dirac-install-tornado-service`` is your friend. This will install a runit component running ``tornado-start-all``.
359341
Nothing is ready yet to install specific tornado service, like the master CS.
360342

343+
Migrate from dips to https
344+
**************************
345+
346+
comment out port, set Protocol = https, change handler
347+
348+
361349
Start the server
362350
****************
363351

‎src/DIRAC/FrameworkSystem/scripts/dirac_install_tornado_service.py‎

Lines changed: 28 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
#!/usr/bin/env python
22
"""
3-
Do the initial installation and configuration of a DIRAC service based on tornado
3+
Do the initial installation and configuration of a DIRAC service based on tornado. If the component is not specified, just install Tornado as such (note that the installation will fail if there are no suited services in the CS)
44
"""
55

66
from __future__ import absolute_import
@@ -62,35 +62,39 @@ def main():
6262
(
6363
"System/Component: Full component name (ie: WorkloadManagement/Matcher)",
6464
"System: Name of the DIRAC system (ie: WorkloadManagement)",
65-
)
65+
),
66+
mandatory=False,
6667
)
6768
Script.registerArgument(" Component: Name of the DIRAC service (ie: Matcher)", mandatory=False)
6869
Script.parseCommandLine()
6970
args = Script.getPositionalArgs()
7071

71-
if len(args) == 1:
72-
args = args[0].split("/")
73-
74-
if len(args) != 2:
75-
Script.showHelp()
76-
DIRACexit(1)
77-
78-
system = args[0]
79-
component = args[1]
80-
81-
result = gComponentInstaller.addDefaultOptionsToCS(
82-
gConfig,
83-
"service",
84-
system,
85-
component,
86-
extensionsByPriority(),
87-
specialOptions=specialOptions,
88-
overwrite=overwrite,
89-
)
72+
# If we specify a service, add its
73+
if args:
74+
# System/Component
75+
if len(args) == 1:
76+
args = args[0].split("/")
77+
78+
if len(args) != 2:
79+
Script.showHelp()
80+
DIRACexit(1)
81+
82+
system = args[0]
83+
component = args[1]
84+
85+
result = gComponentInstaller.addDefaultOptionsToCS(
86+
gConfig,
87+
"service",
88+
system,
89+
component,
90+
extensionsByPriority(),
91+
specialOptions=specialOptions,
92+
overwrite=overwrite,
93+
)
9094

91-
if not result["OK"]:
92-
gLogger.error(result["Message"])
93-
DIRACexit(1)
95+
if not result["OK"]:
96+
gLogger.error(result["Message"])
97+
DIRACexit(1)
9498

9599
result = gComponentInstaller.addTornadoOptionsToCS(gConfig)
96100
if not result["OK"]:

‎src/DIRAC/WorkloadManagementSystem/ConfigTemplate.cfg‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@ Services
1717
Default = authenticated
1818
}
1919
}
20+
##BEGIN JobMonitoring
2021
JobMonitoring
2122
{
2223
Port = 9130
@@ -25,6 +26,8 @@ Services
2526
Default = authenticated
2627
}
2728
}
29+
##END
30+
##BEGIN TornadoJobMonitoring
2831
TornadoJobMonitoring
2932
{
3033
Protocol = https
@@ -33,6 +36,7 @@ Services
3336
Default = authenticated
3437
}
3538
}
39+
##END
3640
JobStateUpdate
3741
{
3842
Port = 9136

0 commit comments

Comments
 (0)