You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Supercomputers are highly heterogeneous infrastructures that can provide non-traditional architectures like:
8
+
9
+
- non-x86 CPUs
10
+
- accelerators (GPUs, FPGAs)
11
+
12
+
They might make use of modern multicore and manycore architectures and are built for applications running fast on a large number of resources, having fast node-interconnectivity.
13
+
They often have different policies from those of HEP Grid Sites:
14
+
15
+
- no internet connectivity
16
+
- no CVMFS
17
+
- VPN access
18
+
19
+
This chapter aims to help administrators to configure DIRAC in order to deal with supercomputers according to their features.
20
+
21
+
22
+
.. toctree::
23
+
:maxdepth:1
24
+
:numbered:
25
+
26
+
.. contents:: Table of contents
27
+
:depth: 4
28
+
29
+
30
+
--------
31
+
Overview
32
+
--------
33
+
34
+
.. image:: ../../_static/hpc_schema.png
35
+
:alt:DIRAC projects interaction overview
36
+
:align:center
37
+
38
+
39
+
We have identified differences between a traditional HEP grid site and supercomputers.
40
+
To run workloads on supercomputers, administrators might need to perform and combine several changes.
41
+
Some can be very easy, such as an update in the CS, while some can require additional actions and analysis such as delivering a subset of CVMFS.
42
+
43
+
---------------------
44
+
Outbound connectivity
45
+
---------------------
46
+
47
+
Multi-core allocations
48
+
----------------------
49
+
50
+
Supercomputers often privilege workloads exploiting many cores in parallel during a short time (HPC).
51
+
This means they allow a small number of large allocations of resources.
52
+
Grid applications are usually not adapted: they are embarrassingly parallel, run on a single core for a long period (HTC).
53
+
54
+
To exploit the manycore nodes of supercomputers and avoid a waste of computing resources, DIRAC can leverage the fat-node partitioning mechanism.
55
+
This consists in submitting a Pilot-Job on a node, which can then fetch and run multiple workloads in parallel.
56
+
To set it up, one has to add two options in the CE configuration: ``LocalCEType=Pool`` and ``NumberOfProcessors=<N>``, ``N`` being the number of cores per worker node.
57
+
58
+
For further details about the CE options, consult :ref:`the Sites section <cs-site>`.
59
+
60
+
61
+
Multi-node allocations
62
+
----------------------
63
+
64
+
In the same way, some supercomputers have specific partitions only accessible for applications exploiting multiple manycore nodes simultaneously in the same allocation.
65
+
To exploit the many-node allocations, DIRAC allows to generate one sub-pilot per node allocated.
66
+
Sub-pilots run in parallel and share the same identifier, output and status.
67
+
68
+
This option is currently only available via :mod:`~DIRAC.Resources.Computing.BatchSystems.SLURM`.
69
+
To use sub-pilots in many node allocations, one has to add an additional options in the CE configuration: ``NumberOfNodes=<min-max>``.
70
+
71
+
For further details about the CE options, consult :ref:`the Sites section <cs-site>`.
72
+
73
+
CVMFS not available
74
+
-------------------
75
+
76
+
Workloads having thousands of dependencies are generally delivered via CVMFS on grid sites.
77
+
By default, CVMFS is not mounted on the nodes of supercomputers.
78
+
One can talk with the system administrators to discuss the possibility of having it mounted on the worker nodes.
79
+
If this is not possible, then one has to use `cvmfs-exec <https://github.com/cvmfs/cvmfsexec>`_.
80
+
It allows mounting CVMFS as an unprivileged user, without the CVMFS package being installed by a system administrator.
81
+
82
+
This action is purely a VO action: the package has to be installed on the worker node before starting the job.
83
+
The solution has not been integrated into DIRAC yet.
84
+
85
+
86
+
LRMS not accessible
87
+
-------------------
88
+
89
+
LRMS, also called Batch System, is the component that orchestrates the worker nodes and the workload on a site.
90
+
On grid sites, a LRMS is often accessible via a CE, and if a CE is not available, then one can interact directly with it via SSH: DIRAC handles both cases.
91
+
92
+
Nevertheless, supercomputers have more restrictive access policies than grid sites and may protect the facility access with a VPN.
93
+
In this situation, one can run a :mod:`~DIRAC.WorkloadManagementSystem.Agent.SiteDirector` combined to a :mod:`~DIRAC.Resources.Computing.LocalComputingElement` directly on the edge node of the supercomputer.
94
+
This allows submitting pilots from the edge of the supercomputer to the worker nodes directly.
95
+
96
+
Supercomputers often do not allow users to execute a program from the edge node for a long time.
97
+
To address this problem, one can call the Site Director in a cron job, executed every N minutes for 1 cycle.
98
+
99
+
Also, to generate pilot proxies, the Site Director has to rely on a host certificate: one has to contact a system administrator for that.
100
+
101
+
----------------------------------
102
+
Only partial outbound connectivity
103
+
----------------------------------
104
+
105
+
This case has not been addressed yet.
106
+
107
+
------------------------
108
+
No outbound connectivity
109
+
------------------------
110
+
111
+
Solutions seen in the previous section cannot work in an environment without external connectivity.
112
+
The well-known Pilot-Job paradigm on which the DIRAC WMS is based does not apply in these circumstances: the Pilot-Jobs cannot fetch jobs from DIRAC.
113
+
Thus, such supercomputers require slightly changes in the WMS: we reintroduced the push model.
114
+
115
+
To leverage the Push model, one has to add the :mod:`~DIRAC.WorkloadManagementSystem.Agent.PushJobAgent` to the ``Systems/WorkloadManagement/<Setup>/Agents``
116
+
CS section, such as::
117
+
118
+
Systems
119
+
PushJobAgent_<Name>
120
+
{
121
+
CEs = <CEs>
122
+
Sites = <Sites>
123
+
CETypes = <CETypes>
124
+
MaxJobsToSubmit = 100
125
+
Module = PushJobAgent
126
+
}
127
+
128
+
:mod:`~DIRAC.WorkloadManagementSystem.Agent.PushJobAgent` inherits from :mod:`~DIRAC.WorkloadManagementSystem.Agent.JobAgent` and proposes a similar structure: it fetches a job from the :mod:`~DIRAC.WorkloadManagementSystem.Service.MatcherHandler` service and submit it to a :mod:`~DIRAC.Resources.Computing.PoolComputingElement`.
129
+
- It provides an additional parameter in ``/LocalSite`` named ``RemoteExecution`` that can be used later in the process to identify computing resources with no external connectivity.
130
+
- There is no ``timeLeft`` attribute: it runs on the DIRAC side as an ``Agent``.
131
+
- ``MaxJobsToSubmit`` corresponds to the maximum number of jobs the agent can handle at the same time.
132
+
- To fetch a job, :mod:`~DIRAC.WorkloadManagementSystem.Agent.PushJobAgent` sends the dictionary of the target CE to the :mod:`~DIRAC.WorkloadManagementSystem.Service.MatcherHandler` service.
133
+
134
+
:mod:`~DIRAC.WorkloadManagementSystem.Agent.PushJobAgent` does not inherit from :mod:`~DIRAC.WorkloadManagementSystem.Agent.SiteDirector` but embeds similar features:
135
+
- It supervises specific Sites/CEs/Queues.
136
+
- If there is an error with a queue, it puts it on hold and waits for a certain number of cycles before trying again.
137
+
138
+
Internally, the workflow modules originally in charge of executing the script/application (:mod:`~DIRAC.Workflow.Modules.Script`) check whether the workload should be
139
+
sent to a remote location before acting.
140
+
:mod:`~DIRAC.WorkloadManagementSystem.Utilities.RemoteRunner` attempts to extract the value from the environment variable initialized by the :mod:`~DIRAC.WorkloadManagementSystem.JobWrapper.JobWrapper`.
141
+
If the variable is not set, then the application is run locally via ``systemCall()``, else the application is submitted to a remote Computing Element such as ARC.
142
+
:mod:`~DIRAC.WorkloadManagementSystem.Utilities.RemoteRunner` wraps the script/application command in an executable, gets all the files of the working directory that correspond to input files and submits the executable along with the input files. It gets the status of the application submitted every 2 minutes until it is finished and finally gets the outputs.
143
+
144
+
145
+
Multi-core/node allocations
146
+
---------------------------
147
+
148
+
This case has not been addressed yet.
149
+
150
+
CVMFS not available
151
+
-------------------
152
+
153
+
Workloads depending on CVMFS cannot run on such infrastructure: the only possibility is to generate a subset of CVMFS, deploy it on the supercomputer,
154
+
and mount it to a container.
155
+
`subCVMFS-builder <https://gitlab.cern.ch/alboyer/subcvmfs-builder>`_ and `subCVMFS-builder-pipeline <https://gitlab.cern.ch/alboyer/subcvmfs-builder-pipeline>`_ are two projects aiming to assist VOs in this process.
156
+
They allow to trace applications of interest, build a subset of CVMFS, test it and deploy it to a remote location.
157
+
158
+
To integrate the subset of CVMFS with the DIRAC workflow, one can leverage the :mod:`~DIRAC.Resources.Computing.ARCComputingElement` ``XRSLExtraString`` option such as::
To mount the subset of CVMFS in the singularity container, one has to contact the ARC administrators to finetune the configuration or has to build a container image containing the subset with `subCVMFS-builder <https://gitlab.cern.ch/alboyer/subcvmfs-builder>`_.
0 commit comments