Skip to content

Commit 7a32220

Browse files
matyasselmecimatyasselmeci
authored
Merge pull request #125 from matyasselmeci/pr/v4/unused
Drop unused environment variables and attributes, and make resource_group mandatory on CEs
2 parents e42b6fc + 2421176 commit 7a32220

File tree

24 files changed

+63
-482
lines changed

24 files changed

+63
-482
lines changed

‎README.md‎

Lines changed: 3 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -170,10 +170,8 @@ This section is contained in `/etc/osg/config.d/10-storage.ini` which is provide
170170

171171
| Option | Values Accepted | Explanation |
172172
|------------------|-----------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
173-
| **se_available** | `True`, `False` | This indicates whether there is an associated SE available. |
174-
| default_se | String | If an SE is available at your cluster, set default_se to the hostname of this SE, otherwise set default_se to UNAVAILABLE. |
175173
| **grid_dir** | String | This setting should point to the directory which holds the files from the OSG worker node package. See note |
176-
| **app_dir** | String | This setting should point to the directory which contains the VO specific applications. See note |
174+
| app_dir | String | This setting should point to the directory which contains the VO specific applications. See note |
177175
| data_dir | String | This setting should point to a directory that can be used to store and stage data in and out of the cluster. See note |
178176
| worker_node_temp | String | This directory should point to a directory that can be used as scratch space on compute nodes. If not set, the default is UNAVAILABLE. See note |
179177
| site_read | String | This setting should be the location or url to a directory that can be read to stage in data via the variable `$OSG_SITE_READ`. This is an url if you are using a SE. If not set, the default is UNAVAILABLE |
@@ -457,6 +455,7 @@ We recommend a name that describes the capabilities of the pilots you accept.
457455
Good names are `singularity_8core`, `gpu`, `bigmem`, `main`.
458456

459457
The following attributes are required:
458+
460459
| Option | Values Accepted | Explanation |
461460
|--------------------------|-----------------------------|------------------------------------------------------------------------------------------------------------------------------------|
462461
| **allowed\_vos** | Comma-separated List or `*` | The VOs that are allowed to run jobs on this resource (autodetected if `*`) |
@@ -467,6 +466,7 @@ The following attributes are required:
467466
Valid values for the **os** option are: `rhel6`, `rhel7`, `rhel8`, or `ubuntu18`.
468467

469468
The following attributes are optional:
469+
470470
| Option | Values Accepted | Explanation |
471471
|---------------------|----------------------|------------------------------------------------------------------------------------------------------------------------------------------|
472472
| **cpucount** | Positive Integer | Number of cores that a job using this type of pilot can get. Default `1`; ignored if **whole\_node** is `True` |
@@ -509,17 +509,3 @@ This section is contained in `/etc/osg/config.d/40-siteinfo.ini` which is provid
509509
| **host\_name** | String | This should be set to be hostname of the CE that is being configured |
510510
| **resource** | String | The resource name of this CE endpoint as registered in Topology. |
511511
| **resource\_group** | String | The resource\_group of this CE as registered in Topology. |
512-
| **sponsor** | String | This should be set to the sponsor of the resource. See note. |
513-
| **site\_policy** | Url | This should be a url pointing to the resource's usage policy |
514-
| **contact** | String | This should be the name of the resource's admin contact |
515-
| **email** | Email address | This should be the email address of the admin contact for the resource |
516-
| **city** | String | This should be the city that the resource is located in |
517-
| **country** | String | This should be two letter country code for the country that the resource is located in. |
518-
| **longitude** | Number | This should be the longitude of the resource. It should be a number between -180 and 180. |
519-
| **latitude** | Number | This should be the latitude of the resource. It should be a number between -90 and 90. |
520-
521-
**Note for sponsor:**<br>
522-
If your resource has multiple sponsors, you can separate them using commas or specify the percentage using the following format:
523-
`osg, atlas, cms` or `osg:10, atlas:45, cms:45`.
524-
The percentages must add up to 100 if multiple sponsors are used.
525-
If you have a sponsor that is not an OSG VO, you can indicate this by using 'local' as the VO.

‎config/10-storage.ini‎

Lines changed: 9 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -10,33 +10,26 @@
1010
; Please refer to the OSG release documentation for an in-depth explanation
1111
; of the various storage models and the requirements for them
1212

13-
; If you have a SE available for your cluster and wish to make it available
14-
; to incoming jobs, set se_available to True, otherwise set it to False
15-
se_available = FALSE
16-
17-
; If you indicated that you have an se available at your cluster, set default_se to
18-
; the hostname of this SE, otherwise set default_se to UNAVAILABLE
19-
default_se = UNAVAILABLE
20-
21-
; The grid_dir setting should point to the directory which holds the files
13+
; The grid_dir setting should point to the directory which holds the files
2214
; from the OSG worker node package, it should be visible on all of the computer
2315
; nodes (read access is required, worker nodes don't need to be able to write)
2416
;
2517
; For RPM installations, /etc/osg/wn-client is the correct location
26-
grid_dir = /etc/osg/wn-client/
18+
;grid_dir = /etc/osg/wn-client/
2719

2820
; The app_dir setting should point to the directory which contains the VO
2921
; specific applications. It corresponds to the OSG_APP environment variable
3022
; in jobs.
3123
;
32-
; If no such directory is available, set app_dir to UNSET.
33-
; UNSET will keep the OSG_APP environment variable from being defined.
24+
; If no such directory is available, set app_dir to UNSET or UNAVAILABLE.
3425
; app_dir may be in OASIS.
3526
; To use an app_dir in OASIS, set it to "/cvmfs/oasis.opensciencegrid.org".
3627
; If not in OASIS, then app_dir should have an etc/ subdirectory, and both
3728
; app_dir and app_dir/etc must be world readable and executable (i.e. at
3829
; least 755 or 1755 permissions).
39-
app_dir = UNAVAILABLE
30+
;
31+
; This setting is optional.
32+
;app_dir = UNAVAILABLE
4033

4134
; The data_dir setting should point to a directory that can be used to store
4235
; and stage data in and out of the cluster. It corresponds to the OSG_DATA
@@ -46,7 +39,7 @@ app_dir = UNAVAILABLE
4639
; nodes.
4740
;
4841
; This setting is optional.
49-
data_dir = UNAVAILABLE
42+
;data_dir = UNAVAILABLE
5043

5144
; The worker_node_temp directory should point to a directory that can be used
5245
; as scratch space on compute nodes, it should allow read and write access on the
@@ -63,10 +56,10 @@ worker_node_temp = UNAVAILABLE
6356
; be read to stage in data, this is an url if you are using a SE
6457
;
6558
; This setting is optional.
66-
site_read = UNAVAILABLE
59+
;site_read = UNAVAILABLE
6760

6861
; The site_write setting should be the location or url to a directory that can
6962
; be write to stage out data, this is an url if you are using a SE
7063
;
7164
; This setting is optional.
72-
site_write = UNAVAILABLE
65+
;site_write = UNAVAILABLE

‎config/40-siteinfo.ini‎

Lines changed: 4 additions & 41 deletions
Original file line numberDiff line numberDiff line change
@@ -20,46 +20,9 @@ host_name = UNAVAILABLE
2020
; YOU WILL NEED TO CHANGE THIS
2121
resource = UNAVAILABLE
2222

23-
; Set "sponsor" to the sponsor(s) for your cluster.
24-
; If your cluster has multiple sponsors, separate them with commas
25-
; and specify the percentages using this format: "osg:10, atlas:45, cms:45".
26-
;
27-
;sponsor =
28-
29-
; Set "site_policy" to a URL that lists your site's usage policy (if there
30-
; is such a URL).
31-
;site_policy = UNAVAILABLE
32-
33-
; Set "contact" to the name of the admin/technical contact for the cluster.
34-
;
35-
; YOU WILL NEED TO CHANGE THIS
36-
contact = UNAVAILABLE
37-
38-
; Set "email" to the email address of the admin/technical contact for the cluster.
39-
;
40-
; YOU WILL NEED TO CHANGE THIS
41-
email = UNAVAILABLE
42-
43-
; Set "city" to the city that the cluster is located in.
44-
;
45-
; YOU WILL NEED TO CHANGE THIS
46-
city = UNAVAILABLE
47-
48-
; Set "country" to the country that the cluster is located in.
49-
;
50-
; YOU WILL NEED TO CHANGE THIS
51-
country = UNAVAILABLE
52-
53-
; Set "longitude" to the longitude of the cluster's location.
54-
; Accepted values are between -180 (west) and 180 (east).
55-
; If you are in the US, this will be negative.
23+
; Set "resource_group" to the name of the resource that you have registered
24+
; in the OSG topology repository at https://github.com/opensciencegrid/topology
5625
;
57-
; YOU WILL NEED TO CHANGE THIS
58-
longitude = UNAVAILABLE
26+
; CEs WILL NEED TO CHANGE THIS
27+
resource_group = UNAVAILABLE
5928

60-
; Set "latitude" to the latitude of the cluster's location.
61-
; Accepted values are between -90 (south) and 90 (north).
62-
; If you are in the US, this will be positive.
63-
;
64-
; YOU WILL NEED TO CHANGE THIS
65-
latitude = UNAVAILABLE

‎osg_configure/configure_modules/gratia.py‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -227,6 +227,9 @@ def configure(self, attributes):
227227
else:
228228
self.options['resource'].value = attributes['OSG_SITE_NAME']
229229

230+
# TODO Why is gratia looking at attributes instead of just using the options
231+
# directly? If it's not necessary, drop the OSG_HOSTNAME attribute from
232+
# siteinformation.py.
230233
if 'OSG_HOSTNAME' not in attributes:
231234
self.log('Hostname of this machine not specified. Please give this '
232235
'in the host_name option in the Site Information section',

‎osg_configure/configure_modules/infoservices.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -235,7 +235,7 @@ def _write_ce_collector_attributes_file(self, attributes_file):
235235

236236
for name, value in [
237237
('OSG_Resource', self.osg_resource),
238-
('OSG_ResourceGroup', self.osg_resource_group or ""),
238+
('OSG_ResourceGroup', self.osg_resource_group),
239239
('OSG_BatchSystems', ",".join(self.enabled_batch_systems))
240240
]:
241241
attributes_file_lines.append("%s = %s" % (name, utilities.classad_quote(value)))

‎osg_configure/configure_modules/siteinformation.py‎

Lines changed: 13 additions & 109 deletions
Original file line numberDiff line numberDiff line change
@@ -33,50 +33,14 @@ def __init__(self, *args, **kwargs):
3333
configfile.Option(name='host_name',
3434
required=MANDATORY_ON_CE,
3535
mapping='OSG_HOSTNAME'),
36-
'sponsor':
37-
configfile.Option(name='sponsor',
38-
required=OPTIONAL,
39-
default_value='',
40-
mapping='OSG_SPONSOR'),
41-
'site_policy':
42-
configfile.Option(name='site_policy',
43-
required=OPTIONAL,
44-
default_value='',
45-
mapping='OSG_SITE_INFO'),
46-
'contact':
47-
configfile.Option(name='contact',
48-
required=MANDATORY_ON_CE,
49-
mapping='OSG_CONTACT_NAME'),
50-
'email':
51-
configfile.Option(name='email',
52-
required=MANDATORY_ON_CE,
53-
mapping='OSG_CONTACT_EMAIL'),
54-
'city':
55-
configfile.Option(name='city',
56-
required=MANDATORY_ON_CE,
57-
mapping='OSG_SITE_CITY'),
58-
'country':
59-
configfile.Option(name='country',
60-
required=MANDATORY_ON_CE,
61-
mapping='OSG_SITE_COUNTRY'),
62-
'longitude':
63-
configfile.Option(name='longitude',
64-
opt_type=float,
65-
required=MANDATORY_ON_CE,
66-
mapping='OSG_SITE_LONGITUDE'),
67-
'latitude':
68-
configfile.Option(name='latitude',
69-
opt_type=float,
70-
required=MANDATORY_ON_CE,
71-
mapping='OSG_SITE_LATITUDE'),
36+
# ^^ the OSG_HOSTNAME mapping is used by the gratia module, though not sure why.
7237
'resource':
7338
configfile.Option(name='resource',
7439
required=MANDATORY,
7540
mapping='OSG_SITE_NAME'),
7641
'resource_group':
7742
configfile.Option(name='resource_group',
78-
default_value='',
79-
required=OPTIONAL)}
43+
required=MANDATORY_ON_CE)}
8044

8145
self.config_section = "Site Information"
8246
self.enabled = True
@@ -96,7 +60,17 @@ def parse_configuration(self, configuration):
9660
self.log('SiteInformation.parse_configuration completed')
9761
return
9862

99-
self.get_options(configuration)
63+
self.get_options(configuration,
64+
ignore_options=[
65+
"city",
66+
"contact",
67+
"country",
68+
"email",
69+
"latitude",
70+
"longitude",
71+
"site_policy",
72+
"sponsor",
73+
])
10074
self.log('SiteInformation.parse_configuration completed')
10175

10276
# pylint: disable-msg=W0613
@@ -134,79 +108,9 @@ def check_attributes(self, attributes):
134108
section=self.config_section,
135109
level=logging.WARNING)
136110

137-
latitude = self.opt_val("latitude")
138-
if not utilities.blank(latitude) and not -90 <= latitude <= 90:
139-
self.log("Latitude must be between -90 and 90, got %s" %
140-
latitude,
141-
section=self.config_section,
142-
option='latitude',
143-
level=logging.ERROR)
144-
attributes_ok = False
145-
146-
longitude = self.opt_val("longitude")
147-
if not utilities.blank(longitude) and not -180 <= longitude <= 180:
148-
self.log("Longitude must be between -180 and 180, got %s" %
149-
longitude,
150-
section=self.config_section,
151-
option='longitude',
152-
level=logging.ERROR)
153-
attributes_ok = False
154-
155-
email = self.opt_val("email")
156-
# make sure the email address has the correct format
157-
if not utilities.blank(email) and not validation.valid_email(email):
158-
self.log("Invalid email address in site information: %s" %
159-
email,
160-
section=self.config_section,
161-
option='email',
162-
level=logging.ERROR)
163-
attributes_ok = False
164-
165-
sponsor = self.opt_val("sponsor")
166-
if not utilities.blank(sponsor):
167-
attributes_ok &= self.check_sponsor(sponsor)
168-
169111
self.log('SiteInformation.check_attributes completed')
170112
return attributes_ok
171113

172-
def check_sponsor(self, sponsor):
173-
attributes_ok = True
174-
percentage = 0
175-
for vo in re.split(r'\s*,?\s*', sponsor):
176-
vo_split = vo.split(':')
177-
if len(vo_split) == 1:
178-
percentage += 100
179-
elif len(vo_split) == 2:
180-
vo_percentage = vo_split[1]
181-
try:
182-
percentage += int(vo_percentage)
183-
except ValueError:
184-
self.log("VO percentage (%s) in sponsor field (%s) not an integer"
185-
% (vo_percentage, vo),
186-
section=self.config_section,
187-
option='sponsor',
188-
level=logging.ERROR,
189-
exception=True)
190-
attributes_ok = False
191-
else:
192-
self.log("VO sponsor field is not formated correctly: %s" % vo,
193-
section=self.config_section,
194-
option='sponsor',
195-
level=logging.ERROR)
196-
self.log("Sponsors should be given as sponsor:percentage "
197-
"separated by a space or comma")
198-
attributes_ok = False
199-
200-
if percentage != 100:
201-
self.log("VO percentages in sponsor field do not add up to 100, got %s"
202-
% percentage,
203-
section=self.config_section,
204-
option='sponsor',
205-
level=logging.ERROR)
206-
attributes_ok = False
207-
208-
return attributes_ok
209-
210114
def module_name(self):
211115
"""Return a string with the name of the module"""
212116
return "SiteInformation"

0 commit comments

Comments
 (0)