ExtraSubmitString for Condor for queue only

[marianne013]

Probably one for @andresailer
I'm trying to commission a GPU queue.
I can submit to it directly using the following sub file:
lx02:2021_Dec_13_1345_lz_EL7 > cat /home/hep/dbauer/dirac_site_tests/uki-lt2-ic-hep/htcondorce_gpu.sub

universe = vanilla
use_x509userproxy = true
+Owner = undefined
+RequestGPUs = 1
executable = pilot_test.sh
output = job.log
error = job.err
log = ce_test.log
delegate_job_GSI_credentials_lifetime = 0
ShouldTransferFiles = YES
WhenToTransferOutput = ON_EXIT
queue

In DIRAC we have Computing/HTCondorCE/ExtraSubmitString = delegate_job_GSI_credentials_lifetime = 0 set.
I tried to set
ExtraSubmitString in ceprod03.grid.hep.ph.ic.ac.uk/Queues/ceprod03.grid.hep.ph.ic.ac.uk-condor-GPU
ExtraSubmitString = RequestGPUs = 1
This resulted in an error:

2022-01-18 08:56:23 UTC WorkloadManagement/SiteDirectorLz/ceprod03.grid.hep.ph.ic.ac.uk DEBUG: ExtraSubmitString:
###
 RequestGPUs = 1
###
2022-01-18 08:56:23 UTC WorkloadManagement/SiteDirectorLz DEBUG: Connected to: dips://dirac01.grid.hep.ph.ic.ac.uk:9135/Configuration/Server
2022-01-18 08:56:23 UTC WorkloadManagement/SiteDirectorLz DEBUG: New connection -> 2a0c:5bc0:c8:2:216:3eff:fe7f:1bf:9135
2022-01-18 08:56:23 UTC WorkloadManagement/SiteDirectorLz/Subprocess VERBOSE: systemCall: ['condor_submit', '-terse', '-pool', u'ceprod03.grid.hep.ph.ic.ac.uk:9619', '-remote', u'ceprod03.grid.hep.ph.ic.ac.uk', '/opt/dir\
ac/condor/HTCondorCE_xt0ECg.sub']
2022-01-18 08:56:23 UTC WorkloadManagement/SiteDirectorLz DEBUG: New version available Updating to version 2022-01-18 08:52:43.493796...
2022-01-18 08:56:24 UTC WorkloadManagement/SiteDirectorLz DEBUG: Updating configuration internals
2022-01-18 08:56:24 UTC WorkloadManagement/SiteDirectorLz DEBUG: Updated to version 2022-01-18 08:52:43.493796
2022-01-18 08:56:24 UTC WorkloadManagement/SiteDirectorLz/ceprod03.grid.hep.ph.ic.ac.uk VERBOSE: {'OK': True, 'Value': (0, u'2589061.0 - 2589061.0\n', u"WARNING: the line 'grid_resource = condor ceprod03.grid.hep.ph.ic.a\
c.uk ceprod03.grid.hep.ph.ic.ac.uk:9619' was unused by condor_submit. Is it a typo?\nWARNING: the line 'RequestGPUs = 1' was unused by condor_submit. Is it a typo?\n")}

I then tried every variation under the sun, trying (as suggested in the header) to insert a '\n'. I tried request_gpus = 1 because I never know what is condor vs HTCondorCE. None of it works. Any idea how this is meant to work ?

I tried catching the .sub file it creates in the act, but it seems to get deleted immediately. Is there a way I can prevent this ?

[andresailer]

What is your condor_version?

In your example file you have +RequestGPUs = 1, does adding a plus in ExtraSubmitString work?
Does request_gpus=1 work for your manual test?

Edit: submitfile is removed here

DIRAC/src/DIRAC/Resources/Computing/HTCondorCEComputingElement.py

Line 285 in 1a6fc08

os.remove(subName)

[marianne013]

I got admonished for the +, apparently that's secret condor code for "just copy it or else". request_gpus showed up in our batch system, but then didn't work. Simon has now found a problem with our setup, so we'll fix that first and then I'll get back to it.

[marianne013]

So I got a bit further.
I made a queue and set the ExtraSubmitString to request_gpus = 1 and this made my job run on the GPU node.
Now I need to add the delegate_job_GSI_credentials_lifetime = 0 condition which I have set under Computing/HTCondorCE for all CEs again, as the default gets overwritten. Currently I am using request_gpus = 1 \n delegate_job_GSI_credentials_lifetime = 0 which seems to work, but seems rather error prone. Is there any chance of a comma separated list for the configuration, similar to other DIRAC options ?

[andresailer]

Fine for me to also allow the ExtraSubmitString to be a list, but keep backward compatible with the newline for a bit.