Not able to submit to ARC CE using REST (CSCS site) #6210

arrabito · 2022-06-27T16:31:25Z

arrabito
Jun 27, 2022

Dear all,

We have a new site in our setup, CSCS, using ARC CE without gridftp.

However, the SiteDirector fails to submit pilots and I get the following message:

2022-06-27 16:27:20 UTC WorkloadManagement/SiteDirectorARCCSCS/arc-noir.cta.cscs.ch VERBOSE: Executable file path: /opt/dirac/work/WorkloadManagement/SiteDirectorARCCSCS/DIRAC__4c5v3rj_pilotwrapper.py
2022-06-27 16:27:20 UTC WorkloadManagement/SiteDirectorARCCSCS/arc-noir.cta.cscs.ch DEBUG: XRSL string submitted : 
&(executable="DIRAC__4c5v3rj_pilotwrapper.py")
(inputFiles=(DIRAC__4c5v3rj_pilotwrapper.py "/opt/dirac/work/WorkloadManagement/SiteDirectorARCCSCS/DIRAC__4c5v3rj_pilotwrapper.py"))
(stdout="4C90DF38.out")
(stderr="4C90DF38.err")
(outputFiles=("4C90DF38.out" "") ("4C90DF38.err" ""))
(queue=normal)

(count = 8)
(countpernode = 8)

      
(runTimeEnvironment="ENV/PROXY") (cpuTime="48h") (memory="3100")
    
2022-06-27 16:27:20 UTC WorkloadManagement/SiteDirectorARCCSCS/arc-noir.cta.cscs.ch DEBUG: DIRAC stamp for job : 4C90DF38
2022-06-27 16:27:20 UTC WorkloadManagement/SiteDirectorARCCSCS/arc-noir.cta.cscs.ch WARN: Failed to submit job because  Job not submitted - incorrect job description? (missing field in XRSL string?)
2022-06-27 16:27:20 UTC WorkloadManagement/SiteDirectorARCCSCS/arc-noir.cta.cscs.ch WARN: Failed to submit job because  some error from the CE - possibly CE problems?
2022-06-27 16:27:20 UTC WorkloadManagement/SiteDirectorARCCSCS/arc-noir.cta.cscs.ch WARN: Failed to submit job because  ... maybe above messages will give a hint.
2022-06-27 16:27:20 UTC WorkloadManagement/SiteDirectorARCCSCS ERROR: Failed submission to queue Queue arc-noir.cta.cscs.ch_normal:
, No pilot references obtained from the ARC job submission

I've tried several XRSL string but it doesn't work.

However, submitting the same XRSL description with arcsub it does work.

I'm using 7.3.12 version.

It seems to the that in LHCb you also use the CSCS site.

Could you please let me know how you succeed to submit there and eventually also send me your configuration for that site?

Thanks a lot

Answered by aldbr

Jul 1, 2022

Here is a new PR for correctly binding ARC6 pilot-jobs with jobs: #6230

It is known bug in ARC client and Alexandre promised to implement workaround for it (_exit).

I added a fix in a previous PR, adding the following line should resolve the issue: https://github.com/DIRACGrid/DIRAC/blob/integration/src/DIRAC/Core/scripts/dirac_agent.py#L41-L43

View full answer

marianne013 · 2022-06-27T18:18:24Z

marianne013
Jun 27, 2022

Hi Luisa, can you submit a job to the site outside of DIRAC ? In my experience the "Job not submitted - incorrect job description?" error is a red-herring and the error is somewhere else, see e.g.:
https://ggus.eu/index.php?mode=ticket_info&ticket_id=155137
Daniela

1 reply

arrabito Jun 28, 2022
Author

Hi Luisa, can you submit a job to the site outside of DIRAC ? In my experience the "Job not submitted - incorrect job description?" error is a red-herring and the error is somewhere else, see e.g.: https://ggus.eu/index.php?mode=ticket_info&ticket_id=155137 Daniela

I confirm that submitting with arcsub does work and I agree with you that the error seems to be elsewhere.

aldbr · 2022-06-28T05:58:30Z

aldbr
Jun 28, 2022
Collaborator

ARCComputingElement does not handle properly the REST and the EMI-ES interface.
We recently introduced a new CE named ARC6ComputingElement that supports all the interfaces (https://github.com/DIRACGrid/DIRAC/blob/rel-v7r3/src/DIRAC/Resources/Computing/ARC6ComputingElement.py).
You can give it a try by replacing CEType = ARC with CEType = ARC6.

Should you have any issue with it, please let us know.

0 replies

arrabito · 2022-06-28T07:03:45Z

arrabito
Jun 28, 2022
Author

Yes I noticed that yesterday and I tried as you said and copying ARC6ComputingElement.py on the server.

After a minor bug fix:

class ARCC6omputingElement(ARCComputingElement):

->
class ARC6ComputingElement(ARCComputingElement):

It worked better, but I still have an error:


2022-06-28 06:59:54 UTC WorkloadManagement/SiteDirectorARCCSCS/arc-noir.cta.cscs.ch DEBUG: Adding target: gsiftp://arc-noir.cta.cscs.ch:2811/jobs (org.nordugrid.gridftpjob)
2022-06-28 06:59:54 UTC WorkloadManagement/SiteDirectorARCCSCS ERROR: Agent exception while calling method <bound method SiteDirector.execute of <DIRAC.WorkloadManagementSystem.Agent.SiteDirector.SiteDirector object at 0x7f21a7d379a0>>
Traceback (most recent call last):
  File "/opt/dirac/versions/v2.0.28-1654240616/Linux-x86_64/lib/python3.9/site-packages/DIRAC/Core/Base/AgentModule.py", line 342, in am_secureCall
    result = functor(*args)
  File "/opt/dirac/versions/v2.0.28-1654240616/Linux-x86_64/lib/python3.9/site-packages/DIRAC/WorkloadManagementSystem/Agent/SiteDirector.py", line 332, in execute
    result = self.submitPilots()
  File "/opt/dirac/versions/v2.0.28-1654240616/Linux-x86_64/lib/python3.9/site-packages/DIRAC/WorkloadManagementSystem/Agent/SiteDirector.py", line 455, in submitPilots
    res = self._submitPilotsToQueue(pilotsToSubmit, ce, queueName)
  File "/opt/dirac/versions/v2.0.28-1654240616/Linux-x86_64/lib/python3.9/site-packages/DIRAC/WorkloadManagementSystem/Agent/SiteDirector.py", line 717, in _submitPilotsToQueue
    submitResult = ce.submitJob(executable, "", pilotsToSubmit)
  File "/opt/dirac/versions/v2.0.28-1654240616/Linux-x86_64/lib/python3.9/site-packages/DIRAC/Resources/Computing/ARC6ComputingElement.py", line 118, in submitJob
    xrslString, diracStamp = self.__writeXRSL(executableFile)
AttributeError: 'ARC6ComputingElement' object has no attribute '_ARC6ComputingElement__writeXRSL'
2022-06-28 06:59:54 UTC WorkloadManagement/SiteDirectorARCCSCS/Monitoring DEBUG: Adding mark to CPU
2022-06-28 06:59:54 UTC WorkloadManagement/SiteDirectorARCCSCS NOTICE: ----------------------------------------
2022-06-28 06:59:54 UTC WorkloadManagement/SiteDirectorARCCSCS NOTICE: Agent module WorkloadManagement/SiteDirectorARCCSCS run summary
2022-06-28 06:59:54 UTC WorkloadManagement/SiteDirectorARCCSCS NOTICE:  Executed 1 times previously
2022-06-28 06:59:54 UTC WorkloadManagement/SiteDirectorARCCSCS NOTICE:  Cycle took 1.77 seconds
2022-06-28 06:59:54 UTC WorkloadManagement/SiteDirectorARCCSCS NOTICE:  Average execution time: 1.77 seconds
2022-06-28 06:59:54 UTC WorkloadManagement/SiteDirectorARCCSCS NOTICE:  Polling time: 60 seconds
2022-06-28 06:59:54 UTC WorkloadManagement/SiteDirectorARCCSCS NOTICE:  Average execution/polling time: 2.95%
2022-06-28 06:59:54 UTC WorkloadManagement/SiteDirectorARCCSCS WARN:  Cycle had an error: Exception while calling <bound method SiteDirector.execute of <DIRAC.WorkloadManagementSystem.Agent.SiteDirector.SiteDirector object at 0x7f21a7d379a0>> method: 'ARC6ComputingElement' object has no attribute '_ARC6ComputingElement__writeXRSL'
2022-06-28 06:59:54 UTC WorkloadManagement/SiteDirectorARCCSCS NOTICE: ----------------------------------------
^C

Any idea?

Thank you,

Luisa

3 replies

aldbr Jun 28, 2022
Collaborator

Sorry, we haven't properly tested it in certification yet.
This PR should solve the issues: #6211

The writeXRSL method was declared as private in ARC and was not accessible from ARC6.
Removing a _ from the name of the method should help: _writeXRSL instead of __writeXRSL.

marianne013 Jun 28, 2022

@aldbr Is there a test for this on the Trello board ? I noticed that there don't seem to be explicit HTCondorCE/ARC(all variant) tests.

aldbr Jun 28, 2022
Collaborator

There is no test explicitly dedicated to HTCondorCE and ARC configurations.
For the moment, the only way to test that is to submit jobs to various CEs and check whether the errors are related to the CE/node/pilot/job.

arrabito · 2022-06-28T08:27:51Z

arrabito
Jun 28, 2022
Author

Yes thank you. It solved that problem, but it's still failing submitting pilots and I don't find any indication from the logs:


2022-06-28 08:23:57 UTC WorkloadManagement/SiteDirectorARCCSCS INFO: Going to submit pilots (a maximum of 2 pilots to arc-noir.cta.cscs.ch_normal queue)
2022-06-28 08:23:58 UTC WorkloadManagement/SiteDirectorARCCSCS VERBOSE: pilotOptions: -S CTA -l CTA --pythonVersion=3 -e CTA -N arc-noir.cta.cscs.ch -Q normal -n ARC.CSCS.ch --userEnvVariables DIRACSYSCONFIG:::pilot.cfg
2022-06-28 08:23:58 UTC WorkloadManagement/SiteDirectorARCCSCS/arc-noir.cta.cscs.ch DEBUG: Set proxy variable X509_USER_PROXY to /tmp/tmp33kapk26
2022-06-28 08:23:58 UTC WorkloadManagement/SiteDirectorARCCSCS/arc-noir.cta.cscs.ch VERBOSE: Executable file path: /opt/dirac/work/WorkloadManagement/SiteDirectorARCCSCS/DIRAC_jriwgnpl_pilotwrapper.py
2022-06-28 08:23:58 UTC WorkloadManagement/SiteDirectorARCCSCS ERROR: Failed submission to queue Queue arc-noir.cta.cscs.ch_normal:
, No pilot references obtained from the ARC job submission
2022-06-28 08:23:58 UTC WorkloadManagement/SiteDirectorARCCSCS VERBOSE: Begin commit
2022-06-28 08:23:58 UTC WorkloadManagement/SiteDirectorARCCSCS INFO: Failed pilot submission Queue: arc-noir.cta.cscs.ch_normal
2022-06-28 08:23:58 UTC WorkloadManagement/SiteDirectorARCCSCS INFO: Total number of pilots submitted in this cycle 0

Could you eventually also share the CE and queue configuration for CSCS site in your instance?
Thank you.

1 reply

aldbr Jun 28, 2022
Collaborator

To get more details about the issue, could you uncomment the following lines please?
https://github.com/DIRACGrid/DIRAC/blob/rel-v7r3/src/DIRAC/Resources/Computing/ARCComputingElement.py#L61-L65

You should get logs from the arc library.

Also, here is the configuration that we have:

arc.lcg.cscs.ch
{
  SubmissionMode = Direct
  OS = CentOS 7.6_0_0
  SI00 = 2500
  MaxRAM = 126000
  CEType = ARC6
  architecture = x86_64
  VO = lhcb
  XRSLExtraString = (runtimeenvironment="ENV/PROXY")   (runtimeenvironment="ENV/GLITE")
  Queues
  {
    nordugrid-slurm-normal
    {
      NumberOfProcessors = 1
      MaxWaitingJobs = 10
      maxCPUTime = 2304
      SI00 = 2552
      VO = lhcb
      MaxTotalJobs = 1000
      ExtraPilotOptions = -M 1
    }
  }
}

arrabito · 2022-06-28T09:37:54Z

arrabito
Jun 28, 2022
Author

Thank you for the configuration.

Here below what I get after uncommeting those lines:


2022-06-28 09:34:08 UTC WorkloadManagement/SiteDirectorARCCSCS INFO: Going to submit pilots (a maximum of 2 pilots to arc-noir.cta.cscs.ch_normal queue)
2022-06-28 09:34:08 UTC WorkloadManagement/SiteDirectorARCCSCS VERBOSE: pilotOptions: -S CTA -l CTA --pythonVersion=3 -e CTA -N arc-noir.cta.cscs.ch -Q normal -n ARC.CSCS.ch --userEnvVariables DIRACSYSCONFIG:::pilot.cfg
2022-06-28 09:34:08 UTC WorkloadManagement/SiteDirectorARCCSCS/arc-noir.cta.cscs.ch DEBUG: Set proxy variable X509_USER_PROXY to /tmp/tmp66iq8mam
2022-06-28 09:34:08 UTC WorkloadManagement/SiteDirectorARCCSCS/arc-noir.cta.cscs.ch VERBOSE: Executable file path: /opt/dirac/work/WorkloadManagement/SiteDirectorARCCSCS/DIRAC_v9nvy7s1_pilotwrapper.py
VERBOSE: LDAPQuery: Initializing connection to arc-noir.cta.cscs.ch:2135
VERBOSE: Failed to bind to ldap server (arc-noir.cta.cscs.ch)
2022-06-28 09:34:08 UTC WorkloadManagement/SiteDirectorARCCSCS ERROR: Failed submission to queue Queue arc-noir.cta.cscs.ch_normal:
, No pilot references obtained from the ARC job submission
2022-06-28 09:34:08 UTC WorkloadManagement/SiteDirectorARCCSCS VERBOSE: Begin commit
2022-06-28 09:34:08 UTC WorkloadManagement/SiteDirectorARCCSCS INFO: Failed pilot submission Queue: arc-noir.cta.cscs.ch_normal

If I understand correctly the problem here is due to the fact that this endpoint is not published in any BDII. I had told to site admins that it was not a requirement....

Do you confirm?

While when using the standard 'ARCComputingElement.py' I was able to create the endpoint and the error was coming after.

Any suggestion?

Thank you.

2 replies

aldbr Jun 28, 2022
Collaborator

Did you just change something?
I don't see the Failed to bind to ldap server error anymore on my side.

Well, I am not an expert of BDII, but I guess it would be necessary with the way it is done in ARC6.
And in the meantime, if this is the problem, then I guess that the arcsub command should not work too.
Could you provide the arcsub command that you used + the output, please?

arrabito Jun 28, 2022
Author

Did you just change something? I don't see the Failed to bind to ldap server error anymore on my side.

I did not change anything

Well, I am not an expert of BDII, but I guess it would be necessary with the way it is done in ARC6.

Yes I guess so

And in the meantime, if this is the problem, then I guess that the arcsub command should not work too. Could you provide the arcsub command that you used + the output, please?

The command I use is:

arcsub -d warning -C arc-noir.cta.cscs.ch ./basic-noir.xrls

with :

$ cat basic-noir.xrls 
&(executable=run.sh)
(inputfiles=
    ('run.sh' './run.sh')
)
(stderr=stderr)
(jobname='basic')
(stdout=stdout)
(gmlog=gmlog)
(wallTime="100")
(count="1")
(memory="1000")
(queue="normal")
(runTimeEnvironment="ENV/PROXY")
(runTimeEnvironment="ENV/GLITE")

Yesterday it was working fine with the pilot reference given as output (I don't have the precise output now), but today it's failing:

$ arcsub -d warning -C arc-noir.cta.cscs.ch ./basic-noir.xrls
ERROR: One or multiple job descriptions was not submitted.

I think that something has changed on the site side. I will contact site admins and let you know.

Thanks.

aldbr · 2022-06-28T11:57:37Z

aldbr
Jun 28, 2022
Collaborator

I did not change anything

And do you still see this error on your side?

Yes I guess so

Would it be a problem? (to publish it in BDII)

Yesterday it was working fine with the pilot reference given as output (I don't have the precise output now), but today it's failing:

Or may be the proxy is not valid anymore?
Just in case, you can renew it with arcproxy -S <vo name>
Then you might need to do export X509_USER_PROXY=<proxy location> (proxy location can be found with arcproxy --info).

1 reply

arrabito Jun 28, 2022
Author

I did not change anything

And do you still see this error on your side?

Yes

Yes I guess so

Would it be a problem? (to publish it in BDII)

I have to ask to site admins

Yesterday it was working fine with the pilot reference given as output (I don't have the precise output now), but today it's failing:

Or may be the proxy is not valid anymore? Just in case, you can renew it with arcproxy -S <vo name> Then you might need to do export X509_USER_PROXY=<proxy location> (proxy location can be found with arcproxy --info).

I don't think so, when the proxy is expired I get:

WARNING: Certificate and key ('usercert.pem' and 'userkey.pem') not found in any of the paths: '/home/dirac/.arc/', '/home/dirac/.globus/', '/opt/dirac/versions/v2.0.28-1654240616/Linux-x86_64/etc/arc/', '/home/dirac/'
WARNING: If the proxy or certificate/key does exist, you can manually specify the locations via environment variables 'X509_USER_CERT'/'X509_USER_KEY' or 'X509_USER_PROXY', or the 'certificatepath'/'proxypath' or 'keypath' attributes in the client configuration file (e.g. '~/.arc/client.conf')
ERROR: Failed configuration initialization

So I setup a proxy by setting the X509_USER_PROXY variable and then I get a different error (the one I mentioned before).

arrabito · 2022-06-28T13:05:53Z

arrabito
Jun 28, 2022
Author

On the site side they fixed some routing problem and now the SiteDirector is able to submit pilots:

2022-06-28 12:48:47 UTC WorkloadManagement/SiteDirectorARCCSCS/arc-noir.cta.cscs.ch DEBUG: XRSL string submitted : 
&(executable="DIRAC_2zt6qat1_pilotwrapper.py")
(inputFiles=(DIRAC_2zt6qat1_pilotwrapper.py "/opt/dirac/work/WorkloadManagement/SiteDirectorARCCSCS/DIRAC_2zt6qat1_pilotwrapper.py") )
(stdout="B3B8D1D0.out")
(stderr="B3B8D1D0.err")
(outputFiles=("B3B8D1D0.out" "") ("B3B8D1D0.err" ""))
(queue=normal)

(count = 8)
(countpernode = 8)

      

(runTimeEnvironment="ENV/PROXY") (runTimeEnvironment="ENV/GLITE") (cpuTime="48h") (memory="3100")
    
2022-06-28 12:48:47 UTC WorkloadManagement/SiteDirectorARCCSCS/arc-noir.cta.cscs.ch DEBUG: DIRAC stamp for job : B3B8D1D0
VERBOSE: String successfully parsed as nordugrid:xrsl.
VERBOSE: Trying to connect arc-noir.cta.cscs.ch(IPv4):443
VERBOSE: Using curve with NID: 716
VERBOSE: Using cipher list: HIGH:!eNULL:!aNULL
VERBOSE: Using cipher: TLS_AES_256_GCM_SHA384
VERBOSE: Peer name: /DC=com/DC=quovadisglobal/DC=grid/DC=UniversitaetBern/DC=hosts/C=CH/ST=Zurich/L=Zurich/O=ETH Zurich/CN=arc-noir.cta.cscs.ch
VERBOSE: Identity name: /DC=com/DC=quovadisglobal/DC=grid/DC=UniversitaetBern/DC=hosts/C=CH/ST=Zurich/L=Zurich/O=ETH Zurich/CN=arc-noir.cta.cscs.ch
VERBOSE: CA name: /C=BM/O=QuoVadis Limited/CN=QuoVadis Root CA 2
VERBOSE: Peer name: /DC=com/DC=quovadisglobal/DC=grid/DC=UniversitaetBern/DC=hosts/C=CH/ST=Zurich/L=Zurich/O=ETH Zurich/CN=arc-noir.cta.cscs.ch
VERBOSE: Identity name: /DC=com/DC=quovadisglobal/DC=grid/DC=UniversitaetBern/DC=hosts/C=CH/ST=Zurich/L=Zurich/O=ETH Zurich/CN=arc-noir.cta.cscs.ch
VERBOSE: CA name: /C=BM/O=QuoVadis Limited/CN=QuoVadis Root CA 2
VERBOSE: Requirement "== ENV/PROXY" satisfied by "ENV/PROXY".
VERBOSE: Requirement "== ENV/GLITE" satisfied by "ENV/GLITE".
VERBOSE: All requirements satisfied.
VERBOSE: Generating emies:adl job description output
VERBOSE: Trying to connect arc-noir.cta.cscs.ch(IPv4):443
VERBOSE: Using curve with NID: 716
VERBOSE: Using cipher list: HIGH:!eNULL:!aNULL
VERBOSE: Using cipher: TLS_AES_256_GCM_SHA384
VERBOSE: Peer name: /DC=com/DC=quovadisglobal/DC=grid/DC=UniversitaetBern/DC=hosts/C=CH/ST=Zurich/L=Zurich/O=ETH Zurich/CN=arc-noir.cta.cscs.ch
VERBOSE: Identity name: /DC=com/DC=quovadisglobal/DC=grid/DC=UniversitaetBern/DC=hosts/C=CH/ST=Zurich/L=Zurich/O=ETH Zurich/CN=arc-noir.cta.cscs.ch
VERBOSE: CA name: /C=BM/O=QuoVadis Limited/CN=QuoVadis Root CA 2
INFO: Transfer from file:/opt/dirac/work/WorkloadManagement/SiteDirectorARCCSCS/DIRAC_2zt6qat1_pilotwrapper.py to https://arc-noir.cta.cscs.ch:443/arex/rest/1.0/jobs/UyZNDmIpPQ1nLZfWYmas2NHqEbqlNmABFKDmMEFKDmPBFKDmjNT2Mn/session/DIRAC_2zt6qat1_pilotwrapper.py
VERBOSE: DataMover: cycle
INFO: Real transfer from file:/opt/dirac/work/WorkloadManagement/SiteDirectorARCCSCS/DIRAC_2zt6qat1_pilotwrapper.py to https://arc-noir.cta.cscs.ch:443/arex/rest/1.0/jobs/UyZNDmIpPQ1nLZfWYmas2NHqEbqlNmABFKDmMEFKDmPBFKDmjNT2Mn/session/DIRAC_2zt6qat1_pilotwrapper.py
VERBOSE: Creating buffer: 1048576 x 2
VERBOSE: DataMove::Transfer: no checksum calculation for file:/opt/dirac/work/WorkloadManagement/SiteDirectorARCCSCS/DIRAC_2zt6qat1_pilotwrapper.py
VERBOSE: Failed to load plugin for URL (empty)
VERBOSE: Failed to load plugin for URL (empty)
INFO: Using buffered transfer method
VERBOSE: Waiting for buffer
VERBOSE: Peer name: /DC=com/DC=quovadisglobal/DC=grid/DC=UniversitaetBern/DC=hosts/C=CH/ST=Zurich/L=Zurich/O=ETH Zurich/CN=arc-noir.cta.cscs.ch
VERBOSE: Identity name: /DC=com/DC=quovadisglobal/DC=grid/DC=UniversitaetBern/DC=hosts/C=CH/ST=Zurich/L=Zurich/O=ETH Zurich/CN=arc-noir.cta.cscs.ch
VERBOSE: CA name: /C=BM/O=QuoVadis Limited/CN=QuoVadis Root CA 2
VERBOSE: Peer name: /DC=com/DC=quovadisglobal/DC=grid/DC=UniversitaetBern/DC=hosts/C=CH/ST=Zurich/L=Zurich/O=ETH Zurich/CN=arc-noir.cta.cscs.ch
VERBOSE: Identity name: /DC=com/DC=quovadisglobal/DC=grid/DC=UniversitaetBern/DC=hosts/C=CH/ST=Zurich/L=Zurich/O=ETH Zurich/CN=arc-noir.cta.cscs.ch
VERBOSE: CA name: /C=BM/O=QuoVadis Limited/CN=QuoVadis Root CA 2
VERBOSE: buffer: read EOF : yes
VERBOSE: buffer: write EOF: yes
VERBOSE: buffer: error    : no, read: no, write: no
VERBOSE: Closing read channel
VERBOSE: Closing write channel
VERBOSE: Checksum not computed
VERBOSE: Generating nordugrid:xrsl job description output
2022-06-28 12:48:49 UTC WorkloadManagement/SiteDirectorARCCSCS/arc-noir.cta.cscs.ch DEBUG: Successfully submitted job https://arc-noir.cta.cscs.ch:443/arex/rest/1.0/jobs/UyZNDmIpPQ1nLZfWYmas2NHqEbqlNmABFKDmMEFKDmPBFKDmjNT2Mn to CE arc-noir.cta.cscs.ch

The only problem is that they remain in Unknown status:

2022-06-28 12:59:11 UTC WorkloadManagement/SiteDirectorARCCSCS/arc-noir.cta.cscs.ch DEBUG: ARC status for job https://arc-noir.cta.cscs.ch:443/arex/rest/1.0/jobs/iApKDm5jPQ1nLZfWYmas2NHqEbqlNmABFKDmMEFKDmKBFKDmTBqD9m is Undefined
2022-06-28 12:59:11 UTC WorkloadManagement/SiteDirectorARCCSCS/arc-noir.cta.cscs.ch DEBUG: DIRAC status for job https://arc-noir.cta.cscs.ch:443/arex/rest/1.0/jobs/iApKDm5jPQ1nLZfWYmas2NHqEbqlNmABFKDmMEFKDmKBFKDmTBqD9m is Unknown

Any idea of the possible reason or how to debug further?

1 reply

aldbr Jun 28, 2022
Collaborator

On the site side they fixed some routing problem and now the SiteDirector is able to submit pilots

Great!

The only problem is that they remain in Unknown status

Do you have any information coming from the arc logs when you try to get the status of a job?
If not, can you get the status of a job submitted with arcsub? (arcstat <url of the job>)

Thanks

arrabito · 2022-06-28T15:57:59Z

arrabito
Jun 28, 2022
Author

arcstat does not work for me:

$ arcstat https://arc-noir.cta.cscs.ch:443/arex/rest/1.0/jobs/CCHNDmt4QQ1nLZfWYmas2NHqEbqlNmABFKDmMEFKDmRBFKDmNUVa4m
Warning: Job not found in job list: https://arc-noir.cta.cscs.ch:443/arex/rest/1.0/jobs/CCHNDmt4QQ1nLZfWYmas2NHqEbqlNmABFKDmMEFKDmRBFKDmNUVa4m
No jobs found, try later

However site admins find that on slurm batch system the jobs finished well.

And what I observe on my side is that these pilots are able to match the payloads that are correctly executed.

However pilots pass from Unknown to Aborted status which is quite annoying.

There is a probably an issue in querying the pilot status. In the logs I found:

2022-06-28 14:09:00 UTC WorkloadManagement/SiteDirectorARCCSCS/arc-noir.cta.cscs.ch DEBUG: Set proxy variable X509_USER_PROXY to /tmp/tmpd6lk3wu3
VERBOSE: Creating and sending job information query request to https://arc-noir.cta.cscs.ch:8443/arex
VERBOSE: Processing a esainfo:GetActivityInfo request
VERBOSE: Trying to connect arc-noir.cta.cscs.ch(IPv4):8443
2022-06-28 14:09:05 UTC WorkloadManagement/SiteDirectorARCCSCS/DIRAC.Core.Tornado.Client.ClientSelector DEBUG: Trying to autodetect client for Accounting/DataStore
2022-06-28 14:09:05 UTC WorkloadManagement/SiteDirectorARCCSCS/DIRAC.Core.Tornado.Client.ClientSelector DEBUG: URL resolved: dips://ccdcta-server04.in2p3.fr:9133/Accounting/DataStore
2022-06-28 14:09:05 UTC WorkloadManagement/SiteDirectorARCCSCS DEBUG: Discovering URL for service Accounting/DataStore -> dips://ccdcta-server04.in2p3.fr:9133/Accounting/DataStore
2022-06-28 14:09:05 UTC WorkloadManagement/SiteDirectorARCCSCS DEBUG: Trying to connect to: dips://ccdcta-server04.in2p3.fr:9133/Accounting/DataStore
2022-06-28 14:09:05 UTC WorkloadManagement/SiteDirectorARCCSCS DEBUG: Connected to: dips://ccdcta-server04.in2p3.fr:9133/Accounting/DataStore
2022-06-28 14:09:05 UTC WorkloadManagement/SiteDirectorARCCSCS DEBUG: New connection -> 134.158.34.54:9133
VERBOSE: Timeout connecting to arc-noir.cta.cscs.ch(IPv4):8443 - 20 s
VERBOSE: Using curve with NID: 716
VERBOSE: Using cipher list: HIGH:!eNULL:!aNULL
VERBOSE: Failed to establish SSL connection
ERROR: Failed to establish connection: TCP: GENERIC_ERROR (Timeout connecting to arc-noir.cta.cscs.ch(IPv4):8443 - 20 s)
VERBOSE: esainfo:GetActivityInfo request to https://arc-noir.cta.cscs.ch:8443/arex failed with response: Failed to send SOAP message: TCP: GENERIC_ERROR (Timeout connecting to arc-noir.cta.cscs.ch(IPv4):8443 - 20 s): TCP: GENERIC_ERROR (Timeout connecting to arc-noir.cta.cscs.ch(IPv4):8443 - 20 s)
VERBOSE: Processing a esainfo:GetActivityInfo request
VERBOSE: Trying to connect arc-noir.cta.cscs.ch(IPv4):8443
VERBOSE: Timeout connecting to arc-noir.cta.cscs.ch(IPv4):8443 - 20 s
VERBOSE: Using curve with NID: 716
VERBOSE: Using cipher list: HIGH:!eNULL:!aNULL
VERBOSE: Failed to establish SSL connection
ERROR: Failed to establish connection: TCP: GENERIC_ERROR (Timeout connecting to arc-noir.cta.cscs.ch(IPv4):8443 - 20 s)
VERBOSE: esainfo:GetActivityInfo request to https://arc-noir.cta.cscs.ch:8443/arex failed with response: Failed to send SOAP message: TCP: GENERIC_ERROR (Timeout connecting to arc-noir.cta.cscs.ch(IPv4):8443 - 20 s): TCP: GENERIC_ERROR (Timeout connecting to arc-noir.cta.cscs.ch(IPv4):8443 - 20 s)
2022-06-28 14:09:40 UTC WorkloadManagement/SiteDirectorARCCSCS/arc-noir.cta.cscs.ch DEBUG: Retrieving status for job [https://arc-noir.cta.cscs.ch:443/arex/rest/1.0/jobs/CCHNDmt4QQ1nLZfWYmas2NHqEbqlNmABFKDmMEFKDmRBFKDmNUVa4m](https://arc-noir.cta.cscs.ch/arex/rest/1.0/jobs/CCHNDmt4QQ1nLZfWYmas2NHqEbqlNmABFKDmMEFKDmRBFKDmNUVa4m)
2022-06-28 14:09:40 UTC WorkloadManagement/SiteDirectorARCCSCS/arc-noir.cta.cscs.ch DEBUG: ARC status for job [https://arc-noir.cta.cscs.ch:443/arex/rest/1.0/jobs/CCHNDmt4QQ1nLZfWYmas2NHqEbqlNmABFKDmMEFKDmRBFKDmNUVa4m](https://arc-noir.cta.cscs.ch/arex/rest/1.0/jobs/CCHNDmt4QQ1nLZfWYmas2NHqEbqlNmABFKDmMEFKDmRBFKDmNUVa4m) is Undefined
2022-06-28 14:09:40 UTC WorkloadManagement/SiteDirectorARCCSCS/arc-noir.cta.cscs.ch DEBUG: DIRAC status for job [https://arc-noir.cta.cscs.ch:443/arex/rest/1.0/jobs/CCHNDmt4QQ1nLZfWYmas2NHqEbqlNmABFKDmMEFKDmRBFKDmNUVa4m](https://arc-noir.cta.cscs.ch/arex/rest/1.0/jobs/CCHNDmt4QQ1nLZfWYmas2NHqEbqlNmABFKDmMEFKDmRBFKDmNUVa4m) is Unknown
2022-06-28 14:09:40 UTC WorkloadManagement/SiteDirectorARCCSCS INFO: Updating status to Unknown for pilot [https://arc-noir.cta.cscs.ch:443/arex/rest/1.0/jobs/CCHNDmt4QQ1nLZfWYmas2NHqEbqlNmABFKDmMEFKDmRBFKDmNUVa4m](https://arc-noir.cta.cscs.ch/arex/rest/1.0/jobs/CCHNDmt4QQ1nLZfWYmas2NHqEbqlNmABFKDmMEFKDmRBFKDmNUVa4m)

which seems to indicate a connection problem but I'm not sure this is the cause.

Also site admins indicate to use:

"The old LDAP method is still open, and although we’d like to phase it out eventually, you can use it to query the CE (the glue interface available thru HTTP doesn’t show this info for everyone):

ldapsearch -LLL -x -h arc-noir.cta.cscs.ch:2135 -b 'nordugrid-job-globalid=gsiftp://arc-noir.cta.cscs.ch:2811/jobs/geuMDmFpPQ1nLZfWYmas2NHqEbqlNmABFKDmMEFKDmOBFKDm1e2xwn,nordugrid-info-group-name=jobs,nordugrid-queue-name=normal,nordugrid-cluster-name=arc-noir.cta.cscs.ch,Mds-Vo-name=local,o=grid' "

but I'm not sure about what should be the correct fix for that.

2 replies

aldbr Jun 28, 2022
Collaborator

arcstat does not work for me

Have you submitted the job with arcsub? Else if it was submitted with the ARC6 CE you need to add the job id to a specific file, but I don't remember which one now.

VERBOSE: Trying to connect arc-noir.cta.cscs.ch(IPv4):8443

Oops indeed, I found the issue. Again a private method that should have been protected (getARCJob).
I'm going to push a fix now, sorry about that.
Just replace __getARCJob with _getARCJob in ARC and ARC6, this should resolve this issue.

At least it is a great way to spot the issues of ARC6 😅

aldbr Jun 28, 2022
Collaborator

The PR: #6216

arrabito · 2022-06-28T17:32:11Z

arrabito
Jun 28, 2022
Author

Thanks a lot with these changes it works better, but there is still some strange behavior.

Now Pilot Status get "Done" and payloads are Matched but in the Pilot Monitor there is no "CurrentJobID" so probably some further fix is needed.

Thanks in advance for your help.

2 replies

aldbr Jun 29, 2022
Collaborator

Indeed, this is a problem I tried to fix here: DIRACGrid/Pilot#159
But this does not work properly because the pilotRef is built differently when the REST interface is used (missing rest/1.0/jobs).
I could fix it in the Pilot and just add the missing part of the pilotRef but this is probably a bad fix: what if some ARC instances use rest/1.0 and some others use rest/1.1 for instance (so far there is no way to know which version of the rest interface is used from the pilot-jobs, even though there is only one version currently).

I have to check whether ARC is able to deal with URL when rest/1.0/jobs is not there.
If it's the case, then I can simply delete this part from the generated pilotRef.
Else, we will go for the fix in the Pilot mentioned above.
I will keep you informed.

Thanks

arrabito Jun 29, 2022
Author

Thanks a lot for these explanations and for keeping me informed.

VladimirRomanovsky · 2022-06-29T08:24:34Z

VladimirRomanovsky
Jun 29, 2022

Be ready to have other issue.
Site director will stuck during finalization.
You have to kill process to finish SiteDirector.
It is known bug in ARC client and Alexandre promised to implement workaround for it (_exit).

1 reply

arrabito Jun 29, 2022
Author

Yes thank you, I've already noticed that...

aldbr · 2022-07-01T08:43:36Z

aldbr
Jul 1, 2022
Collaborator

Here is a new PR for correctly binding ARC6 pilot-jobs with jobs: #6230

It is known bug in ARC client and Alexandre promised to implement workaround for it (_exit).

I added a fix in a previous PR, adding the following line should resolve the issue: https://github.com/DIRACGrid/DIRAC/blob/integration/src/DIRAC/Core/scripts/dirac_agent.py#L41-L43

0 replies

arrabito · 2022-07-01T09:51:19Z

arrabito
Jul 1, 2022
Author

Thank you.
I confirm that both fixes work fine.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Not able to submit to ARC CE using REST (CSCS site) #6210

{{title}}

Replies: 12 comments 14 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Not able to submit to ARC CE using REST (CSCS site) #6210

arrabito Jun 27, 2022

Replies: 12 comments · 14 replies

marianne013 Jun 27, 2022

arrabito Jun 28, 2022 Author

aldbr Jun 28, 2022 Collaborator

arrabito Jun 28, 2022 Author

aldbr Jun 28, 2022 Collaborator

marianne013 Jun 28, 2022

aldbr Jun 28, 2022 Collaborator

arrabito Jun 28, 2022 Author

aldbr Jun 28, 2022 Collaborator

arrabito Jun 28, 2022 Author

aldbr Jun 28, 2022 Collaborator

arrabito Jun 28, 2022 Author

aldbr Jun 28, 2022 Collaborator

arrabito Jun 28, 2022 Author

arrabito Jun 28, 2022 Author

aldbr Jun 28, 2022 Collaborator

arrabito Jun 28, 2022 Author

aldbr Jun 28, 2022 Collaborator

aldbr Jun 28, 2022 Collaborator

arrabito Jun 28, 2022 Author

aldbr Jun 29, 2022 Collaborator

arrabito Jun 29, 2022 Author

VladimirRomanovsky Jun 29, 2022

arrabito Jun 29, 2022 Author

aldbr Jul 1, 2022 Collaborator

arrabito Jul 1, 2022 Author

arrabito
Jun 27, 2022

Replies: 12 comments 14 replies

marianne013
Jun 27, 2022

arrabito Jun 28, 2022
Author

aldbr
Jun 28, 2022
Collaborator

arrabito
Jun 28, 2022
Author

aldbr Jun 28, 2022
Collaborator

aldbr Jun 28, 2022
Collaborator

arrabito
Jun 28, 2022
Author

aldbr Jun 28, 2022
Collaborator

arrabito
Jun 28, 2022
Author

aldbr Jun 28, 2022
Collaborator

arrabito Jun 28, 2022
Author

aldbr
Jun 28, 2022
Collaborator

arrabito Jun 28, 2022
Author

arrabito
Jun 28, 2022
Author

aldbr Jun 28, 2022
Collaborator

arrabito
Jun 28, 2022
Author

aldbr Jun 28, 2022
Collaborator

aldbr Jun 28, 2022
Collaborator

arrabito
Jun 28, 2022
Author

aldbr Jun 29, 2022
Collaborator

arrabito Jun 29, 2022
Author

VladimirRomanovsky
Jun 29, 2022

arrabito Jun 29, 2022
Author

aldbr
Jul 1, 2022
Collaborator

arrabito
Jul 1, 2022
Author