Very long delays in getting FTS jobs updated

[arrabito]

Dear all,

I'm running several 'Moving' transformations using:

from DIRAC.TransformationSystem.Utilities.ReplicationTransformation import createDataTransformation

createDataTransformation(flavour='Moving', ...

and with FTS.

It's working fine, but there are several hours of delay before requests get updated once the FTS transfers are Finished.
And this slows done all the 'Moving' process.

I've also noticed that using dirac-rms-request, I get wrong FTS job status:

$ dirac-rms-request 4926820
Request name='00002637_00000144' ID=4926820 Status='Scheduled'
Created 2022-01-10 09:18:37, Updated 2022-01-10 09:27:35, NotBefore 2022-01-10 09:27:38
Owner: '/O=GRID-FR/C=FR/O=CNRS/OU=LUPM/CN=Luisa Arrabito', Group: cta_prod
  [0] Operation Type='ReplicateAndRegister' ID=22227476 Order=0 Status='Scheduled'

....
FTS3 jobs associated: 
https://fts3-pilot.cern.ch:8449/fts3/ftsmon/#/job/ca69fdd0-71f7-11ec-812f-fa163ec64cc5 (Submitted)

While in the FTS web monitor the transfer is Finished since several hours:

Job finished: 2022-01-10T09:46:20Z

Could this be the reason of the delays I observe?

Could you please tell me which component is responsible to get the updated FTS jobs status, so that I can check what's going on?

Thanks.

[arrabito]

I guess the update is done by the FTS3Agent, and indeed I found some error there:

DataManagement_FTS3Agent/log/current:2022-01-10 15:57:28 UTC DataManagement/FTS3Agent/_monitorJobCallback/1764280 ERROR: Error updating job status {'OK': False, 'Errno': 0, 'Message': 'Error getting the job status Try again: 503', 'CallStack': [' File "/opt/dirac/versions/v2.0.1-1635261124/Linux-x86_64/lib/python3.9/threading.py", line 930, in _bootstrap\n self._bootstrap_inner()\n', ' File "/opt/dirac/versions/v2.0.1-1635261124/Linux-x86_64/lib/python3.9/threading.py", line 973, in _bootstrap_inner\n self.run()\n', ' File "/opt/dirac/versions/v2.0.1-1635261124/Linux-x86_64/lib/python3.9/threading.py", line 910, in run\n self._target(*self._args, **self._kwargs)\n', ' File "/opt/dirac/versions/v2.0.1-1635261124/Linux-x86_64/lib/python3.9/multiprocessing/pool.py", line 125, in worker\n result = (True, func(*args, **kwds))\n', ' File "/opt/dirac/versions/v2.0.1-1635261124/Linux-x86_64/lib/python3.9/site-packages/DIRAC/DataManagementSystem/Agent/FTS3Agent.py", line 223, in _monitorJob\n res = ftsJob.monitor(context=context)\n', ' File "/opt/dirac/versions/v2.0.1-1635261124/Linux-x86_64/lib/python3.9/site-packages/DIRAC/DataManagementSystem/Client/FTS3Job.py", line 151, in monitor\n return S_ERROR("Error getting the job status %s" % e)\n']}

but after restarting the Agent I get:

2022-01-10 16:15:37 UTC DataManagement/FTS3Agent/monitorJobs INFO: 0 jobs to queue for monitoring

which seems to me that it does not attempt to update the status of any jobs.

Could you please suggest how to get the status update done?

Thanks.

[arrabito]

Just to let you know that in principle the problem was due to connection problems to the FTS server when querying the job status.