problems with replication transformation

[zhangxiaomei]

A transformation was created with the following settings:
t.setPlugin("broadcast")
t.setTargetSE("JINR-EOS")
t.setSourceSE("CNAF-STORM")
transBody = '' "

The transformation hasn't submitted the replications although the logs have no error reports. The transformation page has files found but all the files keep in "unused" status:

How to find the problem with this transformation? What problems could be?

[fstagni]

Hi,
you set up a Replication transformation, right? And you want to transfer from CNAF to JINR.

The first question is if there are "tasks" created for this transformation. "tasks" would be created by the TransformationAgent, so logs of it would be helpful.

If tasks are created, then they will need to be submitted, and it this case the agent involved is the RequestTaskAgent.

https://dirac.readthedocs.io/en/latest/AdministratorGuide/Systems/Transformation/index.html

[zhangxiaomei]

This transformation started to create tasks in two days. I looked into the TransformationAgent log to see why tasks-creating is so slow for this transformation and only found:
10861 2025-02-12 02:55:07 UTC Transformation/TransformationAgent/[7710] ._getTransformationFiles INFO: No new 'Unused,ProbInFC' files found for transformation.

Where is the bottleneck to slow down the transformationAgent to create tasks for this transformation? By the way, other transformations are fast at the same time.

The background is: we created two transfers from IHEP->CNAF, JINR daily, and the network IHEP-JINR is not working well, so that the transfer IHEP->CNAF->JINR is created instead. The transformation of CNAF->JINR has problems mentioned here.

[andresailer]

The replica information is cached for 24 hours, so when the files were first found only at IHEP, they were not considered for broadcasting from CNAF to JINR, but once the replica cache expired and replicas at CNAF were found the tasks were created?
You can "Flush" transformations to force a rebuild of the replica cache. See https://dirac.readthedocs.io/en/latest/AdministratorGuide/Systems/Transformation/index.html#actions-on-transformations

When a transformation is flushed also its replica cache will be re-created (instead of after 24 hours).

[zhangxiaomei]

Good suggestion! It did work after flushing. But it is also a pain if I have to do it daily by hand since it is created automatically. One question for the replicas transformation: Why the replicas cache is not based on the availability of files in CNAF instead of all files? If I want to deal only with the files in one of the sites under a directory, all the files under this directory are traversed.

[fstagni]

The replica cache is filled for all the files in input to the transformation. What do you have in the input data query?

[zhangxiaomei]

Input data query is normal:

Additional parameters:

It is better if the replica cache can take "SourceSE=CNAF-STORM“ into account and just fill the files with what SourceSE have.

[chaen]

Make a transformation with 2 consecutive operations. Examples

https://dirac.readthedocs.io/en/latest/AdministratorGuide/Systems/Transformation/index.html#data-management-transformations
https://dirac.readthedocs.io/en/latest/AdministratorGuide/Tutorials/dmsWithTs.html