Support for stored URL in the FileCatalog

[chaen]

Historically, the FileCatalog (whether it was the LFC, DFC, anything else) was storing for a given LFN a URL along side all the other information.

The model later changed because we started having multiple protocols (so a single URL does not make sense anymore), but also because the URL became deterministic (proto + host + base path + lfn). These changes are from v6r12 if my memory serves me well.

When these changes were introduced, great care was taken to still support the old model, and always propagate a URL from the catalog just in case some community would need it (as far s I know, we were not even certain it was needed).

I would like to remove this compatibility layer.

One reason is that in the view of the migration of the DMS to diracx, it will make the transition a lot smoother.
The other reason is that carrying around all these fake URLs has a non negligible performance cost (network, memory etc).

I would like the opinion of
@atsareg : ❓
@marianne013 : ✅
@andresailer : ✅
@iueda: ❓
@arrabito : ❓
@zhangxiaomei : ❓

[andresailer]

Is this URL the PFN stored here?

DIRAC/src/DIRAC/DataManagementSystem/DB/FileCatalogDB.sql

Line 88 in 83e0ec6

PFN VARCHAR(1024),

I am not consciously aware this this is needed for us.

[chaen]

Yes it is. Thanks for your reply

[marianne013]

I had a quick chat with @sfayer and the only place we use it is in some quick and dirty data base hacks (it comes in really handy if you need to generate a list of files on a certain SE), so we can live without it.

[chaen]

Cool thanks. I am not sure how you use that, but I am happy to discuss that use case separately !

[zhangxiaomei]

In JUNO, the jobs with multiple input files are commonly required. Instead of downloading data from WNs, we use dirac-dms-lfn-accessURL to get xrootd url of these files to enable the application read files with xrootd protocol directly. Is the "dirac-dms-lfn-accessURL" command related to what you described? Without direct PFN info, should we maintain one ourselves outside dirac?

[iueda]

It seems to depend on your DIRAC configuration for the StorageElements.

If you have such a parameter set to True, then you rely on this feature (SURLs stored in DB)

/Resources/StorageElements/{self.name}/UseCatalogURL

https://github.com/DIRACGrid/DIRAC/blob/integration/src/DIRAC/Resources/Storage/StorageElement.py#L233

otherwise, SURLs are composed with the SE definition and the LFNs

[zhangxiaomei]

In that case, JUNO is only using a simple SURLs with "SE+LFNs" so far.

[iueda]

I only positively support the idea. I will reconfirm this with my colleagues and will come back to you.

At least I see nowhere the flag UseCatalogURL set in our production DIRAC CS.

Back in the days with LFC, and relying on the PFNs stored in the DB, migration of files at sites (eg. from an old SE to a new SE within the site) caused non-negligible efforts for replacing SURLs in LFC.

With DIRAC and Rucio composing SURLs deterministically, based on the SE endpoint name and the LFN,
such migration is much simpler now -- we just change the parameters (hostname, base path, protocol, ...) in
the SE endpoint definition.

[iueda]

I only positively support the idea. I will reconfirm this with my colleagues and will come back to you.

Now I can tell that Belle II has no problem without PFN/SURL stored in FileCatalog.
(Confirmed together with @cserf)

NOTE: Rucio itself has a feature to work with non-deterministic SURLs, that can be defined SE-by-SE.
It was necessary when raw data files were stored onto TAPE by DAQ/HLT, before their LFNs could be defined, i.e. matching LFNs to PFNs was difficult/impossible.

[atsareg]

This feature of the catalogue was there since the beginning when the LFN <-> PFN convention was a novelty. The PFN construction was seen rather as a convenience feature. Now it is largely accepted as a strict rule. So, the need for a possibility to store PFNs is actually reduced if at all. We had some cases in the past where existing data was imported into the DIRAC DMS and registered in the catalog in a different directory structure than in the physical storage. This is certainly not a general case and can be rather addressed by renaming files on the physical storage (one-time operation). There were also cases where a community decided to change the structure of their logical namespace for some good reasons. In this case the stored PFNs can be handy in order not to affect physical storage (Xiaomei can probably comment for the JUNO's case).
So, there can be some (temporary) occasions where this feature can be useful. On the other hand, this certainly affects the efficiency and adds complexity. Since we have to cope with more and more entries in the catalogues, the latter is more important. I think the proposal to scrap it makes sense. I would consider to use the plug-in mechanism as we did for other things as well for this feature to be possibly included if needed. If not needed, it will not be installed and will not affect the performance.

[zhangxiaomei]

In JUNO, the mapping from LFN to PFN is straightforward (SE+LFNs), not complicated. Just from user point of view, they need a command to access data with xrootd protocol. Therefore, for us it is fine without PFN.

[arrabito]

For CTAO, we don't use this feature at all, so we are totally fine with removing it.