Interacting with File Catalog via HTTP

[seriksen]

Hello,

Is there somewhere which has a description of how to interact with the File Catalog via HTTP?
The only example seems to be for a whoami such as below.

import requests

def _query(url, params):
    capath="/etc/grid-security/certificates"
    cert="/tmp/x509up_u1000"
    with requests.post(url, data=params, cert=cert, verify=capath) as r:
        return r.text

def whoami(server):
    url=f"https://{server}:8444/DataManagement/FileCatalog"
    params = {
        "method": "whoami"
    }
    return _query(url, params)

if __name__ == "__main__":
   #server = "lbcertifdirac70.cern.ch"
    server = "diracdev.grid.hep.ph.ic.ac.uk"
    result = whoami(server)
    print(result)

Any guidance would be very much appreciated, particularly on what methods are supported.

[fstagni]

All existing methods are supported (basically, all symbols starting with export_ of https://github.com/DIRACGrid/DIRAC/blob/integration/src/DIRAC/DataManagementSystem/Service/FileCatalogHandler.py)

For example:

def get_users(server):
  url=f"https://{server}:8444/DataManagement/FileCatalog"
  params = {
      "method": "getUsers"
  }
  return _query(url, params)

def get_file_metadata(server, lfns):
    url=f"https://{server}:8444/DataManagement/FileCatalog"
    params = {
        "method": "getFileMetadata",
        "args": json.dumps([lfns])
    }
    return _query(url, params)

[fstagni]

We have some basic topic doc in https://dirac.readthedocs.io/en/latest/AdministratorGuide/Systems/DataManagement/concepts.html#dms-concepts, would that satisfy your needs?

[seriksen]

that has helped thanks, but I'm still struggling.

I can register a file, but the file isn't actually being uploaded

import json
import os
import zlib
import requests

url = "https://diracdev.grid.hep.ph.ic.ac.uk:8444"
capath = "/users/ak18773/SWIFT_HEP/dev_dirac/diracos/etc/grid-security/certificates"
cert = "/tmp/x509up_u397871"
endpoint = "DataManagement/FileCatalog"
query_url = f"{url}/{endpoint}"
local_file = "/users/ak18773/SWIFT_HEP/dask-dirac/test.py"
remote_file = "/gridpp/user/s/seriksen/test_addFile.py"

with open(local_file, 'rb') as file:
    lfns = {remote_file: {"PFN": local_file,
                          "SE": "UKI-SOUTHGRID-RALPP-disk",
                          "Size": os.path.getsize(local_file),
                          "Checksum": zlib.adler32(file.read())}}

with open(local_file, 'rb') as file:
    file_data = {"file": (file.name, file, "application/octet-stream")}

    print(json.dumps([lfns]))
    print(file_data)
    params = {"method": "addFile", "args": json.dumps([lfns])}
    params["clientSetup"] = params.get("clientSetup", "GridPP")
    result = requests.post(query_url, data=params, files=file_data, cert=cert, verify=capath)
    print(result.json())

Do you know if the file_data needs to be in a different form for this?

[fstagni]

The FileCatalog service is just a service for the FileCatalog, meaning that it will only register LFN->PFN relations and metadata, they will not actually upload files to the destination. For that, there are higher-level interfaces, like the already mentioned DataManager module and specifically the putAndRegister method: https://github.com/DIRACGrid/DIRAC/blob/integration/src/DIRAC/DataManagementSystem/Client/DataManager.py#L424

[seriksen]

Thanks, that's cleared some things up for me.

From my understanding, the way that I would go about doing this all via HTTP would be;

1. Upload a file directory to the storage element (eg to https://mover.pp.rl.ac.uk:2880/pnfs/pp.rl.ac.uk/data/{vo}, equivalent to gfal commands)
2. Register that file (using essentially the same code posted above)

[fstagni]

That would do it, but can ask you what's your goal? Because basically you are creating a procedure outside of DIRAC which would replicate the same code.