`gfal-bringonline` ignores `--pin-lifetime`

[onnozweers]

Dear Gfal developers,

It seems that the gfal-bringonline command ignores the provided pin lifetime when using a WebDAV SURL. The files are staged and pinned, but with the default pin lifetime configured in the dCache Bulk service (2 days) instead of whatever you provide on the gfal-bringonline command line with --pin-lifetime.

When we stage with Ada (the dCache API client), staging succeeds with the specified pin lifetime. So this makes it unlikely that the bug is in dCache.

We've tested it on the dCache at SURF and the dCache at FZ Juelich. We see this behaviour for the Lofar radio telescope project, but I remember that I've seen similar behaviour for one of the CERN experiments (Atlas IIRC).

Here's a debug run of gfal-bringonline:

11:27 loui.grid.surfsara.nl:/home/onno/git/SpiderScripts 
onno$ gfal-bringonline --pin-lifetime $((3600 * 24 * 10)) https://webdav.grid.surfsara.nl:2882/pnfs/grid.sara.nl/data/lofar/ops/projects/lc13_030/770859/L770859_SB180_uv.MS_fbe6d2c2.tar -vvv
DEBUG     no GFAL_CONFIG_DIR env var found, try to load configuration from default directory /etc/gfal2.d/
DEBUG     try to load configuration file /etc/gfal2.d//bdii.conf ...
DEBUG     try to load configuration file /etc/gfal2.d//gfal2_core.conf ...
DEBUG     try to load configuration file /etc/gfal2.d//x509.conf ...
DEBUG     try to load configuration file /etc/gfal2.d//gsiftp_plugin.conf ...
DEBUG     try to load configuration file /etc/gfal2.d//srm_plugin.conf ...
DEBUG     try to load configuration file /etc/gfal2.d//dcap_plugin.conf ...
DEBUG     try to load configuration file /etc/gfal2.d//sftp_plugin.conf ...
DEBUG     try to load configuration file /etc/gfal2.d//xrootd_plugin.conf ...
DEBUG     try to load configuration file /etc/gfal2.d//http_plugin.conf ...
DEBUG     try to load configuration file /etc/gfal2.d//mock_plugin.conf ...
DEBUG    Using credentials from X509_USER_PROXY
DEBUG    Certificate: /tmp/x509up_u31029
DEBUG    Private key: /tmp/x509up_u31029
DEBUG    ... no GFAL_PLUGIN_DIR environment variable specified, try to load plugins in the default directory : /usr/lib64/gfal2-plugins/
DEBUG     [gfal_list_directory_plugins] add plugin to list to load /usr/lib64/gfal2-plugins//libgfal_plugin_file.so 
DEBUG     [gfal_list_directory_plugins] add plugin to list to load /usr/lib64/gfal2-plugins//libgfal_plugin_gridftp.so 
DEBUG     [gfal_list_directory_plugins] add plugin to list to load /usr/lib64/gfal2-plugins//libgfal_plugin_srm.so 
DEBUG     [gfal_list_directory_plugins] add plugin to list to load /usr/lib64/gfal2-plugins//libgfal_plugin_dcap.so 
DEBUG     [gfal_list_directory_plugins] add plugin to list to load /usr/lib64/gfal2-plugins//libgfal_plugin_sftp.so 
DEBUG     [gfal_list_directory_plugins] add plugin to list to load /usr/lib64/gfal2-plugins//libgfal_plugin_xrootd.so 
DEBUG     [gfal_list_directory_plugins] add plugin to list to load /usr/lib64/gfal2-plugins//libgfal_plugin_http.so 
DEBUG     [gfal_list_directory_plugins] add plugin to list to load /usr/lib64/gfal2-plugins//libgfal_plugin_mock.so 
INFO     [gfal_module_load] plugin /usr/lib64/gfal2-plugins//libgfal_plugin_file.so loaded with success 
DEBUG     gfal_plugin loaded successfully : /usr/lib64/gfal2-plugins//libgfal_plugin_file.so
DEBUG     -> [gridftp_plugin] try to load ..
DEBUG     define GSIFTP session re-use to TRUE
DEBUG     -> [gridftp_plugin] loaded ..
INFO     [gfal_module_load] plugin /usr/lib64/gfal2-plugins//libgfal_plugin_gridftp.so loaded with success 
DEBUG     gfal_plugin loaded successfully : /usr/lib64/gfal2-plugins//libgfal_plugin_gridftp.so
INFO     [gfal_module_load] plugin /usr/lib64/gfal2-plugins//libgfal_plugin_srm.so loaded with success 
DEBUG     gfal_plugin loaded successfully : /usr/lib64/gfal2-plugins//libgfal_plugin_srm.so
INFO     [gfal_module_load] plugin /usr/lib64/gfal2-plugins//libgfal_plugin_dcap.so loaded with success 
DEBUG     gfal_plugin loaded successfully : /usr/lib64/gfal2-plugins//libgfal_plugin_dcap.so
INFO     [gfal_module_load] plugin /usr/lib64/gfal2-plugins//libgfal_plugin_sftp.so loaded with success 
DEBUG     gfal_plugin loaded successfully : /usr/lib64/gfal2-plugins//libgfal_plugin_sftp.so
INFO     [gfal_module_load] plugin /usr/lib64/gfal2-plugins//libgfal_plugin_xrootd.so loaded with success 
DEBUG     gfal_plugin loaded successfully : /usr/lib64/gfal2-plugins//libgfal_plugin_xrootd.so
DEBUG    Davix: Enable GRID profile for DAVIX
DEBUG    Davix: Adding CA path /etc/grid-security/certificates/ to valid CA path list
DEBUG    Davix: Using X509_USER_PROXY to supply credentials: /tmp/x509up_u31029
DEBUG    Davix: HTTP/SSL Session caching ENABLED
DEBUG    Davix: Redirection Session caching ENABLED
DEBUG    Davix: libdavix path /lib64/libdavix.so.0, version: 0.8.7
DEBUG    Davix: Enable GRID profile for DAVIX
DEBUG    Davix: Adding CA path /etc/grid-security/certificates/ to valid CA path list
DEBUG    Davix: Using X509_USER_PROXY to supply credentials: /tmp/x509up_u31029
INFO     [gfal_module_load] plugin /usr/lib64/gfal2-plugins//libgfal_plugin_http.so loaded with success 
DEBUG     gfal_plugin loaded successfully : /usr/lib64/gfal2-plugins//libgfal_plugin_http.so
INFO     [gfal_module_load] plugin /usr/lib64/gfal2-plugins//libgfal_plugin_mock.so loaded with success 
DEBUG     gfal_plugin loaded successfully : /usr/lib64/gfal2-plugins//libgfal_plugin_mock.so
DEBUG     plugin priority order: file-2.23.0 -> gridftp-2.23.0 -> srm-2.23.0 -> dcap-2.23.0 -> sftp-2.23.0 -> xrootd-2.23.0 -> http-2.23.0 -> mock-2.23.0 -> 
DEBUG    Davix: Create HttpRequest for https://webdav.grid.surfsara.nl:2882/.well-known/wlcg-tape-rest-api
DEBUG    Impossible to get string_list parameter HTTP:WEBDAV.GRID.SURFSARA.NL:HEADERS, set to a default value (null), err Key file does not have group “HTTP:WEBDAV.GRID.SURFSARA.NL”
DEBUG    Impossible to get string_list parameter HTTP PLUGIN:HEADERS, set to a default value (null), err Key file does not have key “HEADERS” in group “HTTP PLUGIN”
DEBUG    Impossible to get integer parameter HTTP PLUGIN:OPERATION_TIMEOUT, set to default value 1800, err Key file does not have key “OPERATION_TIMEOUT” in group “HTTP PLUGIN”
DEBUG    Using client X509 for HTTPS session authorization
DEBUG    Impossible to get string parameter BEARER:TOKEN, set to default value (null), err Key file does not have group “BEARER”
DEBUG    Davix:  -> executeRequest
DEBUG    Davix:  -> negotiateRequest
DEBUG    Davix: NEON start internal request
DEBUG    Davix: no cached ne_session, create a new one 
DEBUG    Davix: HTTP session to https://webdav.grid.surfsara.nl:2882 begins.
DEBUG    Davix: configure session...
DEBUG    Davix: define connection timeout to 30
DEBUG    Davix: define operation timeout to 1800
DEBUG    Davix: add CA PATH /etc/grid-security/certificates/
DEBUG    Davix: disable login/password authentication
DEBUG    Davix: enable client cert authentication by callback 
DEBUG    Davix: Running pre_send hooks
INFO     Davix: > GET /.well-known/wlcg-tape-rest-api HTTP/1.1
> User-Agent: gfal2-util/1.9.0 gfal2/2.23.0 neon/0.0.29
> Keep-Alive: 
> Connection: Keep-Alive
> TE: trailers
> Host: webdav.grid.surfsara.nl:2882
> 

DEBUG    Davix: Sending request-line and headers:
DEBUG    Davix: Doing DNS lookup on webdav.grid.surfsara.nl...
DEBUG    Davix: clicert callback 
DEBUG    Davix: call client cert callback 
DEBUG    Davix: Request sent; retry is 0.
INFO     Davix: < HTTP/1.1 200 OK
INFO     Davix: < Date: Wed, 19 Nov 2025 10:27:46 GMT
INFO     Davix: < Server: dCache/10.2.16
INFO     Davix: < Content-Type: application/json;charset=utf-8
INFO     Davix: < Transfer-Encoding: chunked
INFO     Davix: < 
DEBUG    Davix: End of headers.
DEBUG    Davix: Running post_headers hooks
DEBUG    Davix:  <- negotiateRequest
DEBUG    Davix: Bad server answer: Content-Length Invalid, impossible to determine answer size
DEBUG    Davix: NEON Read data flow
DEBUG    Davix: [chunk] < 123

DEBUG    Davix: Got chunk size: 291
DEBUG    Davix: Reading 291 bytes of response body.

DEBUG    Davix: Got 291 bytes.
DEBUG    Davix: StandaloneNeonRequestNeonRequest::readBlock read 291 bytes
DEBUG    Davix: NEON Read data flow
DEBUG    Davix: [chunk] < 0

DEBUG    Davix: Got chunk size: 0
DEBUG    Davix: StandaloneNeonRequestNeonRequest::readBlock read 0 bytes
INFO     Davix: < 
DEBUG    Davix: End of headers.
DEBUG    Davix: Running post_send hooks
DEBUG    Davix:  <- executeRequest
DEBUG    Davix: Destroy HttpRequest
DEBUG    Davix: Running destroy hooks.
DEBUG    Davix: Request ends.
DEBUG    Davix: add old session to cache httpswebdav.grid.surfsara.nl:2882
DEBUG    Davix: Create HttpRequest for https://dcacheview.grid.surfsara.nl:22882/api/v1/tape/stage
DEBUG    Impossible to get string_list parameter HTTP:DCACHEVIEW.GRID.SURFSARA.NL:HEADERS, set to a default value (null), err Key file does not have group “HTTP:DCACHEVIEW.GRID.SURFSARA.NL”
DEBUG    Impossible to get string_list parameter HTTP PLUGIN:HEADERS, set to a default value (null), err Key file does not have key “HEADERS” in group “HTTP PLUGIN”
DEBUG    Impossible to get integer parameter HTTP PLUGIN:OPERATION_TIMEOUT, set to default value 1800, err Key file does not have key “OPERATION_TIMEOUT” in group “HTTP PLUGIN”
DEBUG    Using client X509 for HTTPS session authorization
DEBUG    Impossible to get string parameter BEARER:TOKEN, set to default value (null), err Key file does not have group “BEARER”
DEBUG    Davix:  -> executeRequest
DEBUG    Davix:  -> negotiateRequest
DEBUG    Davix: NEON start internal request
DEBUG    Davix: no cached ne_session, create a new one 
DEBUG    Davix: HTTP session to https://dcacheview.grid.surfsara.nl:22882 begins.
DEBUG    Davix: configure session...
DEBUG    Davix: define connection timeout to 30
DEBUG    Davix: define operation timeout to 1800
DEBUG    Davix: add CA PATH /etc/grid-security/certificates/
DEBUG    Davix: disable login/password authentication
DEBUG    Davix: enable client cert authentication by callback 
DEBUG    Davix: Running pre_send hooks
INFO     Davix: > POST /api/v1/tape/stage HTTP/1.1
> User-Agent: gfal2-util/1.9.0 gfal2/2.23.0 neon/0.0.29
> Keep-Alive: 
> Connection: Keep-Alive
> TE: trailers
> Host: dcacheview.grid.surfsara.nl:22882
> Content-Type: application/json
> Content-Length: 116
> 

DEBUG    Davix: Sending request-line and headers:
DEBUG    Davix: Doing DNS lookup on dcacheview.grid.surfsara.nl...
DEBUG    Davix: clicert callback 
DEBUG    Davix: call client cert callback 
DEBUG    Davix: Sending request body:
DEBUG    Davix: Request body sent successfully
DEBUG    Davix: Request sent; retry is 0.
INFO     Davix: < HTTP/1.1 201 Created
INFO     Davix: < Date: Wed, 19 Nov 2025 10:27:46 GMT
INFO     Davix: < Server: dCache/10.2.16
INFO     Davix: < Location: https://dcacheview.grid.surfsara.nl:22882/api/v1/tape/stage/f01c6709-f240-4975-85e5-39fd9cd477ac
INFO     Davix: < Content-Type: application/json
INFO     Davix: < Content-Length: 58
INFO     Davix: < 
DEBUG    Davix: End of headers.
DEBUG    Davix: Running post_headers hooks
DEBUG    Davix:  <- negotiateRequest
DEBUG    Davix: NEON Read data flow
DEBUG    Davix: Reading 58 bytes of response body.

DEBUG    Davix: Got 58 bytes.
DEBUG    Davix: StandaloneNeonRequestNeonRequest::readBlock read 58 bytes
DEBUG    Davix: NEON Read data flow
DEBUG    Davix: StandaloneNeonRequestNeonRequest::readBlock read 0 bytes
DEBUG    Davix: Running post_send hooks
DEBUG    Davix:  <- executeRequest
DEBUG    Davix: Destroy HttpRequest
DEBUG    Davix: Running destroy hooks.
DEBUG    Davix: Request ends.
DEBUG    Davix: add old session to cache httpsdcacheview.grid.surfsara.nl:22882
Bringonline token: f01c6709-f240-4975-85e5-39fd9cd477ac
https://webdav.grid.surfsara.nl:2882/pnfs/grid.sara.nl/data/lofar/ops/projects/lc13_030/770859/L770859_SB180_uv.MS_fbe6d2c2.tar QUEUED
DEBUG    gridftp session cache garbage collection ...
DEBUG    Davix: sess: Destroying session.
DEBUG    Davix: sess: Closing connection.
DEBUG    Davix: sess: Connection closed.
DEBUG    Davix: sess: Destroying session.
DEBUG    Davix: sess: Closing connection.
DEBUG    Davix: sess: Connection closed.

Below is an Ada command, talking directly to the dCache API, which succeeds to set the desired pin lifetime. This debug output contains the exact curl commands that Ada executes towards the dCache API. I hope this will help to understand the issue.

10:25 loui.grid.surfsara.nl:/home/onno/git/SpiderScripts 
onno$ ada/ada --proxy --api https://dcacheview.grid.surfsara.nl:22882/api/v1 --stage /pnfs/grid.sara.nl/data/lofar/ops/projects/lc13_030/770859/L770859_SB180_uv.MS_fbe6d2c2.tar --lifetime 14D --debug
+++ curl --capath /etc/grid-security/certificates --cert /tmp/x509up_u31029 --cacert /tmp/x509up_u31029 -H 'accept: application/json' --fail --silent --show-error --ipv4 -X GET https://dcacheview.grid.surfsara.nl:22882/api/v1/user
+++ jq -r .status
Authenticated.
urlencoding '/pnfs/grid.sara.nl/data/lofar/ops/projects/lc13_030/770859/L770859_SB180_uv.MS_fbe6d2c2.tar' to '%2Fpnfs%2Fgrid.sara.nl%2Fdata%2Flofar%2Fops%2Fprojects%2Flc13_030%2F770859%2FL770859_SB180_uv.MS_fbe6d2c2.tar'
++++ curl --capath /etc/grid-security/certificates --cert /tmp/x509up_u31029 --cacert /tmp/x509up_u31029 -H 'accept: application/json' --silent -X GET https://dcacheview.grid.surfsara.nl:22882/api/v1/namespace/%2Fpnfs%2Fgrid.sara.nl%2Fdata%2Flofar%2Fops%2Fprojects%2Flc13_030%2F770859%2FL770859_SB180_uv.MS_fbe6d2c2.tar
++ curl --capath /etc/grid-security/certificates --cert /tmp/x509up_u31029 --cacert /tmp/x509up_u31029 -H 'accept: application/json' --fail --silent --show-error --ipv4 -H 'content-type: application/json' -X POST https://dcacheview.grid.surfsara.nl:22882/api/v1/bulk-requests -d '{"activity": "PIN", "arguments": {"lifetime": "14", "lifetimeUnit":"DAYS"}, "target": ["//pnfs/grid.sara.nl/data/lofar/ops/projects/lc13_030/770859/L770859_SB180_uv.MS_fbe6d2c2.tar"], "expand_directories": "NONE"}' --write-out '\nHTTP_CODE_%{http_code}' --dump-header -
+ response='HTTP/1.1 201 Created
Date: Wed, 19 Nov 2025 09:27:04 GMT
Server: dCache/10.2.16
request-url: https://dcacheview.grid.surfsara.nl:22882/api/v1/bulk-requests/6adc8816-04f0-455d-a931-1a0fd095710d
Content-Type: application/json
Content-Length: 0


HTTP_CODE_201'
+ true
+ set +x
Returned HTTP status: HTTP_CODE_201
Information about bulk request is logged in /home/onno/.ada/requests.log
To check status of request, paste request URL in browser:
   request-url: https://dcacheview.grid.surfsara.nl:22882/api/v1/bulk-requests/6adc8816-04f0-455d-a931-1a0fd095710d
Or use command:
   ada/ada --stat-request 6adc8816-04f0-455d-a931-1a0fd095710d --api https://dcacheview.grid.surfsara.nl:22882/api/v1 --proxy 

Here's the dCache default setting for pins, used when it does not receive a pin lifetime in the request:
https://github.com/dCache/dcache/blob/c483a3ce65bddb5d48eeb3376467ef16e0a3f297/skel/share/defaults/bulk.properties#L241

Let me know if you need more information.

Kind regards,
Onno

[onnozweers]

I had a quick look at the source. It looks like the pin lifetime is passed on to this method as disk_lifetime:

gfal2/src/plugins/http/gfal_http_plugin_tape.cpp

Line 65 in e301920

std::string stage_request_body(int disk_lifetime, int nbfiles, const char *const *urls, const char *const *metadata) {

But I don't see it being used in that method. Also, I'm wondering if the type casting is consistent (int in one place and time_t in another).

But I'm a humble sysadmin and not a developer, so please correct me if I'm wrong.

[onnozweers]

I tested how the pin lifetime may be added to the /tape/stage request:

{
  "files": [
    {
      "diskLifetime": "P14D",
      "path": "/pnfs/grid.sara.nl/data/users/onno/tape/test"
    }
  ]
}

The diskLifetime is in ISO 8601 format.

I hope this helps.

[UPDATE: I had first posted the /bulk-requests syntax because Ada uses that, but Gfal2 uses the /tape API call so I looked up the syntax for that call.]

[mpatrascoiu]

Hello Onno,

You are right, the Gfal2 pin-lifetime over WebDav protocol is simply ignored altogether.
It's just not there in the HTTP implementation.

Please keep in mind we have stopped Gfal2 development and we're working on providing a replacement for the Grid world. For the moment, we only perform bug fixes.

At this point, best course of action is to leave this ticket open, and even better, formulate a request on the cern-fts/gfal2 project for this capability. I will raise this issue internally as well.

Cheers,
Mihai

[onnozweers]

Hi Mihai,

Thanks for your comment. I hope this issue can be seen as a bug. I will leave this issue open and await the result of the discussion.

Just out of curiosity, what will replace Gfal2? Do you have any more details?

Kind regards,
Onno

[mpatrascoiu]

Hello,

This was presented at the FTS & XRootd Workshop 2025:
Evolution of DMC Clients

Hope the above presentation will provide more details.

In short, there will be a common CLI keeping (most of) the same functionality, but the underlying library will move to something else, likely XRootd library.

Cheers,
Mihai