Check for object existence prior to checksum calculation

[mstfdkmn]

Enforcing a policy to remove specific objects transferred through Globus leads to transfer failures when checksum calculation is enabled in the Globus client. This occurs because the Globus connector attempts to open the removed object for checksum calculation (used as metadata). This results in errors in irods logs and the Globus interface, and can block subsequent transfers.

Could the code at the relevant section https://github.com/irods/irods_client_globus_connector/blob/main/DSI/globus_gridftp_server_iRODS.cpp#L1646 be modified to first check if the object exists before attempting to open it?

[2024-12-19T12:37:45.018Z][icts-t-cloud-rdm-leu-2] {"processname": "irods", "groupname": "irods", "pid": "95", "channel": "stdout", "irods": {"log_category": "legacy", "log_level": "error", "log_message": "[rsDataObjOpen_impl:917] - [OBJ_PATH_DOES_NOT_EXIST: Data object or replica does not exist [error_code=-808000, path=/icts/home/u0137480/globus_test/test/alice.txt].\n\n] [error_code=[-358000], path=[/icts/home/u0137480/globus_test/test/alice.txt], hierarchy=[]", "request_api_name": "DATA_OBJ_OPEN_AN", "request_api_number": 602, "request_api_version": "d", "request_client_user": "u0137480", "request_host": "127.0.0.1", "request_proxy_user": "globus", "request_release_version": "rods4.3.3", "server_host": "icts.irods.t.icts.kuleuven.be", "server_pid": 436834, "server_timestamp": "2024-12-19T12:37:45.018Z", "server_type": "agent", "server_zone": "icts"}}
[2024-12-19T12:38:16.681Z][icts-t-cloud-rdm-leu-2] {"processname": "irods", "groupname": "irods", "pid": "95", "channel": "stdout", "irods": {"log_category": "legacy", "log_level": "error", "log_message": "[-]\t/irods_source/server/core/src/rsApiHandler.cpp:565:int readAndProcClientMsg(rsComm_t *, int) :  status [SYS_HEADER_READ_LEN_ERR]  errno [] -- message [failed to call 'read header']\n\t[-]\t/irods_source/lib/core/src/sockComm.cpp:229:irods::error readMsgHeader(irods::network_object_ptr, msgHeader_t *, struct timeval *) :  status [SYS_HEADER_READ_LEN_ERR]  errno [] -- message [failed to call 'read header']\n\t\t[-]\t/irods_source/plugins/network/src/ssl.cpp:528:irods::error ssl_read_msg_header(irods::plugin_context &, void *, struct timeval *) :  status [SYS_HEADER_READ_LEN_ERR]  errno [] -- message [read 0 expected 4]\n\n", "request_api_name": "GEN_QUERY_AN", "request_api_number": 702, "request_api_version": "d", "request_client_user": "rods", "request_host": "127.0.0.1", "request_proxy_user": "rods", "request_release_version": "rods4.3.0", "server_host": "icts.irods.t.icts.kuleuven.be", "server_pid": 414275, "server_timestamp": "2024-12-19T12:38:16.681Z", "server_type": "agent", "server_zone": "icts"}}
[2024-12-19T12:38:16.683Z][icts-t-cloud-rdm-leu-2] {"processname": "irods", "groupname": "irods", "pid": "95", "channel": "stdout", "irods": {"log_category": "agent", "log_level": "error", "log_message": "Agent [414275] exiting with status = -4000", "request_api_name": "GEN_QUERY_AN", "request_api_number": 702, "request_api_version": "d", "request_client_user": "rods", "request_host": "127.0.0.1", "request_proxy_user": "rods", "request_release_version": "rods4.3.0", "server_host": "icts.irods.t.icts.kuleuven.be", "server_pid": 414275, "server_timestamp": "2024-12-19T12:38:16.683Z", "server_type": "agent", "server_zone": "icts"}}
[2024-12-19T12:38:16.704Z][icts-t-cloud-rdm-leu-2] {"processname": "irods", "groupname": "irods", "pid": "95", "channel": "stdout", "irods": {"log_category": "agent_factory", "log_level": "error", "log_message": "Agent process [414275] exited with status [1].", "server_host": "icts.irods.t.icts.kuleuven.be", "server_pid": 116, "server_timestamp": "2024-12-19T12:38:16.703Z", "server_type": "agent_factory", "server_zone": "icts"}}

[trel]

What should iRODS (or the connector) return if Globus asks for a checksum of something that does not exist?

It currently returns a -1 when the file cannot be opened.

[mstfdkmn]

If the file does not exist, the appropriate error (expected by Globus) or the same could be returned, and the Globus client should skip that file. If it's a single-file transfer, the entire transfer should be cancelled; otherwise, the transfer should continue for the remaining (non-removed) files. That is my simple approach :)

With that in mind, it is be great to get any recommendations for implementing a generic policy (applicable whether checksums are enabled or disabled) to remove unwanted files transferred via Globus. Thanks.