🐛 FileNotFoundError related to monitors #7086
Description
While "running" a workflow, I am reliably running into the following error:
FileNotFoundError: [Errno 2] No such file
From the full traceback (see below), the issue seems to be related to the monitors. In the end, the workflow does complete successfully, but the issue produces a lot of noise that will worry users.
I tested both core.ssh and core.ssh_async, and am running into the issue for both.
Full Traceback
11/05/2025 11:52:01 AM <1716> aiida.engine.transports: [ERROR] Exception whilst using transport:
Traceback (most recent call last):
File "/Users/mbercx/project/defect/git/aiida-core/src/aiida/engine/transports.py", line 106, in request_transport
yield transport_request.future
File "/Users/mbercx/project/defect/git/aiida-core/src/aiida/engine/processes/calcjobs/tasks.py", line 257, in do_monitor
return monitors.process(node, transport)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/mbercx/project/defect/git/aiida-core/src/aiida/engine/processes/calcjobs/monitors.py", line 204, in process
monitor_result = monitor_function(node, transport, **monitor.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/mbercx/project/defect/git/aiida-vasp/src/aiida_vasp/calcs/monitors.py", line 66, in monitor_stdout
file_stat = transport.get_attribute(stdout_path)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/mbercx/project/defect/git/aiida-core/src/aiida/transports/plugins/ssh.py", line 1201, in get_attribute
paramiko_attr = self.lstat(path)
^^^^^^^^^^^^^^^^
File "/Users/mbercx/project/defect/git/aiida-core/src/aiida/transports/plugins/ssh.py", line 662, in lstat
return self.sftp.lstat(path)
^^^^^^^^^^^^^^^^^^^^^
File "/Users/mbercx/.aiida_venvs/defect/lib/python3.12/site-packages/paramiko/sftp_client.py", line 511, in lstat
t, msg = self._request(CMD_LSTAT, path)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/mbercx/.aiida_venvs/defect/lib/python3.12/site-packages/paramiko/sftp_client.py", line 857, in _request
return self._read_response(num)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/mbercx/.aiida_venvs/defect/lib/python3.12/site-packages/paramiko/sftp_client.py", line 909, in _read_response
self._convert_status(msg)
File "/Users/mbercx/.aiida_venvs/defect/lib/python3.12/site-packages/paramiko/sftp_client.py", line 938, in _convert_status
raise IOError(errno.ENOENT, text)
FileNotFoundError: [Errno 2] No such file
11/05/2025 11:52:01 AM <1716> aiida.orm.nodes.process.calculation.calcjob.CalcJobNode: [ERROR] iteration 1 of do_monitor excepted, retrying after 20 seconds
Traceback (most recent call last):
File "/Users/mbercx/project/defect/git/aiida-core/src/aiida/engine/utils.py", line 205, in exponential_backoff_retry
result = await coro()
^^^^^^^^^^^^
File "/Users/mbercx/project/defect/git/aiida-core/src/aiida/engine/processes/calcjobs/tasks.py", line 257, in do_monitor
return monitors.process(node, transport)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/mbercx/project/defect/git/aiida-core/src/aiida/engine/processes/calcjobs/monitors.py", line 204, in process
monitor_result = monitor_function(node, transport, **monitor.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/mbercx/project/defect/git/aiida-vasp/src/aiida_vasp/calcs/monitors.py", line 66, in monitor_stdout
file_stat = transport.get_attribute(stdout_path)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/mbercx/project/defect/git/aiida-core/src/aiida/transports/plugins/ssh.py", line 1201, in get_attribute
paramiko_attr = self.lstat(path)
^^^^^^^^^^^^^^^^
File "/Users/mbercx/project/defect/git/aiida-core/src/aiida/transports/plugins/ssh.py", line 662, in lstat
return self.sftp.lstat(path)
^^^^^^^^^^^^^^^^^^^^^
File "/Users/mbercx/.aiida_venvs/defect/lib/python3.12/site-packages/paramiko/sftp_client.py", line 511, in lstat
t, msg = self._request(CMD_LSTAT, path)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/mbercx/.aiida_venvs/defect/lib/python3.12/site-packages/paramiko/sftp_client.py", line 857, in _request
return self._read_response(num)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/mbercx/.aiida_venvs/defect/lib/python3.12/site-packages/paramiko/sftp_client.py", line 909, in _read_response
self._convert_status(msg)
File "/Users/mbercx/.aiida_venvs/defect/lib/python3.12/site-packages/paramiko/sftp_client.py", line 938, in _convert_status
raise IOError(errno.ENOENT, text)
FileNotFoundError: [Errno 2] No such file