Timeout for file flush

[erlendkopp]

If connection to HSDS (or the internet) is lost during a file flush, the process hangs indefinitely. There should be some timeout logic, preferably set by a parameter to the flush function, that allows the HTTP request to stop after a certain amount of time.

On a device, like an IoT device, pulling the plug on the network connection mid-flush results in this sample script below hanging indefinitely.

import h5pyd

path = '/test/timeout.h5'

with h5pyd.File(path, 'w') as f:
    dset = f.create_dataset('data', (1000000000,), dtype='i')
    print('Start flush PUT')
    f.flush()
    print('Successful flush PUT')

Here is a full traceback after KeyboardInterrupt:

Start flush PUT
^CTraceback (most recent call last):
  File "/home/user/repo/test_timeout.py", line 8, in <module>
    f.flush()
  File "/home/user/.local/lib/python3.10/site-packages/h5pyd/_hl/files.py", line 775, in flush
    rsp = self.PUT(req, body=body)
  File "/home/user/.local/lib/python3.10/site-packages/h5pyd/_hl/base.py", line 1017, in PUT
    rsp = self._id._http_conn.PUT(req, body=body, params=params, format=format)
  File "/home/user/.local/lib/python3.10/site-packages/h5pyd/_hl/httpconn.py", line 580, in PUT
    rsp = s.put(
  File "/home/user/.local/lib/python3.10/site-packages/requests/sessions.py", line 649, in put
    return self.request("PUT", url, data=data, **kwargs)
  File "/home/user/.local/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/user/.local/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/home/user/.local/lib/python3.10/site-packages/requests/adapters.py", line 644, in send
    resp = conn.urlopen(
  File "/home/user/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    response = self._make_request(
  File "/home/user/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 534, in _make_request
    response = conn.getresponse()
  File "/home/user/.local/lib/python3.10/site-packages/urllib3/connection.py", line 565, in getresponse
    httplib_response = super().getresponse()
  File "/usr/lib/python3.10/http/client.py", line 1375, in getresponse
    response.begin()
  File "/usr/lib/python3.10/http/client.py", line 318, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.10/http/client.py", line 279, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib/python3.10/socket.py", line 705, in readinto
    return self._sock.recv_into(b)
  File "/usr/lib/python3.10/ssl.py", line 1274, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/lib/python3.10/ssl.py", line 1130, in read
    return self._sslobj.read(len, buffer)
KeyboardInterrupt

Since you are using the requests library, which as far as I understand already have a timeout parameter, which you can add in h5pyd/_hl/httpconn.py", line 580, in PUT, rsp = s.put(, you can hopefully add timeout as a parameter in file.flush() and pass it to the put request.

[jreadey]

Thanks for calling attention to this!
I've made changes to httpconn.py so that the timeout value is used for PUT, POST, and DELETE as well as GET.
I didn't add a timeout parameter to file.flush(), but you can set the timeout the parameter in h5pyd.File(). Or if not set, it will default to 180 seconds.
Note that unless you set retries to 0 (default is 10), on a timeout, h5pyd will sleep a bit and then retry the same request up to the retry limit.

Please try out the fix and let me know how it goes.