WORKER_TIMEOUT on data-lifecycle

[BenGalewsky]

See slack message:

2026-03-05 16:00:01 INFO servicex servicex_app Data lifecycle ops request using 2026-03-03T16:00:01 Timestamp to reduce by 0 extra: {'asctime': '2026-03-05 16:00:01'}
Bucket 'ba52cb32-34a5-446f-abd8-d3761e56ced2' deleted successfully.
Bucket 'a730cb44-e688-41c1-af1e-a4c6ec20561a' deleted successfully.
Bucket '21d01ce0-70e6-4b65-81b4-73fb1bc20e48' deleted successfully.
[2026-03-05 16:02:02 +0000] [1] [CRITICAL] WORKER TIMEOUT (pid:260)
[2026-03-05 16:02:02 +0000] [260] [ERROR] Error handling request /servicex/internal/data-lifecycle?cutoff_timestamp=2026-03-03T16:00:01
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/gunicorn/workers/sync.py", line 134, in handle
    self.handle_request(listener, req, client, addr)
  File "/usr/local/lib/python3.10/site-packages/gunicorn/workers/sync.py", line 177, in handle_request
    respiter = self.wsgi(environ, resp.start_response)
  File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 2213, in __call__
    return self.wsgi_app(environ, start_response)
  File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 2190, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 1484, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 1469, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
  File "/usr/local/lib/python3.10/site-packages/flask_restful/__init__.py", line 489, in wrapper
    resp = resource(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/flask/views.py", line 109, in view
    return current_app.ensure_sync(self.dispatch_request)(**kwargs)
  File "/usr/local/lib/python3.10/site-packages/flask_restful/__init__.py", line 604, in dispatch_request
    resp = meth(*args, **kwargs)
  File "/home/servicex/servicex_app/resources/internal/data_lifecycle_ops.py", line 165, in post
    deleted_log = self.delete_expired_transforms(
  File "/home/servicex/servicex_app/resources/internal/data_lifecycle_ops.py", line 73, in delete_expired_transforms
    object_store.delete_bucket_and_contents(transform.request_id)
  File "/home/servicex/servicex_app/object_store_manager.py", line 59, in delete_bucket_and_contents
    self.minio_client.remove_object(bucket_name, obj.object_name)
  File "/usr/local/lib/python3.10/site-packages/minio/api.py", line 2173, in remove_object
    self._execute(
  File "/usr/local/lib/python3.10/site-packages/minio/api.py", line 441, in _execute
    return self._url_open(
  File "/usr/local/lib/python3.10/site-packages/minio/api.py", line 303, in _url_open
    response = self._http.urlopen(
  File "/usr/local/lib/python3.10/site-packages/urllib3/poolmanager.py", line 457, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    response = self._make_request(
  File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 534, in _make_request
    response = conn.getresponse()
  File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 571, in getresponse
    httplib_response = super().getresponse()
  File "/usr/local/lib/python3.10/http/client.py", line 1375, in getresponse
    response.begin()
  File "/usr/local/lib/python3.10/http/client.py", line 318, in begin
    version, status, reason = self._read_status()
  File "/usr/local/lib/python3.10/http/client.py", line 279, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/local/lib/python3.10/socket.py", line 717, in readinto
    return self._sock.recv_into(b)
  File "/usr/local/lib/python3.10/site-packages/gunicorn/workers/base.py", line 204, in handle_abort
    sys.exit(1)
SystemExit: 1

[ponyisi]

This is presumably due to the cleanup requests taking too long to be handled in a synchronous HTTP request. Should this just be put into a task queue and run asynchronously?