s3-integrator stuck in config-changed event IF no connection to S3 host #70
Description
Steps to reproduce
Deploy and configure charm without connectivity to S3 storage (e.g. PS6 and AWS S3).
Expected behavior
Charm went into blocked/error state telling about lack of S3 connectivity, forcing user to check networking and react with a reconfiguration or networking fix. Better ideas are welcome.
P.S. better to handle the huge exception/trace in debug-log.
Actual behavior
The unit has stuck in event:
App Version Status Scale Charm Channel Rev Exposed Message
s3-integrator active 1 s3-integrator 2/edge 246 no
...
Unit Workload Agent Machine Public address Ports Message
s3-integrator/1* active executing 4 10.142.137.98 (config-changed)
...
Error in log
unit-s3-integrator-1: 14:03:54 DEBUG unit.s3-integrator/1.juju-log Starting new HTTPS connection (2): s3.amazonaws.com:443
unit-s3-integrator-1: 14:11:54 DEBUG unit.s3-integrator/1.juju-log Exception received when sending HTTP request.
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-s3-integrator-1/charm/venv/lib/python3.12/site-packages/urllib3/connection.py", line 198, in _new_conn
sock = connection.create_connection(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/lib/juju/agents/unit-s3-integrator-1/charm/venv/lib/python3.12/site-packages/urllib3/util/connection.py", line 85, in create_connection
raise err
File "/var/lib/juju/agents/unit-s3-integrator-1/charm/venv/lib/python3.12/site-packages/urllib3/util/connection.py", line 73, in create_connection
sock.connect(sa)
TimeoutError: timed out
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-s3-integrator-1/charm/venv/lib/python3.12/site-packages/botocore/httpsession.py", line 465, in send
urllib_response = conn.urlopen(
^^^^^^^^^^^^^
File "/var/lib/juju/agents/unit-s3-integrator-1/charm/venv/lib/python3.12/site-packages/urllib3/connectionpool.py", line 841, in urlopen
retries = retries.increment(
^^^^^^^^^^^^^^^^^^
File "/var/lib/juju/agents/unit-s3-integrator-1/charm/venv/lib/python3.12/site-packages/urllib3/util/retry.py", line 449, in increment
raise reraise(type(error), error, _stacktrace)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/lib/juju/agents/unit-s3-integrator-1/charm/venv/lib/python3.12/site-packages/urllib3/util/util.py", line 39, in reraise
raise value
File "/var/lib/juju/agents/unit-s3-integrator-1/charm/venv/lib/python3.12/site-packages/urllib3/connectionpool.py", line 787, in urlopen
response = self._make_request(
^^^^^^^^^^^^^^^^^^^
File "/var/lib/juju/agents/unit-s3-integrator-1/charm/venv/lib/python3.12/site-packages/urllib3/connectionpool.py", line 488, in _make_request
raise new_e
File "/var/lib/juju/agents/unit-s3-integrator-1/charm/venv/lib/python3.12/site-packages/urllib3/connectionpool.py", line 464, in _make_request
self._validate_conn(conn)
File "/var/lib/juju/agents/unit-s3-integrator-1/charm/venv/lib/python3.12/site-packages/urllib3/connectionpool.py", line 1093, in _validate_conn
conn.connect()
File "/var/lib/juju/agents/unit-s3-integrator-1/charm/venv/lib/python3.12/site-packages/urllib3/connection.py", line 753, in connect
self.sock = sock = self._new_conn()
^^^^^^^^^^^^^^^^
File "/var/lib/juju/agents/unit-s3-integrator-1/charm/venv/lib/python3.12/site-packages/urllib3/connection.py", line 207, in _new_conn
raise ConnectTimeoutError(
urllib3.exceptions.ConnectTimeoutError: (<botocore.awsrequest.AWSHTTPSConnection object at 0x7e0e88b702f0>, 'Connection to s3.amazonaws.com timed out. (connect timeout=60)')
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-s3-integrator-1/charm/venv/lib/python3.12/site-packages/botocore/endpoint.py", line 279, in _do_get_response
http_response = self._send(request)
^^^^^^^^^^^^^^^^^^^
File "/var/lib/juju/agents/unit-s3-integrator-1/charm/venv/lib/python3.12/site-packages/botocore/endpoint.py", line 383, in _send
return self.http_session.send(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/lib/juju/agents/unit-s3-integrator-1/charm/venv/lib/python3.12/site-packages/botocore/httpsession.py", line 500, in send
raise ConnectTimeoutError(endpoint_url=request.url, error=e)
botocore.exceptions.ConnectTimeoutError: Connect timeout on endpoint URL: "https://s3.amazonaws.com/data-charms-testing?list-type=2&prefix=alutay_1762350927%2F&encoding-type=url"
unit-s3-integrator-1: 14:11:54 DEBUG unit.s3-integrator/1.juju-log Event needs-retry.s3.ListObjectsV2: calling handler <function _update_status_code at 0x7e0e89af4040>
unit-s3-integrator-1: 14:11:54 DEBUG unit.s3-integrator/1.juju-log Event needs-retry.s3.ListObjectsV2: calling handler <botocore.retryhandler.RetryHandler object at 0x7e0e88f95a90>
unit-s3-integrator-1: 14:11:54 DEBUG unit.s3-integrator/1.juju-log retry needed, retryable exception caught: Connect timeout on endpoint URL: "https://s3.amazonaws.com/data-charms-testing?list-type=2&prefix=alutay_1762350927%2F&encoding-type=url"
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-s3-integrator-1/charm/venv/lib/python3.12/site-packages/urllib3/connection.py", line 198, in _new_conn
sock = connection.create_connection(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/lib/juju/agents/unit-s3-integrator-1/charm/venv/lib/python3.12/site-packages/urllib3/util/connection.py", line 85, in create_connection
raise err
File "/var/lib/juju/agents/unit-s3-integrator-1/charm/venv/lib/python3.12/site-packages/urllib3/util/connection.py", line 73, in create_connection
sock.connect(sa)
TimeoutError: timed out
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-s3-integrator-1/charm/venv/lib/python3.12/site-packages/botocore/httpsession.py", line 465, in send
urllib_response = conn.urlopen(
^^^^^^^^^^^^^
File "/var/lib/juju/agents/unit-s3-integrator-1/charm/venv/lib/python3.12/site-packages/urllib3/connectionpool.py", line 841, in urlopen
retries = retries.increment(
^^^^^^^^^^^^^^^^^^
File "/var/lib/juju/agents/unit-s3-integrator-1/charm/venv/lib/python3.12/site-packages/urllib3/util/retry.py", line 449, in increment
raise reraise(type(error), error, _stacktrace)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/lib/juju/agents/unit-s3-integrator-1/charm/venv/lib/python3.12/site-packages/urllib3/util/util.py", line 39, in reraise
raise value
File "/var/lib/juju/agents/unit-s3-integrator-1/charm/venv/lib/python3.12/site-packages/urllib3/connectionpool.py", line 787, in urlopen
response = self._make_request(
^^^^^^^^^^^^^^^^^^^
File "/var/lib/juju/agents/unit-s3-integrator-1/charm/venv/lib/python3.12/site-packages/urllib3/connectionpool.py", line 488, in _make_request
raise new_e
File "/var/lib/juju/agents/unit-s3-integrator-1/charm/venv/lib/python3.12/site-packages/urllib3/connectionpool.py", line 464, in _make_request
self._validate_conn(conn)
File "/var/lib/juju/agents/unit-s3-integrator-1/charm/venv/lib/python3.12/site-packages/urllib3/connectionpool.py", line 1093, in _validate_conn
conn.connect()
File "/var/lib/juju/agents/unit-s3-integrator-1/charm/venv/lib/python3.12/site-packages/urllib3/connection.py", line 753, in connect
self.sock = sock = self._new_conn()
^^^^^^^^^^^^^^^^
File "/var/lib/juju/agents/unit-s3-integrator-1/charm/venv/lib/python3.12/site-packages/urllib3/connection.py", line 207, in _new_conn
raise ConnectTimeoutError(
urllib3.exceptions.ConnectTimeoutError: (<botocore.awsrequest.AWSHTTPSConnection object at 0x7e0e88b702f0>, 'Connection to s3.amazonaws.com timed out. (connect timeout=60)')
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-s3-integrator-1/charm/venv/lib/python3.12/site-packages/botocore/retryhandler.py", line 307, in _should_retry
return self._checker(
^^^^^^^^^^^^^^
File "/var/lib/juju/agents/unit-s3-integrator-1/charm/venv/lib/python3.12/site-packages/botocore/retryhandler.py", line 363, in __call__
checker_response = checker(
^^^^^^^^
File "/var/lib/juju/agents/unit-s3-integrator-1/charm/venv/lib/python3.12/site-packages/botocore/retryhandler.py", line 247, in __call__
return self._check_caught_exception(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/lib/juju/agents/unit-s3-integrator-1/charm/venv/lib/python3.12/site-packages/botocore/retryhandler.py", line 416, in _check_caught_exception
raise caught_exception
File "/var/lib/juju/agents/unit-s3-integrator-1/charm/venv/lib/python3.12/site-packages/botocore/endpoint.py", line 279, in _do_get_response
http_response = self._send(request)
^^^^^^^^^^^^^^^^^^^
File "/var/lib/juju/agents/unit-s3-integrator-1/charm/venv/lib/python3.12/site-packages/botocore/endpoint.py", line 383, in _send
return self.http_session.send(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/lib/juju/agents/unit-s3-integrator-1/charm/venv/lib/python3.12/site-packages/botocore/httpsession.py", line 500, in send
raise ConnectTimeoutError(endpoint_url=request.url, error=e)
botocore.exceptions.ConnectTimeoutError: Connect timeout on endpoint URL: "https://s3.amazonaws.com/data-charms-testing?list-type=2&prefix=alutay_1762350927%2F&encoding-type=url"
unit-s3-integrator-1: 14:11:54 DEBUG unit.s3-integrator/1.juju-log Retry needed, action of: 0.42334853669811046
unit-s3-integrator-1: 14:11:54 DEBUG unit.s3-integrator/1.juju-log Event needs-retry.s3.ListObjectsV2: calling handler
Versions
Operating system: 24.04
Juju CLI: 3.6.11
Juju agent: 3.6.11
Charm revision: 2/edge (r246)
Log output
See above.