DALServiceError on cosmodc2 notebook

[jkrick]

When trying to figure out what is going on in astropy/astropy#144

I am seeing the below full traceback for the following code.

adql = f"""
SELECT TOP 100 redshift
FROM {tablename}
WHERE CONTAINS(
    POINT('ICRS', ra, dec),
    CIRCLE('ICRS', 54.2, -37.4, 0.05)
) = 1
"""
job = service.submit_job(adql)
job.run()
if wait_for_job(job, timeout=300):
    spatial_results = job.fetch_result()
    print("Rows:", len(results))
else:
    print("Job did NOT finish — cancelled or failed.")

Judith is watching this query on the database and tells me it is still 'EXECUTING' even after I got this error. I don't understand that. @bsipocz what are your thoughts/suggestions for how to move forward?

---------------------------------------------------------------------------
TimeoutError                              Traceback (most recent call last)
File [/opt/envs/python3/lib/python3.12/site-packages/urllib3/connectionpool.py:534](https://console.fornax.sciencecloud.nasa.gov/opt/envs/python3/lib/python3.12/site-packages/urllib3/connectionpool.py#line=533), in HTTPConnectionPool._make_request(self, conn, method, url, body, headers, retries, timeout, chunked, response_conn, preload_content, decode_content, enforce_content_length)
    533 try:
--> 534     response = conn.getresponse()
    535 except (BaseSSLError, OSError) as e:

File [/opt/envs/python3/lib/python3.12/site-packages/urllib3/connection.py:565](https://console.fornax.sciencecloud.nasa.gov/opt/envs/python3/lib/python3.12/site-packages/urllib3/connection.py#line=564), in HTTPConnection.getresponse(self)
    564 # Get the response from http.client.HTTPConnection
--> 565 httplib_response = super().getresponse()
    567 try:

File [/opt/envs/base/lib/python3.12/http/client.py:1428](https://console.fornax.sciencecloud.nasa.gov/opt/envs/base/lib/python3.12/http/client.py#line=1427), in HTTPConnection.getresponse(self)
   1427 try:
-> 1428     response.begin()
   1429 except ConnectionError:

File [/opt/envs/base/lib/python3.12/http/client.py:331](https://console.fornax.sciencecloud.nasa.gov/opt/envs/base/lib/python3.12/http/client.py#line=330), in HTTPResponse.begin(self)
    330 while True:
--> 331     version, status, reason = self._read_status()
    332     if status != CONTINUE:

File [/opt/envs/base/lib/python3.12/http/client.py:292](https://console.fornax.sciencecloud.nasa.gov/opt/envs/base/lib/python3.12/http/client.py#line=291), in HTTPResponse._read_status(self)
    291 def _read_status(self):
--> 292     line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
    293     if len(line) > _MAXLINE:

File [/opt/envs/base/lib/python3.12/socket.py:707](https://console.fornax.sciencecloud.nasa.gov/opt/envs/base/lib/python3.12/socket.py#line=706), in SocketIO.readinto(self, b)
    706 try:
--> 707     return self._sock.recv_into(b)
    708 except timeout:

File [/opt/envs/base/lib/python3.12/ssl.py:1252](https://console.fornax.sciencecloud.nasa.gov/opt/envs/base/lib/python3.12/ssl.py#line=1251), in SSLSocket.recv_into(self, buffer, nbytes, flags)
   1249         raise ValueError(
   1250           "non-zero flags not allowed in calls to recv_into() on %s" %
   1251           self.__class__)
-> 1252     return self.read(nbytes, buffer)
   1253 else:

File [/opt/envs/base/lib/python3.12/ssl.py:1104](https://console.fornax.sciencecloud.nasa.gov/opt/envs/base/lib/python3.12/ssl.py#line=1103), in SSLSocket.read(self, len, buffer)
   1103 if buffer is not None:
-> 1104     return self._sslobj.read(len, buffer)
   1105 else:

TimeoutError: The read operation timed out

The above exception was the direct cause of the following exception:

ReadTimeoutError                          Traceback (most recent call last)
File [/opt/envs/python3/lib/python3.12/site-packages/requests/adapters.py:644](https://console.fornax.sciencecloud.nasa.gov/opt/envs/python3/lib/python3.12/site-packages/requests/adapters.py#line=643), in HTTPAdapter.send(self, request, stream, timeout, verify, cert, proxies)
    643 try:
--> 644     resp = conn.urlopen(
    645         method=request.method,
    646         url=url,
    647         body=request.body,
    648         headers=request.headers,
    649         redirect=False,
    650         assert_same_host=False,
    651         preload_content=False,
    652         decode_content=False,
    653         retries=self.max_retries,
    654         timeout=timeout,
    655         chunked=chunked,
    656     )
    658 except (ProtocolError, OSError) as err:

File [/opt/envs/python3/lib/python3.12/site-packages/urllib3/connectionpool.py:841](https://console.fornax.sciencecloud.nasa.gov/opt/envs/python3/lib/python3.12/site-packages/urllib3/connectionpool.py#line=840), in HTTPConnectionPool.urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, preload_content, decode_content, **response_kw)
    839     new_e = ProtocolError("Connection aborted.", new_e)
--> 841 retries = retries.increment(
    842     method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2]
    843 )
    844 retries.sleep()

File [/opt/envs/python3/lib/python3.12/site-packages/urllib3/util/retry.py:474](https://console.fornax.sciencecloud.nasa.gov/opt/envs/python3/lib/python3.12/site-packages/urllib3/util/retry.py#line=473), in Retry.increment(self, method, url, response, error, _pool, _stacktrace)
    473 if read is False or method is None or not self._is_method_retryable(method):
--> 474     raise reraise(type(error), error, _stacktrace)
    475 elif read is not None:

File [/opt/envs/python3/lib/python3.12/site-packages/urllib3/util/util.py:39](https://console.fornax.sciencecloud.nasa.gov/opt/envs/python3/lib/python3.12/site-packages/urllib3/util/util.py#line=38), in reraise(tp, value, tb)
     38         raise value.with_traceback(tb)
---> 39     raise value
     40 finally:

File [/opt/envs/python3/lib/python3.12/site-packages/urllib3/connectionpool.py:787](https://console.fornax.sciencecloud.nasa.gov/opt/envs/python3/lib/python3.12/site-packages/urllib3/connectionpool.py#line=786), in HTTPConnectionPool.urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, preload_content, decode_content, **response_kw)
    786 # Make the request on the HTTPConnection object
--> 787 response = self._make_request(
    788     conn,
    789     method,
    790     url,
    791     timeout=timeout_obj,
    792     body=body,
    793     headers=headers,
    794     chunked=chunked,
    795     retries=retries,
    796     response_conn=response_conn,
    797     preload_content=preload_content,
    798     decode_content=decode_content,
    799     **response_kw,
    800 )
    802 # Everything went great!

File [/opt/envs/python3/lib/python3.12/site-packages/urllib3/connectionpool.py:536](https://console.fornax.sciencecloud.nasa.gov/opt/envs/python3/lib/python3.12/site-packages/urllib3/connectionpool.py#line=535), in HTTPConnectionPool._make_request(self, conn, method, url, body, headers, retries, timeout, chunked, response_conn, preload_content, decode_content, enforce_content_length)
    535 except (BaseSSLError, OSError) as e:
--> 536     self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
    537     raise

File [/opt/envs/python3/lib/python3.12/site-packages/urllib3/connectionpool.py:367](https://console.fornax.sciencecloud.nasa.gov/opt/envs/python3/lib/python3.12/site-packages/urllib3/connectionpool.py#line=366), in HTTPConnectionPool._raise_timeout(self, err, url, timeout_value)
    366 if isinstance(err, SocketTimeout):
--> 367     raise ReadTimeoutError(
    368         self, url, f"Read timed out. (read timeout={timeout_value})"
    369     ) from err
    371 # See the above comment about EAGAIN in Python 3.

ReadTimeoutError: HTTPSConnectionPool(host='irsawebops1.ipac.caltech.edu', port=443): Read timed out. (read timeout=10.0)

During handling of the above exception, another exception occurred:

ReadTimeout                               Traceback (most recent call last)
File [/opt/envs/python3/lib/python3.12/site-packages/pyvo/dal/tap.py:715](https://console.fornax.sciencecloud.nasa.gov/opt/envs/python3/lib/python3.12/site-packages/pyvo/dal/tap.py#line=714), in AsyncTAPJob._update(self, wait_for_statechange, timeout)
    714 else:
--> 715     response = self._session.get(self.url, stream=True, timeout=timeout)
    716 response.raise_for_status()

File [/opt/envs/python3/lib/python3.12/site-packages/requests/sessions.py:602](https://console.fornax.sciencecloud.nasa.gov/opt/envs/python3/lib/python3.12/site-packages/requests/sessions.py#line=601), in Session.get(self, url, **kwargs)
    601 kwargs.setdefault("allow_redirects", True)
--> 602 return self.request("GET", url, **kwargs)

File [/opt/envs/python3/lib/python3.12/site-packages/requests/sessions.py:589](https://console.fornax.sciencecloud.nasa.gov/opt/envs/python3/lib/python3.12/site-packages/requests/sessions.py#line=588), in Session.request(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json)
    588 send_kwargs.update(settings)
--> 589 resp = self.send(prep, **send_kwargs)
    591 return resp

File [/opt/envs/python3/lib/python3.12/site-packages/requests/sessions.py:703](https://console.fornax.sciencecloud.nasa.gov/opt/envs/python3/lib/python3.12/site-packages/requests/sessions.py#line=702), in Session.send(self, request, **kwargs)
    702 # Send the request
--> 703 r = adapter.send(request, **kwargs)
    705 # Total elapsed time of the request (approximately)

File [/opt/envs/python3/lib/python3.12/site-packages/requests/adapters.py:690](https://console.fornax.sciencecloud.nasa.gov/opt/envs/python3/lib/python3.12/site-packages/requests/adapters.py#line=689), in HTTPAdapter.send(self, request, stream, timeout, verify, cert, proxies)
    689 elif isinstance(e, ReadTimeoutError):
--> 690     raise ReadTimeout(e, request=request)
    691 elif isinstance(e, _InvalidHeader):

ReadTimeout: HTTPSConnectionPool(host='irsawebops1.ipac.caltech.edu', port=443): Read timed out. (read timeout=10.0)

During handling of the above exception, another exception occurred:

DALServiceError                           Traceback (most recent call last)
Cell In[19], line 12
     10 job = service.submit_job(adql)
     11 job.run()
---> 12 if wait_for_job(job, timeout=300):
     13     spatial_results = job.fetch_result()
     14     print("Rows:", len(results))

Cell In[5], line 23, in wait_for_job(job, timeout, poll_interval)
     20 start = time.time()
     22 while True:
---> 23     phase = job.phase
     24     print("Phase:", phase)
     26     # Normal success

File [/opt/envs/python3/lib/python3.12/site-packages/pyvo/dal/tap.py:753](https://console.fornax.sciencecloud.nasa.gov/opt/envs/python3/lib/python3.12/site-packages/pyvo/dal/tap.py#line=752), in AsyncTAPJob.phase(self)
    748 @property
    749 def phase(self):
    750     """
    751     the current query phase
    752     """
--> 753     self._update()
    754     return self._job.phase

File [/opt/envs/python3/lib/python3.12/site-packages/pyvo/dal/tap.py:718](https://console.fornax.sciencecloud.nasa.gov/opt/envs/python3/lib/python3.12/site-packages/pyvo/dal/tap.py#line=717), in AsyncTAPJob._update(self, wait_for_statechange, timeout)
    716     response.raise_for_status()
    717 except requests.RequestException as ex:
--> 718     raise DALServiceError.from_except(ex, self.url)
    720 # requests doesn't decode the content by default
    721 response.raw.read = partial(response.raw.read, decode_content=True)

DALServiceError: HTTPSConnectionPool(host='irsawebops1.ipac.caltech.edu', port=443): Read timed out. (read timeout=10.0)

[troyraen]

The traceback indicates that the read is timing out after 10 seconds. I think we should increase that limit. That may fix this issue altogether, but even if it doesn't it may allow the "real" issue to surface. This 10-second read timeout limit is common in astropy/astroquery/pyvo and we've dealt with it in other issues/notebooks.

(I see this line at the top: if wait_for_job(job, timeout=300):, but that doesn't appear to be the time limit that's getting hit here.)

[bsipocz]

The 10s is the timeout for the update check (as this is run as an async query). That value is baked in, and I would be surprised if it would be the source of our issues.

[bsipocz]

I mean is that normal for an async service to timeout on this?

I pull in @tomdonaldson to chime in in case he think it's reasonably to make this configurable within pyvo.

code xref: https://github.com/astropy/pyvo/blob/main/pyvo/dal/tap.py#L703

[jkrick]

Sorry, the wait_for_job() is below. It is my own wrapper designed to kill the jobs if they don't finish in what I think is a reasonable amount of time while I am testing out this TAP query notebook. I can run this exact same query twice and once have it run for a while until I abort just fine, and the second time throw this error. Also, at times, while testing, I can get this error after just a minute, and at times after 2 minutes. I can't figure out what triggers it.

def wait_for_job(job, timeout=60, poll_interval=5.0):
"""
Wait until a TAP job reaches COMPLETED or ERROR.
If it is still EXECUTING after timeout seconds, cancel it.

Parameters
----------
job : pyvo.dal.AsyncTAPJob
    The TAP job object from service.submit_job().
timeout : int or float
    Max time (seconds) to wait before auto-cancelling.
poll_interval : float
    Seconds between job.phase checks.

Returns
-------
bool
    True if COMPLETED; False if ERROR, ABORTED, or timeout-cancelled.
"""
start = time.time()

while True:
    phase = job.phase
    print("Phase:", phase)

    # Normal success
    if phase == "COMPLETED":
        return True

    # Hard failure
    if phase in ("ERROR", "ABORTED", "UNKNOWN"):
        return False

    # Timeout reached — cancel it
    if time.time() - start > timeout:
        print(f"Timeout reached ({timeout} s). Aborting TAP job...")
        job.abort()
        return False

    time.sleep(poll_interval)

[jkrick]

I've done a lot of testing for this, and ended up just submitting a bug issue to astropy. That is here: astropy/pyvo#713

Chatgpt thinks it is the 10s timeout. I have pretty much ruled out a TAP problem and backend problem. (although I think those are there in this notebook, just not causing this error).