Skip to content

Commit f68832c

Browse files
author
Tom Augspurger
authored
[goes-glm]: Try to fix with socket timeout (#183)
In a recent run, we experience the hanging goes-glm task again. Here's the backtrace from py-spy on the hanging process. ``` Thread 1 (active): "MainThread" read (ssl.py:1134) recv_into (ssl.py:1278) readinto (socket.py:706) read (http/client.py:466) _fp_read (urllib3/response.py:533) read (urllib3/response.py:567) stream (urllib3/response.py:628) generate (requests/models.py:816) __next__ (core/pipeline/transport/_requests_basic.py:173) process_content (blob/_download.py:52) _initial_request (blob/_download.py:435) __init__ (blob/_download.py:349) download_blob (blob/_blob_client.py:848) wrapper_use_tracer (core/tracing/decorator.py:76) <lambda> (core/storage/blob.py:514) with_backoff (core/utils/backoff.py:142) download_file (core/storage/blob.py:513) create_item (goes_glm.py:38) create_items (dataset/items/task.py:117) run (dataset/items/task.py:153) parse_and_run (task/task.py:53) run_task (task/run.py:138) run_cmd (task/_cli.py:32) run_cmd (task/cli.py:50) new_func (click/decorators.py:26) invoke (click/core.py:760) invoke (click/core.py:1404) invoke (click/core.py:1657) invoke (click/core.py:1657) main (click/core.py:1055) __call__ (click/core.py:1130) cli (cli/cli.py:140) <module> (pctasks:8) Thread 13 (idle): "fsspecIO" select (selectors.py:468) _run_once (asyncio/base_events.py:1884) run_forever (asyncio/base_events.py:607) run (threading.py:975) _bootstrap_inner (threading.py:1038) _bootstrap (threading.py:995) Thread 14 (active): "asyncio_0" _worker (concurrent/futures/thread.py:81) run (threading.py:975) _bootstrap_inner (threading.py:1038) _bootstrap (threading.py:995) ``` I think we only care about the `MainThread`. Setting the `timeout_seconds` on the `download_blob` request was apparently not sufficient. I don't know if setting the default socket timeout is sufficient, but it's worth a shot.
1 parent 16e666f commit f68832c

File tree

3 files changed

+10
-4
lines changed

3 files changed

+10
-4
lines changed

datasets/goes/goes-glm/dataset.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
id: goes_glm
2-
image: ${{ args.registry }}/pctasks-goes-glm:2023.4.17.0
2+
image: ${{ args.registry }}/pctasks-goes-glm:2023.4.24.0
33

44
args:
55
- registry

datasets/goes/goes-glm/goes_glm.py

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
1+
import socket
12
from pathlib import Path
23
from tempfile import TemporaryDirectory
34
from typing import List, Union
@@ -24,6 +25,11 @@ class GoesGlmCollection(Collection):
2425
def create_item(
2526
cls, asset_uri: str, storage_factory: StorageFactory
2627
) -> Union[List[pystac.Item], WaitTaskResult]:
28+
# Last-ditch attempt to enforce timeouts in the socket / ssl code
29+
timeout = socket.getdefaulttimeout()
30+
if timeout is None:
31+
socket.setdefaulttimeout(60)
32+
2733
nc_storage, nc_asset_path = storage_factory.get_storage_for_file(asset_uri)
2834

2935
if isinstance(nc_storage, pctasks.core.storage.blob.BlobStorage):

datasets/goes/goes-glm/workflows/goes-glm-update.yaml

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ jobs:
1414
id: create-splits
1515
tasks:
1616
- id: create-splits
17-
image: ${{ args.registry }}/pctasks-goes-glm:2023.4.17.0
17+
image: ${{ args.registry }}/pctasks-goes-glm:2023.4.24.0
1818
code:
1919
src: datasets/goes/goes-glm/goes_glm.py
2020
task: goes_glm:GoesGlmCollection.create_splits_task
@@ -53,7 +53,7 @@ jobs:
5353
id: create-chunks
5454
tasks:
5555
- id: create-chunks
56-
image: ${{ args.registry }}/pctasks-goes-glm:2023.4.17.0
56+
image: ${{ args.registry }}/pctasks-goes-glm:2023.4.24.0
5757
code:
5858
src: datasets/goes/goes-glm/goes_glm.py
5959
task: pctasks.dataset.chunks.task:create_chunks_task
@@ -74,7 +74,7 @@ jobs:
7474
id: process-chunk
7575
tasks:
7676
- id: create-items
77-
image: ${{ args.registry }}/pctasks-goes-glm:2023.4.17.0
77+
image: ${{ args.registry }}/pctasks-goes-glm:2023.4.24.0
7878
code:
7979
src: datasets/goes/goes-glm/goes_glm.py
8080
task: goes_glm:GoesGlmCollection.create_items_task

0 commit comments

Comments
 (0)