⚗️ Changed `periodic_task_result` internals in the attempt of solving issue #7032

GitHK · 2025-01-13T16:19:27Z

What do these changes do?

Somehow after the periodic_task_result is done polling for the status it recovers the result and then raises an error error relative to the status retrieval done in the coroutine awaited by await_for.

This implementation is equivalent and tries to avoid the issue by making the situation impossible to occur.

"2025-01-13T10:15:07.608Z","osparc-srv03","log_level=INFO | log_timestamp=2025-01-13 10:15:07,608 | log_source=httpx:_send_single_request(1773) | log_uid=None | log_oec=None| log_trace_id=0 | log_span_id=0 | log_resource.service.name= | log_trace_sampled=False] | log_msg=HTTP Request: GET http://dy-sidecar_20a82fa9-0614-4a3b-84e0-48dd5935003c:8000/task/simcore_service_dynamic_sidecar.modules.long_running_tasks.task_ports_inputs_pull.8d6c2a62-4df6-43db-8310-0e5f6e6d4567 ""HTTP/1.1 200 OK"""
"2025-01-13T10:15:07.822Z","osparc-srv03","log_level=INFO | log_timestamp=2025-01-13 10:15:07,822 | log_source=httpx:_send_single_request(1773) | log_uid=None | log_oec=None| log_trace_id=b13b450573dfec1a316d59a11a490299 | log_span_id=fa2efb5f572b4a82 | log_resource.service.name= | log_trace_sampled=True] | log_msg=HTTP Request: GET http://dy-sidecar_20a82fa9-0614-4a3b-84e0-48dd5935003c:8000/task/simcore_service_dynamic_sidecar.modules.long_running_tasks.task_ports_inputs_pull.8d6c2a62-4df6-43db-8310-0e5f6e6d4567 ""HTTP/1.1 200 OK"""
"2025-01-13T10:15:07.833Z","osparc-srv03","log_level=INFO | log_timestamp=2025-01-13 10:15:07,833 | log_source=httpx:_send_single_request(1773) | log_uid=None | log_oec=None| log_trace_id=b13b450573dfec1a316d59a11a490299 | log_span_id=fa2efb5f572b4a82 | log_resource.service.name= | log_trace_sampled=True] | log_msg=HTTP Request: GET http://dy-sidecar_20a82fa9-0614-4a3b-84e0-48dd5935003c:8000/task/simcore_service_dynamic_sidecar.modules.long_running_tasks.task_ports_inputs_pull.8d6c2a62-4df6-43db-8310-0e5f6e6d4567/result ""HTTP/1.1 200 OK"""
"2025-01-13T10:15:08.629Z","osparc-srv03","log_level=INFO | log_timestamp=2025-01-13 10:15:08,629 | log_source=httpx:_send_single_request(1773) | log_uid=None | log_oec=None| log_trace_id=0 | log_span_id=0 | log_resource.service.name= | log_trace_sampled=False] | log_msg=HTTP Request: GET http://dy-sidecar_20a82fa9-0614-4a3b-84e0-48dd5935003c:8000/task/simcore_service_dynamic_sidecar.modules.long_running_tasks.task_ports_inputs_pull.8d6c2a62-4df6-43db-8310-0e5f6e6d4567 ""HTTP/1.1 404 Not Found"""
"2025-01-13T10:15:08.641Z","osparc-srv03","log_level=ERROR | log_timestamp=2025-01-13 10:15:08,634 | log_source=simcore_service_director_v2.modules.dynamic_sidecar.scheduler._core._observer:observing_single_service(155) | log_uid=19937 | log_oec=OEC:140568355058400| log_trace_id=0 | log_span_id=0 | log_resource.service.name= | log_trace_sampled=False] | log_msg=This service (dy-sidecar_20a82fa9-0614-4a3b-84e0-48dd5935003c) unexpectedly failed. Our team has recorded the issue and is working to resolve it as quickly as possible. Thank you for your patience..
{'context': ""{'action': 'getting_status',
""
            "" 'body': ""
            '\'{""code"":""long_running_task.task_not_found"",""message"":""No task '
            ""with '
""
            '         '
            ""'simcore_service_dynamic_sidecar.modules.long_running_tasks.task_ports_inputs_pull.8d6c2a62-4df6-43db-8310-0e5f6e6d4567 ""
            ""'
""
            '         \'found""}\',
'
            "" 'code': 'long_running_task.client.generic_error',
""
            ' \'message\': ""Unexpected error while \'getting_status\' for ""
'
            '            '
            '""\'simcore_service_dynamic_sidecar.modules.long_running_tasks.task_ports_inputs_pull.8d6c2a62-4df6-43db-8310-0e5f6e6d4567\': '
            '""
'
            ""            'status=404 '
""
            '            '
            '\'body={""code"":""long_running_task.task_not_found"",""message"":""No '
            ""'
""
            ""            'task with '
""
            '            '
            ""'simcore_service_dynamic_sidecar.modules.long_running_tasks.task_ports_inputs_pull.8d6c2a62-4df6-43db-8310-0e5f6e6d4567 ""
            ""'
""
            '            \'found""}\',
'
            "" 'service_name': ""
            ""'dy-sidecar_20a82fa9-0614-4a3b-84e0-48dd5935003c',
""
            "" 'status': 404,
""
            "" 'task_id': ""
            ""'simcore_service_dynamic_sidecar.modules.long_running_tasks.task_ports_inputs_pull.8d6c2a62-4df6-43db-8310-0e5f6e6d4567',
""
            "" 'user_id': 19937}"",
 'error_code': 'OEC:140568355058400',
 'exception_details': ""Unexpected error while 'getting_status' for ""
                      ""'simcore_service_dynamic_sidecar.modules.long_running_tasks.task_ports_inputs_pull.8d6c2a62-4df6-43db-8310-0e5f6e6d4567': ""
                      'status=404 '
                      'body={""code"":""long_running_task.task_not_found"",""message"":""No '
                      'task with '
                      'simcore_service_dynamic_sidecar.modules.long_running_tasks.task_ports_inputs_pull.8d6c2a62-4df6-43db-8310-0e5f6e6d4567 '
                      'found""}',
 'exception_type': '<class '
                   ""'servicelib.long_running_tasks._errors.GenericClientError'>"",
 'tip': 'Observation of '
        ""service_name='dy-sidecar_20a82fa9-0614-4a3b-84e0-48dd5935003c' ""
        'unexpectedly failed'}
Traceback (most recent call last):
  File ""/home/scu/.venv/lib/python3.11/site-packages/simcore_service_director_v2/modules/dynamic_sidecar/scheduler/_core/_observer.py"", line 139, in observing_single_service
    await _apply_observation_cycle(scheduler, scheduler_data)
  File ""/home/scu/.venv/lib/python3.11/site-packages/simcore_service_director_v2/modules/dynamic_sidecar/scheduler/_core/_observer.py"", line 69, in _apply_observation_cycle
    await dynamic_scheduler_event.action(app, scheduler_data)
  File ""/home/scu/.venv/lib/python3.11/site-packages/simcore_service_director_v2/modules/dynamic_sidecar/scheduler/_core/_events.py"", line 205, in action
    await create_user_services(app, scheduler_data)
  File ""/home/scu/.venv/lib/python3.11/site-packages/simcore_service_director_v2/modules/dynamic_sidecar/scheduler/_core/_events_user_services.py"", line 233, in create_user_services
    await sidecars_client.pull_service_input_ports(dynamic_sidecar_endpoint)
  File ""/home/scu/.venv/lib/python3.11/site-packages/simcore_service_director_v2/modules/dynamic_sidecar/api_client/_public.py"", line 411, in pull_service_input_ports
    transferred_bytes = await self._await_for_result(
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ""/home/scu/.venv/lib/python3.11/site-packages/simcore_service_director_v2/modules/dynamic_sidecar/api_client/_public.py"", line 306, in _await_for_result
    async with periodic_task_result(
  File ""/usr/local/lib/python3.11/contextlib.py"", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File ""/home/scu/.venv/lib/python3.11/site-packages/servicelib/fastapi/long_running_tasks/_context_manager.py"", line 114, in periodic_task_result
    await asyncio.wait_for(_wait_task_completion(), timeout=task_timeout)
  File ""/usr/local/lib/python3.11/asyncio/tasks.py"", line 489, in wait_for
    return fut.result()
           ^^^^^^^^^^^^
  File ""/home/scu/.venv/lib/python3.11/site-packages/servicelib/fastapi/long_running_tasks/_context_manager.py"", line 111, in _wait_task_completion
    task_status = await _status_update()
                  ^^^^^^^^^^^^^^^^^^^^^^
  File ""/home/scu/.venv/lib/python3.11/site-packages/servicelib/fastapi/long_running_tasks/_context_manager.py"", line 98, in _status_update
    task_status: TaskStatus = await client.get_task_status(task_id)
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ""/home/scu/.venv/lib/python3.11/site-packages/servicelib/fastapi/long_running_tasks/_client.py"", line 97, in request_wrapper
    async for attempt in AsyncRetrying(
  File ""/home/scu/.venv/lib/python3.11/site-packages/tenacity/asyncio/__init__.py"", line 166, in __anext__
    do = await self.iter(retry_state=self._retry_state)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ""/home/scu/.venv/lib/python3.11/site-packages/tenacity/asyncio/__init__.py"", line 153, in iter
    result = await action(retry_state)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File ""/home/scu/.venv/lib/python3.11/site-packages/tenacity/_utils.py"", line 99, in inner
    return call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File ""/home/scu/.venv/lib/python3.11/site-packages/tenacity/__init__.py"", line 398, in <lambda>
    self._add_action_func(lambda rs: rs.outcome.result())
                                     ^^^^^^^^^^^^^^^^^^^
  File ""/usr/local/lib/python3.11/concurrent/futures/_base.py"", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File ""/usr/local/lib/python3.11/concurrent/futures/_base.py"", line 401, in __get_result
    raise self._exception
  File ""/home/scu/.venv/lib/python3.11/site-packages/servicelib/fastapi/long_running_tasks/_client.py"", line 106, in request_wrapper
    return await request_func(zelf, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ""/home/scu/.venv/lib/python3.11/site-packages/servicelib/fastapi/long_running_tasks/_client.py"", line 150, in get_task_status
    raise GenericClientError(
servicelib.long_running_tasks._errors.GenericClientError: Unexpected error while 'getting_status' for 'simcore_service_dynamic_sidecar.modules.long_running_tasks.task_ports_inputs_pull.8d6c2a62-4df6-43db-8310-0e5f6e6d4567': status=404 body={""code"":""long_running_task.task_not_found"",""message"":""No task with simcore_service_dynamic_sidecar.modules.long_running_tasks.task_ports_inputs_pull.8d6c2a62-4df6-43db-8310-0e5f6e6d4567 found""}"

Related issue/s

How to test

Dev-ops checklist

No ENV changes or I properly updated ENV (read the instruction)

codecov · 2025-01-13T16:28:33Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 86.76%. Comparing base (4d732df) to head (57f6c32).
Report is 1 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #7032      +/-   ##
==========================================
- Coverage   87.56%   86.76%   -0.80%     
==========================================
  Files        1629     1299     -330     
  Lines       63454    55099    -8355     
  Branches     2047     1010    -1037     
==========================================
- Hits        55566    47809    -7757     
+ Misses       7552     7096     -456     
+ Partials      336      194     -142

Flag	Coverage Δ
integrationtests	`64.48% <ø> (+0.71%)`	⬆️
unittests	`84.51% <100.00%> (-1.45%)`	⬇️

Components	Coverage Δ
api	`∅ <ø> (∅)`
pkg_aws_library	`∅ <ø> (∅)`
pkg_dask_task_models_library	`∅ <ø> (∅)`
pkg_models_library	`∅ <ø> (∅)`
pkg_notifications_library	`∅ <ø> (∅)`
pkg_postgres_database	`∅ <ø> (∅)`
pkg_service_integration	`∅ <ø> (∅)`
pkg_service_library	`73.89% <100.00%> (ø)`
pkg_settings_library	`∅ <ø> (∅)`
pkg_simcore_sdk	`85.38% <ø> (ø)`
agent	`96.45% <ø> (ø)`
api_server	`90.54% <ø> (ø)`
autoscaling	`96.09% <ø> (ø)`
catalog	`90.66% <ø> (ø)`
clusters_keeper	`99.24% <ø> (ø)`
dask_sidecar	`91.26% <ø> (ø)`
datcore_adapter	`93.18% <ø> (ø)`
director	`76.51% <ø> (+0.08%)`	⬆️
director_v2	`91.30% <ø> (ø)`
dynamic_scheduler	`97.21% <ø> (ø)`
dynamic_sidecar	`89.75% <ø> (ø)`
efs_guardian	`90.46% <ø> (ø)`
invitations	`93.44% <ø> (ø)`
osparc_gateway_server	`∅ <ø> (∅)`
payments	`92.66% <ø> (ø)`
resource_usage_tracker	`89.52% <ø> (+0.11%)`	⬆️
storage	`89.57% <ø> (ø)`
webclient	`∅ <ø> (∅)`
webserver	`84.10% <ø> (-1.84%)`	⬇️

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4d732df...57f6c32. Read the comment docs.

sanderegg

ok

sonarqubecloud · 2025-01-17T06:22:47Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

refactor

631be1a

GitHK added this to the Singularity milestone Jan 13, 2025

GitHK self-assigned this Jan 13, 2025

GitHK added bug buggy, it does not work as expected t:maintenance Some planned maintenance work labels Jan 13, 2025

GitHK changed the title ~~♻️ Refactor periodic_task_result internals in the attempt of solving issue~~ ⚗️ Changed periodic_task_result internals in the attempt of solving issue Jan 13, 2025

GitHK marked this pull request as ready for review January 14, 2025 06:17

GitHK requested a review from pcrespov as a code owner January 14, 2025 06:17

GitHK requested review from giancarloromeo, matusdrobuliak66 and sanderegg January 14, 2025 06:17

sanderegg approved these changes Jan 14, 2025

View reviewed changes

giancarloromeo approved these changes Jan 14, 2025

View reviewed changes

pcrespov approved these changes Jan 14, 2025

View reviewed changes

Merge branch 'master' into pr-osparc-werner-issue

48b0c39

GitHK enabled auto-merge (squash) January 14, 2025 13:17

odeimaiz and others added 2 commits January 15, 2025 08:41

Merge branch 'master' into pr-osparc-werner-issue

f86e947

Merge branch 'master' into pr-osparc-werner-issue

57f6c32

GitHK merged commit 17f3111 into ITISFoundation:master Jan 17, 2025
91 of 93 checks passed

GitHK deleted the pr-osparc-werner-issue branch January 17, 2025 07:58

matusdrobuliak66 mentioned this pull request Jan 30, 2025

🚀 Pre-release master -> staging_Singularity1 #7082

Closed

16 tasks

matusdrobuliak66 mentioned this pull request Mar 6, 2025

🚀 Release v1.80.0 #7081

Closed

63 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

⚗️ Changed `periodic_task_result` internals in the attempt of solving issue #7032

⚗️ Changed `periodic_task_result` internals in the attempt of solving issue #7032

Uh oh!

GitHK commented Jan 13, 2025 •

edited

Loading

Uh oh!

codecov bot commented Jan 13, 2025 •

edited

Loading

Uh oh!

sanderegg left a comment

Uh oh!

sonarqubecloud bot commented Jan 17, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

⚗️ Changed periodic_task_result internals in the attempt of solving issue #7032

⚗️ Changed periodic_task_result internals in the attempt of solving issue #7032

Uh oh!

Conversation

GitHK commented Jan 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What do these changes do?

Related issue/s

How to test

Dev-ops checklist

Uh oh!

codecov bot commented Jan 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

sanderegg left a comment

Choose a reason for hiding this comment

Uh oh!

sonarqubecloud bot commented Jan 17, 2025

Quality Gate passed

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

⚗️ Changed `periodic_task_result` internals in the attempt of solving issue #7032

⚗️ Changed `periodic_task_result` internals in the attempt of solving issue #7032

GitHK commented Jan 13, 2025 •

edited

Loading

codecov bot commented Jan 13, 2025 •

edited

Loading