Skip to content

Follow up to osparc.io incident #8491

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 16 commits into from
Oct 9, 2025
Merged

Follow up to osparc.io incident #8491

Changes from 1 commit
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
9e1a28d
use exponential backoff when polling long running tasks in api-server
bisgaard-itis Oct 9, 2025
1631ee8
add method for recording number of tasks in event loop in the prometh…
bisgaard-itis Oct 9, 2025
b8c2a10
record number of tasks in event loop in aiohttp and fastapi metrics h…
bisgaard-itis Oct 9, 2025
3950ea8
@wvangeit use wit_random_exponential
bisgaard-itis Oct 9, 2025
be831a3
add test in aiohttp case
bisgaard-itis Oct 9, 2025
3394eee
add test in fastapi case
bisgaard-itis Oct 9, 2025
cb7c615
Merge branch 'master' into use-backoff-polling-strategy-and-expose-ev…
bisgaard-itis Oct 9, 2025
df0e3bc
@pcrespov several metrics of event loop saturation
bisgaard-itis Oct 9, 2025
165dcd9
remove computation of number of running tasks as it can be intensive
bisgaard-itis Oct 9, 2025
087c578
@pcrespov change method name
bisgaard-itis Oct 9, 2025
5e6510d
@pcrespov remove asserts
bisgaard-itis Oct 9, 2025
fea9443
improve docs
bisgaard-itis Oct 9, 2025
160da14
Merge branch 'master' into use-backoff-polling-strategy-and-expose-ev…
bisgaard-itis Oct 9, 2025
3bfa20d
Merge branch 'master' into use-backoff-polling-strategy-and-expose-ev…
bisgaard-itis Oct 9, 2025
4e8098b
Merge branch 'master' into use-backoff-polling-strategy-and-expose-ev…
bisgaard-itis Oct 9, 2025
bba912f
Merge branch 'master' into use-backoff-polling-strategy-and-expose-ev…
bisgaard-itis Oct 9, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions services/api-server/src/simcore_service_api_server/services_http/webserver.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@
from tenacity.asyncio import AsyncRetrying
from tenacity.before_sleep import before_sleep_log
from tenacity.stop import stop_after_delay
from tenacity.wait import wait_fixed
from tenacity.wait import wait_exponential

from ..core.settings import WebServerSettings
from ..exceptions.backend_errors import (
Expand Down Expand Up @@ -253,7 +253,7 @@ async def _wait_for_long_running_task_results(self, lrt_response: httpx.Response

# GET task status now until done
async for attempt in AsyncRetrying(
wait=wait_fixed(0.5),
wait=wait_exponential(multiplier=0.5, min=0.5, max=30),
stop=stop_after_delay(_POLL_TIMEOUT),
reraise=True,
retry=retry_if_exception_type(TryAgain),
Expand Down
Loading