LHS sampling failed on osparc.io

[JavierGOrdonnez]

Long Story Short
Registered a new function (ID c99b206a-3f44-4c28-8ee6-5796a10cdc45) on osparc.io and tried to run an LHS sampling for it. After ~1min I got this error response: "Something went wrong on our end. We've been notified and will resolve this issue as soon as possible. Thank you for your patience. [OEC:f44bb58b61b7-1758818548120]

Will investigate & tag the appropriate people.

[JavierGOrdonnez]

Found it in Graylog, this was an error in the API server, please investigate @wvangeit @sanderegg @bisgaard-itis . Note that this function (ID above) is an osparc python runner, a 10MB zip file, and two jsonifiers -- should be rather lightweight to copy.

Something went wrong on our end. We've been notified and will resolve this issue as soon as possible. Thank you for your patience. [OEC:f44bb58b61b7-1758818548120].
{
  "exception_type": "<class 'TimeoutError'>",
  "exception_string": "",
  "exception_causes": "[CancelledError]''",
  "error_code": "OEC:f44bb58b61b7-1758818548120",
  "context": {},
  "tip": "Unexpected error"
}
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/asyncio/tasks.py", line 500, in wait_for
    return fut.result()
           ^^^^^^^^^^^^
  File "/home/scu/.venv/lib/python3.11/site-packages/aio_pika/patterns/rpc.py", line 416, in call
    return await future
           ^^^^^^^^^^^^
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/scu/.venv/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "/home/scu/.venv/lib/python3.11/site-packages/starlette/middleware/base.py", line 182, in __call__
    with recv_stream, send_stream, collapse_excgroups():
  File "/usr/local/lib/python3.11/contextlib.py", line 158, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/scu/.venv/lib/python3.11/site-packages/starlette/_utils.py", line 83, in collapse_excgroups
    raise exc
  File "/home/scu/.venv/lib/python3.11/site-packages/starlette/middleware/base.py", line 184, in __call__
    response = await self.dispatch_func(request, call_next)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/scu/.venv/lib/python3.11/site-packages/servicelib/fastapi/monitoring.py", line 56, in dispatch
    response = await call_next(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/scu/.venv/lib/python3.11/site-packages/starlette/middleware/base.py", line 159, in call_next
    raise app_exc
  File "/home/scu/.venv/lib/python3.11/site-packages/starlette/middleware/base.py", line 144, in coro
    await self.app(scope, receive_or_disconnect, send_no_error)
  File "/home/scu/.venv/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 63, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/home/scu/.venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/home/scu/.venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/home/scu/.venv/lib/python3.11/site-packages/starlette/routing.py", line 716, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/home/scu/.venv/lib/python3.11/site-packages/starlette/routing.py", line 736, in app
    await route.handle(scope, receive, send)
  File "/home/scu/.venv/lib/python3.11/site-packages/starlette/routing.py", line 290, in handle
    await self.app(scope, receive, send)
  File "/home/scu/.venv/lib/python3.11/site-packages/starlette/routing.py", line 78, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/home/scu/.venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/home/scu/.venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/home/scu/.venv/lib/python3.11/site-packages/starlette/routing.py", line 75, in app
    response = await f(request)
               ^^^^^^^^^^^^^^^^
  File "/home/scu/.venv/lib/python3.11/site-packages/fastapi/routing.py", line 302, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/scu/.venv/lib/python3.11/site-packages/fastapi/routing.py", line 213, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/scu/.venv/lib/python3.11/site-packages/simcore_service_api_server/api/routes/functions_routes.py", line 454, in map_function
    return await web_api_rpc_client.register_function_job_collection(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/scu/.venv/lib/python3.11/site-packages/simcore_service_api_server/services_rpc/wb_api_server.py", line 656, in register_function_job_collection
    return await functions_rpc_interface.register_function_job_collection(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/scu/.venv/lib/python3.11/site-packages/servicelib/logging_utils.py", line 536, in _async_wrapper
    result = await func_or_coro(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/scu/.venv/lib/python3.11/site-packages/servicelib/rabbitmq/rpc_interfaces/webserver/functions/functions_rpc_interface.py", line 507, in register_function_job_collection
    result = await rabbitmq_rpc_client.request(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/scu/.venv/lib/python3.11/site-packages/servicelib/rabbitmq/_client_rpc.py", line 100, in request
    return await asyncio.wait_for(awaitable, timeout=timeout_s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/tasks.py", line 502, in wait_for
    raise exceptions.TimeoutError() from exc
TimeoutError

[JavierGOrdonnez]

I will try:

• Running less samples (in case that originated the timeout)
• Creating the study & running it on sim4life.io -- in case it is about the capacity of the webserver

[JavierGOrdonnez]

More news - the sampling seems to be running correctly. Therefore, I dont know what the timeout is related to.

[wvangeit]

did you only press the run sampling button once? (i.e. was this the first try?)

[JavierGOrdonnez]

Yes

[JavierGOrdonnez]

Btw, I did try again with a 10-jobs sample and worked without issues.

[wvangeit]

Which version and type of the MMUX service is this?

[JavierGOrdonnez]

Now I just run again another 50-samples campaign and this one did submit successfully in about 40 seconds.

@wvangeit MMUX MOGA v1.5.5 on osparc.io

[JavierGOrdonnez]

I am also very often getting the error "could not fetch functions" in the first screen - so it seems the webserver in osparc.io is being a bit flaky in general.