-
Notifications
You must be signed in to change notification settings - Fork 32
Open
Labels
a:pipeline-servicesissues on non-core services in user's pipeline (use osparc-services repo for specifics)issues on non-core services in user's pipeline (use osparc-services repo for specifics)a:simcore-sdka:webserverwebserver's codebase. Assigning the area is particularly useful for bugswebserver's codebase. Assigning the area is particularly useful for bugs
Description
#7725 brought a fix in that postgres pg_notify system cannot pass a too large payload. Nevertheless there are many shortcomings:
- the system runs in a special webserver replica that runs alone and cannot be duplicated --> solution is to use an exclusive locking mechanism that ensures only one run at a time but in multiple replicas of the standard webserver
- it still uses
aiopgand it should be brought toasyncpg - in case of a burst of changes (which happens often), there will be as many notifications coming in, and then as many call to update the project tables, and as many socket.io events which is a waste of resources
Another question is that if the legacy services were finally gone, one could replace that database notification system with one using a real message broker such as RabbitMQ.
- simcore-sdk would need to send an event notification in the RabbitMQ whenever the outputs is modified
- webserver would listen to these changes in a distributed way, therefore sharing the load
- burst calls would still be an issue though
Metadata
Metadata
Labels
a:pipeline-servicesissues on non-core services in user's pipeline (use osparc-services repo for specifics)issues on non-core services in user's pipeline (use osparc-services repo for specifics)a:simcore-sdka:webserverwebserver's codebase. Assigning the area is particularly useful for bugswebserver's codebase. Assigning the area is particularly useful for bugs