Add semi-automatic migration from classic to modern dy-services by mrnicegyu11 · Pull Request #160 · ITISFoundation/osparc-services

mrnicegyu11 · 2025-04-17T14:51:16Z

What do these changes do?

Adds makefiles, .osparc. folder with jinja2 templates and further tooling
Migrates jupyter-base-notebook, raw-graphs and tissue-properties

Documentation on some peculiarities is needed:

input, output, state path mapping needs to be done manually
the folder name in the repo must match the docker image name
there is no validation / testing at this point (manually on osparc-master-speag)
some images have unpinned dependencies, this must be addressed / fixed manually

Related issue number

Checklist

I think the code is well written
I updated the table of contents (make toc)
Unit tests for the changes exist
Documentation reflects the changes
Runs in the swarm
If you design a new module, add your user to .github/CODEOWNERS

pcrespov · 2025-04-19T11:00:58Z

.osparc/jupyter-base-notebook/Dockerfile

+# ----------------------------------------------------------------
+# install simcore-sdk
+
+RUN pip install git+https://github.com/ITISFoundation/osparc-simcore.git@9143165172586f205d0fae63aa2c4ebffa79428b#subdirectory=packages/service-library &&\


I understand that the goal to upgrade to "modern dy-services" is to remove the libraries that communicate directly with the syste. Then could you clarify why these dependencies (simcore-sdk, service-library, ...) are still required? Additionally, could you provide more information on the necessity of the simcore-stack env vars like POSTGRES_* above?

Ah good point, I didn't think it through from this perspective. We might be able to remove those libraries all together maybe

yeah all that stuff should not be needed

odeimaiz · 2025-04-23T16:24:15Z

I didn't go through all the changes... but it's mostly about moving files around and renamings, right?

I guess that in the next iteration we will need to go service by service and make them work with the new filesystem hierarchy.

sanderegg

this starts to look very nice. Thanks for the effort!

sanderegg · 2025-04-22T06:27:26Z

.osparc/jupyter-base-notebook/Dockerfile

+ENV SIMCORE_NODE_UUID="-1" \
+    SIMCORE_USER_ID="-1" \
+    SIMCORE_NODE_BASEPATH="" \
+    SIMCORE_NODE_APP_STATE_PATH="/home/jovyan/notebooks" \
+    STORAGE_ENDPOINT="=1" \
+    S3_ENDPOINT="=1" \
+    S3_ACCESS_KEY="-1" \
+    S3_SECRET_KEY="-1" \
+    S3_BUCKET_NAME="-1" \
+    POSTGRES_ENDPOINT="-1" \
+    POSTGRES_USER="-1" \
+    POSTGRES_PASSWORD="-1" \
+    POSTGRES_DB="-1"


if it becomes dy-sidecar then all these should disappear I believe

sanderegg · 2025-04-22T06:29:18Z

.osparc/jupyter-base-notebook/Dockerfile

+# ----------------------------------------------------------------
+# install simcore-sdk
+
+RUN pip install git+https://github.com/ITISFoundation/osparc-simcore.git@9143165172586f205d0fae63aa2c4ebffa79428b#subdirectory=packages/service-library &&\


yeah all that stuff should not be needed

sanderegg · 2025-04-22T06:30:58Z

.osparc/raw-graphs/Dockerfile

+# install simcore-sdk
+RUN apk add --no-cache --virtual .build-deps \
+    g++ \
+    git \
+    python3-dev \
+    postgresql-dev && \
+    pip install --no-cache-dir git+https://github.com/ITISFoundation/osparc-simcore.git@ba8279cc7927929c7aa96232bc3ce1fd6ea6be55#subdirectory=packages/service-library &&\
+    pip install --no-cache-dir git+https://github.com/ITISFoundation/osparc-simcore.git@ba8279cc7927929c7aa96232bc3ce1fd6ea6be55#subdirectory=packages/simcore-sdk &&\
+    pip install --no-cache-dir git+https://github.com/ITISFoundation/osparc-simcore.git@ba8279cc7927929c7aa96232bc3ce1fd6ea6be55#subdirectory=services/storage/client-sdk/python &&\
+    pip install --no-cache-dir git+https://github.com/ITISFoundation/osparc-simcore.git@ba8279cc7927929c7aa96232bc3ce1fd6ea6be55#subdirectory=packages/postgres-database &&\
+    pip install --no-cache-dir git+https://github.com/ITISFoundation/osparc-simcore.git@ba8279cc7927929c7aa96232bc3ce1fd6ea6be55#subdirectory=packages/models-library &&\
+    apk del --no-cache .build-deps
+


sanderegg · 2025-04-22T06:31:08Z

.osparc/raw-graphs/Dockerfile

+ENV SIMCORE_NODE_UUID="-1" \
+    SIMCORE_USER_ID="-1" \
+    SIMCORE_NODE_BASEPATH="/raw" \
+    STORAGE_ENDPOINT="=1" \
+    RAWGRAPHS_INPUT_PATH="../inputs" \
+    RAWGRAPHS_OUTPUT_PATH="../outputs" \
+    S3_ENDPOINT="=1" \
+    S3_ACCESS_KEY="-1" \
+    S3_SECRET_KEY="-1" \
+    S3_BUCKET_NAME="-1" \
+    POSTGRES_ENDPOINT="-1" \
+    POSTGRES_USER="-1" \
+    POSTGRES_PASSWORD="-1" \
+    POSTGRES_DB="-1"


sanderegg · 2025-04-22T06:32:20Z

.osparc/raw-graphs/Dockerfile

+# --------------- Healthcheck -------------------
+COPY --chown=scu:scu scripts/docker/healthcheck.py $HOME/healthcheck/healthcheck.py
+# will start to run after interval every interval. fails after timeout. fail do not count if during start-period. will do # retries
+HEALTHCHECK --interval=10s --timeout=30s --start-period=1s --retries=3 CMD [ "python3", "/home/scu/healthcheck/healthcheck.py", "http://localhost:4000" ]


if you are setting the same healthcheck everywhere. I would suggest a start-period of 3-5 seconds. As soon as the healthcheck returns a 0 exit code, the start period is completed. so there is no drawback of setting a larger value

sanderegg · 2025-04-27T19:36:33Z

.osparc/3d-viewer-gpu/Dockerfile

+# set up oSparc env variables
+ENV SIMCORE_NODE_UUID="-1" \
+    SIMCORE_USER_ID="-1" \
+    SIMCORE_NODE_BASEPATH="" \
+    SIMCORE_NODE_APP_STATE_PATH=${PARAVIEW_INPUT_PATH} \
+    SIMCORE_HOST_NAME="localhost" \
+    STORAGE_ENDPOINT="=1" \
+    S3_ENDPOINT="=1" \
+    S3_ACCESS_KEY="-1" \
+    S3_SECRET_KEY="-1" \
+    S3_BUCKET_NAME="-1" \
+    POSTGRES_ENDPOINT="-1" \
+    POSTGRES_USER="-1" \
+    POSTGRES_PASSWORD="-1" \
+    POSTGRES_DB="-1"


all this should disappear

sanderegg · 2025-04-27T19:38:13Z

.osparc/3d-viewer-gpu/Dockerfile

+#------------------------------------------
+FROM common AS production
+# will start to run after interval every interval. fails after timeout. fail do not count if during start-period. will do # retries
+HEALTHCHECK --interval=10s --timeout=30s --start-period=1s --retries=3 CMD [ "python3", "/home/root/docker/healthcheck_curl_host.py", "http://localhost", "/" ]


--start-period=1s without changing --start-interval makes no sense. I would suggest --start-period=20s --start-interval=1s that means during 20s it will check every 1s if the service answers. If yes then it goes into standard 10s with timeout 30 3x times.

sanderegg · 2025-04-27T19:39:41Z

.osparc/jupyter-base-notebook/Dockerfile

+ENV NB_UID=8004
+ENV NB_GID=100
+RUN usermod  -u $NB_UID $NB_USER


are you sure about these? the notebooks used to have a jovyan user.

sanderegg · 2025-04-27T19:40:17Z

.osparc/jupyter-base-notebook/Dockerfile

-COPY --chown=jovyan:users scripts/docker/healthcheck_curl_host.py /healthcheck/healthcheck_curl_host.py
+COPY --chown=1000:100 scripts/docker/healthcheck_curl_host.py /healthcheck/healthcheck_curl_host.py
 # will start to run after interval every interval. fails after timeout. fail do not count if during start-period. will do # retries
 HEALTHCHECK --interval=10s --timeout=30s --start-period=1s --retries=3 CMD [ "python3", "/healthcheck/healthcheck_curl_host.py", "http://localhost:8888" ]


healthcheck should be adjusted (see my previous comment)

sanderegg · 2025-04-27T19:41:13Z

.osparc/raw-graphs/Dockerfile

+RUN apk add --no-cache --virtual .build-deps \
+    g++ \
+    git \
+    python3-dev \
+    postgresql-dev && \
+    pip install --no-cache-dir git+https://github.com/ITISFoundation/osparc-simcore.git@ba8279cc7927929c7aa96232bc3ce1fd6ea6be55#subdirectory=packages/service-library &&\
+    pip install --no-cache-dir git+https://github.com/ITISFoundation/osparc-simcore.git@ba8279cc7927929c7aa96232bc3ce1fd6ea6be55#subdirectory=packages/simcore-sdk &&\
+    pip install --no-cache-dir git+https://github.com/ITISFoundation/osparc-simcore.git@ba8279cc7927929c7aa96232bc3ce1fd6ea6be55#subdirectory=services/storage/client-sdk/python &&\
+    pip install --no-cache-dir git+https://github.com/ITISFoundation/osparc-simcore.git@ba8279cc7927929c7aa96232bc3ce1fd6ea6be55#subdirectory=packages/postgres-database &&\
+    pip install --no-cache-dir git+https://github.com/ITISFoundation/osparc-simcore.git@ba8279cc7927929c7aa96232bc3ce1fd6ea6be55#subdirectory=packages/models-library &&\
+    apk del --no-cache .build-deps


this should go away

mrnicegyu11 added 4 commits April 17, 2025 11:21

init: tissue properties

90d17b9

init: tissue properties

db13562

Upgrade raw-graphs and tissue-properties

cc8a919

do jupyter-base

b357813

mrnicegyu11 requested review from odeimaiz, pcrespov and sanderegg as code owners April 17, 2025 14:51

pcrespov reviewed Apr 19, 2025

View reviewed changes

mrnicegyu11 added 4 commits April 23, 2025 13:09

Progress

5d2ba4e

Fix jupyter base notebook permissions

349f468

Make 3D Viewer build

2e917f3

3d viewer changes

3072ec2

odeimaiz approved these changes Apr 23, 2025

View reviewed changes

add tissue properties fixes

248f24c

sanderegg reviewed Apr 27, 2025

View reviewed changes

mrnicegyu11 added 3 commits May 5, 2025 16:22

Fix tissue properties inputs outputs

cfba726

Fix 3d-viewer-gpu

53e2042

MAG change request

94cb5b5

Conversation

mrnicegyu11 commented Apr 17, 2025

What do these changes do?

Related issue number

Checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

odeimaiz commented Apr 23, 2025

Uh oh!

sanderegg left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants