Properly allocate resources for helper containers (egress proxy, tracing, rclone) in dynamic services

[GitHK]

What do these changes do?

The problem

When a user starts an interactive service in oSPARC, the platform doesn't just start that one container — it also quietly starts several "helper" containers behind the scenes to make things work correctly:

• a proxy to be reachd form the internet
• 0 or more network proxies that control which external websites/services the container is allowed to reach ("egress proxy")
• monitoring/tracing collectors, if tracing is enabled
• a file-syncing helper for services that need to mount remote storage (rclone)

These helper containers use real CPU and memory on the compute node they run on, but until now the platform only reserved resources for the user services, not for its helpers — in other words, it under-counted how much room a service actually needs. This under-counting means the cluster can end up placing more services on a node than it actually has capacity for, which can lead to services running slower than expected or, in busier deployments, containers being killed by the system for using more resources than were accounted for.

What changed

• The platform now calculates the real, total resource footprint of a service — the main container plus all its helper containers — and reserves that full amount up front. This gives the cluster's scheduler accurate information to place services where they'll actually fit.
• The egress-proxy and tracing-collector helpers each contribute a fixed, pre-configured CPU/RAM footprint to that total (one entry per proxy/collector instance actually spawned).
• The rclone file-syncing helper no longer contributes a single fixed footprint for every service — its memory footprint is now computed per-service (see below), and that computed value is what's summed into the same total described above.
• Updated and added tests to confirm resources are now calculated and reserved correctly for all three helper types.

How the rclone helper's memory limit is sized

Previously, the rclone mount container always got the same fixed memory allocation, no matter how small or large the service it supported was — this could waste memory reservation on small services and still be too little for services that shuffle very large files.

Now, rclone's memory limit scales with the service it's attached to:

1. Take the memory limit of the largest container in that service (e.g. if a service is made up of multiple containers via docker-compose, the biggest one).
2. Use 10% of that value as the starting point.
3. Clamp the result between a floor of 0.5 GB and a ceiling of 10 GB.
4. CPU stays fixed at 1 core regardless of service size — rclone's mount/sync workload is I/O-bound, not CPU-bound, so scaling CPU with the service doesn't help.

Worked examples:

• Service's biggest container has a 2 GB limit → 10% = 200 MB → below the floor → rclone gets 0.5 GB.
• Service's biggest container has a 20 GB limit → 10% = 2 GB → within range → rclone gets 2 GB.
• Service's biggest container has a 200 GB limit → 10% = 20 GB → above the ceiling → rclone gets 10 GB.

This keeps small services from over-reserving memory for a helper they barely use, while still giving large, data-heavy services enough headroom for their file cache — all without ever exceeding a sane absolute ceiling.

User-facing impact

No visible change to how services look or behave day-to-day. This is a behind-the-scenes reliability and capacity-planning improvement: services get the resources they truly need — main container plus every helper it spawns — reducing the risk of slowdowns or out-of-memory/CPU-throttling issues caused by underestimated resource usage in production.

Related issue/s

• https://github.com/ITISFoundation/private-issues/issues/8

How to test

Dev-ops

[codecov]

Codecov Report

❌ Patch coverage is 60.86957% with 81 lines in your changes missing coverage. Please review.
✅ Project coverage is 85.26%. Comparing base (e2d7ce2) to head (30de5b9).

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #9397      +/-   ##
==========================================
- Coverage   87.68%   85.26%   -2.42%     
==========================================
  Files        2124     2126       +2     
  Lines       83922    84039     +117     
  Branches     1569     1569              
==========================================
- Hits        73583    71657    -1926     
- Misses       9913    11956    +2043     
  Partials      426      426              

Flag	Coverage Δ
integrationtests	63.71% <56.68%> (-0.01%)	⬇️
unittests	85.30% <69.56%> (-1.29%)	⬇️

Components	Coverage Δ
pkg_aws_library	95.33% <ø> (ø)
pkg_celery_library	76.60% <ø> (ø)
pkg_dask_task_models_library	90.73% <ø> (ø)
pkg_models_library	92.60% <100.00%> (-0.01%)	⬇️
pkg_notifications_library	∅ <ø> (∅)
pkg_postgres_database	90.13% <ø> (ø)
pkg_service_integration	72.99% <ø> (ø)
pkg_service_library	70.59% <0.00%> (-0.17%)	⬇️
pkg_settings_library	91.33% <100.00%> (+0.40%)	⬆️
pkg_simcore_sdk	86.30% <ø> (ø)
agent	93.43% <ø> (ø)
api_server	92.95% <ø> (ø)
autoscaling	95.29% <ø> (ø)
catalog	92.47% <ø> (ø)
clusters_keeper	98.48% <ø> (ø)
dask_sidecar	93.22% <ø> (ø)
datcore_adapter	98.08% <ø> (ø)
director	79.15% <100.00%> (+0.13%)	⬆️
director_v2	78.59% <56.52%> (-13.10%)	⬇️
dynamic_scheduler	95.82% <ø> (ø)
dynamic_sidecar	88.33% <100.00%> (-0.04%)	⬇️
efs_guardian	89.86% <ø> (ø)
invitations	91.63% <ø> (ø)
payments	92.17% <ø> (ø)
resource_usage_tracker	91.89% <ø> (-0.16%)	⬇️
storage	87.05% <ø> (ø)
webclient	∅ <ø> (∅)
webserver	82.77% <100.00%> (-4.34%)	⬇️

---

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e2d7ce2...30de5b9. Read the comment docs.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[pcrespov]

I'm surprised this wasn't handled from the start. I assumed accounting for helper container resources would be a core, extensible part of the dynamic-sidecar design (i.e. new "helper" services added later, like rclone, would automatically be factored into the resource calculation).

My mental model was: the director spawns a dy-sidecar, which starts a docker-compose bundle including the core, the "real" helper containers (rclone for volumes/copy, the reverse proxy for the front-end, egress proxy, tracing) and the "main" user service(s), at least one of which exposes a web server the proxy connects to. Since the dy-sidecar is part of the swarm, it should reserve resources for the whole bundle it brings along, not just the main service.

Some open questions before this goes out:

• Tier allocation: we have pre-defined machine tiers (S, M, L). What percentage of each tier is allocated to each component (main service, rclone, reverse proxy, egress proxy, tracing, and whatever comes later)?
• Impact on constrained tiers: if rclone is capped tightly on tier S, does that slow down copying? Does it still work reliably? I'm assuming it has minimal specs, but does this mean the tiers themselves need to be revisited (and by how much)?

I'd want these questions addressed before release to avoid surprises in production.
I highty encourange you to critically review this with @mguidon @Konohana0608 since they are the main service creators and anticipate pbolems!!

[pcrespov]

@GitHK , if you do not re-request for review, probably @sanderegg will never notice your question ...

[GitHK]

There are currently two resource allocations strategies that can be adopted

1. subtract from total from the resources allocated to the user service subtract requirements for all the helpers

• 👍 service fits on the same machine
• 👎 rclone depending on the service, rclone might get too little resources and have issues (this will manifest while the service is running)
• 👎 can cause existing services to behave unexpectedly due to too many resources used (hard to detect)

2. add on top of total allocate extra resources for each helper

• 👎 uses more resources, service might not fit on the same machine types (easy to detect, service will not start or less services fit on the same machine)
• 👍 service works as before, no further testing is required
• 👍 rclone can be better controlled by enforcing a reasonable minimum amount of resources, no longer risks user service resource depletion

I have considered your feedback @pcrespov and @sanderegg and compiled a list of pros and cons for the two viable approaches for construing helper services.

I think the second approach is better.
PC's point about fitting services on machines is still valid, but is more preferable than the alternatives.
With this approach less services will fit on the machines, but overall the likelihood of errors is reduced.

[mguidon]

So from my perspective, two things are affected by this:

1. Service that are provisioned with a dedicate machine (sim4life/s4l-jupyter). Here we should properly recalculate the actual available resources. They surface on the Tiers description via the Pricing Plans. In those cases we typically have enough CPU resources but potentialy lack enough RAM. So its good that we cap rclone. I assume the hard limit is not gonna kill the container but you also pass on this limit to rclone itself?
2. Services that share resources (s4l-lite). Here we need to double check how many services we can run concurrently on the ec2 instances. It used to be exactly 6 but I already saw that it maxes out with 5. Additionally, we removed one sub-service from the sim4life stack. Also, I think we can skip rclone on the s4l-lite deployment enirely in case this eats too much RAM.

[GitHK]

@mguidon let me reply here before I forget

So from my perspective, two things are affected by this:

1. Service that are provisioned with a dedicate machine (sim4life/s4l-jupyter). Here we should properly recalculate the actual available resources. They surface on the Tiers description via the Pricing Plans. In those cases we typically have enough CPU resources but potentialy lack enough RAM. So it's good that we cap rclone. I assume the hard limit is not gonna kill the container but you also pass on this limit to rclone itself?

We assign the entire footprint (user services + helper containers) to the dynamic-sidecar, from this we remove what is allocated for each container, which means that if the service starts on the machine, there are no more possible surprises regarding resource allocation. Everything is now limited.

2. Services that share resources (s4l-lite). Here we need to double check how many services we can run concurrently on the ec2 instances. It used to be exactly 6 but I already saw that it maxes out with 5. Additionally, we removed one sub-service from the sim4life stack. Also, I think we can skip rclone on the s4l-lite deployment enirely in case this eats too much RAM.

I agree with you here that we can cut rclone, especially since we also limit the disk for them to 5GB.
Also, yes we need to check how many will fit on the machine since the egress proxies will require a bit of resources as well.

[GitHK]

@sanderegg I'm missing one point here that I think I did not manage to get across.

When a dynamic service is started, its resources are provided by the catalog. What happens with an autoscaled instance?
Does it get RAM & CPU equal to the amount available on the host, minus the ones reserved for the OPS services (that is already deducted)? If this is true then the service will never fit, since I add on top of these more resources so the machine's physical size will be smaller than the one requested by the service.

The question would be also, what would be the ideal way to deal with it if you are a creator. What would you like to see? I guess MAG can answer here.

[sanderegg]

@sanderegg I'm missing one point here that I think I did not manage to get across.

When a dynamic service is started, its resources are provided by the catalog. What happens with an autoscaled instance? Does it get RAM & CPU equal to the amount available on the host, minus the ones reserved for the OPS services (that is already deducted)? If this is true then the service will never fit, since I add on top of these more resources so the machine's physical size will be smaller than the one requested by the service.

The question would be also, what would be the ideal way to deal with it if you are a creator. What would you like to see? I guess MAG can answer here.

@GitHK

Autoscaling does not change whatever is asked from the docker services ever. What it does is:

• in billable mode, the sidecar defines what EC2 type it requires + reservations/limits. The autoscaling service then only checks that the available resources on that machine (minus the System/OPS parts - the common code that is currently in use and that you should still use) are enough for what the sidecar requires. The director-v2 in this case adjusts the resources given by the catalog.
• in non-billable mode, the sidecar only defines reservations/limits. The autoscaling service looks for the smallest EC2 available that can fit the requirements if there is no other EC2 already running that can fit the case. In this case, the director-v2 does NOT adjust anything.

[sanderegg]

this looks much better. Still some questions on the high overview.

[sanderegg]

in a sidecar pattern, I believe these are all sidecars actually.
I propose to either call these SIDECARS or SIDECAR_HELPERS to reduce confusion

[sanderegg]

can we discuss these noqa that you add here. I believe we should generally not require these anywhere. only for exceptional parts.

[sanderegg]

same can we check together?

[sanderegg]

? what is this?

[sanderegg]

what is the _range function for? this looks re-inventing stuff that the faker already does

[sanderegg]

minor: I guess since this is a wrapper we do not really need to know that caddy is inside

[sanderegg]

this is still a problem here

[sanderegg]

why you need to compute it twice?

[sanderegg]

this docstring is not very necessary I guess?

[sanderegg]

maybe adding a warning could be nice here.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud