Handling Garbage Collector's ServiceWaitingForManualInterventionError and Stale DS-Scheduler State and Directorv2 Interaction

[pcrespov]

Which deploy/s?

production aws (e.g. osparc.io)

Current Behavior

• The garbage collector is reporting the following error dynamic_scheduler.errors.ServiceWaitingForManualInterventionError on a specific service while running GC on the orphan services.
• Looking at the scheduler monitoring (e.g. https://monitoring.*/dynamic-scheduler/) we find
  
• Q: we do not know why? UNEXEPECTED_OUTME!?? How to debug this?
• Portainer shows that this service was constantly pending
• 
  • probably the machine was removed but the scheduler does not updated its state?
• We remove the service from the scheduler. The monitoring does not show any problem with the service anymore
• But the GC continues logging error
• We restart the directorv2
• No the GC does not log error anymore
• It is strange since we see that the GC is getting this information from the scheduler and not the directorv2
• We wonder whether the scheduler state still depends on something living int he drierctorv2 which is definitively misbehaving

An error related to garbage collection (GC) of orphan services was observed in the system. The root cause appeared to be a stale scheduler state that was not cleared properly, potentially due to inconsistencies between the dynamic-scheduler and directorv2 components. Manual intervention and a component restart were ultimately required to resolve the issue.

• Initial Symptom

  • The garbage collector reported the following recurring error:

    dynamic_scheduler.errors.ServiceWaitingForManualInterventionError
    

  • This occurred during GC runs on orphan services.
    

• Monitoring Clues

  • Inspection of the scheduler monitoring dashboard (https://monitoring.*/dynamic-scheduler/) revealed a failed service in an unexpected state:
    

    UNEXPECTED_OUTCOME!
    

  • This status was not self-explanatory and raised questions about its origin and meaning. We did not know how to debug this!??

• Container State

  • Portainer indicated that the service in question remained in a "pending" state indefinitely.
  • This suggested a possible issue where the node hosting the service might have been removed, yet the scheduler had not updated its internal state accordingly.
    

• Manual Cleanup Attempt

  • The affected service was manually removed from the scheduler.
  • After this action, the scheduler’s monitoring interface no longer displayed any issues related to the service.

• Persistence of GC Errors

  • Despite the manual cleanup, the garbage collector continued logging the same ServiceWaitingForManualInterventionError.

• Directorv2 Restart

  • Restarting the directorv2 component led to the GC error disappearing.
  • This was unexpected, given that GC operations were believed to rely solely on the scheduler’s state and not directorv2.

---

Analysis and Open Questions

• The error seems rooted in a mismatch or stale state between the dynamic-scheduler and directorv2.
• Although the scheduler was manually corrected, the GC continued to fail, suggesting some latent state still depended on directorv2.
• The resolution via directorv2 restart implies that the scheduler might query or synchronize with directorv2, directly or indirectly, for orphan service validation.
• The meaning and origin of the UNEXPECTED_OUTCOME! state is unclear and requires further information to diagnose the unexpected event! (e.g. well defined exepction handling and logging)

Recommendations

• Clarify and document the interaction between dynamic-scheduler, directorv2, and the GC system.
• Investigate how GC fetches service state and whether it caches any data from directorv2.
• Improve error messages like UNEXPECTED_OUTME! to be more informative or link them to internal state reasons
• Consider adding automatic reconciliation logic to detect and handle removed/missing nodes. (instead of "manual intervention")

[pcrespov]

Garbage Collector Core Task

The webserver's garbage collector runs as a background task with three main steps, implemented in collect_garbage() function 1 . The third step handles orphan service detection and removal.

Orphan Service Detection

The orphan detection logic is implemented in remove_orphaned_services() 2 . Here's how it works:

1. Gets running services: First queries the dynamic scheduler to get all currently running services 3
2. Identifies opened projects: Scans the Redis registry to find all projects that have active user sessions 4
3. Gets expected services: For each opened project, retrieves the list of node IDs that should be running 5
4. Finds orphans: Computes the difference between running services and expected services to identify orphans 6

Service Removal Process

When orphan services are detected, the garbage collector removes them through _remove_service() 7 . This function:

1. Determines save state: Checks if the service state should be saved based on user permissions and project access 8
2. Stops the service: Calls the dynamic scheduler to stop the service 9

Communication Targets

When closing orphan services, the garbage collector communicates with:

• Dynamic Scheduler Service: The primary communication target for stopping services via dynamic_scheduler_service.stop_dynamic_service() 9
• Redis Registry: For tracking user sessions and project states 10
• Database: For checking user roles and project access permissions 11

Deployment Configuration

The garbage collector runs as a separate service instance wb-garbage-collector in the Docker Swarm deployment 12 , configured with access to the same databases and message queues as the main webserver.

Notes

The garbage collector uses a conservative approach - it skips orphan removal if it cannot list nodes for any project to avoid accidentally removing legitimate services. The process runs with limited concurrency (max 2 concurrent calls) to avoid overwhelming the system during cleanup operations.

Wiki pages you might want to explore:

• Overview (ITISFoundation/osparc-simcore)
• Testing Framework (ITISFoundation/osparc-simcore)