install plugin in cluster env, InitPythonEnvironment will delete .venv dir causing race conditions and failures

[avtion]

Self Checks

To make sure we get to you in time, please check the following :)

• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:)
• "Please do not modify this template :) and fill in all the required fields."

Versions

1. dify-plugin-daemon Version 0.5.1
2. dify-api Version 1.10.1-fix.1

Describe the bug
When running multiple pods of dify-plugin-daemon in k8s, the InitPythonEnvironment() function can be executed concurrently by different pods for the same plugin. This race condition may cause the deleteVirtualEnvironment() function to be triggered while another pod is still initializing the Python virtual environment, resulting in the virtual environment being deleted during the initialization process. This leads to initialization failures and unpredictable behavior.

Root Cause
The issue is triggered by the handleNewLocalPlugins() function in the cluster mode:

1. Each pod runs startLocalMonitor() which calls handleNewLocalPlugins() every 30 seconds
2. handleNewLocalPlugins() lists all installed plugins from the shared installedBucket (cross-pod shared storage)
3. Each pod checks if the plugin is already running locally (c.localPluginRuntimes.Exists())
4. If not running locally, the pod calls LaunchLocalPlugin() -> InitEnvironment() -> InitPythonEnvironment()
5. Although LaunchLocalPlugin() has a localPluginInstallationLock, this lock is in-memory and only prevents concurrent launches within the same pod
6. Multiple pods can simultaneously pass this local lock check and all attempt to initialize the same plugin's Python environment in the shared storage
7. This causes race conditions where one pod's deleteVirtualEnvironment() destroys the environment being built by another pod

To Reproduce
Steps to reproduce the behavior:

1. Deploy dify-plugin-daemon with multiple replicas (e.g., 3+ pods) in a Kubernetes cluster
2. Share the same persistent volume or storage for plugin working directories across all pods
3. Install or update a plugin that triggers Python environment initialization (it take about 1 minute to initialize)
4. Observe that multiple pods attempt to initialize the Python environment simultaneously
5. See error: virtual environment gets deleted while being initialized, causing "virtual environment is invalid" errors

Expected behavior

• Only one pod should be allowed to initialize the Python environment for a specific plugin at a time
• Other pods should wait for the initialization to complete or timeout gracefully
• The initialization process should be protected by a distributed lock mechanism
• Log messages should be emitted when the virtual environment is being deleted and recreated

[dosubot]

Hi @avtion! I'm Dosu and I’m helping the dify-plugin-daemon team.

This is a known issue in cluster environments: when multiple dify-plugin-daemon pods share storage, they can concurrently run InitPythonEnvironment, which deletes and recreates the .venv directory. The in-memory lock (localPluginInstallationLock) only prevents concurrent initialization within a single pod, not across pods, so race conditions occur and environments get deleted mid-initialization, causing failures and "virtual environment is invalid" errors. There is currently no distributed locking for environment initialization—only a temporary distributed lock (using Redis) for plugin installation tasks, not for the Python environment setup itself (code reference, issue #527).

The open-source version does not officially support multi-instance cluster mode; this is acknowledged by maintainers, and the enterprise edition may have different features (issue #132). There are no built-in workarounds or best practices documented for this scenario. To avoid these race conditions, you can: run only a single replica, avoid sharing plugin working directories across pods, or implement your own distributed lock (e.g., using Redis) around environment initialization.

If you need true multi-pod support with shared storage, a distributed locking mechanism is required to serialize environment setup for each plugin.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

  

[avtion]

@dosu this is not about multi install or update plugin request, this is a new promble for 0.5.0 version ControlPanel watchdog functions.

[dosubot]

You're right—this isn't about concurrent install/update requests, but a new issue introduced by the ControlPanel watchdog logic in 0.5.0. The watchdog (specifically, startLocalMonitor and handleNewLocalPlugins) now scans for and auto-launches plugins every 30 seconds across all pods. In a cluster, this means multiple pods can simultaneously try to initialize the same plugin environment, each running InitPythonEnvironment and deleteVirtualEnvironment without any distributed lock. This causes race conditions where one pod deletes the .venv directory while another is still initializing it, leading to failures and "virtual environment is invalid" errors.

There’s currently no built-in distributed locking or coordination for this watchdog-triggered environment setup—only for plugin install/upgrade tasks. The only mitigations are to run a single replica, avoid shared plugin directories, or implement your own distributed lock around environment initialization. No config flags or hooks exist to disable or coordinate the watchdog in multi-pod setups, and no recent PRs address this gap.

If you need true multi-pod support, a distributed lock (e.g., via Redis) is required to serialize environment setup for each plugin.
Relevant code reference

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

  

[Yeuoly]

Hi, In a cluster environment, you should never share the cwd directory among pods, it definitely cases the race condition error, instead, you may separate them, only share directories like plugins plugin_packages etc.

[avtion]

@Yeuoly Thanks for your reply. Not sharing the cwd folder between pods is indeed a way to avoid concurrent operations by multiple pods, but it also introduces new issues.

If the cwd folder is not shared, it means that each Pod needs to individually unzip the difypkg file, download dependencies, and precompile during startup or when installing new plugins. In cases with a large number of plugins, this could lead to significantly longer startup times for each Pod.

At the same time, in the current Helm deployment method recommended by Dify, PLUGIN_WORKING_PATH usually points to /app/storage/cwd, while the mounted Persistent Volume directory is /app/storage. Splitting the cwd directory does not seem to be an easy task for previously deployed clusters.

https://github.com/douban/charts/blob/master/charts/dify/templates/deployment.yaml#L503
https://github.com/douban/charts/blob/master/charts/dify/templates/deployment.yaml#L503

Perhaps could consider introducing support for distributed locks in the InitPythonEnvironment process to avoid issues that may arise from multiple pods coexisting and potentially triggering deleteVirtualEnvironment when sharing the cwd directory.

[Yeuoly]

@avtion , it's by design, adding locks here the there causes so much issues in distributed systems, in cases:

• What if a plugin was initialised successfully on pod A but failed on pod B?
• What if a pod was killed by host, the cwd were created but not completed already?
• What if the cwd was deleted by pod A but pod B were still using it?
• ...

There were so much issues we need to handle, as a result, you need to use https://github.com/langgenius/dify-plugin-daemon/blob/main/docs/runtime/sri.md if you are hosting a huge system which requires many plugins or you can contact business service.

As the secondary solution, the local plugin runtime requires a long time startup, but it works, I guess you may increase the parallel startup counts here https://github.com/langgenius/dify-plugin-daemon/blob/main/internal/types/app/config.go#L103C2-L103C32, and use rolling updates on k8s