[Q&A] How to handle gem version conflicts in fluentd?

[RamHaridas]

What is a problem?

Scenario: Let's say that I have installed plugins at two separate paths /path/to/default and /path/to/custom.

• I have installed pluginA at /path/to/default that is dependent on gemA version 1.5 (less than 2.0)
• Then, I installed pluginB at /path/to/custom that is dependent on gemA version 2.3 (greater than 2.0)
• Now, I update the GEM_PATH environment variable GEM_PATH=/path/to/custom:/path/to/default
• This encourages fluentd to load both pluginA and pluginB at runtime.

But the default behavior of fluentd to load gems, would default it to go with the max version of gemA which is 2.3 (as confirmed here #5158 by @Watson1978).
Now, this may work well for pluginB but not for pluginA

I wanted to understand that what steps should be taken to overcome this dependency conflict and how we can overcome it?

Describe the configuration of Fluentd

No response

Describe the logs of Fluentd

No response

Environment

- Fluentd version: 1.19.0
- Fluent Package version: 6.0.0
- Operating system: Rocky 8
- Kernel version: 5.14

[Watson1978]

Basically, due to Ruby's language specification, there is no built-in mechanism to isolate namespaces if they conflict within the same process. All loaded gems share the same runtime environment.
(Since Ruby 4.0, it has the Namespace feature, it might be able to avoid the conflict)

Therefore, it is the responsibility of the plugin (or library) developers to implement their code within proper, unique namespaces to avoid such collisions.

[renancvitor]

Answer

This is a known limitation of how RubyGems resolves dependencies at runtime, and Fluentd inherits this behavior.

When multiple versions of the same gem exist in GEM_PATH, RubyGems will resolve and activate a single version per process, usually the highest version that satisfies any dependency. Once a gem is activated, another version of the same gem cannot be loaded in the same Ruby process. This is why pluginA ends up breaking when gemA 2.3 is selected.

Unfortunately, Fluentd does not provide native per-plugin dependency isolation, so this conflict cannot be fully solved within a single Fluentd process.

Below are the recommended and practical ways to handle this.

---

1. Use Bundler to lock dependencies (recommended)
   The most reliable approach is to run Fluentd under Bundler with a controlled Gemfile and Gemfile.lock.

This ensures:

• A deterministic set of gem versions
• No accidental resolution to newer incompatible versions

Example:

source "https://rubygems.org"

gem "fluentd", "~> 1.19.0"
gem "pluginA", "~> x.y"
gem "pluginB", "~> z.w"

Then install and run Fluentd via:

bundle install
bundle exec fluentd -c fluent.conf

This avoids relying on GEM_PATH resolution order and prevents version drift.

---

2. Run plugins in separate Fluentd instances
   If pluginA and pluginB require incompatible versions of the same dependency, the safest architectural solution is process isolation.

Options include:

• Run two Fluentd instances, each with its own gem set
• Separate them by:
  • Different users
  • Different directories
  • Different containers (Docker / Kubernetes)

Each instance can then safely load its required gem versions without conflict.

---

3. Containerize per dependency set (strong isolation)
   Using containers provides clean dependency boundaries:

• One container with:
  • pluginA + gemA < 2.0
• Another container with:
  • pluginB + gemA >= 2.0

This is the most future-proof approach in environments where plugins evolve independently.

---

4. Avoid relying on GEM_PATH ordering
   While manipulating GEM_PATH may appear to work, it is not a reliable solution. RubyGems dependency activation is global per process, and ordering does not guarantee compatibility once a gem is activated.

This approach is fragile and may break with:

• Plugin load order changes
• Fluentd upgrades
• RubyGems updates

Conclusion

In summary:

• Fluentd cannot load multiple versions of the same gem in one process
• This is a RubyGems limitation, not a Fluentd bug
• Proper solutions require dependency locking (Bundler) or process isolation

If plugin dependencies cannot be aligned, running separate Fluentd instances (or containers) is the only safe and supported solution.