In a Docker container, gems at a path can't be installed

[mrj]

When running bundle install in a Docker container, only Gemfile entries at a URL can be installed — Gems at a local path can't unless pre-copied, forcing a (possibly-premature) remote commit.

This is expected, because even if the container has been given access to the local file system, bundle is unaware of it.

One solution would be to add a --home-path option to bundle install, which provides the path into a mounted file system equivalent to the non-container Gemfile location, to which Gem paths are relative.

For example, if an entry in the Gemfile in the myapp directory is

gem 'xyz', path: '../xyz'

Then by mapping this myapp parent directory as the root of a file system mapped into the container at path /local

bundle install --home-path=/local/myapp

can now install paths like the above, now relative to the home-path setting.

Absolute paths would require mounting the whole file system. Interpreting ~ home directory path leaders would be helpful.

[deivid-rodriguez]

This does not seem specific to Bundler, right? Any script/code that references another file relatively will need the structure respected in the cointainer. If you don't want to copy it, can you not mount your local ../xyz as a ../xyz directory relative to the working directory where you are running bundle install?

[mrj]

Thanks David for looking at this.

You're right that if all path gems are in or below the Gemfile's parent directory, and the Gemfile has been copied into the container below its root — like Kamal-Rails does by putting it at /rails/Gemfile — then the copied Gemfile will be able to install the path gems if each of them has been mounted to the container root.

But this is not possible if any gem is located higher than the parent directory (you can't go above /), plus you'd like to only mount once rather than once for each path or path gem. Kamal could get around this by rewriting the Gemfile paths, but it's not nice for all packages that use bundler to have to do this.

So yes, because the paths are in the Gemfile, which is a bundler thing, bundler should be responsible for a solution.

Another advantage of bundler knowing about and having access to the ex-container Gemfile directory is that bundler can copy the Gemfile and Gemfile.lock files itself instead of Kamal having to do it.

[deivid-rodriguez]

You're right that if all path gems are in or below the Gemfile's parent directory, and the Gemfile has been copied into the container below its root — like Kamal-Rails does by putting it at /rails/Gemfile — then the copied Gemfile will be able to install the path gems if each of them has been mounted to the container root.

Note that path gems are not really installed, they just need to be available in the location pointed to by the Gemfile.

So I still don't understand what's wrong with my solution of mounting the necessary directory from the host into the container (docker run ... --mount type=bind,source=/local/myapp,target=/xyz ...), and I'm also unclear what you expect bundle install --home-path=/local/myapp would do.

[deivid-rodriguez]

But this is not possible if any gem is located higher than the parent directory (you can't go above /), plus you'd like to only mount once rather than once for each path or path gem. Kamal could get around this by rewriting the Gemfile paths, but it's not nice for all packages that use bundler to have to do this.

Alternatively I don't see the issue with copying dependent path gems at build time either. Can you explain again what would be the problem with copying directories outside of the Gemfile directory? What kind of error do you get?

[mrj]

David, Kamal copies the Gemfile and Gemfile.lock to the root of the container filesystem, then, with / as the working directory, bundle install is run to install the gems to /usr/local/bundle in the container.

If a gem in the Gemfile is localed at a path (in the external filesytem), the only mount that would make the same path work for both the external development app and the bundle install container command would be to have a path_gems directory in the same directory as the Gemfile (the app root), which Kamal must mount to the container root.

Now:

gem 'gem_name', path: 'path_gems/gem_name'

Will work both for development and for installation into the production container.

I suppose with symbolic links, gems from anywhere on the filesysytem can be handled this way.

Nicer would be have bundler manage this so that Gemfile paths can remain arbitrary. My suggestion of a --home-path option would allow mounting of any point in the external filesystem to any point in the container, while still allowing for external and container path-equivalence by stripping off this home-path container directory prefix.