Method to execute commands/scripts before packaging the new gem

[sronsiek]

We need to provide pre-compiled gems without internet access. We download from rubygems.org, compile using gem compiler, and push the compiled gems to our local geminabox server. From here the gems are downloaded onto target machines that do not have compilers.

My problem is the passenger gem. This requires unpacking, then running a cmd which builds mod_passeneger.so. I want to include this file in the compiled gem.

gem compiler unpacks in a tmp area in a dir containing a time stamp - so I cannot unpack, compile & run gem compiler on the same dir hoping it will include mod_passenger.so.

Ideally I need a hook to be able to insert a file into the unpacked area before it is repackaged.

I can use gem unpack - but can I& place a file in the unpacked area and perhaps gem-compile the unpacked gem dir?

Or can I unpack & re-pack the gem-compiled gem file??

[luislavena]

Hello @sronsiek, thank you for your interest in gem-compiler.

I'm not familiar with passenger gem and where it compiles and places the binaries it produces, so not sure if it will be possible pack those binaries back into the gem.

From what I can deduce by looking at Passenger's gemspec (not the one in the repository, but the one obtained using gem spec gemfile), it uses extension trick to download some binaries, which are later extracted and wrapped into compilation by passenger-install-* commands.

From that, I don't think is possible package binaries for Passenger, at least not with this approach. gem-compiler works great when there is no extra steps to be performed for generating those binaries.

Please note that my experience with Passenger and it's setup is limited, so I might be missing something but this will require several things:

• A hook to run something (should be called before extensions are compiled or after?)
• Determine in which context these scripts need to be run (what about other gem dependencies?)
• An extra argument that allow specify which extra patterns to be added to the resulting gem

What you can do to prove this theory is perform the manual steps involved in this:

• Download: gem fetch passenger
• Extract: gem unpack passeger-X.Y.Z.gem
• Generate the spec (based on downloaded gem): gem spec --ruby passenger-X.Y.Z > passenger-X.Y.Z/passenger.gemspec
• Determine which extensions needs compilation: gem spec passenger-X.Y.Z extensions
• Compile extensions (cd into the directory and invoke the extconf.rb from previous step)
• Here, run now the command you wanted to automate (via the hooks)
• See if that command can be automated (no user interaction)
• Check the resulting artifacts and determine which files needs to be added to the gemspec.
• Add the files to the files section of the generated gemspec and remove the extensions listed.
• Package the gem: gem build passenger.gemspec
• Try the resulting gem in your system.

If that works, specially the manual hook and that can be automated, then we can figure out extra options to be added to gem-compiler.

If that doesn't work, then it means Passenger cannot produce pre-compiled binaries, in which case I will recommend reaching out to Passenger developers for assistance.

Will keep this open to hear your feedback once you try the above options.

Looking forward your commends.

[mpapis]

@FooBarWidget - would it possible to have a version of passenger gem that does the passenger-install-* during gem installation? This way gem-compiler would work without any changes ... maybe except of adding post install messages.

[sronsiek]

Hi Luis,

Thanks very much for your very informative post. I'm not an expert on passenger, but I can tell you that the passenger gem seems to have a two-part compilation process:

1. The standard gem compilation done automatically by gem compile or bundler. This, as you say attempts to download some archives, in my case:

Attempting to download https://oss-binaries.phusionpassenger.com/binaries/passenger/by_release/5.0.15/rubyext-ruby-2.1.2-x86_64-linux.tar.gz into /home/app_user/gem_build/cache/d20150831-18514-1u5772h/passenger-5.0.15/download_cache
*** Could not download https://oss-binaries.phusionpassenger.com/binaries/passenger/by_release/5.0.15/rubyext-ruby-2.1.2-x86_64-linux.tar.gz: The requested URL returned error: 404 Not Found
Attempting to download https://s3.amazonaws.com/phusion-passenger/binaries/passenger/by_release/5.0.15/rubyext-ruby-2.1.2-x86_64-linux.tar.gz into /home/app_user/gem_build/cache/d20150831-18514-1u5772h/passenger-5.0.15/download_cache
*** Could not download https://s3.amazonaws.com/phusion-passenger/binaries/passenger/by_release/5.0.15/rubyext-ruby-2.1.2-x86_64-linux.tar.gz: The requested URL returned error: 403 Forbidden
Attempting to download https://oss-binaries.phusionpassenger.com/binaries/passenger/by_release/5.0.15/nginx-1.8.0-x86_64-linux.tar.gz into /home/app_user/gem_build/cache/d20150831-18514-1u5772h/passenger-5.0.15/download_cache
Attempting to download https://oss-binaries.phusionpassenger.com/binaries/passenger/by_release/5.0.15/agent-x86_64-linux.tar.gz into /home/app_user/gem_build/cache/d20150831-18514-1u5772h/passenger-5.0.15/download_cache

However I have found that if these downloads fail (in my case they do fail because there are no pre-built archives for my ruby version), they are later built locally - so download failure does in fact not matter.

2 . In addition to the output from 1., the mod_passenger.so lib is needed - this seems deliberately excluded from the default build, and is performed via the passenger-install- commands. I'm not sure why it is done like this, you can specify apache or the nginx build (part of the script name) and you specify support for one or more different language supports eg ruby, python ... (run-time args). Maybe these are the reasons. Build results, incl quite a few intermediate object files are placed in the buildout directory tree within the unpacked gem structure:

passenger-5.0.15/buildout/apache2/mod_passenger.so

Just for info - the following is compiled as part of the extension is included in a gem-compiled gem:

passenger-5.0.15/buildout/support-binaries/PassengerAgent

So, in order to automate the entire process, one of the following should do the trick:

• Either a hook to run something (after ext compilation)
• or a hook to add files / dirs to the gem to be packaged (I saw a loop over artifacts in yr code - I assume this is something else.

The cmd in my case was:

cd passenger-5.0.15/bin
./passenger-install-apache2-module --auto --languages ruby,python,nodejs

The --auto option provides non-interactive execution.

I'm not familiar with gemspecs - do you need to modify this to add extra files?

However - in my particular case you have provided all the cmd steps for me to run individually when I encounter the passenger gem - so for now I'll try that approach, adding the extra cmd where required.

Having said that - I think many people would appreciate an all-in-one build soln!

cheers,
stefan

[FooBarWidget]

I'm currently on a trip in China, so it'll take a while before I can come back with an elaborate answer. But it's tricky, to say the least. The thing with Passenger is that we support a ton of different integrations with the OS.

• The user might be interested in Apache, or in Nginx, or in Standalone... there is no way to tell which one they want until they've declared their intention by running one of the commands.
• Worse, integration with Apache and Nginx requires further parameters, such as which Apache they want to compile for, where to install Nginx to, etc.
• The required dependencies are also different depending on the integration mode.
• And depending on the user's OS, it might be easier for them to use our Debian/RPM packages which also provides deeper OS integration (e.g. SELinux support, an Nginx package compatible with the OS version's).

To give you a good answer, I have to first look into what gem-compiler does and what you exactly want to achieve with the passenger gem. Until then, my above comments may serve as some food for thought.

[sronsiek]

I'm failing on the last step: gem build <gemspec> complains about the syntax of the gemspec file that was produced by gem spec:

 --- !ruby/object:Gem::Specification
                 ^
passenger-5.0.15/passenger.gemspec:2: syntax error, unexpected ':', expecting end-of-input
name: passenger

requiring yaml in irb returns true. Not sure what is wrong here ... gem version is 2.4.8

[luislavena]

Sorry for that, please try saving the gemspec again using "--ruby" option.

Forgot that the default is YAML but we need ruby code for "gem build" to
work.

[sronsiek]

Ok - I have it working - although the exact handling of passenger varies with version. I'm supporting 3.0.13 & 5.0.15. I followed Luis' instructions which were a very good guide - here my exact commands.
Assumption is the gem file exists in the current directory (collected with gem fetch).

GEM=passenger-5.0.15.gem   # passenger-3.0.13
PKG=`echo $GEM | sed 's/\.gem//'`

gem spec $GEM --ruby > ${PKG}/passenger.gemspec
if [ -n "`echo $GEM | grep 'passenger-3.0.13'`" ]; then
  # Apply some specific patches for as per:
  # https://github.com/phusion/passenger/commit/39cf482d1f2c3118021076362bd3f0a41750faa6
  # https://code.google.com/p/phusion-passenger/issues/detail?id=784
fi

# Run passenger build / install script:
cd $PKG/bin
if [ -n "`./passenger-install-apache2-module --help | grep '\-\-languages'`" ]; then
  ./passenger-install-apache2-module --auto --languages ruby,python,nodejs 
else
  ./passenger-install-apache2-module --auto
fi
cd ../..

# Ruby script to mod the gemspec file (see below)
cp ${HOME}/inputs/update_passenger_gemspec.rb .
./update_passenger_gemspec.rb $GEM

# Now build the compiled platform specific gem:
cd $PKG
gem build passenger.gemspec

# Move it to the orig working dir & remove the source gem & it's debris: 
mv *.gem ..
cd ..
rm -rf $PKG $GEM update_passenger_gemspec.rb

The ruby script looks as follows:

require "fileutils"
require "rubygems/installer"

gem =  ARGV.shift  # Skipped validation checks here
gem_dir = gem.sub /\.gem$/, ''
gem_name = gem_dir.sub( /\-[0-9.]*?$/, '')

Dir.chdir gem_dir

gs = Gem::Specification.load( gemspec_file )

gs.platform = Gem::Platform::CURRENT

if gs.extensions.any?
  target_dir = Dir.pwd
  installer = Gem::Installer.new( "../#{gem}" )
  installer.spec.full_gem_path = target_dir

  # Following just contains make log after build
  installer.spec.extension_dir = File.join(target_dir, "lib")

  installer.build_extensions
end


gs.extensions.clear

if gs.version.to_s =~ /^5\./
  gs.files << [ "buildout/apache2/mod_passenger.so",
                "buildout/apache2/module_libboost_oxt.a",
                "buildout/ruby/ruby-2.1.2-x86_64-linux/passenger_native_support.so",
                "buildout/support-binaries/PassengerAgent",
                "buildout/libev/.libs/libev.a",
                "buildout/libuv/.libs/libuv.a" ]
elsif gs.version.to_s =~ /^3\./

  gs.files << [ "ext/apache2/mod_passenger.so",
                "ext/apache2/module_libboost_oxt.a",
                "ext/ruby/ruby-2.1.2-x86_64-linux-gnu/passenger_native_support.so",
                "ext/libev/.libs/libev.a" ]
end

# gs.executables << "PassengerAgent"

# Overwite the source gemspec with the modified version:
File.open( gemspec_file, "w") { |file| file.write gs.to_ruby }

So basically - yes - a hook where you described would cater for this solution!

Now it's time for a beer!