Latest update breaking ruby 2.1.9 and mysql

[sAnti09]

Used ruby-build as part of the docker sequence commands in building a rails image. The ci was fine until July 20, where it produced the error below:

> /root/.rbenv/versions/2.1.9/lib/ruby/gems/2.1.0/extensions/x86_64-linux/2.1.0-static/mysql2-0.3.21/mysql2/mysql2.so: [BUG] Segmentation fault at 0x00000000000000
> ruby 2.1.9p490 (2016-03-30 revision 54437) [x86_64-linux]
>
> 
> -- Control frame information -----------------------------------------------
> c:0018 p:-17543132468098 s:0061 e:000060 TOP    [FINISH]
> c:0017 p:---- s:0059 e:000058 CFUNC  :require
> c:0016 p:0265 s:0055 e:000054 TOP    /root/.rbenv/versions/2.1.9/lib/ruby/gems/2.1.0/gems/mysql2-0.3.21/lib/mysql2.rb:31 [FINISH]
> c:0015 p:---- s:0052 e:000051 CFUNC  :require
> c:0014 p:0039 s:0048 e:000047 BLOCK  /root/.rbenv/versions/2.1.9/lib/ruby/gems/2.1.0/gems/bundler-1.17.3/lib/bundler/runtime.rb:81 [FINISH]
> c:0013 p:---- s:0044 e:000043 CFUNC  :each
> c:0012 p:0053 s:0041 e:000040 BLOCK  /root/.rbenv/versions/2.1.9/lib/ruby/gems/2.1.0/gems/bundler-1.17.3/lib/bundler/runtime.rb:76 [FINISH]
> c:0011 p:---- s:0035 e:000034 CFUNC  :each
> c:0010 p:0032 s:0032 e:000031 METHOD /root/.rbenv/versions/2.1.9/lib/ruby/gems/2.1.0/gems/bundler-1.17.3/lib/bundler/runtime.rb:65
> c:0009 p:0013 s:0028 e:000027 METHOD /root/.rbenv/versions/2.1.9/lib/ruby/gems/2.1.0/gems/bundler-1.17.3/lib/bundler.rb:114
> c:0008 p:0100 s:0024 e:000023 TOP    /app/config/application.rb:10 [FINISH]
> c:0007 p:---- s:0022 e:000021 CFUNC  :require
> c:0006 p:0014 s:0018 e:000017 BLOCK  /root/.rbenv/versions/2.1.9/lib/ruby/gems/2.1.0/gems/railties-3.2.22.5/lib/rails/commands.rb:53 [FINISH]
> c:0005 p:---- s:0015 e:000014 CFUNC  :tap
> c:0004 p:0504 s:0012 e:000011 TOP    /root/.rbenv/versions/2.1.9/lib/ruby/gems/2.1.0/gems/railties-3.2.22.5/lib/rails/commands.rb:50 [FINISH]
> c:0003 p:---- s:0008 e:000007 CFUNC  :require
> c:0002 p:0045 s:0004 E:001888 EVAL   script/rails:6 [FINISH]
> c:0001 p:0000 s:0002 E:001ca8 TOP    [FINISH]
> 
> -- Ruby level backtrace information ----------------------------------------
> script/rails:6:in `<main>'
> script/rails:6:in `require'
> /root/.rbenv/versions/2.1.9/lib/ruby/gems/2.1.0/gems/railties-3.2.22.5/lib/rails/commands.rb:50:in `<top (required)>'
> /root/.rbenv/versions/2.1.9/lib/ruby/gems/2.1.0/gems/railties-3.2.22.5/lib/rails/commands.rb:50:in `tap'
> /root/.rbenv/versions/2.1.9/lib/ruby/gems/2.1.0/gems/railties-3.2.22.5/lib/rails/commands.rb:53:in `block in <top (required)>'
> /root/.rbenv/versions/2.1.9/lib/ruby/gems/2.1.0/gems/railties-3.2.22.5/lib/rails/commands.rb:53:in `require'
> /app/config/application.rb:10:in `<top (required)>'
> /root/.rbenv/versions/2.1.9/lib/ruby/gems/2.1.0/gems/bundler-1.17.3/lib/bundler.rb:114:in `require'
> /root/.rbenv/versions/2.1.9/lib/ruby/gems/2.1.0/gems/bundler-1.17.3/lib/bundler/runtime.rb:65:in `require'
> /root/.rbenv/versions/2.1.9/lib/ruby/gems/2.1.0/gems/bundler-1.17.3/lib/bundler/runtime.rb:65:in `each'
> /root/.rbenv/versions/2.1.9/lib/ruby/gems/2.1.0/gems/bundler-1.17.3/lib/bundler/runtime.rb:76:in `block in require'
> /root/.rbenv/versions/2.1.9/lib/ruby/gems/2.1.0/gems/bundler-1.17.3/lib/bundler/runtime.rb:76:in `each'
> /root/.rbenv/versions/2.1.9/lib/ruby/gems/2.1.0/gems/bundler-1.17.3/lib/bundler/runtime.rb:81:in `block (2 levels) in require'
> /root/.rbenv/versions/2.1.9/lib/ruby/gems/2.1.0/gems/bundler-1.17.3/lib/bundler/runtime.rb:81:in `require'
> /root/.rbenv/versions/2.1.9/lib/ruby/gems/2.1.0/gems/mysql2-0.3.21/lib/mysql2.rb:31:in `<top (required)>'
> /root/.rbenv/versions/2.1.9/lib/ruby/gems/2.1.0/gems/mysql2-0.3.21/lib/mysql2.rb:31:in `require'

Versions

• Ubuntu 18.04.6 LTS
• OpenSSL 1.1.1 11 Sep 2018
• rbenv 1.2.0-16-gc4395e5
• ruby 2.1.9p490 (2016-03-30 revision 54437) [x86_64-linux]
• Rails 3.2.22.5
• mysql2-0.3.21

Temporary solution

Updated the command sequence w/c was part of building the image via rbenv and ruby-build

git clone https://github.com/rbenv/rbenv.git ~/.rbenv
&& echo 'export PATH="$PATH"' >> ~/.bashrc
&& echo 'eval "$(rbenv init -)"' >> ~/.bashrc
&& /bin/bash -c "source ~/.bashrc"
&& git clone --branch v20220630 https://github.com/rbenv/ruby-build.git ~/.rbenv/plugins/ruby-build
&& RUBY_CONFIGURE_OPTS="$RUBY_CONFIGURE_OPTS" rbenv install $RBENV_VERSION
&& rbenv global $RBENV_VERSION
&& echo "gem: --no-document" > ~/.gemrc
&& gem install bundler -v $BUNDLER_VERSION

[eregon]

Could you provide a Dockerfile to reproduce this?
Otherwise it's likely too difficult to reproduce.

2.1.* is long EOL and unsupported, so if this turns out to be an issue in CRuby there is no chance to fix it.

The 2.1.0-static sounds interesting, I didn't expect to see static in the path there. Maybe some special flag when installing the gem/installing ruby?

[sAnti09]

Yes, unfortunately this project is an old one and we have yet to upgrade to the latest version of ruby on rails

FROM ubuntu:18.04 as base

ENV TZ=America/Los_Angeles
ENV DEBIAN_FRONTEND=noninteractive
ENV PATH="/root/.rbenv/shims:/root/.rbenv/bin:$PATH"
ENV RUBY_CONFIGURE_OPTS="--disable-install-doc --with-jemalloc"
ENV RBENV_VERSION=2.1.9
ENV BUNDLER_VERSION=1.17.3

RUN apt-get update -qq && apt-get install -y libjemalloc-dev && rm -rf /var/lib/apt/lists/*
ENV LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.1

RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime \
    && echo $TZ > /etc/timezone \
    && apt-get update -qq \
    && apt-get install -y --no-install-recommends \
        ca-certificates \
        gnupg2 \
        autoconf \
        bison \
        build-essential \
        git \
        libcups2-dev \
        libffi-dev \
        libgdbm-dev \
        libncurses5-dev \
        libreadline-dev \
        libssl1.0-dev \
        libyaml-dev \
        libxslt1-dev \
        wget \
        zlib1g-dev \
        curl \
        tzdata \
        ntp \
    && git clone https://github.com/rbenv/rbenv.git ~/.rbenv \
    && echo 'export PATH="$PATH"' >> ~/.bashrc \
    && echo 'eval "$(rbenv init -)"' >> ~/.bashrc \
    && /bin/bash -c "source ~/.bashrc" \
    && git clone --branch v20220630 https://github.com/rbenv/ruby-build.git ~/.rbenv/plugins/ruby-build \
    && RUBY_CONFIGURE_OPTS="$RUBY_CONFIGURE_OPTS" rbenv install $RBENV_VERSION \
    && rbenv global $RBENV_VERSION \
    && echo "gem: --no-document" > ~/.gemrc \
    && gem install bundler -v $BUNDLER_VERSION \
    && apt-get install -y --no-install-recommends libmysqlclient-dev mysql-client

WORKDIR /app

COPY vendor/cache ./vendor/cache
COPY Gemfile Gemfile.lock ./

RUN bundle install --local

COPY . ./
RUN touch .dockerized

FROM base as builder
RUN MYSQL_ADAPTER=nulldb bundle exec rake assets:precompile

[eregon]

This works for me locally (it prints Mysql2):

FROM ubuntu:18.04 as base

ENV TZ=America/Los_Angeles
ENV DEBIAN_FRONTEND=noninteractive
ENV PATH="/root/.rbenv/shims:/root/.rbenv/bin:$PATH"
ENV RUBY_CONFIGURE_OPTS="--disable-install-doc --with-jemalloc"
ENV RBENV_VERSION=2.1.9
ENV BUNDLER_VERSION=1.17.3

RUN apt-get update -qq && apt-get install -y libjemalloc-dev && rm -rf /var/lib/apt/lists/*
ENV LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.1

RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime \
    && echo $TZ > /etc/timezone \
    && apt-get update -qq \
    && apt-get install -y --no-install-recommends \
        ca-certificates \
        gnupg2 \
        autoconf \
        bison \
        build-essential \
        git \
        libcups2-dev \
        libffi-dev \
        libgdbm-dev \
        libncurses5-dev \
        libreadline-dev \
        libssl1.0-dev \
        libyaml-dev \
        libxslt1-dev \
        wget \
        zlib1g-dev \
        curl \
        tzdata \
        ntp \
    && apt-get install -y --no-install-recommends libmysqlclient-dev mysql-client

RUN git clone https://github.com/rbenv/rbenv.git ~/.rbenv \
    && echo 'export PATH="$PATH"' >> ~/.bashrc \
    && echo 'eval "$(rbenv init -)"' >> ~/.bashrc \
    && /bin/bash -c "source ~/.bashrc" \
    && echo "gem: --no-document" > ~/.gemrc

RUN git clone --branch master https://github.com/rbenv/ruby-build.git ~/.rbenv/plugins/ruby-build \
    && RUBY_CONFIGURE_OPTS="$RUBY_CONFIGURE_OPTS" rbenv install $RBENV_VERSION \
    && rbenv global $RBENV_VERSION \
    && gem install bundler -v $BUNDLER_VERSION

WORKDIR /app

# COPY vendor/cache ./vendor/cache
# COPY Gemfile Gemfile.lock ./
COPY Gemfile ./

RUN bundle install

FROM base as builder
RUN MYSQL_ADAPTER=nulldb bundle exec ruby -rmysql2 -e 'p Mysql2'

I would suggest to clean and repopulate your vendor/cache, that could be the cause of the segfault.

Also you both using both --with-jemalloc and LD_PRELOAD I think that's not supported (just do LD_PRELOAD IIRC is correct).

Re 2.1.0-static, that's because enable_shared is only used on Ruby 2.4+ by ruby-build, so not surprising.

What ultimately changed here is that openssl 1.0.2 gets compiled, even though the system has it. I will investigate why.

[eregon]

What ultimately changed here is that openssl 1.0.2 gets compiled, even though the system has it. I will investigate why.

So on ubuntu:18.04 with the apt-get installs above:

# openssl version
OpenSSL 1.1.1  11 Sep 2018
# printf '#include <openssl/opensslv.h>\nOPENSSL_VERSION_TEXT\n' | cc -xc -E - 2>/dev/null | tail -n 1
"OpenSSL 1.1.1  11 Sep 2018"

So the system openssl version is considered 1.1.1, and there seems to be almost nothing to tell us there is also a system openssl 1.0 installed. I'm not sure how CRuby even find the openssl 1.0 headers.
I could find the binary /usr/lib/ssl1.0/openssl (after apt-get install openssl1.0) and lib /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0 but not the headers, there seems to be no opensslv.h or ssl.h somehow.

As https://www.openssl.org/source/ says openssl 1.0.2 is EOL, and you are using it on a more recent system where it's not the "system default openssl".

---

In the Dockerfile above you install libssl1.0-dev, which removes libssl-dev.
But then later you do apt-get install -y --no-install-recommends libmysqlclient-dev mysql-client which removes libssl1.0-dev and installs libssl-dev (1.1.1).
That's never going to work, you must not mix openssl versions like that.

So your system mysql package says I can't work with openssl 1.0 I need 1.1,
and Ruby 2.1.9 says I can't use openssl 1.1 only 1.0.
So there is no way for this to work with system packages.
ruby-build installs its own openssl, which seems the right thing here.

[eregon]

To be clear, the problem is mysql needs libssl 1.1 and ruby 2.1.9 needs libssl 1.0, so you end up loading two different openssl versions in the same process, and that typically ends in a segfault.
I don't think it ever worked correctly on ubuntu 18.04, or if it did it's extremely lucky and likely to crash at some point. libssl has no binary compatibility guarantees whatsoever between such versions, and never supports loading 2 versions in the same process (AFAIK).

To solve this you would need a mysql package which is compiled against libssl 1.0 e.g. from an older OS which has a default libssl 1.0 (or build it yourself), or you would need to use Ruby >= 2.4 which compiles against libssl 1.1.

root@a7ad26467017:/app# ldd /root/.rbenv/versions/2.1.9/lib/ruby/2.1.0/x86_64-linux/openssl.so
	linux-vdso.so.1 (0x00007ffec1ee7000)
	/usr/lib/x86_64-linux-gnu/libjemalloc.so.1 (0x00007f56bf087000)
	libssl.so.1.0.0 => /root/.rbenv/versions/2.1.9/openssl/lib/libssl.so.1.0.0 (0x00007f56bee16000)
	libcrypto.so.1.0.0 => /root/.rbenv/versions/2.1.9/openssl/lib/libcrypto.so.1.0.0 (0x00007f56be9b0000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f56be791000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f56be3a0000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f56bf512000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f56be19c000)

root@a7ad26467017:/app# ldd /usr/bin/mysql
	linux-vdso.so.1 (0x00007fff2d91c000)
	/usr/lib/x86_64-linux-gnu/libjemalloc.so.1 (0x00007f84a0835000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f84a0616000)
	libedit.so.2 => /usr/lib/x86_64-linux-gnu/libedit.so.2 (0x00007f84a03df000)
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f84a01c2000)
	libssl.so.1.1 => /usr/lib/x86_64-linux-gnu/libssl.so.1.1 (0x00007f849ff35000)
	libcrypto.so.1.1 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007f849fa6a000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f849f866000)
	libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f849f4dd000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f849f2c5000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f849eed4000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f84a0a6b000)
	libtinfo.so.5 => /lib/x86_64-linux-gnu/libtinfo.so.5 (0x00007f849ecaa000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f849e90c000)

[eregon]

Here is a minimal reproduction of this "half libssl1.0 half libssl1.1" state on Ubuntu 18.04:

$ docker run --rm -it ubuntu:18.04
# apt-get update
# apt-get install --no-install-recommends ca-certificates libssl1.0-dev
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  libssl1.0.0 libssl1.1 openssl
The following NEW packages will be installed:
  ca-certificates libssl1.0-dev libssl1.0.0 libssl1.1 openssl

# openssl version
OpenSSL 1.1.1  11 Sep 2018

# apt-get install gcc
# printf '#include <openssl/opensslv.h>\nOPENSSL_VERSION_TEXT\n' | cc -xc -E - 2>/dev/null | tail -n 1
"OpenSSL 1.0.2n  7 Dec 2017"

Fun/horrible mix of openssl versions.
So we should trust the header more than openssl version. I think we probably shouldn't even look at openssl version.
I'll change that: #2015
It will restore the previous behavior in this case.
The use-case here is hopelessly broken though, one can't mix openssl versions at runtime, but it does show us the better way to get the system openssl version.

[eregon]

v20220725 released and finds the correct openssl version in this situation.