Skip to content

Building Bazel

Jump to bottom
linuxonz edited this page Oct 26, 2023 · 34 revisions

Building Bazel

Following instructions specify the steps to build Bazel version 6.3.2 on Linux on IBM Z:

  • Ubuntu (20.04, 22.04, 23.04, 23.10)

General Notes:

  • When following the steps below please use a standard permission user unless otherwise specified.
  • A directory /<source_root>/ will be referred to in these instructions, this is a temporary writable directory anywhere you'd like to place it.

Step 1 : Building Bazel

1.1) Build using script

If you'd like to build Bazel using the manual steps, please go to STEP 1.2.

Use the following commands to build Bazel using the build script. Please make sure you have wget installed.

  wget -q https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/Bazel/6.3.2/build_bazel.sh

  # Build Bazel
  bash build_bazel.sh   [Provide -t option for executing build with tests]

In case of any errors, check logs for more details or go to STEP 1.2 to follow the manual build steps.

1.2) Install dependencies  

  • Set environment variable SOURCE_ROOT:
  export SOURCE_ROOT=/<source_root>/
  sudo apt-get update
  • Ubuntu 20.04
  sudo apt-get install -y --no-install-recommends bind9-host build-essential coreutils curl dnsutils ed expect file git gnupg2 iproute2 iputils-ping mkisofs lcov less libssl-dev lsb-release netcat-openbsd openjdk-11-jdk-headless zip zlib1g-dev unzip wget python2 python2-dev python-is-python3 python3 python3-dev python3-pip python3-requests python3-setuptools python3-six python3-wheel python3-yaml
  • Ubuntu 22.04
  sudo apt-get install -y --no-install-recommends bind9-host build-essential coreutils curl dnsutils ed expect file git gnupg2 iproute2 iputils-ping mkisofs lcov less libssl-dev lsb-release netcat-openbsd openjdk-11-jdk-headless zip zlib1g-dev unzip wget python2 python2-dev
  • Ubuntu 23.04
  sudo apt-get install -y --no-install-recommends bind9-host build-essential coreutils curl dnsutils ed expect file git gnupg2 iproute2 iputils-ping lcov less libssl-dev lsb-release netcat-openbsd openjdk-11-jdk-headless zip zlib1g-dev unzip wget mkisofs libbz2-dev libdb-dev libffi-dev liblzma-dev libncurses-dev libreadline-dev libsqlite3-dev
  • Ubuntu 23.10
  sudo apt-get install -y --no-install-recommends bind9-host build-essential coreutils curl dnsutils ed expect file git gnupg2 iproute2 iputils-ping lcov less libssl-dev lsb-release netcat-openbsd openjdk-11-jdk-headless zip zlib1g-dev unzip wget mkisofs libbz2-dev libdb-dev libffi-dev liblzma-dev libncurses-dev libreadline-dev libsqlite3-dev gcc-12 g++-12

  sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-12 20
  sudo update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-12 20
  sudo update-alternatives --install /usr/bin/gcov gcov /usr/bin/gcov-12 20
  sudo update-alternatives --install /usr/bin/gcov-tool gcov-tool /usr/bin/gcov-tool-12 20
  sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-13 13
  sudo update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-13 13
  sudo update-alternatives --install /usr/bin/gcov gcov /usr/bin/gcov-13 13
  sudo update-alternatives --install /usr/bin/gcov-tool gcov-tool /usr/bin/gcov-tool-13 13
  • Set JAVA_HOME:
  export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-s390x
  export PATH=$JAVA_HOME/bin:$PATH

1.3) Build

  • Build and install Python 2.7 (Only on Ubuntu (23.04, 23.10)):
  cd $SOURCE_ROOT
  wget https://www.python.org/ftp/python/2.7.18/Python-2.7.18.tar.xz
  tar -xvf Python-2.7.18.tar.xz
  cd Python-2.7.18
  ./configure
  make
  sudo make install
  sudo python2 -m ensurepip
  • Build and install Python 3.9 (Only on Ubuntu (22.04, 23.04, 23.10)):
  cd $SOURCE_ROOT
  wget -q https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/Python3/3.9.7/build_python3.sh
  sed -i 's/20.04/22.04/g' build_python3.sh #Only on Ubuntu 22.04
  sed -i 's/20.04/23.04/g' build_python3.sh #Only on Ubuntu 23.04
  sed -i 's/20.04/23.10/g' build_python3.sh #Only on Ubuntu 23.10
  sudo apt-get remove python3 -y
  bash build_python3.sh -y
  rm -f build_python3.sh
  sudo update-alternatives --install /usr/bin/python python /usr/local/bin/python3 40
  sudo python3 -m pip install requests setuptools six wheel
  • Download and patch rules_java 5.5.0:
  cd $SOURCE_ROOT
  git clone -b 5.5.0 https://github.com/bazelbuild/rules_java.git
  cd rules_java
  curl -sSL https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/Bazel/6.3.2/patch/rules_java_5.5.0.patch | git apply
  • Download Bazel 6.3.2 distribution archive:
  cd $SOURCE_ROOT
  wget https://github.com/bazelbuild/bazel/releases/download/6.3.2/bazel-6.3.2-dist.zip
  mkdir -p dist/bazel && cd dist/bazel
  unzip -q ../../bazel-6.3.2-dist.zip
  chmod -R +w .
  • Apply the patch and build the bootstrap bazel:
  cd ${SOURCE_ROOT}/dist/bazel
  curl -sSL https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/Bazel/6.3.2/patch/dist-md5.patch | git apply
  env EXTRA_BAZEL_ARGS="--tool_java_runtime_version=local_jdk" bash ./compile.sh
  • Checkout the Bazel 6.3.2 source code and apply the patches:
  cd $SOURCE_ROOT
  git clone https://github.com/bazelbuild/bazel.git
  cd bazel
  git checkout 6.3.2
  curl -sSLO https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/Bazel/6.3.2/patch/bazel.patch
  sed -i "s#RULES_JAVA_ROOT_PATH#${SOURCE_ROOT}#g" bazel.patch
  patch -p1 < bazel.patch
  rm -f bazel.patch

Note: Netty and netty-tcnative jar are needed to fix test case failures related to remote execution. If remote execution functionality is not required then directly go to Build Bazel section.

  • Install and patch netty-tcnative 2.0.51:
  cd $SOURCE_ROOT
  sudo apt-get install -y ninja-build cmake perl golang libssl-dev libapr1-dev autoconf automake libtool make tar git wget maven
  git clone https://github.com/netty/netty-tcnative.git
  cp -r netty-tcnative netty-tcnative_2.0.50
  cd netty-tcnative
  git checkout netty-tcnative-parent-2.0.51.Final
  curl -sSL https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/Bazel/6.0.0/patch/netty-tcnative_2.0.51.patch | patch -p1
  • Apply the netty-tcnative gcc patch on Ubuntu 22.04, 23.04 and 23.10 only
  cd $SOURCE_ROOT/netty-tcnative
  curl -sSL https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/Bazel/6.0.0/patch/netty-tcnative-gcc_2.0.51.patch | patch -p1
  • Build netty-tcnative 2.0.51:
  cd $SOURCE_ROOT/netty-tcnative
  mvn install
  • Install and patch netty-tcnative 2.0.50 (required by netty 4.1.75):
  cd $SOURCE_ROOT
  cd netty-tcnative_2.0.50
  git checkout netty-tcnative-parent-2.0.50.Final
  curl -sSL https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/Bazel/6.0.0/patch/netty-tcnative_2.0.50.patch | patch -p1
  • Apply the netty-tcnative gcc patch on Ubuntu 22.04, 23.04 and 23.10 only
  cd $SOURCE_ROOT/netty-tcnative_2.0.50
  curl -sSL https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/Bazel/6.0.0/patch/netty-tcnative-gcc_2.0.50.patch | patch -p1
  • Build netty-tcnative 2.0.50:
  cd $SOURCE_ROOT/netty-tcnative_2.0.50
  mvn install
  • Build netty 4.1.75:
  cd $SOURCE_ROOT
  git clone https://github.com/netty/netty.git
  cd netty
  git checkout netty-4.1.75.Final
  ./mvnw clean install -DskipTests
  • Copy the netty and netty-tcnative jar into the respective bazel directory and apply the patch to use them:
  cp $SOURCE_ROOT/netty-tcnative/boringssl-static/target/netty-tcnative-boringssl-static-2.0.51.Final-linux-s390_64.jar \
    $SOURCE_ROOT/bazel/third_party/netty_tcnative/
  cp $SOURCE_ROOT/netty/buffer/target/netty-buffer-4.1.75.Final.jar \
      $SOURCE_ROOT/netty/codec/target/netty-codec-4.1.75.Final.jar \
      $SOURCE_ROOT/netty/codec-http/target/netty-codec-http-4.1.75.Final.jar \
      $SOURCE_ROOT/netty/codec-http2/target/netty-codec-http2-4.1.75.Final.jar \
      $SOURCE_ROOT/netty/common/target/netty-common-4.1.75.Final.jar \
      $SOURCE_ROOT/netty/handler/target/netty-handler-4.1.75.Final.jar \
      $SOURCE_ROOT/netty/handler-proxy/target/netty-handler-proxy-4.1.75.Final.jar \
      $SOURCE_ROOT/netty/resolver/target/netty-resolver-4.1.75.Final.jar \
      $SOURCE_ROOT/netty/resolver-dns/target/netty-resolver-dns-4.1.75.Final.jar \
      $SOURCE_ROOT/netty/transport/target/netty-transport-4.1.75.Final.jar \
      $SOURCE_ROOT/netty/transport-sctp/target/netty-transport-sctp-4.1.75.Final.jar \
      $SOURCE_ROOT/netty/transport-native-unix-common/target/netty-transport-native-unix-common-4.1.75.Final.jar \
      $SOURCE_ROOT/netty/transport-native-unix-common/target/netty-transport-native-unix-common-4.1.75.Final-linux-s390_64.jar \
      $SOURCE_ROOT/netty/transport-native-kqueue/target/netty-transport-native-kqueue-4.1.75.Final.jar \
      $SOURCE_ROOT/netty/transport-native-epoll/target/netty-transport-native-epoll-4.1.75.Final.jar \
      $SOURCE_ROOT/netty/transport-native-epoll/target/netty-transport-native-epoll-4.1.75.Final-linux-s390_64.jar \
      $SOURCE_ROOT/bazel/third_party/netty/
  cd $SOURCE_ROOT/bazel
  curl -sSL https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/Bazel/6.3.2/patch/bazel-netty.patch | patch -p1
  • Build Bazel:
  cd $SOURCE_ROOT/bazel
  ${SOURCE_ROOT}/dist/bazel/output/bazel build -c opt --stamp --embed_label "6.3.2" //src:bazel //src:bazel_jdk_minimal //src:test_repos //src/main/java/...
  mkdir output
  cp bazel-bin/src/bazel output/bazel
  • Bazel version:
  cd $SOURCE_ROOT/bazel
  ./output/bazel --version
  bazel 6.3.2

Step 2: Execute Test Suite (Optional)

  • Run complete test suite:
  cd $SOURCE_ROOT/bazel
  ./output/bazel --host_jvm_args=-Xmx2g test -c opt --build_tests_only --flaky_test_attempts=3 --test_timeout=3600 --show_progress_rate_limit=5 --terminal_columns=143 --show_timestamps --verbose_failures \
    -- //scripts/... //src/java_tools/... //src/test/... //src/tools/execlog/... //src/tools/singlejar/... //src/tools/workspacelog/... //third_party/ijar/... -//tools/android/... //tools/aquery_differ/... //tools/python/... \
    -//src/java_tools/import_deps_checker/... -//src/test/shell/bazel/android/... -//src/test/java/com/google/devtools/build/android/... -//src/test/shell/bazel:bazel_determinism_test
  • Run individual test:
  cd $SOURCE_ROOT/bazel
  ./output/bazel test -c opt --cache_test_results=no --build_tests_only --flaky_test_attempts=3 --test_timeout=3600 --show_progress_rate_limit=5 --show_timestamps --verbose_failures -- //<module_name>:<testcase_name>

For example:

  cd $SOURCE_ROOT/bazel
  ./output/bazel test -c opt --cache_test_results=no --build_tests_only --flaky_test_attempts=3 --test_timeout=3600 --show_progress_rate_limit=5 --show_timestamps --verbose_failures -- //src/test/cpp/util:md5_test

Notes:

1. Some tests have been excluded in the bazel test command above:

  • //src/java_tools/import_deps_checker/... fails on all platforms
  • //src/test/shell/bazel/android/... fails because an android sdk is required
  • //src/test/java/com/google/devtools/build/android/... fails because an android sdk is required
  • //src/test/shell/bazel:bazel_determinism_test fails because some archive files which must be rebuilt during the test on s390x result in new file sizes and md5sum values

2. The following test would be skipped, since prebuilt ijar and singlejar binaries haven't been released for s390x:

  • //src/test/shell/bazel:bazel_java_test_defaults_prebuilt

3. The following test would fail unless ipv6 is enabled on the system:

  • //src/test/java/com/google/devtools/build/lib/bazel/bzlmod:BzlmodTests

4. The following test may fail with an out of memory error if building on a machine with less than 4GB of memory:

  • //src/test/java/com/google/devtools/build/lib/graph:GraphTests

5. More information on running Bazel in a low memory environment:

  • https://bazel.build/advanced/performance/memory

Step 3: Use the new module-focused external dependencies management system in Bazel (Optional)

The new module-focused MODULE.bazel system can be enabled by the flag --enable_bzlmod, for example:

  cd $SOURCE_ROOT/bazel
  ./output/bazel build --enable_bzlmod //src:bazel_nojdk

References:

The information provided in this article is accurate at the time of writing, but on-going development in the open-source projects involved may make the information incorrect or obsolete. Please open issue or contact us on IBM Z Community if you have any questions or feedback.

Clone this wiki locally