Skip to content

Building GlusterFS

Jump to bottom
aborkar-ibm edited this page Aug 10, 2022 · 39 revisions

Building GlusterFS

Following versions of GlusterFS are available in respective distributions at the time of creation of these build instructions:

  • RHEL (8.4, 8.6) has 6.0
  • SLES 15 SP3 has 3.12.3
  • Ubuntu 18.04 has 3.13.2
  • Ubuntu 20.04 has 7.2
  • Ubuntu 22.04 has 10.1

The instructions provided below specify the steps to build or install GlusterFS 10.2 on Linux on IBM Z for following distributions:

  • RHEL (8.4, 8.6)
  • SLES 15 SP3

Note: For Ubuntu (18.04, 20.04, 22.04), please refer to this link to install GlusterFS from the community GlusterFS PPA (ppa:gluster/glusterfs-10).

General Notes:

  • When following the steps below please use a super user unless otherwise specified
  • A directory /<source root>/ will be referred to in these instructions, this is a temporary writable directory anywhere you'd like to place it

Step 1: Build and Install GlusterFS

Step 1.1: Build using script

If you want to build GlusterFS using manual steps, go to Step 1.2.

Use the following commands to build GlusterFS using the build script. Please make sure you have wget installed.

wget -q https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/GlusterFS/10.2/build_glusterfs.sh
# Build GlusterFS
bash build_glusterfs.sh   [Provide -h option to check usage]

If the build completes successfully, go to STEP 3. In case of error, check logs for more details or go to STEP 1.2 to follow manual build steps.

Step 1.2: Install the dependencies

  • Set environment variables:

    export SOURCE_ROOT=/<source_root>/
export PREFIX=/usr/local
export PATCH_URL=https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/GlusterFS/10.2/patch

PATH=${PREFIX}/bin:${PREFIX}/sbin${PATH:+:${PATH}}
export PATH

PKG_CONFIG_PATH=${PREFIX}/lib/pkgconfig:${PREFIX}/lib64/pkgconfig${PKG_CONFIG_PATH:+:${PKG_CONFIG_PATH}}
export PKG_CONFIG_PATH

LD_LIBRARY_PATH=${PREFIX}/lib:${PREFIX}/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
export LD_LIBRARY_PATH

LD_RUN_PATH=${PREFIX}/lib:${PREFIX}/lib64${LD_RUN_PATH:+:${LD_RUN_PATH}}
export LD_RUN_PATH

  • RHEL (8.4, 8.6)

    yum install -y autoconf automake bison bzip2 flex fuse-devel gcc-c++ git glib2-devel libacl-devel libaio-devel libibverbs-devel librdmacm-devel libtool libxml2-devel libuuid-devel liburing-devel lvm2 make binutils openssl-devel pkgconfig python3 readline-devel wget zlib-devel tar gzip libtirpc-devel patch rpcgen userspace-rcu-devel which diffutils xz gperftools gperf

  • SLES (15 SP3)

    zypper install -y autoconf automake bison cmake flex fuse-devel gcc-c++ git-core glib2-devel libacl-devel libaio-devel librdmacm1 libtool liburcu-devel libuuid-devel libxml2-devel lvm2 make pkg-config python3 python3-xattr rdma-core-devel readline-devel openssl-devel zlib-devel which gawk dmraid popt-devel gperftools-devel gperf gperftools libtirpc-devel rpcgen

Step 1.3: Build and Install GCC 7.5.0 (Only on RHEL 8.x)

cd $SOURCE_ROOT
mkdir gcc
cd gcc
wget --no-check-certificate https://ftpmirror.gnu.org/gcc/gcc-7.5.0/gcc-7.5.0.tar.xz
tar -xf gcc-7.5.0.tar.xz
cd gcc-7.5.0
./contrib/download_prerequisites
mkdir objdir
cd objdir
../configure --prefix=${PREFIX} --enable-languages=c,c++ --with-arch=zEC12 --with-long-double-128 --build=s390x-linux-gnu --host=s390x-linux-gnu --target=s390x-linux-gnu --enable-threads=posix --with-system-zlib --disable-multilib
make -j 8
make install

update-alternatives --install /usr/bin/cc cc ${PREFIX}/bin/gcc 40
update-alternatives --install /usr/bin/gcc gcc ${PREFIX}/bin/gcc 40
update-alternatives --install /usr/bin/g++ g++ ${PREFIX}/bin/g++ 40
update-alternatives --install /usr/bin/c++ c++ ${PREFIX}/bin/g++ 40

Note : This step is to fix test case failures ./tests/basic/fencing/afr-lock-heal-basic.t and ./tests/basic/fencing/afr-lock-heal-advanced.t, see issue#1468.

Step 1.4: Download, Configure and Build GlusterFS

  • Download and configure source

    Note : If git clone fails with SSL certificate issue, execute git config --global http.sslVerify false command and then do git clone.

    cd $SOURCE_ROOT
git clone https://github.com/gluster/glusterfs
cd glusterfs
git checkout v10.2
./autogen.sh
./configure --enable-gnfs                                              # For RHEL 
./configure --enable-gnfs --disable-linux-io_uring                     # For SLES 15 SP3

  • Patch xlators/performance/io-threads/src/io-threads.h:

    This patch fixes the ./tests/features/weighted-rebalance.t test case failure, see issue#3346.

    cd $SOURCE_ROOT/glusterfs
wget --no-check-certificate $PATCH_URL/io-threads.h.diff
git apply io-threads.h.diff

  • Patch xlators/features/bit-rot/src/stub/bit-rot-stub.c:

    This patch fixes the ./tests/bitrot/bug-1207627-bitrot-scrub-status.t test case failure, see issue#3347.

    cd $SOURCE_ROOT/glusterfs
wget --no-check-certificate $PATCH_URL/bit-rot-stub.c.diff
git apply bit-rot-stub.c.diff

  • Patch xlators/nfs/server/src/nfs.c and xlators/nfs/server/src/nfs.h:

    This patch fixes test case failure ./tests/bugs/nfs/bug-1157223-symlink-mounting.t on both RHEL 8.x and SLES 15 SP3 and the following test case failures on RHEL 8.x, see issue#3604.

    • ./tests/basic/afr/arbiter-mount.t
    • ./tests/basic/afr/tarissue.t
    • ./tests/basic/all_squash.t
    • ./tests/basic/ec/nfs.t
    • ./tests/basic/mount.t
    • ./tests/basic/nufa.t
    • ./tests/basic/quota-anon-fd-nfs.t
    • ./tests/basic/quota-nfs.t
    • ./tests/basic/quota.t
    • ./tests/basic/stats-dump.t
    • ./tests/basic/volume-status.t
    • ./tests/bitrot/br-stub.t
    • ./tests/bitrot/bug-1244613.t
    • ./tests/bugs/access-control/bug-958691.t
    • ./tests/bugs/bug-1258069.t
    • ./tests/bugs/core/bug-927616.t
    • ./tests/bugs/distribute/bug-1125824.t
    • ./tests/bugs/distribute/bug-1161156.t
    • ./tests/bugs/distribute/bug-1190734.t
    • ./tests/bugs/ec/bug-1187474.t
    • ./tests/bugs/fuse/many-groups-for-acl.t
    • ./tests/bugs/glusterd/bug-1070734.t
    • ./tests/bugs/glusterd/optimized-basic-testcases.t
    • ./tests/bugs/glusterfs-server/bug-864222.t
    • ./tests/bugs/glusterfs-server/bug-887145.t
    • ./tests/bugs/glusterfs-server/bug-904300.t
    • ./tests/bugs/glusterfs/bug-867253.t
    • ./tests/bugs/glusterfs/bug-872923.t
    • ./tests/bugs/nfs/bug-1053579.t
    • ./tests/bugs/nfs/bug-1143880-fix-gNFSd-auth-crash.t
    • ./tests/bugs/nfs/bug-1161092-nfs-acls.t
    • ./tests/bugs/nfs/bug-1166862.t
    • ./tests/bugs/nfs/bug-1210338.t
    • ./tests/bugs/nfs/bug-877885.t
    • ./tests/bugs/nfs/bug-904065.t
    • ./tests/bugs/nfs/bug-915280.t
    • ./tests/bugs/nfs/bug-974972.t
    • ./tests/bugs/nfs/showmount-many-clients.t
    • ./tests/bugs/nfs/socket-as-fifo.t
    • ./tests/bugs/nfs/subdir-trailing-slash.t
    • ./tests/bugs/nfs/zero-atime.t
    • ./tests/bugs/quota/bug-1087198.t
    • ./tests/bugs/quota/bug-1153964.t
    • ./tests/bugs/quota/bug-1243798.t
    • ./tests/bugs/replicate/bug-1340623-mkdir-fails-remove-brick-started.t
    • ./tests/bugs/replicate/bug-830665.t
    • ./tests/bugs/rpc/bug-1043886.t
    • ./tests/bugs/rpc/bug-847624.t
    • ./tests/bugs/rpc/bug-921072.t
    • ./tests/bugs/snapshot/bug-1260848.t
    • ./tests/gfid2path/gfid2path_nfs.t
    cd $SOURCE_ROOT/glusterfs
wget --no-check-certificate $PATCH_URL/nfs.diff
git apply nfs.diff

  • Build and install

    make
make install

Step 2: Run the test cases (optional)

2.1) Install the testing dependencies

  • RHEL (8.4, 8.6)

    yum install -y acl attr bc bind-utils boost-devel docbook-style-xsl expat-devel gdb net-tools nfs-utils psmisc vim xfsprogs yajl redhat-rpm-config python3-devel python3-pyxattr python3-prettytable perl-Test-Harness popt-devel procps-ng

  • SLES (15 SP3)

    zypper install -y acl attr bc bind-utils gdb libxml2-tools net-tools-deprecated nfs-utils psmisc thin-provisioning-tools vim xfsprogs python3-xattr python3-PrettyTable libselinux-devel selinux-tools popt-devel

  • Link pstack command (For SLES only)

    ln -sf `which gstack` /usr/bin/pstack

  • Build DBENCH

    cd $SOURCE_ROOT
git clone https://github.com/sahlberg/dbench
cd dbench
git checkout caa52d347171f96eef5f8c2d6ab04a9152eaf113
./autogen.sh
./configure --datadir=/usr/local/share/doc/loadfiles/
make
make install

  • Build thin-provisioning-tools (For RHEL only)

    cd $SOURCE_ROOT
git clone https://github.com/jthornber/thin-provisioning-tools
cd thin-provisioning-tools
git checkout v0.7.6
autoreconf
./configure
make
make install

  • Build yajl (For SLES only)

    cd $SOURCE_ROOT
git clone https://github.com/lloyd/yajl
cd yajl
git checkout 2.1.0
./configure
make install

2.2) Patch and run tests

  • The following patch fixes the test cases listed below that fail due to hardcoded hash values for little-endian systems, see issue#3311:

    • ./tests/bugs/posix/bug-1619720.t
    • ./tests/bugs/replicate/bug-1655854-support-dist-to-rep3-arb-conversion.t
    • ./tests/bugs/distribute/bug-1600379.t
    • ./tests/bugs/glusterd/bug-1699339.t
    • ./tests/bugs/distribute/bug-1786679.t
    • ./tests/bugs/glusterfs/bug-902610.t
    • ./tests/basic/changelog/changelog-rename.t
    • ./tests/basic/distribute/dir-heal.t
    • ./tests/basic/distribute/spare_file_rebalance.t
    • ./tests/basic/namespace.t
    cd $SOURCE_ROOT/glusterfs
wget --no-check-certificate $PATCH_URL/hash-tests.diff
git apply hash-tests.diff

  • The following patch fixes test cases failing due to system configuration:

    • ./tests/basic/distribute/throttle-rebal.t
    • ./tests/bugs/nfs/bug-847622.t
    • ./tests/00-geo-rep/00-georep-verify-non-root-setup.t
    cd $SOURCE_ROOT/glusterfs
wget --no-check-certificate $PATCH_URL/test-patch.diff
git apply test-patch.diff

  • Fix the ./tests/00-geo-rep/00-georep-verify-non-root-setup.t test case failure (For RHEL only):

    ln -s /usr/local/lib/libgfchangelog.so.0 /lib64/libgfchangelog.so
ldconfig /usr/local/lib
ldconfig /usr/local/lib64

  • Run the test cases:

    cd $SOURCE_ROOT/glusterfs
./run-tests.sh

Note:

  • If you want to use the kernel NFS server or client on a Gluster storage server, make sure to disable the Gluster NFS server:

    gluster volume set <VOLUME> nfs.disable true

  • For geo-replication test cases:

    • Password-less SSH login has to be set up. Refer to the Setting Up the Environment for Geo-replication in the documentation.
    • The locale of the system must be UTF-8 for the test cases to pass.

  • For ./tests/bugs/glusterfs/bug-866459.t test case: This test is under investigation, refer to this issue for details.

  • For ./tests/bugs/glusterfs-server/bug-861542.t test case: If the hostname of the machine is long, the test case will fail. The hostname of the machine can be shortened temporarily via the hostname command to get the test case to pass.

  • Tests under ./tests/bugs/snapshot require the snapd service to be stopped (systemctl stop snapd) if installed:.

  • The following test cases fail on RHEL 8.x on both Intel and s390x:

    • ./tests/basic/uss.t
    • ./tests/bugs/snapshot/bug-1166197.t

  • Test case ./tests/bugs/replicate/bug-1728770-pass-xattrs.t fails on RHEL 8.6 on both Intel and s390x.

  • The following test cases fail on SLES 15 SP3 on both Intel and s390x:

    • ./tests/00-geo-rep/00-georep-verify-non-root-setup.t
    • ./tests/bugs/shard/bug-1251824.t
    • ./tests/bugs/shard/bug-1468483.t

  • Test cases such as ./tests/basic/ctime/ctime-utimesat.t fail intermittently. They pass when running individually through prove -vf <testcase>.

  • Additional information on the GlusterFS test framework can be found here.

Step 3: Run the GlusterFS Daemon (Optional)

glusterd

Similar logs should be output in /var/log/glusterfs/glusterd.log:

[2022-06-22 18:38:51.350381 +0000] I [MSGID: 100030] [glusterfsd.c:2769:main] 0-glusterd: Started running version [{arg=glusterd}, {version=10.2}, {cmdlinestr=glusterd}]

Step 4: Test the GlusterFS volume (Optional)

Refer to Documentation for instructions.

References

The information provided in this article is accurate at the time of writing, but on-going development in the open-source projects involved may make the information incorrect or obsolete. Please open issue or contact us on IBM Z Community if you have any questions or feedback.

Clone this wiki locally