Skip to content

Building GlusterFS

Jump to bottom
aborkar-ibm edited this page Jun 8, 2023 · 39 revisions

Building GlusterFS

Following versions of GlusterFS are available in respective distributions at the time of creation of these build instructions:

  • RHEL (8.6, 8.7, 9.0, 9.1) has 6.0
  • SLES 15 SP4 has 9.3
  • Ubuntu 20.04 has 7.2
  • Ubuntu 22.04 has 10.1
  • Ubuntu 22.10 has 10.2

The instructions provided below specify the steps to build or install GlusterFS 10.3 on Linux on IBM Z for following distributions:

  • RHEL (7.8, 7.9, 8.6, 8.7, 9.0, 9.1)
  • SLES (12 SP5, 15 SP4)

Note: For Ubuntu (20.04, 22.04, 22.10), please refer to this link to install GlusterFS from the community GlusterFS PPA (ppa:gluster/glusterfs-10).

General Notes:

  • When following the steps below please use a super user unless otherwise specified
  • A directory /<source root>/ will be referred to in these instructions, this is a temporary writable directory anywhere you'd like to place it

Step 1: Build and Install GlusterFS

Step 1.1: Build using script

If you want to build GlusterFS using manual steps, go to Step 1.2.

Use the following commands to build GlusterFS using the build script. Please make sure you have wget installed.

wget -q https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/GlusterFS/10.3/build_glusterfs.sh
# Build GlusterFS
bash build_glusterfs.sh    [Provide -h option to check usage,  -t for executing build with tests]

If the build completes successfully, go to STEP 3. In case of error, check logs for more details or go to STEP 1.2 to follow manual build steps.

Step 1.2: Install the dependencies

  • Set environment variables:

    export SOURCE_ROOT=/<source_root>/
export PREFIX=/usr/local
export PATCH_URL=https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/GlusterFS/10.3/patch

PATH=${PREFIX}/bin:${PREFIX}/sbin${PATH:+:${PATH}}
export PATH

PKG_CONFIG_PATH=${PREFIX}/lib/pkgconfig:${PREFIX}/lib64/pkgconfig${PKG_CONFIG_PATH:+:${PKG_CONFIG_PATH}}
export PKG_CONFIG_PATH

LD_LIBRARY_PATH=${PREFIX}/lib:${PREFIX}/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
export LD_LIBRARY_PATH

LD_RUN_PATH=${PREFIX}/lib:${PREFIX}/lib64${LD_RUN_PATH:+:${LD_RUN_PATH}}
export LD_RUN_PATH

  • RHEL (7.8, 7.9)

    yum install -y autoconf automake bison bzip2 flex fuse-devel gcc-c++ git glib2-devel libacl-devel libaio-devel libibverbs-devel librdmacm-devel libtool libxml2-devel libuuid-devel lvm2 make binutils openssl openssl-devel pkgconfig python3 readline-devel wget zlib-devel tar gzip libtirpc-devel patch which diffutils xz gperftools gperf

  • RHEL (8.6, 8.7)

    yum install -y autoconf automake bison bzip2 flex fuse-devel gcc-c++ git glib2-devel libacl-devel libaio-devel libibverbs-devel librdmacm-devel libtool libxml2-devel libuuid-devel liburing-devel lvm2 make binutils openssl-devel pkgconfig python3 readline-devel wget zlib-devel tar gzip libtirpc-devel patch rpcgen userspace-rcu-devel which diffutils xz gperftools gperf

  • RHEL (9.0, 9.1)

    yum install -y autoconf automake bison bzip2 flex fuse-devel gcc-c++ git glib2-devel libacl-devel libaio-devel libibverbs-devel librdmacm-devel libtool libxml2-devel libuuid-devel lvm2 make binutils openssl-devel pkgconfig python3 readline-devel wget zlib-devel tar gzip libtirpc-devel patch rpcgen userspace-rcu-devel which diffutils xz gperftools gperf

  • SLES (12 SP5)

    zypper install -y autoconf automake bison cmake flex fuse-devel gcc-c++ git-core glib2-devel libacl-devel libaio-devel librdmacm1 libtool libuuid-devel libxml2-devel lvm2 make pkg-config python3 python3-xattr rdma-core-devel readline-devel openssl-devel zlib-devel which gawk dmraid popt-devel gperftools-devel gperf gperftools libtirpc-devel

  • SLES (15 SP4)

    zypper install -y autoconf automake bison cmake flex fuse-devel gcc-c++ git-core glib2-devel libacl-devel libaio-devel librdmacm1 libtool liburcu-devel libuuid-devel libxml2-devel lvm2 make pkg-config python3 python3-xattr rdma-core-devel readline-devel openssl-devel zlib-devel which gawk dmraid popt-devel gperftools-devel gperf gperftools libtirpc-devel rpcgen liburing-devel

Step 1.3: Build and install dependencies

  • Build and Install GCC 7.5.0 (Only on RHEL 8.x)

    cd $SOURCE_ROOT
mkdir gcc
cd gcc
wget --no-check-certificate https://ftpmirror.gnu.org/gcc/gcc-7.5.0/gcc-7.5.0.tar.xz
tar -xf gcc-7.5.0.tar.xz
cd gcc-7.5.0
./contrib/download_prerequisites
mkdir objdir
cd objdir
../configure --prefix=${PREFIX} --enable-languages=c,c++ --with-arch=zEC12 --with-long-double-128 --build=s390x-linux-gnu --host=s390x-linux-gnu --target=s390x-linux-gnu --enable-threads=posix --with-system-zlib --disable-multilib
make -j 8
make install

update-alternatives --install /usr/bin/cc cc ${PREFIX}/bin/gcc 40
update-alternatives --install /usr/bin/gcc gcc ${PREFIX}/bin/gcc 40
update-alternatives --install /usr/bin/g++ g++ ${PREFIX}/bin/g++ 40
update-alternatives --install /usr/bin/c++ c++ ${PREFIX}/bin/g++ 40

Note : This step is to fix test case failures ./tests/basic/fencing/afr-lock-heal-basic.t and ./tests/basic/fencing/afr-lock-heal-advanced.t, see issue#1468.

  • Install userspace-rcu (Only on RHEL 7.x and SLES12SP5)

    git clone git://git.liburcu.org/userspace-rcu.git
cd userspace-rcu/
git checkout v0.14.0
./bootstrap
./configure
make
make install
ldconfig

  • Install liburing-devel (Only on RHEL 9.x)

    cd $SOURCE_ROOT
git clone https://github.com/axboe/liburing.git
sudo yum groupinstall -y "Development Tools"
cd liburing
make
sudo make install
export PKG_CONFIG_PATH=/usr/lib/pkgconfig

Step 1.4: Download, Configure and Build GlusterFS

  • Download and configure source

    Note : If git clone fails with SSL certificate issue, execute git config --global http.sslVerify false command and then do git clone.

    cd $SOURCE_ROOT
git clone https://github.com/gluster/glusterfs
cd glusterfs
git checkout v10.3
./autogen.sh
./configure --enable-gnfs  ##For RHEL 8.x, 9.x and SLES15SP4
./configure --enable-gnfs --disable-linux-io_uring  ##For RHEL 7.x
./configure --enable-gnfs --disable-linux-io_uring --disable-events  ##For SLES12SP5

  • Patch xlators/performance/io-threads/src/io-threads.h:

    This patch fixes the ./tests/features/weighted-rebalance.t test case failure. This failure is fixed on devel branch by this commit

    cd $SOURCE_ROOT/glusterfs
wget --no-check-certificate $PATCH_URL/io-threads.h.diff
git apply io-threads.h.diff

  • Patch xlators/features/bit-rot/src/stub/bit-rot-stub.c:

    This patch fixes the ./tests/bitrot/bug-1207627-bitrot-scrub-status.t test case failure. This failure is fixed on devel branch by this commit.

    cd $SOURCE_ROOT/glusterfs
wget --no-check-certificate $PATCH_URL/bit-rot-stub.c.diff
git apply bit-rot-stub.c.diff

  • Patch xlators/nfs/server/src/nfs.c and xlators/nfs/server/src/nfs.h:

    This patch fixes test case failure ./tests/bugs/nfs/bug-1157223-symlink-mounting.t on both RHEL and SLES and the following test case failures on RHEL. These failures are fixed on devel branch by this commit.

    • ./tests/basic/afr/arbiter-mount.t
    • ./tests/basic/afr/tarissue.t
    • ./tests/basic/all_squash.t
    • ./tests/basic/ec/nfs.t
    • ./tests/basic/mount.t
    • ./tests/basic/nufa.t
    • ./tests/basic/quota-anon-fd-nfs.t
    • ./tests/basic/quota-nfs.t
    • ./tests/basic/quota.t
    • ./tests/basic/stats-dump.t
    • ./tests/basic/volume-status.t
    • ./tests/bitrot/br-stub.t
    • ./tests/bitrot/bug-1244613.t
    • ./tests/bugs/access-control/bug-958691.t
    • ./tests/bugs/bug-1258069.t
    • ./tests/bugs/core/bug-927616.t
    • ./tests/bugs/distribute/bug-1125824.t
    • ./tests/bugs/distribute/bug-1161156.t
    • ./tests/bugs/distribute/bug-1190734.t
    • ./tests/bugs/ec/bug-1187474.t
    • ./tests/bugs/fuse/many-groups-for-acl.t
    • ./tests/bugs/glusterd/bug-1070734.t
    • ./tests/bugs/glusterd/optimized-basic-testcases.t
    • ./tests/bugs/glusterfs-server/bug-864222.t
    • ./tests/bugs/glusterfs-server/bug-887145.t
    • ./tests/bugs/glusterfs-server/bug-904300.t
    • ./tests/bugs/glusterfs/bug-867253.t
    • ./tests/bugs/glusterfs/bug-872923.t
    • ./tests/bugs/nfs/bug-1053579.t
    • ./tests/bugs/nfs/bug-1143880-fix-gNFSd-auth-crash.t
    • ./tests/bugs/nfs/bug-1161092-nfs-acls.t
    • ./tests/bugs/nfs/bug-1166862.t
    • ./tests/bugs/nfs/bug-1210338.t
    • ./tests/bugs/nfs/bug-877885.t
    • ./tests/bugs/nfs/bug-904065.t
    • ./tests/bugs/nfs/bug-915280.t
    • ./tests/bugs/nfs/bug-974972.t
    • ./tests/bugs/nfs/showmount-many-clients.t
    • ./tests/bugs/nfs/socket-as-fifo.t
    • ./tests/bugs/nfs/subdir-trailing-slash.t
    • ./tests/bugs/nfs/zero-atime.t
    • ./tests/bugs/quota/bug-1087198.t
    • ./tests/bugs/quota/bug-1153964.t
    • ./tests/bugs/quota/bug-1243798.t
    • ./tests/bugs/replicate/bug-1340623-mkdir-fails-remove-brick-started.t
    • ./tests/bugs/replicate/bug-830665.t
    • ./tests/bugs/rpc/bug-1043886.t
    • ./tests/bugs/rpc/bug-847624.t
    • ./tests/bugs/rpc/bug-921072.t
    • ./tests/bugs/snapshot/bug-1260848.t
    • ./tests/gfid2path/gfid2path_nfs.t
    cd $SOURCE_ROOT/glusterfs
wget --no-check-certificate $PATCH_URL/nfs.diff
git apply nfs.diff

  • Build and install

    make
make install

Step 2: Run the test cases (optional)

2.1) Install the testing dependencies

  • RHEL (7.8, 7.9)

    yum install -y acl attr bc bind-utils boost-devel docbook-style-xsl expat-devel gdb net-tools nfs-utils psmisc vim xfsprogs yajl redhat-rpm-config perl-Test-Harness popt-devel procps-ng
yum install -y pyxattr python3-pip
pip3 install prettytable

  • RHEL (8.6, 8.7)

    yum install -y acl attr bc bind-utils boost-devel docbook-style-xsl expat-devel gdb net-tools nfs-utils psmisc vim xfsprogs yajl redhat-rpm-config python3-devel python3-pyxattr python3-prettytable perl-Test-Harness popt-devel procps-ng

  • RHEL (9.0, 9.1)

    yum install -y acl attr bc bind-utils boost-devel docbook-style-xsl expat-devel gdb net-tools nfs-utils psmisc vim xfsprogs yajl redhat-rpm-config python3-devel python3-pyxattr python3-prettytable perl-Test-Harness popt-devel procps-ng dbench

  • SLES (12 SP5)

    zypper install -y acl attr bc bind-utils gdb libxml2-tools net-tools nfs-utils psmisc vim xfsprogs python3-xattr libselinux-devel selinux-tools popt-devel python3-pip
pip3 install prettytable

  • SLES (15 SP4)

    zypper install -y acl attr bc bind-utils gdb libxml2-tools net-tools-deprecated nfs-utils psmisc thin-provisioning-tools vim xfsprogs python3-xattr python3-PrettyTable libselinux-devel selinux-tools popt-devel

  • Link pstack command (For SLES only)

    ln -sf `which gstack` /usr/bin/pstack

  • Build DBENCH (For RHEL 7.x, 8.x and SLES 12SP5, 15SP4 only)

    cd $SOURCE_ROOT
git clone https://github.com/sahlberg/dbench
cd dbench
git checkout caa52d347171f96eef5f8c2d6ab04a9152eaf113
./autogen.sh
./configure --datadir=/usr/local/share/doc/loadfiles/
make
make install

  • Build thin-provisioning-tools (For RHEL and SLES12SP5 only)

    cd $SOURCE_ROOT
git clone https://github.com/jthornber/thin-provisioning-tools
cd thin-provisioning-tools
git checkout v0.7.6
autoreconf
./configure
make
make install

  • Build yajl (For SLES only)

    cd $SOURCE_ROOT
git clone https://github.com/lloyd/yajl
cd yajl
git checkout 2.1.0
./configure
make install

2.2) Patch and run tests

  • Patch run-tests.sh (On SLES 12 SP5 only, see this issue):

    cd $SOURCE_ROOT/glusterfs
wget --no-check-certificate $PATCH_URL/run-tests.sh.diff
git apply run-tests.sh.diff

  • The following patch fixes the test cases listed below that fail due to hardcoded hash values for little-endian systems. These failures are fixed on devel branch by this commit:

    • ./tests/bugs/posix/bug-1619720.t
    • ./tests/bugs/replicate/bug-1655854-support-dist-to-rep3-arb-conversion.t
    • ./tests/bugs/distribute/bug-1600379.t
    • ./tests/bugs/glusterd/bug-1699339.t
    • ./tests/bugs/distribute/bug-1786679.t
    • ./tests/bugs/glusterfs/bug-902610.t
    • ./tests/basic/changelog/changelog-rename.t
    • ./tests/basic/distribute/dir-heal.t
    • ./tests/basic/distribute/sparse_file_rebalance.t
    • ./tests/basic/namespace.t
    cd $SOURCE_ROOT/glusterfs
wget --no-check-certificate $PATCH_URL/hash-tests.diff
git apply hash-tests.diff

  • The following patch fixes test cases failing due to system configuration:

    • ./tests/basic/distribute/throttle-rebal.t
    • ./tests/bugs/nfs/bug-847622.t
    • ./tests/00-geo-rep/00-georep-verify-non-root-setup.t
    cd $SOURCE_ROOT/glusterfs
wget --no-check-certificate $PATCH_URL/test-patch.diff
git apply test-patch.diff

  • Fix the ./tests/00-geo-rep/00-georep-verify-non-root-setup.t test case failure (For RHEL only):

    ln -s /usr/local/lib/libgfchangelog.so.0 /lib64/libgfchangelog.so
ldconfig /usr/local/lib
ldconfig /usr/local/lib64

  • Run the test cases:

    cd $SOURCE_ROOT/glusterfs
./run-tests.sh

Note:

  • If you want to use the kernel NFS server or client on a Gluster storage server, make sure to disable the Gluster NFS server:

    gluster volume set <VOLUME> nfs.disable true

  • For geo-replication test cases:

    • Password-less SSH login has to be set up. Refer to the Setting Up the Environment for Geo-replication in the documentation.
    • The locale of the system must be UTF-8 for the test cases to pass.

  • For ./tests/bugs/glusterfs/bug-866459.t test case: This is fixed in devel branch through this commit.

  • Test cases which fail due to snapshot create failed error are fixed in devel branch through this commit.

  • For ./tests/bugs/glusterfs-server/bug-861542.t test case: If the hostname of the machine is long, the test case will fail. The hostname of the machine can be shortened temporarily via the hostname command to get the test case to pass.

  • For ./tests/bugs/glusterd/brick-order-check-add-brick.t test case: If it fails, add force at the end of the failing command in the test case.

  • For ./tests/bugs/glusterd/brick-mux-validation-in-cluster.t test case: If it fails, add sleep 5 before the last command in the test case.

  • The following test cases fail on RHEL 8.x on both Intel and s390x:

    • ./tests/basic/uss.t
    • ./tests/bugs/snapshot/bug-1166197.t
    • ./tests/bugs/replicate/bug-1728770-pass-xattrs.t

  • The test case ./tests/basic/gfapi/gfapi-copy-file-range.t fails for SLES (12 SP5, 15 SP4) on both x86_64 and s390x due to a system call failure. This is beacuse of the unsupported xfs reflink, it can be enabled via echo 'Y' > /sys/module/xfs/parameters/allow_unsupported.

  • The following test cases fail on SLES 15 SP4 and SLES 12 SP5 on both Intel and s390x:

    • ./tests/00-geo-rep/00-georep-verify-non-root-setup.t
    • ./tests/bugs/shard/bug-1251824.t
    • ./tests/bugs/shard/bug-1468483.t

  • The test case ./tests/basic/distribute/directory-rebalance.t fails on SLES 12 SP5 on both Intel and s390x:

  • Test cases such as ./tests/basic/ctime/ctime-utimesat.t fail intermittently. They pass when running individually through prove -vf <testcase>.

  • Certain test cases fail in SLES12SP5 which are being investigated in v11.0

    • ./tests/00-geo-rep/georep-basic-dr-tarssh-arbiter.t
    • ./tests/00-geo-rep/georep-basic-dr-tarssh.t
    • ./tests/bugs/gfapi/bug-1447266/1460514.t

  • ./tests/basic/fencing/afr-lock-heal-basic.t and ./tests/basic/fencing/afr-lock-heal-advanced.t test cases in RHEL9.0, 9.1 and the following test cases in RHEL9.0 fail which are being investigated in v11.0

    • ./tests/bugs/access-control/bug-958691.t
    • ./tests/bugs/bug-1258069.t
    • ./tests/bugs/distribute/bug-1125824.t
    • ./tests/bugs/glusterfs-server/bug-904300.t
    • ./tests/bugs/glusterfs/bug-866459.t
    • ./tests/bugs/glusterfs/bug-867253.t
    • ./tests/bugs/glusterfs/bug-872923.t
    • ./tests/bugs/nfs/bug-1143880-fix-gNFSd-auth-crash.t
    • ./tests/bugs/nfs/bug-1157223-symlink-mounting.t
    • ./tests/bugs/nfs/bug-847622.t
    • ./tests/bugs/nfs/bug-904065.t
    • ./tests/bugs/nfs/bug-915280.t
    • ./tests/bugs/rpc/bug-921072.t

  • Additional information on the GlusterFS test framework can be found here.

Step 3: Run the GlusterFS Daemon (Optional)

glusterd

Similar logs should be output in /var/log/glusterfs/glusterd.log:

[2023-03-21 11:47:58.738745 +0000] I [MSGID: 100030] [glusterfsd.c:2769:main] 0-glusterd: Started running version [{arg=glusterd}, {version=10.3}, {cmdlinestr=glusterd}]

Step 4: Test the GlusterFS volume (Optional)

Refer to Documentation for instructions.

References

The information provided in this article is accurate at the time of writing, but on-going development in the open-source projects involved may make the information incorrect or obsolete. Please open issue or contact us on IBM Z Community if you have any questions or feedback.

Clone this wiki locally