Skip to content

Building GlusterFS

Jump to bottom
aborkar-ibm edited this page Mar 15, 2019 · 39 revisions

Building GlusterFS

Below versions of GlusterFS are available in respective distributions at the time of creation of these build instructions:

  • Ubuntu 16.04 - 3.7.6
  • Ubuntu 18.04 - 3.13.2

The instructions provided below specify the steps to build GlusterFS 5.3 on Linux on IBM Z for following distributions:

  • RHEL (7.4, 7.5, 7.6)
  • SLES (12 SP3, 15)
  • Ubuntu (16.04, 18.04)

General Note:

  • When following the steps below please use a super user unless otherwise specified
  • A directory /<source root>/ will be referred to in these instructions, this is a temporary writable directory anywhere you'd like to place it

Building and Installing GlusterFS

Step 1: Build using script

If you want to build GlusterFS using manual steps, go to Step 2.

Use the following commands to build GlusterFS using the build script. Please make sure you have wget installed.

wget -q https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/GlusterFS/build_glusterfs.sh
# Build GlusterFS
bash build_glusterfs.sh   [Provide -t option for executing build with tests]

If the build completes successfully, go to STEP 5. In case of error, check logs for more details or go to STEP 2 to follow manual build steps.

Step 2: Install the dependencies

  • RHEL (7.4, 7.5, 7.6)

    yum install wget git make gcc-c++ libaio-devel boost-devel expat-devel autoconf autoheader automake libtool flex bison openssl-devel libacl-devel sqlite-devel libxml2-devel python-devel python pyxattr attr yajl-devel nfs-utils xfsprogs-devel popt-static sysvinit-tools psmisc libibverbs-devel librdmacm-devel readline-devel lvm2-devel glib2-devel fuse-devel bc libuuid-devel net-tools vim-common gdb

    • Building urcu

      cd /<source_root>/
git clone git://git.liburcu.org/userspace-rcu.git
cd userspace-rcu
git checkout v0.10.2
./bootstrap
./configure
make
make install
ldconfig

    • Building thin-provisioning-tools

      cd /<source_root>/
git clone https://github.com/jthornber/thin-provisioning-tools
cd thin-provisioning-tools
git checkout v0.7.6
autoreconf
./configure
make
make install

  • SLES 12 SP3

    zypper install wget which git make gcc-c++ libaio-devel boost-devel autoconf automake cmake libtool flex bison lvm2-devel libacl-devel python-devel python python-xattr attr xfsprogs-devel sysvinit-tools psmisc bc libopenssl-devel libxml2-devel sqlite3 sqlite3-devel popt-devel nfs-utils python-xml net-tools libuuid-devel liburcu-devel libyajl-devel gdb acl

  • SLES 15

    zypper install wget which git make gcc-c++ libaio-devel boost-devel autoconf automake cmake libtool flex bison lvm2-devel libacl-devel python3-devel python3 python3-xattr attr xfsprogs-devel sysvinit-tools psmisc bc libopenssl-devel libxml2-devel sqlite3 sqlite3-devel popt-devel nfs-utils python-xml net-tools-deprecated libuuid-devel liburcu-devel libyajl-devel gdb acl

  • Ubuntu (16.04, 18.04)

    apt-get update
apt-get install make automake autoconf libtool flex bison pkg-config libssl-dev libxml2-dev python3-dev libaio-dev libibverbs-dev librdmacm-dev libreadline-dev liblvm2-dev libglib2.0-dev liburcu-dev libcmocka-dev libsqlite3-dev libacl1-dev wget tar dbench git xfsprogs attr nfs-common yajl-tools sqlite3 libxml2-utils thin-provisioning-tools bc uuid-dev net-tools vim-common

Step 3: Download , Configure and Build GlusterFS

  • Download and configure source

    cd /<source_root>/
git clone https://github.com/gluster/glusterfs
cd glusterfs
git checkout v5.3
./autogen.sh
./configure --enable-gnfs			# for RHEL, SLES 15 and Ubuntu
./configure --enable-gnfs --disable-events	# for SLES 12 SP3

  • Update rculist-extra.h (Only for RHEL)

    rm contrib/userspace-rcu/rculist-extra.h
cp /usr/local/include/urcu/rculist.h contrib/userspace-rcu/rculist-extra.h

  • Create a patch file patch_io-threads.diff with the following contents: (Only for SLES & Ubuntu)

    diff --git a/xlators/performance/io-threads/src/io-threads.h b/xlators/performance/io-threads/src/io-threads.h
index 949d1d1b1..be5c66d7b 100644
--- a/xlators/performance/io-threads/src/io-threads.h
+++ b/xlators/performance/io-threads/src/io-threads.h
@@ -34,7 +34,7 @@ struct iot_conf;
 #define IOT_DEFAULT_THREADS 16
 #define IOT_MAX_THREADS 64

-#define IOT_THREAD_STACK_SIZE ((size_t)(256 * 1024))
+#define IOT_THREAD_STACK_SIZE ((size_t)(512 * 1024))

 typedef struct {
     struct list_head clients;

    Apply the patch using the command below:

    patch -p1 < patch_io-threads.diff

  • Build and install

    make
make install

  • Set environment variables

    export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH
ldconfig

Step 4: Run the test cases (optional)

  • Building DBENCH (Only for RHEL and SLES)

    cd /<source_root>/
git clone https://github.com/sahlberg/dbench
cd dbench
./autogen.sh
./configure
make
make install

  • Building YAJL (Only for SLES)

    cd /<source_root>/
git clone https://github.com/lloyd/yajl
cd yajl
git checkout 2.1.0
./configure
make install

  • Create a patch file patch_tests.diff with the following contents:

    diff --git a/tests/basic/distribute/throttle-rebal.t b/tests/basic/distribute/throttle-rebal.t
index f4823cf4f..1cc3cac5d 100644
--- a/tests/basic/distribute/throttle-rebal.t
+++ b/tests/basic/distribute/throttle-rebal.t
@@ -17,7 +17,7 @@ function set_throttle {
}

 #Determine number of cores
-cores=$(cat /proc/cpuinfo | grep processor | wc -l)
+cores=$(cat /proc/cpuinfo | grep -w processor | wc -l)
 if [ "$cores" == "" ]; then
         echo "Could not get number of cores available"
 fi
diff --git a/tests/basic/mount-nfs-auth.t b/tests/basic/mount-nfs-auth.t
index 0d4e01abb..d6e11e810 100755
--- a/tests/basic/mount-nfs-auth.t
+++ b/tests/basic/mount-nfs-auth.t
@@ -15,16 +15,16 @@ TEST glusterd
 TEST pidof glusterd
 TEST $CLI volume info

-H0IP=$(ip addr show |grep -w inet |grep -v 127.0.0.1|awk '{ print $2 }'| cut -d "/" -f 1)
+H0IP=$(hostname -i | awk '{ print $1 }')
 H0IP6=$(host $HOSTNAME | grep IPv6 | awk '{print $NF}')

 # Export variables for allow & deny
-EXPORT_ALLOW="/$V0 $H0(sec=sys,rw,anonuid=0) @ngtop(sec=sys,rw,anonuid=0)"
-EXPORT_ALLOW_SLASH="/$V0/ $H0(sec=sys,rw,anonuid=0) @ngtop(sec=sys,rw,anonuid=0)"
+EXPORT_ALLOW="/$V0 $H0IP(sec=sys,rw,anonuid=0) @ngtop(sec=sys,rw,anonuid=0)"
+EXPORT_ALLOW_SLASH="/$V0/ $H0IP(sec=sys,rw,anonuid=0) @ngtop(sec=sys,rw,anonuid=0)"
 EXPORT_DENY="/$V0 1.2.3.4(sec=sys,rw,anonuid=0) @ngtop(sec=sys,rw,anonuid=0)"

 # Netgroup variables for allow & deny
-NETGROUP_ALLOW="ngtop ng1000\nng1000 ng999\nng999 ng1\nng1 ng2\nng2 ($H0,,)"
+NETGROUP_ALLOW="ngtop ng1000\nng1000 ng999\nng999 ng1\nng1 ng2\nng2 ($H0IP,,)"
 NETGROUP_DENY="ngtop ng1000\nng1000 ng999\nng999 ng1\nng1 ng2\nng2 (1.2.3.4,,)"

 V0L1="$V0/L1"
@@ -32,8 +32,8 @@ V0L2="$V0L1/L2"
 V0L3="$V0L2/L3"

 # Other variations for allow & deny
-EXPORT_ALLOW_RO="/$V0 $H0(sec=sys,ro,anonuid=0) @ngtop(sec=sys,ro,anonuid=0)"
-EXPORT_ALLOW_L1="/$V0L1 $H0(sec=sys,rw,anonuid=0) @ngtop(sec=sys,rw,anonuid=0)"
+EXPORT_ALLOW_RO="/$V0 $H0IP(sec=sys,ro,anonuid=0) @ngtop(sec=sys,ro,anonuid=0)"
+EXPORT_ALLOW_L1="/$V0L1 $H0IP(sec=sys,rw,anonuid=0) @ngtop(sec=sys,rw,anonuid=0)"
 EXPORT_WILDCARD="/$V0 *(sec=sys,rw,anonuid=0) @ngtop(sec=sys,rw,anonuid=0)"

 function build_dirs () {
diff --git a/tests/bugs/nfs/bug-847622.t b/tests/bugs/nfs/bug-847622.t
index 3b836745a..f0b3e0f3d 100755
--- a/tests/bugs/nfs/bug-847622.t
+++ b/tests/bugs/nfs/bug-847622.t
@@ -28,7 +28,7 @@ cd $N0

 # simple getfacl setfacl commands
 TEST touch testfile
-TEST setfacl -m u:14:r testfile
+TEST setfacl -m u:14:r $B0/brick0/testfile
 TEST getfacl testfile

 cd
diff --git a/tests/bugs/distribute/bug-1161311.t b/tests/bugs/distribute/bug-1161311.t
index d88642e..f01da99 100755
--- a/tests/bugs/distribute/bug-1161311.t
+++ b/tests/bugs/distribute/bug-1161311.t
@@ -89,6 +89,8 @@ done
 ## into separate bricks when brick count is 3
 TEST mv $M0/dir1/FILE2 $M0/dir1/FILE1

+brick_loc=$(get_backend_paths $M0/dir1/FILE1)
+
 # unmount and remount the volume
 EXPECT_WITHIN $UMOUNT_TIMEOUT "Y" force_umount $M0
 TEST glusterfs -s $H0 --volfile-id $V0 $M0;
@@ -98,7 +100,7 @@ TEST $CLI volume rebalance $V0 start force

 # Wait for FILE to get the sticky bit on, so that file is under
 # active rebalance, before creating the links
-TEST checksticky $B0/${V0}3/dir1/FILE1
+TEST checksticky $brick_loc

 # Create the links
 ## FILE3 FILE5 FILE7 have hashes, c8c91469 566d26ce 22ce7eba
@@ -119,7 +121,7 @@ cd /

 # Ideally for this test to have done its job, the file should still be
 # under migration, so check the sticky bit again
-TEST checksticky $B0/${V0}3/dir1/FILE1
+TEST checksticky $brick_loc

 # Wait for rebalance to complete
 EXPECT_WITHIN $REBALANCE_TIMEOUT "completed" rebalance_status_field $V0
diff --git a/tests/bugs/distribute/bug-1193636.t b/tests/bugs/distribute/bug-1193636.t
index ccde02e..4e8e886 100644
--- a/tests/bugs/distribute/bug-1193636.t
+++ b/tests/bugs/distribute/bug-1193636.t
@@ -2,6 +2,7 @@

 . $(dirname $0)/../../include.rc
 . $(dirname $0)/../../volume.rc
+. $(dirname $0)/../../dht.rc


 checksticky () {
@@ -41,11 +42,13 @@ dd if=/dev/zero of=$M0/dir1/FILE2 bs=64k count=10240
 # act on the file
 TEST mv $M0/dir1/FILE2 $M0/dir1/FILE1

+brick_loc=$(get_backend_paths $M0/dir1/FILE1)
+
 build_tester $(dirname $0)/bug-1193636.c

 TEST $CLI volume rebalance $V0 start force

-TEST checksticky $B0/${V0}3/dir1/FILE1
+TEST checksticky $brick_loc

 TEST setfattr -n "user.test1" -v "test1" $M0/dir1/FILE1
 TEST setfattr -n "user.test2" -v "test1" $M0/dir1/FILE1
diff --git a/tests/basic/namespace.t b/tests/basic/namespace.t
index 25a6b98b4..91cd60163 100644
--- a/tests/basic/namespace.t
+++ b/tests/basic/namespace.t
@@ -6,9 +6,10 @@
 
# These hashes are a result of calling SuperFastHash
# on the corresponding folder names.
-NAMESPACE_HASH=28153613
-NAMESPACE2_HASH=3926991974
-NAMESPACE3_HASH=3493960770
+NAMESPACE_HASH=3253352021
+NAMESPACE2_HASH=458775276
+NAMESPACE3_HASH=1268089390
+
 
function check_brick_multiplex() {
  	local ret=$($CLI volume info|grep "cluster.brick-multiplex"|cut -d" " -f2)
@@ -110,12 +111,12 @@ EXPECT_WITHIN 10 "Y" check_samples LOOKUP $NAMESPACE_HASH /namespace patchy0

mkdir -p $N0/namespace{2,3}
EXPECT_WITHIN 10 "Y" check_samples MKDIR $NAMESPACE2_HASH /namespace2 patchy0
-EXPECT_WITHIN 10 "Y" check_samples MKDIR $NAMESPACE3_HASH /namespace3 patchy0
+EXPECT_WITHIN 10 "Y" check_samples MKDIR $NAMESPACE3_HASH /namespace3 patchy6

touch $N0/namespace2/file
touch $N0/namespace3/file
EXPECT_WITHIN 10 "Y" check_samples CREATE $NAMESPACE2_HASH /namespace2/file patchy0
-EXPECT_WITHIN 10 "Y" check_samples CREATE $NAMESPACE3_HASH /namespace3/file patchy0
+EXPECT_WITHIN 10 "Y" check_samples CREATE $NAMESPACE3_HASH /namespace3/file patchy6
 
truncate -s 0 $N0/namespace/bar
EXPECT_WITHIN 10 "Y" check_samples TRUNCATE $NAMESPACE_HASH /namespace/bar patchy0

    Apply the patch using the command below:

    patch -p1 < patch_tests.diff

  • Create a patch file patch_run-tests.diff with the following contents:
    (Only for Ubuntu 16.04 and SLES 12 SP3, reference to the issue.)

    diff --git a/run-tests.sh b/run-tests.sh
index 4e722d211..d7dcac03d 100755
--- a/run-tests.sh
+++ b/run-tests.sh
@@ -118,10 +118,10 @@ function check_dependencies()
     fi

     # Check for netstat
-    env netstat --version > /dev/null 2>&1
-    if [ $? -ne 0 ]; then
-        MISSING="$MISSING netstat"
-    fi
+    #env netstat --version > /dev/null 2>&1
+    #if [ $? -ne 0 ]; then
+    #    MISSING="$MISSING netstat"
+    #fi

     # Check for the Perl Test Harness
     env prove --version > /dev/null 2>&1
@@ -155,10 +155,10 @@ function check_dependencies()
     fi

     # Check for netstat
-    env netstat --version > /dev/null 2>&1
-    if [ $? -ne 0 ]; then
-        MISSING="$MISSING netstat"
-    fi
+    #env netstat --version > /dev/null 2>&1
+    #if [ $? -ne 0 ]; then
+    #    MISSING="$MISSING netstat"
+    #fi

     # check for psutil python package
     test `uname -s` == "Darwin" || test `uname -s` == "FreeBSD" && {

    Apply the patch using the command below:

    patch -p1 < patch_run-tests.diff

  • Run the test cases

    cd /<source_root>/glusterfs
./run-tests.sh

Note:

  • If any of the tests fail with error volume create: patchy: failed: Host is not in 'Peer in Cluster' state, make changes to file /etc/hosts by copying the host details entry that is present in the beginning to the section IPv6 capable hosts and replace 0.0.0.0 with the IP address of the host. It should be similar to : 0.0.0.0 .xyz.com hostname

  • If you want to use the kernel NFS server or client on a Gluster storage server, make sure to disable the Gluster NFS server:

    gluster volume set <VOLUME> nfs.disable true

  • For geo-replication test case : Password-less SSH login has to be set up. Refer to the Setting Up the Environment for Geo-replication in the documentation.

  • The patch for io-threads.h will allow weighted-rebalance.t test case to pass.

  • The test failures below are observed on s390x due to the little endian/big endian specific hash values calculations and can be ignored. Community is looking into it. Refer to this issue for further info.
    /tests/basic/changelog/changelog-rename.t
    /tests/bugs/posix/bug-1619720.t
    /tests/features/trash.t

  • For bug-1207627-bitrot-scrub-status.t test case : This test failure is under investigation, refer to this issue for details.

  • For ssl-ciphers.t test case (Only for SLES 15) : The test case is failing on s390x as well as on x86. Refer issue for details.

  • This test is under investigation : /tests/bugs/glusterfs/bug-866459.t.Refer issue for details.

  • For /tests/bugs/glusterfs/bug-902610.t test case : This feature is not supported by Gluster anymore and it will be removed in the later version. Refer to this issue for details.

  • Additional information on the GlusterFS test framework can be found here.

Step 5: Run the GlusterFS Daemon (Optional)

glusterd

Step 6: Testing the GlusterFS volume (Optional)

Refer Documentation for instructions.

Reference

https://www.gluster.org/
https://github.com/gluster/glusterfs
http://docs.gluster.org/en/latest/Developer-guide/Building-GlusterFS/

The information provided in this article is accurate at the time of writing, but on-going development in the open-source projects involved may make the information incorrect or obsolete. Please open issue or contact us on IBM Z Community if you have any questions or feedback.

Clone this wiki locally