Passing special mount options through glusteropts option ( enable-ino32 )

[kybeg]

Hi,

i'm setting up a cockroachDB cluster and am using this plugin for glusterFS access. Everything works great until I discovered, that the db won't start due to a missing mount option.

I have verified that this will work:

glusterfs -s 192.168.128.107 -s 192.168.128.240 -s 192.168.128.97 --enable-ino32=on   --volfile-id=db1 --volfile-server=192.168.128.107 /mnt

And then manually start a container with the folder attached as a volume:

docker run -ti -v /mnt:/mnt roach:v2

However, when I try this using docker-compose and this plugin it won't work.

This is how the volume is specified in the compose file:

volumes:
  codb-1:
     driver: glusterfs
     driver_opts:
            glusteropts: "-s 192.168.128.107 -s 192.168.128.240 -s 192.168.128.97 --enable-ino32=on   --volfile-id=db1 --volfile-server=192.168.128.107 "

The log messages i get on the host where this starts are:

Mar 28 11:13:05 imt3003worker-3-2 dockerd[9247]: time="2019-03-28T11:13:05Z" level=error msg="Entering go-plugins-helpers mountPath" plugin=c268b167dd63d1515e32b0020743eb2614414266c7aa4e5e8505cfda6b703885
Mar 28 11:13:05 imt3003worker-3-2 dockerd[9247]: time="2019-03-28T11:13:05Z" level=error msg="[-s 192.168.128.107 -s 192.168.128.240 -s 192.168.128.97 --enable-ino32=on   --volfile-id=db1 --volfile-server=192.168.128.107  /var/lib/docker-volumes/800abb5a25f11b755e27d019c3ad781644fca0ce14916efee4aee77a1f11ae13]" plugin=c268b167dd63d1515e32b0020743eb2614414266c7aa4e5e8505cfda6b703885
Mar 28 11:13:05 imt3003worker-3-2 dockerd[9247]: time="2019-03-28T11:13:05Z" level=info msg="Command output: Usage: glusterfs [OPTION...] --volfile-server=SERVER [MOUNT-POINT]" plugin=c268b167dd63d1515e32b0020743eb2614414266c7aa4e5e8505cfda6b703885
Mar 28 11:13:05 imt3003worker-3-2 dockerd[9247]: time="2019-03-28T11:13:05Z" level=info msg="  or:  glusterfs [OPTION...] --volfile=VOLFILE [MOUNT-POINT]" plugin=c268b167dd63d1515e32b0020743eb2614414266c7aa4e5e8505cfda6b703885
Mar 28 11:13:05 imt3003worker-3-2 dockerd[9247]: time="2019-03-28T11:13:05Z" level=info msg="Try `glusterfs --help' or `glusterfs --usage' for more information." plugin=c268b167dd63d1515e32b0020743eb2614414266c7aa4e5e8505cfda6b703885
Mar 28 11:13:05 imt3003worker-3-2 dockerd[9247]: time="2019-03-28T11:13:05Z" level=info plugin=c268b167dd63d1515e32b0020743eb2614414266c7aa4e5e8505cfda6b703885
Mar 28 11:13:05 imt3003worker-3-2 dockerd[9247]: time="2019-03-28T11:13:05.570778395Z" level=error msg="fatal task error" error="VolumeDriver.Mount: error mounting roach_codb-1: exit status 64" module=node/agent/taskmanager node.id=ovme1tww944ctcvltyv3eu4ab service.id=09gkviz967ljg4atnzadl4brk task.id=pgy4fel3e9kmv9zrr10iexxay

Also, the README uses glusterfsopts, but the code seems to refer to glusteropts, which appears to be working for me.

I understand that the glusterfs client used in the plugin ( CentOS) is not the same as I'm using locally ( Ubuntu ). Any advice on how to proceed would be greatly appreciated.

[trajano]

You're looking for --enable-ino32=on ?

[trajano]

Also we're using 3.x GlusterFS in this plugin. It may be the cause of your issues.

[kybeg]

Yes, its the --enable-ino32=on option, I can use it fine when I use the glusterfs client manually, but when i use it in the docker-compose.yml file it fails. However, I'm not sure it is the option per se, that causes this, or some other issue with the way i'm specifying it.

[trajano]

I think it may be a GlusterFS version mismatch

[chrisbecke]

I'm just looking at at the extra spaces after the --enable-ino32 option in the docker-compose glusteropts. The plugin, for the most part, seems to exec out with exactly the the command line its given, so any glusterfs command that works from the cli should work from the plugin.

Also -s is an alias for --volfile-server, which means you are passing the same server twice.