Can't mount volume

[renich]

I tried the following in a vanilla centos7:

# remove any old version
yum -y remove docker docker-client docker-client-latest docker-common docker-latest docker-latest-logrotate docker-logrotate
docker-engine

# isntall it
yum install -y yum-utils   device-mapper-persistent-data lvm2
yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
yum -y install docker-ce docker-ce-cli containerd.io
systemctl enable --now docker

# enable plugin
plugin_alias='glusterfs-plugin'
servers='glusterfs0,glusterfs1,glusterfs2'

docker plugin install --alias $plugin_alias trajano/glusterfs-volume-plugin --grant-all-permissions --disable
docker plugin set $plugin_alias SERVERS="$servers"
docker plugin enable $plugin_alias

# create volume
docker volume create -d glusterfs-plugin:latest test

# try it
docker run -it -v test:/mnt alpine

If fails like this:

[root@glusterfs0 ~]# docker run -it -v test:/mnt alpine
^[[3~docker: Error response from daemon: VolumeDriver.Mount: error mounting test: exit status 1.

And the logs say:

Apr 11 01:36:16 glusterfs0.ditas.mia.cloudsigma.com dockerd[19413]: time="2019-04-11T01:36:16Z" level=error msg="Entering go-plugins-helpers capabilitiesPath" plugin=8d4372842780942f912a8fe61ea2d31c2fc253eeba7a1f4f86968bba9a59063b
Apr 11 01:36:16 glusterfs0.ditas.mia.cloudsigma.com dockerd[19413]: time="2019-04-11T01:36:16Z" level=error msg="Entering go-plugins-helpers getPath" plugin=8d4372842780942f912a8fe61ea2d31c2fc253eeba7a1f4f86968bba9a59063b
Apr 11 01:36:16 glusterfs0.ditas.mia.cloudsigma.com dockerd[19413]: time="2019-04-11T01:36:16Z" level=error msg="Entering go-plugins-helpers capabilitiesPath" plugin=8d4372842780942f912a8fe61ea2d31c2fc253eeba7a1f4f86968bba9a59063b
Apr 11 01:36:16 glusterfs0.ditas.mia.cloudsigma.com dockerd[19413]: time="2019-04-11T01:36:16Z" level=error msg="Entering go-plugins-helpers getPath" plugin=8d4372842780942f912a8fe61ea2d31c2fc253eeba7a1f4f86968bba9a59063b
Apr 11 01:36:16 glusterfs0.ditas.mia.cloudsigma.com dockerd[19413]: time="2019-04-11T01:36:16Z" level=error msg="Entering go-plugins-helpers capabilitiesPath" plugin=8d4372842780942f912a8fe61ea2d31c2fc253eeba7a1f4f86968bba9a59063b
Apr 11 01:36:16 glusterfs0.ditas.mia.cloudsigma.com dockerd[19413]: time="2019-04-11T01:36:16Z" level=error msg="Entering go-plugins-helpers mountPath" plugin=8d4372842780942f912a8fe61ea2d31c2fc253eeba7a1f4f86968bba9a59063b
Apr 11 01:36:16 glusterfs0.ditas.mia.cloudsigma.com dockerd[19413]: time="2019-04-11T01:36:16Z" level=error msg="[-s glusterfs0 -s glusterfs1 -s glusterfs2 --volfile-id=test /var/lib/docker-volumes/d4781c258094fd2c306d11a9922bf0410b7b6d151bf50cf80f6949a50c97b044]" plugin=8d4372842780942f912a8fe61ea2d31c2fc253eeba7a1f4f86968bba9a59063b
Apr 11 01:36:16 glusterfs0.ditas.mia.cloudsigma.com dockerd[19413]: time="2019-04-11T01:36:16Z" level=info msg="Command output: " plugin=8d4372842780942f912a8fe61ea2d31c2fc253eeba7a1f4f86968bba9a59063b
Apr 11 01:36:16 glusterfs0.ditas.mia.cloudsigma.com dockerd[19413]: time="2019-04-11T01:36:16.141200212Z" level=error msg="Handler for POST /v1.39/containers/create returned error: VolumeDriver.Mount: error mounting test: exit status 1"
Apr 11 01:36:25 glusterfs0.ditas.mia.cloudsigma.com dhclient[3334]: DHCPREQUEST on eth0 to 10.20.241.23 port 67 (xid=0xc9099cc)

Also, check this out:

[root@glusterfs0 ~]# docker volume list
DRIVER                    VOLUME NAME
glusterfs-plugin:latest   test

[root@glusterfs0 ~]# docker volume inspect test
[
    {
        "CreatedAt": "0001-01-01T00:00:00Z",
        "Driver": "glusterfs-plugin:latest",
        "Labels": {},
        "Mountpoint": "",
        "Name": "test",
        "Options": {},
        "Scope": "global",
        "Status": {
            "args": [
                "-s",
                "glusterfs0",
                "-s",
                "glusterfs1",
                "-s",
                "glusterfs2",
                "--volfile-id=test"
            ],
            "mounted": false
        }
    }
]

[root@glusterfs0 ~]# gluster volume list
No volumes present in cluster

[root@glusterfs0 ~]# gluster peer status
Number of Peers: 2

Hostname: glusterfs2
Uuid: ab2cdee4-7257-4238-8aa7-a9a3068f2398
State: Peer in Cluster (Connected)

Hostname: glusterfs1
Uuid: fb556123-992c-4021-a169-2eafe0a7ec00
State: Peer in Cluster (Connected)

[trajano]

https://github.com/trajano/docker-volume-plugins/tree/master/glusterfs-volume-plugin#just-the-name the name should not be test but should be volume/subdir

[renich]

I don't understand. In this example: https://github.com/trajano/docker-volume-plugins/tree/master/glusterfs-volume-plugin#testing-outside-the-swarm you do exactly this, don't you?

[trajano]

The approach you specified assumes that you have a glusterfs volume called trajano.

In your case you're using the volume name of test but when you listed your volumes in glusterfs test is not present.

The glusterfs volumes need to exist before connecting to it using this plugin.

[renich]

Oh, so I need to create the volume? OK, I will try that then.

[renich]

So, I've created the volume and started it:

gluster volume create test glusterfs0:/srv/glusterfs/brick0/test glusterfs1:/srv/glusterfs/brick0/test glusterfs2:/srv/glusterfs/brick0/test
gluster volume start test

# list it (it shows)
gluster volume list

# try to mount it into docker
docker run -it -v test:/mnt alpine

I get these errors (journald)

Apr 12 06:39:03 glusterfs0.ditas.mia.cloudsigma.com dockerd[19413]: time="2019-04-12T06:39:03Z" level=error msg="Entering go-plugins-helpers capabilitiesPath" plugin=8d4372842780942f912a8fe61ea2d31c2fc253eeba7a1f4f86968bba9a59063b
Apr 12 06:39:03 glusterfs0.ditas.mia.cloudsigma.com dockerd[19413]: time="2019-04-12T06:39:03Z" level=error msg="Entering go-plugins-helpers getPath" plugin=8d4372842780942f912a8fe61ea2d31c2fc253eeba7a1f4f86968bba9a59063b
Apr 12 06:39:03 glusterfs0.ditas.mia.cloudsigma.com dockerd[19413]: time="2019-04-12T06:39:03Z" level=error msg="Entering go-plugins-helpers capabilitiesPath" plugin=8d4372842780942f912a8fe61ea2d31c2fc253eeba7a1f4f86968bba9a59063b
Apr 12 06:39:03 glusterfs0.ditas.mia.cloudsigma.com dockerd[19413]: time="2019-04-12T06:39:03Z" level=error msg="Entering go-plugins-helpers getPath" plugin=8d4372842780942f912a8fe61ea2d31c2fc253eeba7a1f4f86968bba9a59063b
Apr 12 06:39:03 glusterfs0.ditas.mia.cloudsigma.com dockerd[19413]: time="2019-04-12T06:39:03Z" level=error msg="Entering go-plugins-helpers capabilitiesPath" plugin=8d4372842780942f912a8fe61ea2d31c2fc253eeba7a1f4f86968bba9a59063b
Apr 12 06:39:03 glusterfs0.ditas.mia.cloudsigma.com dockerd[19413]: time="2019-04-12T06:39:03Z" level=error msg="Entering go-plugins-helpers mountPath" plugin=8d4372842780942f912a8fe61ea2d31c2fc253eeba7a1f4f86968bba9a59063b
Apr 12 06:39:03 glusterfs0.ditas.mia.cloudsigma.com dockerd[19413]: time="2019-04-12T06:39:03Z" level=error msg="[-s glusterfs0 -s glusterfs1 -s glusterfs2 --volfile-id=test /var/lib/docker-volumes/38c93ca685f2f6d1542893ae2fcf9c9481039097e5332d24bd4dcdd36f5f9f6c]" plugin=8d4372842780942f912a8fe61ea2d31c2fc253eeba7a1f4f86968bba9a59063b
Apr 12 06:39:03 glusterfs0.ditas.mia.cloudsigma.com dockerd[19413]: time="2019-04-12T06:39:03Z" level=info msg="Command output: " plugin=8d4372842780942f912a8fe61ea2d31c2fc253eeba7a1f4f86968bba9a59063b
Apr 12 06:39:03 glusterfs0.ditas.mia.cloudsigma.com dockerd[19413]: time="2019-04-12T06:39:03.977326820Z" level=error msg="Handler for POST /v1.39/containers/create returned error: VolumeDriver.Mount: error mounting test: exit status 1"

On the other hand, this works: mount -t glusterfs glusterfs0:/test /mnt.

Any ideas?

[trajano]

the resulting command should be something like (I changed the mount point so you can test)

glusterfs -s glusterfs0 -s glusterfs1 -s glusterfs2 --volfile-id=test /mnt

FYI the plugin still uses glusterfs 3.x

[renich]

Thank you for the personalized example. I am using 6.0-1... :S

In any case, I did the following an dit didn't work either:

# undo
docker volume rm trajano
docker plugin disable trajano/glusterfs-volume-plugin
docker plugin rm trajano/glusterfs-volume-plugin

# do
docker plugin install trajano/glusterfs-volume-plugin --grant-all-permissions
docker plugin enable trajano/glusterfs-volume-plugin
docker volume create -d trajano/glusterfs-volume-plugin --opt servers=glusterfs0,glusterfs1,glusterfs2 trajano
docker run -it -v trajano:/mnt alpine

It didn't work. Will try the glusterfs -s command to see if it works. Sorry for all the trouble. I don't think I should bother you anymore.

[codedge]

I am running GlusterFS 6.5 and can confirm that volume creating and mounting works - at least on the command line.

WORKS

$ docker volume create -d glusterfs:latest lei01/export
$ docker run -it -v lei01/export alpine

DOES NOT WORK
What does not work for me is when running it via docker-compose inside a Docker Swarm via Portainer:

version: "3.4"

services:
  foo:
    image: alpine
    command: ping localhost
    networks:
      - net
    volumes:
      - vol1:/tmp
    deploy:
      placement:
        constraints: [node.role == worker]

networks:
  net:
    driver: overlay

volumes:
  vol1:
    driver: glusterfs
    name: "lei01/export"

Then I always get the VolumeDriver.Mount: error mounting... error message.

Any idea?

Same issue here

[trajano]

The plugin is using glusterfs 3.x so it may not work with newer versions of Gluster.