glusterfs on kubernetes stock on Error waiting for job 'heketi-storage-copy-job' to complete. stage

[ehsanrocket]

hi all
I have set up a kubernetes cluster with the following specs:
kubeadm version 1.18.2
8 nodes with ubuntu 18.04 LTS

I have edited the kube-templates scripts for Deployment apiVersion to be apps/v1 and Service apiVersion to v1 and also added the spec.selector: to wokrs on kubernetes version 1.18.2
the glusterfs-server glusterfs-common and glusterfs-client packags are installed on the nodes

for glusterfs storage I have followed the instuctions, edit the topology file and add the needed firewall rules
but it stock at the Error waiting for job 'heketi-storage-copy-job' to complete. stage

here are the heketi-storage-copy-job events:

Events:
  Type     Reason       Age        From                        Message
  ----     ------       ----       ----                        -------
  Normal   Scheduled    <unknown>  default-scheduler           Successfully assigned glusterfs/heketi-storage-copy-job-ftqps to server-xxx
  Warning  FailedMount  19m        kubelet, server-xxx  MountVolume.SetUp failed for volume "heketi-storage" : mount failed: mount failed: exit status 1
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/2aca16e0-9080-440e-8340-0ab1ad8ffbff/volumes/kubernetes.io~glusterfs/heketi-stor
age --scope -- mount -t glusterfs -o auto_unmount,backup-volfile-servers=10.0.0.1:10.0.0.2:10.0.0.3,log-file=/var/lib/kubelet/plugins/kubernetes.io/
glusterfs/heketi-storage/heketi-storage-copy-job-ftqps-glusterfs.log,log-level=ERROR 10.0.0.2:heketidbstorage /var/lib/kubelet/pods/2aca16e0-9080-440e-8340-0ab1a
d8ffbff/volumes/kubernetes.io~glusterfs/heketi-storage
Output: Running scope as unit: run-r7c281513866a4b04b6ca2737d65e886f.scope
Mount failed. Please check the log file for more details.
, the following error information was pulled from the glusterfs log to help diagnose this issue:
[2020-05-17 16:40:22.243907] E [MSGID: 100026] [glusterfsd.c:2403:glusterfs_process_volfp] 0-: failed to construct the graph
[2020-05-17 16:40:22.244187] E [graph.c:1102:glusterfs_graph_destroy] (-->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x537) [0x556eaf57fe77] -->/usr/sbin/glusterfs(glusterfs
_process_volfp+0x180) [0x556eaf579ea0] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(glusterfs_graph_destroy+0x6b) [0x7f29a9580eab] ) 0-graph: invalid argument: graph
[Invalid argument]
  Warning  FailedMount  19m  kubelet, srvarvan-xxx  MountVolume.SetUp failed for volume "heketi-storage" : mount failed: mount failed: exit status 1
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/2aca16e0-9080-440e-8340-0ab1ad8ffbff/volumes/kubernetes.io~glusterfs/heketi-stor
age --scope -- mount -t glusterfs -o auto_unmount,backup-volfile-servers=10.0.0.1:10.0.0.2:10.0.0.3,log-file=/var/lib/kubelet/plugins/kubernetes.io/
glusterfs/heketi-storage/heketi-storage-copy-job-ftqps-glusterfs.log,log-level=ERROR 10.0.0.2:heketidbstorage /var/lib/kubelet/pods/2aca16e0-9080-440e-8340-0ab1a
d8ffbff/volumes/kubernetes.io~glusterfs/heketi-storage
Output: Running scope as unit: run-rd71a89eb73ad41e4a9c2502f46a6121c.scope
Mount failed. Please check the log file for more details.
, the following error information was pulled from the glusterfs log to help diagnose this issue:
[2020-05-17 16:40:22.933491] E [MSGID: 100026] [glusterfsd.c:2403:glusterfs_process_volfp] 0-: failed to construct the graph
[2020-05-17 16:40:22.933844] E [graph.c:1102:glusterfs_graph_destroy] (-->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x537) [0x56267e02ce77] -->/usr/sbin/glusterfs(glusterfs
_process_volfp+0x180) [0x56267e026ea0] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(glusterfs_graph_destroy+0x6b) [0x7ff942e3deab] ) 0-graph: invalid argument: graph
[Invalid argument]
  Warning  FailedMount  19m  kubelet, srvarvan-xxx  MountVolume.SetUp failed for volume "heketi-storage" : mount failed: mount failed: exit status 1
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/2aca16e0-9080-440e-8340-0ab1ad8ffbff/volumes/kubernetes.io~glusterfs/heketi-stor
age --scope -- mount -t glusterfs -o auto_unmount,backup-volfile-servers=10.0.0.1:10.0.0.2:10.0.0.3,log-file=/var/lib/kubelet/plugins/kubernetes.io/
glusterfs/heketi-storage/heketi-storage-copy-job-ftqps-glusterfs.log,log-level=ERROR 10.0.0.3:heketidbstorage /var/lib/kubelet/pods/2aca16e0-9080-440e-8340-0ab1a
d8ffbff/volumes/kubernetes.io~glusterfs/heketi-storage
Output: Running scope as unit: run-rd71a89eb73ad41e4a9c2502f46a6121c.scope
Mount failed. Please check the log file for more details.
, the following error information was pulled from the glusterfs log to help diagnose this issue:
[2020-05-17 16:40:22.933491] E [MSGID: 100026] [glusterfsd.c:2403:glusterfs_process_volfp] 0-: failed to construct the graph
[2020-05-17 16:40:22.933844] E [graph.c:1102:glusterfs_graph_destroy] (-->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x537) [0x56267e02ce77] -->/usr/sbin/glusterfs(glusterfs
_process_volfp+0x180) [0x56267e026ea0] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(glusterfs_graph_destroy+0x6b) [0x7ff942e3deab] ) 0-graph: invalid argument: graph
.......
 Warning  FailedMount  17m                 kubelet, srvarvan-xxx  Unable to attach or mount volumes: unmounted volumes=[heketi-storage], unattached volumes=[heke
ti-storage-secret default-token-55czn heketi-storage]: timed out waiting for the condition                                  
  Warning  FailedMount  68s (x2 over 12m)   kubelet, srvarvan-xxx  Unable to attach or mount volumes: unmounted volumes=[heketi-storage], unattached volumes=[heke
ti-storage heketi-storage-secret default-token-55czn]: timed out waiting for the condition                                                                             
  Warning  FailedMount  42s (x14 over 16m)  kubelet, srvarvan-xxx  (combined from similar events): MountVolume.SetUp failed for volume "heketi-storage" : mount fa
iled: mount failed: exit status 1                                                                                                                                      
Mounting command: systemd-run

what am I doing wrong ?