Kubernetes cannot mount Glusterfs volumes - kubernetes

Question

I have a GlusterFS cluster and a k8s cluster. On any of the nodes in the k8s cluster, I can execute the following command:
mount -t glusterfs data1:/sbnstore /mnt/data/ -o xlator-option="transport.address-family=inet6"
and this works perfectly. I can create and access files under /mnt/data and they appear in the GlusterFS cluster and replicate among the peers. Note that the GlusterFS peers do not have IPv4 addresses. Only IPv6.
However creating a Kubernetes Pod hangs in the container creation stage. The following is the manifest:
apiVersion: v1
kind: Pod
metadata:
name: shell-testg
labels:
alan: testg
spec:
containers:
- name: ubuntu
image: ubuntu:latest
command: ["/bin/sleep", "3650d"]
volumeMounts:
- name: glusterfs-volume
mountPath: /data
volumes:
- name: glusterfs-volume
glusterfs:
endpoints: glusterfs-cluster
path: sbnstore
readOnly: no
When this manifest is applied, the pod that is created is permanently stuck in ContainerCreating status.
$ kc get pod
NAME READY STATUS RESTARTS AGE
shell-testg 0/1 ContainerCreating 0 12m
Digging for more info (extraneous lines omitted):
$ kc describe pod shell-testg
Name: shell-testg
Namespace: default
...
Mounts:
/data from glusterfs-volume (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-kxktp (ro)
...
Volumes:
glusterfs-volume:
Type: Glusterfs (a Glusterfs mount on the host that shares a pod's lifetime)
EndpointsName: glusterfs-cluster
Path: sbnstore
ReadOnly: false
kube-api-access-kxktp:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedMount 109s (x6 over 17m) kubelet Unable to attach or mount volumes: unmounted volumes=[glusterfs-volume], unattached volumes=[glusterfs-volume kube-api-access-kxktp]: timed out waiting for the condition
Warning FailedMount 84s kubelet MountVolume.SetUp failed for volume "glusterfs-volume" : mount failed: mount failed: exit status 1
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/snap/microk8s/common/var/lib/kubelet/pods/2ef51502-7327-443c-abc1-2a0f1ee1b177/volumes/kubernetes.io~glusterfs/glusterfs-volume --scope -- mount -t glusterfs -o auto_unmount,backup-volfile-servers=2001:470:1:999:1:0:30:100:2001:470:1:999:1:0:30:200:2001:470:1:999:1:0:30:300,log-file=/var/snap/microk8s/common/var/lib/kubelet/plugins/kubernetes.io/glusterfs/glusterfs-volume/shell-testg-glusterfs.log,log-level=ERROR 2001:470:1:841:1:0:30:200:sbnstore /var/snap/microk8s/common/var/lib/kubelet/pods/2ef51502-7327-443c-abc1-2a0f1ee1b177/volumes/kubernetes.io~glusterfs/glusterfs-volume
Output: Running scope as unit: run-rd7adcd2329cb4ab2a694d02925df1a2f.scope
[2022-06-23 18:32:35.342172] E [glusterfsd.c:833:gf_remember_backup_volfile_server] 0-glusterfs: failed to set volfile server: File exists
...
22-06-23 18:32:35.342443] E [glusterfsd.c:833:gf_remember_backup_volfile_server] 0-glusterfs: failed to set volfile server: File exists
Mounting glusterfs on /var/snap/microk8s/common/var/lib/kubelet/pods/2ef51502-7327-443c-abc1-2a0f1ee1b177/volumes/kubernetes.io~glusterfs/glusterfs-volume failed.
So now I am kind of stuck. Poking through the github issues board this kind of error seems to appear when there is a version incompatibility. My k8s is 1.23 and gluster is 7.2.
Any suggestions of what to try next?
UPDATE
As it turns out, the issue is that the glusterfs module within Kubernetes is not able to use IPv6. So I have a workaround which is to convert all the Endpoints to IPv4 addresses.
Not working:
apiVersion: v1
kind: Endpoints
metadata:
name: glusterfs-cluster
subsets:
- addresses:
- ip: 2001:470:1:999:1:0:30:100
ports:
- port: 49152
- addresses:
- ip: 2001:470:1:999:1:0:30:200
ports:
- port: 49152
- addresses:
- ip: 2001:470:1:999:1:0:30:300
ports:
- port: 49152
Working:
apiVersion: v1
kind: Endpoints
metadata:
name: glusterfs-cluster
subsets:
- addresses:
- ip: 10.2.12.1
ports:
- port: 49152
- addresses:
- ip: 10.2.12.2
ports:
- port: 49152
- addresses:
- ip: 10.2.12.3
ports:
- port: 49152
It is interesting to note that automatically provisioned PV/PVCs using the Heketi package do work over IPv6.
I haven't been able to find any way to specify configuration parameters to the glusterfs module that might provide a real solution. So if anyone has any pointers to that please provide it in an answer to this posting.
Hopefully this update will help someone else avoid this learning curve.