Ignite not discoverable in kubernetes cluster with TcpDiscoveryKubernetesIpFinder - kubernetes

Question

I am trying to make ignite deployed in k8s discoverable using TcpDiscoveryKubernetesIpFinder. I have also used all the deployment configurations as recommended in apache ignite documentation to make it discoverable. Ignite version is v2.6. When I try to access the ignite from another service inside the cluster(and namespace), it fails giving below error.
. . instance-14292nccv10-74997cfdff-kqdqh] Caused by:
java.io.IOException: Server returned HTTP response code: 403 for URL:
https://kubernetes.default.svc.cluster.local:443/api/v1/namespaces/my-namespace/endpoints/ignite-service
[instance-14292nccv10-74997cfdff-kqdqh] at
sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)
~[na:1.8.0_151] [instance-14292nccv10-74997cfdff-kqdqh] at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)
~[na:1.8.0_151] [instance-14292nccv10-74997cfdff-kqdqh] at
sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:263)
~[na:1.8.0_151] [instance-14292nccv10-74997cfdff-kqdqh] . .
My ignite configurations to make it discoverable are as follows,
apiVersion: v1
kind: ServiceAccount
metadata:
name: ignite-service
namespace: my-namespace
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: ignite-service
namespace: my-namespace
rules:
- apiGroups:
- ""
resources:
- pods
- endpoints
verbs:
- get
- list
- watch
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: ignite-service
roleRef:
kind: ClusterRole
name: ignite-service
apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
name: ignite-service
namespace: my-namespace
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: ignite-service-volume-claim-blr3
namespace: my-namespace
spec:
storageClassName: ssd
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
---
apiVersion: v1
kind: Secret
metadata:
name: ignite-files
namespace: my-namespace
data:
ignite-config.xml: PGJlYW5zIHhtbG5zID0gImh0dHA6Ly93d3cuc3ByaW5nZnJhbWV3b3JrLm9yZy9zY2hlbWEvYmVhbnMiCiAgICAgICB4bWxuczp4c2kgPSAiaHR0cDovL3d3dy53My5vcmcvMjAwMS9YTUxTY2hlbWEtaW5zdGFuY2UiCiAgICAgICB4bWxuczp1dGlsID0gImh0dHA6Ly93d3cuc3ByaW5nZnJhbWV3b3JrLm9yZy9zY2hlbWEvdXRpbCIKICAgICAgIHhzaTpzY2hlbWFMb2NhdGlvbiA9ICIKICAgICAgIGh0dHA6Ly93d3cuc3ByaW5nZnJhbWV3b3JrLm9yZy9zY2hlbWEvYmVhbnMKICAgICAgIGh0dHA6Ly93d3cuc3ByaW5nZnJhbWV3b3JrLm9yZy9zY2hlbWEvYmVhbnMvc3ByaW5nLWJlYW5zLnhzZAogICAgICAgaHR0cDovL3d3dy5zcHJpbmdmcmFtZXdvcmsub3JnL3NjaGVtYS91dGlsCiAgICAgICBodHRwOi8vd3d3LnNwcmluZ2ZyYW1ld29yay5vcmcvc2NoZW1hL3V0aWwvc3ByaW5nLXV0aWwueHNkIj4KCiAgICA8YmVhbiBjbGFzcyA9ICJvcmcuYXBhY2hlLmlnbml0ZS5jb25maWd1cmF0aW9uLklnbml0ZUNvbmZpZ3VyYXRpb24iPgogICAgICAgIDxwcm9wZXJ0eSBuYW1lID0gImRpc2NvdmVyeVNwaSI+CiAgICAgICAgICAgIDxiZWFuIGNsYXNzID0gIm9yZy5hcGFjaGUuaWduaXRlLnNwaS5kaXNjb3ZlcnkudGNwLlRjcERpc2NvdmVyeVNwaSI+CiAgICAgICAgICAgICAgICA8cHJvcGVydHkgbmFtZSA9ICJpcEZpbmRlciI+CiAgICAgICAgICAgICAgICAgICAgPGJlYW4gY2xhc3MgPSAib3JnLmFwYWNoZS5pZ25pdGUuc3BpLmRpc2NvdmVyeS50Y3AuaXBmaW5kZXIua3ViZXJuZXRlcy5UY3BEaXNjb3ZlcnlLdWJlcm5ldGVzSXBGaW5kZXIiPgogICAgICAgICAgICAgICAgICAgICAgICA8cHJvcGVydHkgbmFtZT0ibmFtZXNwYWNlIiB2YWx1ZT0ibXktbmFtZXNwYWNlIi8+CiAgICAgICAgICAgICAgICAgICAgICAgIDxwcm9wZXJ0eSBuYW1lPSJzZXJ2aWNlTmFtZSIgdmFsdWU9Imlnbml0ZS1zZXJ2aWNlIi8+CiAgICAgICAgICAgICAgICAgICAgPC9iZWFuPgogICAgICAgICAgICAgICAgPC9wcm9wZXJ0eT4KICAgICAgICAgICAgPC9iZWFuPgogICAgICAgIDwvcHJvcGVydHk+CiAgICAgICAgPCEtLSBFbmFibGluZyBBcGFjaGUgSWduaXRlIG5hdGl2ZSBwZXJzaXN0ZW5jZS4gLS0+CiAgICAgICAgPHByb3BlcnR5IG5hbWUgPSAiZGF0YVN0b3JhZ2VDb25maWd1cmF0aW9uIj4KICAgICAgICAgICAgPGJlYW4gY2xhc3MgPSAib3JnLmFwYWNoZS5pZ25pdGUuY29uZmlndXJhdGlvbi5EYXRhU3RvcmFnZUNvbmZpZ3VyYXRpb24iPgogICAgICAgICAgICAgICAgPHByb3BlcnR5IG5hbWUgPSAiZGVmYXVsdERhdGFSZWdpb25Db25maWd1cmF0aW9uIj4KICAgICAgICAgICAgICAgICAgICA8YmVhbiBjbGFzcyA9ICJvcmcuYXBhY2hlLmlnbml0ZS5jb25maWd1cmF0aW9uLkRhdGFSZWdpb25Db25maWd1cmF0aW9uIj4KICAgICAgICAgICAgICAgICAgICAgICAgPHByb3BlcnR5IG5hbWUgPSAicGVyc2lzdGVuY2VFbmFibGVkIiB2YWx1ZSA9ICJ0cnVlIi8+CiAgICAgICAgICAgICAgICAgICAgPC9iZWFuPgogICAgICAgICAgICAgICAgPC9wcm9wZXJ0eT4KICAgICAgICAgICAgICAgIDxwcm9wZXJ0eSBuYW1lID0gInN0b3JhZ2VQYXRoIiB2YWx1ZSA9ICIvZGF0YS9pZ25pdGUvc3RvcmFnZSIvPgogICAgICAgICAgICAgICAgPHByb3BlcnR5IG5hbWUgPSAid2FsUGF0aCIgdmFsdWUgPSAiL2RhdGEvaWduaXRlL2RiL3dhbCIvPgogICAgICAgICAgICAgICAgPHByb3BlcnR5IG5hbWUgPSAid2FsQXJjaGl2ZVBhdGgiIHZhbHVlID0gIi9kYXRhL2lnbml0ZS9kYi93YWwvYXJjaGl2ZSIvPgogICAgICAgICAgICA8L2JlYW4+CiAgICAgICAgPC9wcm9wZXJ0eT4KICAgIDwvYmVhbj4KPC9iZWFucz4=
type: Opaque
---
apiVersion: v1
kind: Service
metadata:
# Name of Ignite Service used by Kubernetes IP finder.
# The name must be equal to TcpDiscoveryKubernetesIpFinder.serviceName.
name: ignite-service
namespace: my-namespace
spec:
clusterIP: None # custom value.
ports:
- port: 9042 # custom value.
selector:
# Must be equal to one of the labels set in Ignite pods'
# deployement configuration.
app: ignite-service
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
# Custom Ignite cluster's name.
name: ignite-service
namespace: my-namespace
spec:
# A number of Ignite pods to be started by Kubernetes initially.
replicas: 1
template:
metadata:
labels:
# This label has to be added to the selector's section of
# ignite-service.yaml so that the Kubernetes Ignite lookup service
# can easily track all Ignite pods available deployed so far.
app: ignite-service
spec:
serviceAccountName: ignite-service
volumes:
# Custom name for the storage that holds Ignite's configuration
# which is example-kube.xml.
- name: ignite-storage
persistentVolumeClaim:
# Must be equal to the PersistentVolumeClaim created before.
claimName: ignite-service-volume-claim-blr3
- name: ignite-files
secret:
secretName: ignite-files
containers:
# Custom Ignite pod name.
- name: ignite-node
# Ignite Docker image. Kubernetes IP finder is supported starting from
# Apache Ignite 2.6.0
image: apacheignite/ignite:2.6.0
lifecycle:
postStart:
exec:
command: ['/bin/sh', '/opt/ignite/apache-ignite-fabric/bin/control.sh', '--activate']
env:
# Ignite's Docker image parameter. Adding the jar file that
# contain TcpDiscoveryKubernetesIpFinder implementation.
- name: OPTION_LIBS
value: ignite-kubernetes
# Ignite's Docker image parameter. Passing the Ignite configuration
# to use for an Ignite pod.
- name: CONFIG_URI
value: file:///etc/ignite-files/ignite-config.xml
- name: ENV
value: my-namespace
ports:
# Ports to open.
# Might be optional depending on your Kubernetes environment.
- containerPort: 11211 # REST port number.
- containerPort: 47100 # communication SPI port number.
- containerPort: 47500 # discovery SPI port number.
- containerPort: 49112 # JMX port number.
- containerPort: 10800 # SQL port number.
volumeMounts:
# Mounting the storage with the Ignite configuration.
- mountPath: "/data/ignite"
name: ignite-storage
- name: ignite-files
mountPath: "/etc/ignite-files"
I saw some links in stackoverflow with similar issue, followed the proposed solution but that doesn't work either. Any pointers on this will be of great help!

Answer 1

According to the URL, the IP finder tries to use a service named ignite, while you create it by name ignite-service.
You should provide both namespace and service name in the IP finder configuration:
<bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder">
<property name="namespace" value="my-namespace"/>
<property name="serviceName" value="ignite-service"/>
</bean>

Answer 2

You need to make sure you have the following locked down and handled.
Creation of your namespace in kubernetes
Creation of your service account in kubernetes
Permissions set for your service account in your namespace in your cluster.
service account permissions
https://kubernetes.io/docs/reference/access-authn-authz/rbac/#service-account-permissions