Akka Kubernetes API is not able to access Service account token - kubernetes

Question

We are working with Akka Kubernetes API to connect to other pods in the Kubernetes cluster(AWS EKS) to form a cluster. But we are getting the following error when the token is being read
[ERROR] 2022-12-01 12:05:53.635+0000 [cloud-poi-ha-akka.actor.default-dispatcher-11] a.d.k.KubernetesApiServiceDiscovery - Error reading api-token from /var/run/secrets/kubernetes.io/serviceaccount/token
java.nio.file.AccessDeniedException: /var/run/secrets/kubernetes.io/serviceaccount/token
When we checked the token that's gets mounted on the pod, we observed it's not the token from the service account we intend to use but the default token. This causes the discovery of other pods fail with error 403.
[INFO ] 2022-12-01 12:07:19.999+0000 [cloud-poi-ha-akka.actor.default-dispatcher-14] a.d.k.KubernetesApiServiceDiscovery - Querying for pods with label selector: [app=cloud-poi]. Namespace: [fdnbug-cloud-poi-ha-bugfix-def-85666]. Port: [None]
[WARN ] 2022-12-01 12:07:20.011+0000 [cloud-poi-ha-akka.actor.default-dispatcher-16] a.d.k.KubernetesApiServiceDiscovery - Forbidden to communicate with Kubernetes API server; check RBAC settings. Response: [{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods is forbidden: User \"system:anonymous\" cannot list resource \"pods\" in API group \"\" in the namespace \"fdnbug-cloud-poi-ha-bugfix-def-85666\"","reason":"Forbidden","details":{"kind":"pods"},"code":403}
]
[WARN ] 2022-12-01 12:07:20.013+0000 [cloud-poi-ha-akka.actor.default-dispatcher-12] a.m.c.b.i.BootstrapCoordinator - Resolve attempt failed! Cause: akka.discovery.kubernetes.KubernetesApiServiceDiscovery$KubernetesApiException: Forbidden when communicating with the Kubernetes API. Check RBAC settings.
Our Statefulset and ServiceAccount manifests looks like:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: sa-ne-ha
namespace: somenamespace
labels:
componentName: ne
app: some-app
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: ne-r-cluster-managers
namespace: somenamespace
labels:
componentName: ne
app: some-app
rules:
- apiGroups:
- ""
resources:
- pods
- pods/status
- pods/log
- statefulsets
verbs:
- get
- list
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: ne-rb-cluster-managers
namespace: somenamespace
labels:
componentName: ne
app: someapp
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: ne-r-cluster-managers
subjects:
- kind: ServiceAccount
name: sa-ne-ha
namespace: somenamespace
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: ne-ha
namespace: somenamespace
labels:
componentName: ne-ha
app: someapp
version: someversion
annotations:
prometheus.io/scrape: "true"
prometheus.io/path: "/metrics"
prometheus.io/port: "9252"
spec:
serviceName: ne-headless
replicas: 2
selector:
matchLabels:
componentName: ne-ha
app: someapp
updateStrategy:
type: RollingUpdate
template:
metadata:
labels:
componentName: ne-ha
app: someapp
spec:
serviceAccountName: sa-ne-ha
containers:
- name: ne-ha
image: ${DOCKER_IMAGE}
imagePullPolicy: Always
resources:
requests:
cpu: ${CONTAINERS_RESOURCES_CPU_REQUESTS}
memory: ${CONTAINERS_RESOURCES_MEMORY_REQUESTS}
limits:
cpu: ${CONTAINERS_RESOURCES_CPU_LIMITS}
memory: ${CONTAINERS_RESOURCES_MEMORY_LIMITS}
command:
- "/microservice/bin"
env:
ports:
- name: http
containerPort: 8080
protocol: TCP
readinessProbe:
httpGet:
path: /ready
port: 8080
initialDelaySeconds: 15
periodSeconds: 30
failureThreshold: 3
timeoutSeconds: 30
livenessProbe:
httpGet:
path: /alive
port: 8080
initialDelaySeconds: 130
periodSeconds: 30
failureThreshold: 3
timeoutSeconds: 5
---
Have someone encountered a similar issue and was able to resolve it?