Service discovery with Scala akka inside kubernetes - scala

Question

I am facing issues with working with service discovery using akka. I recently migrated my application from Kubernetes version 1.18 to 1.21 . Service discovery is working fine on the older version but it doesn't seem to work on version 1.21. Below are the error log:-
play.api.UnexpectedException: Unexpected exception[IllegalStateException: Service ${serviceName} was not found by service locator]
at play.api.http.HttpErrorHandlerExceptions$.throwableToUsefulException(HttpErrorHandler.scala:358)
at play.api.http.DefaultHttpErrorHandler.onServerError(HttpErrorHandler.scala:264)
at play.core.server.AkkaHttpServer$$anonfun$2.applyOrElse(AkkaHttpServer.scala:430)
at play.core.server.AkkaHttpServer$$anonfun$2.applyOrElse(AkkaHttpServer.scala:422)
at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:454)
at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:63)
at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:100)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94)
at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:100)
Config file looks like:-
akka.management {
cluster.bootstrap {
contact-point-discovery {
discovery-method = kubernetes-api
service-name = ${serviceName}
required-contact-point-nr = ${REQUIRED_CONTACT_POINT_NR}
}
}

Answer 1

First, make sure that your Role and Rolebinding are correct, as documented. This is, by far, the most common mistake: not giving the discovery mechanism the proper permissions.
Secondly, try manually specifying the pod-label-selector rather than specifying a service-name. This is what I've done recently and what the defaults are in the docs. So maybe you are in a situation where you don't have permission to query the headless service, but you can query the pods directly via the pod-label-selector.
Fundamentally what it's doing is trivial. It's running an extremely simple query (and that you can specify manually) against the K8S API and getting a list of Pod IPs back. (Which it then passes into the Akka Cluster mechanism where it does much more complex things to establish and modify the cluster, but this query is super simple.) So fundamentally, the only things that can be wrong are:
The query is wrong. (Either the thing your querying doesn't exist or there is a typo in your query or implied query.)
The permissions on your query are wrong such that your serviceaccount can execute the query.