AKKA `Http().singleRequest` may stuck on one node - scala

Question

Short story:
In a 2-node Kubernetes AKKA clustering setup, each node runs a piece of code that calls a Kubeless function as follows:
log.trace("Calling URL [{}] with bytes response.", address)
Http().singleRequest(
request = HttpRequest(
method = HttpMethods.POST,
uri = address,
entity = HttpEntity(contentType, data)
),
connectionContext = noCertificateCheckContext
)
The above code generates the following log on the good node:
Calling URL [http://10.233.54.59:8080] with bytes response.
[t-dispatcher-22] [36mr$$anonfun$receive$1[m: Creating pool.
[t-dispatcher-22] [36mr$$anonfun$receive$1[m: Dispatching request [POST Strict(14160 bytes)] to pool
[t-dispatcher-21] [36mr$$anonfun$receive$1[m: [0 (Unconnected)]Dispatching request [POST / Strict(14160 bytes)]
[t-dispatcher-21] [36mr$$anonfun$receive$1[m: [0 (Unconnected)]Before event [onNewRequest] In state [Unconnected] for [0 ms]
[t-dispatcher-21] [36mr$$anonfun$receive$1[m: [0 (Unconnected)]Establishing connection
[t-dispatcher-21] [36mr$$anonfun$receive$1[m: [0 (Connecting)]After event [onNewRequest] State change [Unconnected] -> [Connecting]
[t-dispatcher-14] [36mr$$anonfun$receive$1[m: Resolving 10.233.54.59 before connecting
[...]
...and each of these calls succeeds.
The above code generates the following code on the bad node:
Calling URL [http://10.233.54.59:8080] with bytes response.
...and literally nothing after that. The bad node can accept incoming requests through its HTTP API written in AKKA and can do all things, but not Http().singleRequest, which is used to call Kubeless functions.
The only difference between the good and the bad node is that the bad node first starts an AKKA Cluster object for a short amount of time, does some stuff with it, then tears it down.
system.terminate()
Await.result(system.whenTerminated, timeout)
...and then it starts it again. The cluster can then be formed between the two nodes. Both nodes report that they are in a cluster and can gossip properly.
The bad node can not perform HTTP requests even from the first one. HTTP entities are properly subscribed using response.entity.toStrict(20 seconds) or response.discardEntityBytes().
Hints or ideas on how to fix/debug the problem further are much appreciated!
More details:
I start one AKKA cluster on Kubernetes using the following dependencies:
"com.typesafe.akka" %% "akka-actor" % "2.6.16",
"com.typesafe.akka" %% "akka-slf4j" % "2.6.16",
"com.typesafe.akka" %% "akka-stream" % "2.6.16",
"com.typesafe.akka" %% "akka-cluster" % "2.6.16",
"com.typesafe.akka" %% "akka-cluster-tools" % "2.6.16",
"com.typesafe.akka" %% "akka-cluster-metrics" % "2.6.16",
"com.typesafe.akka" %% "akka-stream-typed" % "2.6.16",
"com.typesafe.akka" %% "akka-actor-typed" % "2.6.16",
"com.typesafe.akka" %% "akka-cluster-typed" % "2.6.16",
"com.typesafe.akka" %% "akka-cluster-sharding-typed" % "2.6.16",
"com.typesafe.akka" %% "akka-stream-typed" % "2.6.16",
"com.typesafe.akka" %% "akka-discovery" % "2.6.16",
"com.lightbend.akka.management" %% "akka-management-cluster-bootstrap" % "1.1.1",
"com.lightbend.akka.discovery" %% "akka-discovery-kubernetes-api" % "1.1.1",
"com.typesafe.akka" %% "akka-stream-testkit" % "2.6.16",
"com.typesafe.akka" %% "akka-stream-testkit" % "2.6.16" % "test",
"com.typesafe.akka" %% "akka-http" % "10.2.6",
"com.typesafe.akka" %% "akka-http-core" % "10.2.6",
"com.typesafe.akka" %% "akka-http-testkit" % "10.2.6",
"com.typesafe.akka" %% "akka-http-spray-json" % "10.2.6",
"com.typesafe.akka" %% "akka-http-testkit" % "10.2.6" % "test",
Below is the default configuration.
akka {
test {
timefactor = 3.0
}
loglevel = "DEBUG"
stdout-loglevel = "DEBUG"
loggers = ["akka.event.slf4j.Slf4jLogger"]
logging-filter = "akka.event.slf4j.Slf4jLoggingFilter"
log-config-on-start = on
http {
parsing {
uri-parsing-mode = strict
error-logging-verbosity = full
}
server {
server-header = ""
verbose-error-messages = on
idle-timeout = 60s
max-connections = 2048
request-timeout = 30s
websocket.periodic-keep-alive-mode = pong
}
host-connection-pool {
max-connections = 32
max-open-requests = 64
max-connection-lifetime = 10 min
keep-alive-timeout = 10 min
response-entity-subscription-timeout = 2.second
}
routing {
verbose-error-messages = on
}
}
discovery {
method = config
}
management {
cluster.bootstrap {
contact-point-discovery {
service-name = "local-cluster"
discovery-method = config
required-contact-point-nr = 1
}
}
}
actor {
allow-java-serialization = on
warn-about-java-serializer-usage = off
provider = "cluster"
blocking-dispatcher {
type = "Dispatcher"
executor = "fork-join-executor"
fork-join-executor {
parallelism-min = 32
parallelism-max = 128
}
throughput = 1
}
}
cluster {
min-nr-of-members = 1
log-info-verbose = off
sharding {
passivate-idle-entity-after = off
}
}
remote {
log-remote-lifecycle-events = off
artery {
transport = tcp
canonical.port = 0
bind.port = 0
bind.bind-timeout = 10s
}
}
extensions = [
"akka.cluster.metrics.ClusterMetricsExtension",
"akka.cluster.pubsub.DistributedPubSub"
]
}
On Kubernetes, the configuration is overwritten with the following:
akka {
discovery {
kubernetes-api {
pod-label-selector = "app=my-app"
pod-domain = "cluster.local"
}
}
management {
cluster.bootstrap {
contact-point-discovery {
discovery-method = kubernetes-api
service-name = ${?AKKA_CLUSTER_BOOTSTRAP_SERVICE_NAME}
service-namespace = "staging.svc.cluster.local"
interval = 10 seconds
required-contact-point-nr = 0
}
}
}
cluster {
configuration-compatibility-check.enforce-on-join = off
shutdown-after-unsuccessful-join-seed-nodes = 120s
downing-provider-class = "akka.cluster.sbr.SplitBrainResolverProvider"
sharding {
use-lease = "akka.coordination.lease.kubernetes"
}
split-brain-resolver {
active-strategy = "lease-majority"
lease-majority {
lease-implementation = "akka.coordination.lease.kubernetes"
}
}
}
}