Hazelcast map on kubernetes not in sync - kubernetes

Question

Writing a spring based application written in kotlin with hazelcast I encounter issue when deploying to kubernetes.
For hazelcast on kubernetes I use DNS lookup mode for discovery.
I have the following hazelcast configuration:
hazelcast:
network:
join:
multicast:
enabled: false
kubernetes:
enabled: true
service-dns: my-application-hs
And the following service.yaml for the kubernetes deployment:
apiVersion: v1
kind: Service
metadata:
name: my-application
spec:
type: ClusterIP
selector:
component: my-application
ports:
- name: http
port: 80
protocol: TCP
targetPort: http
---
apiVersion: v1
kind: Service
metadata:
name: my-application-hs
spec:
type: ClusterIP
clusterIP: None
selector:
component: my-application
ports:
- name: hazelcast
port: 5701
A hazelcast map is used like this:
#Component
class CacheClientImplHazelcast(){
private val hz: HazelcastInstance
init{
val serializer = SerializerConfig()
.setTypeClass(MyDto::class.java)
.setImplementation(MyDtoSerializer())
val config = Config()
config.serializationConfig.addSerializerConfig(serializer)
hz = Hazelcast.newHazelcastInstance(config)
}
fun getAllData(): List<MyDto> {
val map: IMap<String, MyDto> = hz.getMap("my-map")
return map.values.toList()
}
fun putData(key:String, myDto: MyDto) {
val map: IMap<String, MyDto> = hz.getMap("my-map")
map.put(key, myDto)
}
override fun clear() {
val map: IMap<String, MyDto> = hz.getMap("my-map")
map.clear()
}
}
When running 3 instances on kubernetes the logs from hazelcast always show me 4 entries, something like this:
Members {size:4, ver:53} [
Member [10.4.2.32]:5701 - c1e70d6f-a62d-4924-9815-36bb1f98f141
Member [10.4.3.25]:5702 - be96c292-8847-4f56-ae32-f27f380d7c5b
Member [10.4.2.32]:5702 - 6ca96bfd-eb74-4149-8630-a2488e76d97d
Member [10.4.11.41]:5702 - 7e8b0bc9-ad2b-41eb-afbf-b7af9ed497bd this
]
(Side question 1: why do I see 4 here instead of 3?)
Now even though it seems the member are connected (at least the logs of the nodes show all the same member uuids) when I write data on one node it is not available on the other nodes. Calls to getAllData only show data that has been put into the hazelcast map on this node. When I send requests to the individual nodes (curl on shell) I only see a fraction of the data. When I send a request on the normal url of the pod then with round-robin I get data of the different nodes, which is not synchronized.
If I run the very same application locally with the following hazelcast.yaml:
hazelcast:
network:
join:
multicast:
enabled: false
Then it all works correctly and the cache really seems to be "synchronized" across different server instances running locally. For this test I start the application on different ports.
However what is also strange, even if I run 2 instances locally, the logs I see from hazelcast indicate there are 4 members:
Members {size:4, ver:4} [
Member [192.168.68.106]:5701 - 01713c9f-7718-4ed4-b532-aaf62443c425
Member [192.168.68.106]:5702 - 09921004-88ef-4fe5-9376-b88869fde2bc
Member [192.168.68.106]:5703 - cea4b13f-d538-48f1-b0f2-6c53678c5823 this
Member [192.168.68.106]:5704 - 44d84e70-5b68-4c69-a45b-fee39bd75554
]
(Side questions 2: Why do I see 4 members even I only started 2 locally?)
The main question now is: Why does this setup not work on kubernetes? Why does each node have a separated map that is not in sync with the other nodes?
Here some log message that could be relevant but I was not able to identify an issue with:
2022-04-19T12:29:31.414588357Z2022-04-19 12:29:31.414 INFO 1 --- [cached.thread-7] c.h.i.server.tcp.TcpServerConnector : [10.4.11.41]:5702 [dev] [5.1.1] Connecting to /10.4.3.25:5702, timeout: 10000, bind-any: true
2022-04-19T12:29:31.414806473Z2022-04-19 12:29:31.414 INFO 1 --- [.IO.thread-in-2] c.h.i.server.tcp.TcpServerConnection : [10.4.11.41]:5702 [dev] [5.1.1] Initialized new cluster connection between /10.4.11.41:5702 and /10.4.3.25:46475
2022-04-19T12:29:31.414905573Z2022-04-19 12:29:31.414 INFO 1 --- [cached.thread-4] c.h.i.server.tcp.TcpServerConnector : [10.4.11.41]:5702 [dev] [5.1.1] Connecting to /10.4.2.32:5702, timeout: 10000, bind-any: true
2022-04-19T12:29:31.416520854Z2022-04-19 12:29:31.416 INFO 1 --- [.IO.thread-in-0] c.h.i.server.tcp.TcpServerConnection : [10.4.3.25]:5702 [dev] [5.1.1] Initialized new cluster connection between /10.4.3.25:5702 and /10.4.11.41:40455
2022-04-19T12:29:31.416833551Z2022-04-19 12:29:31.416 INFO 1 --- [.IO.thread-in-1] c.h.i.server.tcp.TcpServerConnection : [10.4.2.32]:5702 [dev] [5.1.1] Initialized new cluster connection between /10.4.2.32:54433 and /10.4.11.41:5702
2022-04-19T12:29:31.417377114Z2022-04-19 12:29:31.417 INFO 1 --- [.IO.thread-in-0] c.h.i.server.tcp.TcpServerConnection : [10.4.11.41]:5702 [dev] [5.1.1] Initialized new cluster connection between /10.4.11.41:40455 and /10.4.3.25:5702
2022-04-19T12:29:31.417545174Z2022-04-19 12:29:31.417 INFO 1 --- [.IO.thread-in-2] c.h.i.server.tcp.TcpServerConnection : [10.4.2.32]:5702 [dev] [5.1.1] Initialized new cluster connection between /10.4.2.32:5702 and /10.4.11.41:53547
2022-04-19T12:29:31.418541840Z2022-04-19 12:29:31.418 INFO 1 --- [.IO.thread-in-1] c.h.i.server.tcp.TcpServerConnection : [10.4.11.41]:5702 [dev] [5.1.1] Initialized new cluster connection between /10.4.11.41:53547 and /10.4.2.32:5702
2022-04-19T12:29:31.419763311Z2022-04-19 12:29:31.419 INFO 1 --- [.IO.thread-in-2] c.h.i.server.tcp.TcpServerConnection : [10.4.3.25]:5702 [dev] [5.1.1] Initialized new cluster connection between /10.4.3.25:46475 and /10.4.11.41:5702
2022-04-19T12:29:31.676218042Z2022-04-19 12:29:31.675 INFO 1 --- [gulis.migration] c.h.i.partition.impl.MigrationManager : [10.4.2.32]:5701 [dev] [5.1.1] Repartitioning cluster data. Migration tasks count: 271

Answer 1

There are two issues here but only one problem.
why the extra instances
Spring (Boot) will create a Hazelcast instance for you if it finds a Hazelcast config file and no HazelcastInstance #Bean. You can fix by excluding the HazelcastAutoConfiguration.class or by returning the instance you create in your component class as a #Bean.
why the data sync issue on kubernetes
Each pod accidentally has 2 Hazelcast nodes, one on 5701 and one on 5702. But your ClusterIP only lists 5701. Some instances in the pod can't be reached from outside. This will go away when you fix the first issue.