Hazelcast doesn't connect to pods when using Headless Service - kubernetes

Question

The hazelcast members are able to communicate with each other in non-kube environments. However, the same is not happening in Kube environments. The pod is not able to resolve its own domain.
I am working with Spring Boot 2.1.8 and Hazelcast 3.11.4 (TcpIp config)
Hazelcast configuration:
Config config = new Config();
config.setInstanceName("hazelcast-instance")
.setGroupConfig(new GroupConfig(hazelcastGroupName, hazelcastGroupPassword))
.setProperties(hzProps)
.setNetworkConfig(
new NetworkConfig()
.setPort(hazelcastNetworkPort)
.setPortAutoIncrement(hazelcastNetworkPortAutoIncrement)
.setJoin(
new JoinConfig()
.setMulticastConfig(new MulticastConfig().setEnabled(false))
.setAwsConfig(new AwsConfig().setEnabled(false))
.setTcpIpConfig(new TcpIpConfig().setEnabled(true).setMembers(memberList))))
.addMapConfig(initReferenceDataMapConfig());
return config;
Members definition in Statefulset config file:
- name: HC_NETWORK_MEMBERS
value: project-0.project-ha, project-1.project-ha
Headless service config:
---
apiVersion: v1
kind: Service
metadata:
namespace: {{ kuber_namespace }}
labels:
app: project
name: project-ha
spec:
clusterIP: None
selector:
app: project
Errors:
Resolving domain name 'project-0.project-ha' to address(es): [10.42.30.215]
Cannot resolve hostname: 'project-1.project-ha'
Members {size:1, ver:1} [
Member [10.42.11.173]:5001 - d928de2c-b4ff-4f6d-a324-5487e33ca037 this
]
The other pod has a similar error.
Fun facts:
It works fine when I set the full hostname in the HC_NETWORK_MEMBERS var. For example: project-0.project-ha.namespace.svc.cluster.local, project-1.project-ha.namespace.svc.cluster.local

Answer 1

There is a discovery plugin for Kubernetes, see here, and varies "how to" guides for Kubernetes here
If you use that, most of your problems should go away. The plugin looks up the member list from Kubernete's DNS.
If you can, it's worth upgrading to Hazelcast 4.2, as 3.11 is not the latest. Make sure to get a matching version of the plugin.
On 4.2, discovery has auto-detection, so will do it's best to help.

Answer 2

Please check the related guides:
Hazelcast Guide: Hazelcast for Kubernetes
Hazelcast Guide: Embedded Hazelcast on Kubernetes
For the embedded Hazelcast, you need to use the Hazelcast Kubernetes plugin. For the client-server, you can use TCP-IP configuration on the member side and use Hazelcast service name as the static DNS. It will be resolved automatically.