No available Hazelcast instance"HazelcastCachingProvider.HAZELCAST_CONFIG_LOCATION" - kubernetes

Question

any idea about this error:
No available Hazelcast instance. Please specify your Hazelcast configuration file path via "HazelcastCachingProvider.HAZELCAST_CONFIG_LOCATION"
Working fine with this cfg in a local kubernetes, but always getting this error when i set Kubernetes = true and multicast to false. I'm trying to use it for Liberty in IBMCloud Kubernetes.
<hazelcast xmlns="http://www.hazelcast.com/schema/config"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.hazelcast.com/schema/config
https://hazelcast.com/schema/config/hazelcast-config-3.12.xsd">
<group>
<name>cluster</name>
</group>
<network>
<join>
<multicast enabled="true"/>
<kubernetes enabled="false"/>
</join>
</network>
</hazelcast>
server.xml
<httpSessionCache libraryRef="jCacheVendorLib"
uri="file:${server.config.dir}hazelcast-config.xml" />
<library id="jCacheVendorLib">
<file name="${shared.config.dir}/lib/global/hazelcast-3.12.6.jar" />
</library>
This is what I done:
I have a docker image using liberty, in the liberty configuration I set the following configuration to use hazelcast:
<server>
<featureManager>
...
<feature>sessionCache-1.0</feature>
...
</featureManager>
...
<httpSessionCache libraryRef="jCacheVendorLib"
uri="file:${server.config.dir}hazelcast-config.xml" />
<library id="jCacheVendorLib">
<file name="${shared.config.dir}/lib/global/hazelcast-3.12.6.jar" />
</library>
...
</server>
Then I set the configuration in hazelcast-config.xml. I only get the error when I set kubernetes=true and multicast=false. If I left kubernetes = false and multicast = true works fine on my local kubernetes, but hazelcast can't find other pods when I deployed it on cloud (looks like ips are on a different network)
<hazelcast xmlns="http://www.hazelcast.com/schema/config"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.hazelcast.com/schema/config
https://hazelcast.com/schema/config/hazelcast-config-3.12.xsd">
<group>
<name>cluster</name>
</group>
<network>
<join>
<multicast enabled="true"/>
<kubernetes enabled="false"/>
</join>
</network>
</hazelcast>
Also I ran the RBAC yaml.
And run the following yaml to deploy it:
apiVersion: apps/v1
kind: Deployment
metadata:
name: employee-service
labels:
app: employee-service
spec:
replicas: 3
selector:
matchLabels:
app: employee-service
template:
metadata:
labels:
app: employee-service
spec:
containers:
- name: myapp
image: myapp
ports:
- name: http
containerPort: 8080
- name: multicast
containerPort: 5701
------------------------------
apiVersion: v1
kind: Service
metadata:
name: service
spec:
type: NodePort
selector:
app: employee-service
ports:
- protocol: TCP
port: 9080
targetPort: 9080
nodePort: 31234

Answer 1

If you use the Hazelcast Kubernetes plugin for the discovery, please make sure that you configured RBAC, for example with the following command.
kubectl apply -f https://raw.githubusercontent.com/hazelcast/hazelcast-kubernetes/master/rbac.yaml
Please also make sure that the default parameters work for you (you run your Hazelcast in the same namespace, etc.).
If that does not help, please share the full StackTrace logs.