Hazelcast failed to start in Kubernetes - kubernetes

Question

I have a HA Kubernetes cluster that initialized with custom certificates. I want to run Hazelcast on it, but there is an error in discovering Hazelcast members using Kubernetes API.
This is my deploy file:
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: hazelcast
labels:
app: hazelcast
spec:
replicas: 3
serviceName: hazelcast-service
selector:
matchLabels:
app: hazelcast
template:
metadata:
labels:
app: hazelcast
spec:
imagePullSecrets:
- name: nexuspullsecret
containers:
- name: hazelcast
image: 192.168.161.187:9050/hazelcast-custom:4.0.2
imagePullPolicy: "Always"
ports:
- name: hazelcast
containerPort: 5701
livenessProbe:
httpGet:
path: /hazelcast/health/node-state
port: 5701
initialDelaySeconds: 30
periodSeconds: 10
timeoutSeconds: 5
successThreshold: 1
failureThreshold: 3
readinessProbe:
httpGet:
path: /hazelcast/health/node-state
port: 5701
initialDelaySeconds: 30
periodSeconds: 10
timeoutSeconds: 1
successThreshold: 1
failureThreshold: 1
resources:
requests:
memory: "0"
cpu: "0"
limits:
memory: "2048Mi"
cpu: "500m"
volumeMounts:
- name: hazelcast-storage
mountPath: /data/hazelcast
env:
- name: JAVA_OPTS
value: "-Dhazelcast.rest.enabled=true -Dhazelcast.config=/data/hazelcast/hazelcast.xml"
volumes:
- name: hazelcast-storage
configMap:
name: hazelcast-configuration
---
apiVersion: v1
kind: Service
metadata:
name: hazelcast-service
spec:
type: ClusterIP
selector:
app: hazelcast
ports:
- protocol: TCP
port: 5701
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: hazelcast-cluster-role
rules:
- apiGroups: [""]
resources: ["endpoints", "pods", "nodes"]
verbs: ["get", "watch", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: hazelcast-cluster-role-binding
subjects:
- kind: ServiceAccount
name: default
namespace: default
roleRef:
kind: ClusterRole
name: hazelcast-cluster-role
apiGroup: rbac.authorization.k8s.io
---
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
name: hazelcast
namespace: default
spec:
maxUnavailable: 0
selector:
matchLabels:
app: hazelcast
---
apiVersion: v1
kind: ConfigMap
metadata:
name: hazelcast-configuration
data:
hazelcast.xml: |-
<?xml version="1.0" encoding="UTF-8"?>
<hazelcast xmlns="http://www.hazelcast.com/schema/config"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.hazelcast.com/schema/config
http://www.hazelcast.com/schema/config/hazelcast-config-4.0.xsd">
<network>
<rest-api enabled="true"></rest-api>
<join>
<!-- deactivate normal discovery -->
<multicast enabled="false"/>
<tcp-ip enabled="false" />
<!-- activate the Kubernetes plugin -->
<kubernetes enabled="true">
<service-name>hazelcast-service</service-name>
<namespace>default</namespace>
<kubernetes-api-retries>20</kubernetes-api-retries>
</kubernetes>
</join>
</network>
<user-code-deployment enabled="true">
<class-cache-mode>ETERNAL</class-cache-mode>
<provider-mode>LOCAL_AND_CACHED_CLASSES</provider-mode>
</user-code-deployment>
<reliable-topic name="ConfirmationTimeout">
<read-batch-size>10</read-batch-size>
<topic-overload-policy>DISCARD_OLDEST</topic-overload-policy>
<statistics-enabled>true</statistics-enabled>
</reliable-topic>
<ringbuffer name="ConfirmationTimeout">
<capacity>10000</capacity>
<backup-count>1</backup-count>
<async-backup-count>0</async-backup-count>
<time-to-live-seconds>0</time-to-live-seconds>
<in-memory-format>BINARY</in-memory-format>
<merge-policy batch-size="100">com.hazelcast.spi.merge.PutIfAbsentMergePolicy</merge-policy>
</ringbuffer>
<scheduled-executor-service name="ConfirmationTimeout">
<capacity>100</capacity>
<capacity-policy>PER_NODE</capacity-policy>
<pool-size>32</pool-size>
<durability>3</durability>
<merge-policy batch-size="100">com.hazelcast.spi.merge.PutIfAbsentMergePolicy</merge-policy>
</scheduled-executor-service>
<cp-subsystem>
<cp-member-count>3</cp-member-count>
<group-size>3</group-size>
<session-time-to-live-seconds>300</session-time-to-live-seconds>
<session-heartbeat-interval-seconds>5</session-heartbeat-interval-seconds>
<missing-cp-member-auto-removal-seconds>14400</missing-cp-member-auto-removal-seconds>
<fail-on-indeterminate-operation-state>false</fail-on-indeterminate-operation-state>
<raft-algorithm>
<leader-election-timeout-in-millis>15000</leader-election-timeout-in-millis>
<leader-heartbeat-period-in-millis>5000</leader-heartbeat-period-in-millis>
<max-missed-leader-heartbeat-count>10</max-missed-leader-heartbeat-count>
<append-request-max-entry-count>100</append-request-max-entry-count>
<commit-index-advance-count-to-snapshot>10000</commit-index-advance-count-to-snapshot>
<uncommitted-entry-count-to-reject-new-appends>100</uncommitted-entry-count-to-reject-new-appends>
<append-request-backoff-timeout-in-millis>100</append-request-backoff-timeout-in-millis>
</raft-algorithm>
<locks>
<fenced-lock>
<name>TimeoutLock</name>
<lock-acquire-limit>1</lock-acquire-limit>
</fenced-lock>
</locks>
</cp-subsystem>
<metrics enabled="true">
<management-center>
<retention-seconds>30</retention-seconds>
</management-center>
<jmx enabled="false"/>
<collection-frequency-seconds>10</collection-frequency-seconds>
</metrics>
</hazelcast>
I have tested this deploy file on non-custom certificate ssl HA Kubernetes cluster and it works without any problems.
This is log files:
########################################
# JAVA_OPTS=-Djava.net.preferIPv4Stack=true -Djava.util.logging.config.file=/opt/hazelcast/logging.properties -XX:MaxRAMPercentage=80.0 -XX:+UseParallelGC --add-modules java.se --add-exports java.base/jdk.internal.ref=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.nio=ALL-UNNAMED --add-opens java.base/sun.nio.ch=ALL-UNNAMED --add-opens java.management/sun.management=ALL-UNNAMED --add-opens jdk.management/com.sun.management.internal=ALL-UNNAMED -Dhazelcast.rest.enabled=true -Dhazelcast.config=/data/hazelcast/hazelcast.xml
# CLASSPATH=/opt/hazelcast/*:/opt/hazelcast/lib/*:/opt/hazelcast/user-lib/*
# CLASSPATH_DEFAULT=/opt/hazelcast/*:/opt/hazelcast/lib/*:/opt/hazelcast/user-lib/*
# starting now....
########################################
+ exec java -server -Djava.net.preferIPv4Stack=true -Djava.util.logging.config.file=/opt/hazelcast/logging.properties -XX:MaxRAMPercentage=80.0 -XX:+UseParallelGC --add-modules java.se --add-exports java.base/jdk.internal.ref=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.nio=ALL-UNNAMED --add-opens java.base/sun.nio.ch=ALL-UNNAMED --add-opens java.management/sun.management=ALL-UNNAMED --add-opens jdk.management/com.sun.management.internal=ALL-UNNAMED -Dhazelcast.rest.enabled=true -Dhazelcast.config=/data/hazelcast/hazelcast.xml com.hazelcast.core.server.HazelcastMemberStarter
Sep 08, 2020 7:08:36 AM com.hazelcast.internal.config.AbstractConfigLocator
INFO: Loading configuration '/data/hazelcast/hazelcast.xml' from System property 'hazelcast.config'
Sep 08, 2020 7:08:37 AM com.hazelcast.internal.config.AbstractConfigLocator
INFO: Using configuration file at /data/hazelcast/hazelcast.xml
Sep 08, 2020 7:08:40 AM com.hazelcast.instance.AddressPicker
INFO: [LOCAL] [dev] [4.0.2] Prefer IPv4 stack is true, prefer IPv6 addresses is false
Sep 08, 2020 7:08:40 AM com.hazelcast.instance.AddressPicker
INFO: [LOCAL] [dev] [4.0.2] Picked [10.42.128.11]:5701, using socket ServerSocket[addr=/0.0.0.0,localport=5701], bind any local is true
Sep 08, 2020 7:08:40 AM com.hazelcast.system
INFO: [10.42.128.11]:5701 [dev] [4.0.2] Hazelcast 4.0.2 (20200702 - 2de3027) starting at [10.42.128.11]:5701
Sep 08, 2020 7:08:40 AM com.hazelcast.system
INFO: [10.42.128.11]:5701 [dev] [4.0.2] Copyright (c) 2008-2020, Hazelcast, Inc. All Rights Reserved.
Sep 08, 2020 7:08:42 AM com.hazelcast.spi.impl.operationservice.impl.BackpressureRegulator
INFO: [10.42.128.11]:5701 [dev] [4.0.2] Backpressure is disabled
Sep 08, 2020 7:08:43 AM com.hazelcast.spi.discovery.integration.DiscoveryService
INFO: [10.42.128.11]:5701 [dev] [4.0.2] Kubernetes Discovery properties: { service-dns: null, service-dns-timeout: 5, service-name: hazelcast-service, service-port: 0, service-label: null, service-label-value: true, namespace: default, pod-label: null, pod-label-value: null, resolve-not-ready-addresses: true, use-node-name-as-external-address: false, kubernetes-api-retries: 20, kubernetes-master: https://kubernetes.default.svc}
Sep 08, 2020 7:08:43 AM com.hazelcast.spi.discovery.integration.DiscoveryService
INFO: [10.42.128.11]:5701 [dev] [4.0.2] Kubernetes Discovery activated with mode: KUBERNETES_API
Sep 08, 2020 7:08:43 AM com.hazelcast.instance.impl.Node
INFO: [10.42.128.11]:5701 [dev] [4.0.2] Activating Discovery SPI Joiner
Sep 08, 2020 7:08:43 AM com.hazelcast.cp.CPSubsystem
INFO: [10.42.128.11]:5701 [dev] [4.0.2] CP Subsystem is enabled with 3 members.
Sep 08, 2020 7:08:44 AM com.hazelcast.spi.impl.operationexecutor.impl.OperationExecutorImpl
INFO: [10.42.128.11]:5701 [dev] [4.0.2] Starting 2 partition threads and 3 generic threads (1 dedicated for priority tasks)
Sep 08, 2020 7:08:44 AM com.hazelcast.internal.diagnostics.Diagnostics
INFO: [10.42.128.11]:5701 [dev] [4.0.2] Diagnostics disabled. To enable add -Dhazelcast.diagnostics.enabled=true to the JVM arguments.
Sep 08, 2020 7:08:45 AM com.hazelcast.core.LifecycleService
INFO: [10.42.128.11]:5701 [dev] [4.0.2] [10.42.128.11]:5701 is STARTING
Sep 08, 2020 7:08:47 AM com.hazelcast.kubernetes.RetryUtils
WARNING: Couldn't discover Hazelcast members using Kubernetes API, [1] retrying in 1 seconds...
Sep 08, 2020 7:08:49 AM com.hazelcast.kubernetes.RetryUtils
WARNING: Couldn't discover Hazelcast members using Kubernetes API, [2] retrying in 2 seconds...
Sep 08, 2020 7:08:51 AM com.hazelcast.kubernetes.RetryUtils
WARNING: Couldn't discover Hazelcast members using Kubernetes API, [3] retrying in 3 seconds...
Sep 08, 2020 7:08:54 AM com.hazelcast.kubernetes.RetryUtils
WARNING: Couldn't discover Hazelcast members using Kubernetes API, [4] retrying in 5 seconds...
Sep 08, 2020 7:09:00 AM com.hazelcast.kubernetes.RetryUtils
WARNING: Couldn't discover Hazelcast members using Kubernetes API, [5] retrying in 7 seconds...
Sep 08, 2020 7:09:07 AM com.hazelcast.kubernetes.RetryUtils
WARNING: Couldn't discover Hazelcast members using Kubernetes API, [6] retrying in 11 seconds...
Sep 08, 2020 7:09:12 AM com.hazelcast.internal.ascii.rest.HttpPostCommandProcessor
WARNING: [10.42.128.11]:5701 [dev] [4.0.2] An error occurred while handling request HttpCommand [HTTP_GET]{uri='/hazelcast/health/node-state'}AbstractTextCommand[HTTP_GET]{requestId=0}
java.lang.NullPointerException
at com.hazelcast.internal.ascii.rest.HttpGetCommandProcessor.handleHealthcheck(HttpGetCommandProcessor.java:137)
at com.hazelcast.internal.ascii.rest.HttpGetCommandProcessor.handle(HttpGetCommandProcessor.java:79)
at com.hazelcast.internal.ascii.rest.HttpGetCommandProcessor.handle(HttpGetCommandProcessor.java:47)
at com.hazelcast.internal.ascii.TextCommandServiceImpl$CommandExecutor.run(TextCommandServiceImpl.java:396)
at com.hazelcast.internal.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:217)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
at com.hazelcast.internal.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:64)
at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:80)
Sep 08, 2020 7:09:14 AM com.hazelcast.internal.ascii.rest.HttpPostCommandProcessor
WARNING: [10.42.128.11]:5701 [dev] [4.0.2] An error occurred while handling request HttpCommand [HTTP_GET]{uri='/hazelcast/health/node-state'}AbstractTextCommand[HTTP_GET]{requestId=0}
java.lang.NullPointerException
at com.hazelcast.internal.ascii.rest.HttpGetCommandProcessor.handleHealthcheck(HttpGetCommandProcessor.java:137)
at com.hazelcast.internal.ascii.rest.HttpGetCommandProcessor.handle(HttpGetCommandProcessor.java:79)
at com.hazelcast.internal.ascii.rest.HttpGetCommandProcessor.handle(HttpGetCommandProcessor.java:47)
at com.hazelcast.internal.ascii.TextCommandServiceImpl$CommandExecutor.run(TextCommandServiceImpl.java:396)
at com.hazelcast.internal.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:217)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
at com.hazelcast.internal.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:64)
at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:80)
Sep 08, 2020 7:09:19 AM com.hazelcast.kubernetes.RetryUtils
WARNING: Couldn't discover Hazelcast members using Kubernetes API, [7] retrying in 17 seconds...
Sep 08, 2020 7:09:22 AM com.hazelcast.internal.ascii.rest.HttpPostCommandProcessor
WARNING: [10.42.128.11]:5701 [dev] [4.0.2] An error occurred while handling request HttpCommand [HTTP_GET]{uri='/hazelcast/health/node-state'}AbstractTextCommand[HTTP_GET]{requestId=0}
java.lang.NullPointerException
at com.hazelcast.internal.ascii.rest.HttpGetCommandProcessor.handleHealthcheck(HttpGetCommandProcessor.java:137)
at com.hazelcast.internal.ascii.rest.HttpGetCommandProcessor.handle(HttpGetCommandProcessor.java:79)
at com.hazelcast.internal.ascii.rest.HttpGetCommandProcessor.handle(HttpGetCommandProcessor.java:47)
at com.hazelcast.internal.ascii.TextCommandServiceImpl$CommandExecutor.run(TextCommandServiceImpl.java:396)
at com.hazelcast.internal.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:217)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
at com.hazelcast.internal.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:64)
at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:80)
Sep 08, 2020 7:09:24 AM com.hazelcast.internal.ascii.rest.HttpPostCommandProcessor
WARNING: [10.42.128.11]:5701 [dev] [4.0.2] An error occurred while handling request HttpCommand [HTTP_GET]{uri='/hazelcast/health/node-state'}AbstractTextCommand[HTTP_GET]{requestId=0}
java.lang.NullPointerException
at com.hazelcast.internal.ascii.rest.HttpGetCommandProcessor.handleHealthcheck(HttpGetCommandProcessor.java:137)
at com.hazelcast.internal.ascii.rest.HttpGetCommandProcessor.handle(HttpGetCommandProcessor.java:79)
at com.hazelcast.internal.ascii.rest.HttpGetCommandProcessor.handle(HttpGetCommandProcessor.java:47)
at com.hazelcast.internal.ascii.TextCommandServiceImpl$CommandExecutor.run(TextCommandServiceImpl.java:396)
at com.hazelcast.internal.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:217)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
at com.hazelcast.internal.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:64)
at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:80)
Sep 08, 2020 7:09:32 AM com.hazelcast.internal.ascii.rest.HttpPostCommandProcessor
WARNING: [10.42.128.11]:5701 [dev] [4.0.2] An error occurred while handling request HttpCommand [HTTP_GET]{uri='/hazelcast/health/node-state'}AbstractTextCommand[HTTP_GET]{requestId=0}
java.lang.NullPointerException
at com.hazelcast.internal.ascii.rest.HttpGetCommandProcessor.handleHealthcheck(HttpGetCommandProcessor.java:137)
at com.hazelcast.internal.ascii.rest.HttpGetCommandProcessor.handle(HttpGetCommandProcessor.java:79)
at com.hazelcast.internal.ascii.rest.HttpGetCommandProcessor.handle(HttpGetCommandProcessor.java:47)
at com.hazelcast.internal.ascii.TextCommandServiceImpl$CommandExecutor.run(TextCommandServiceImpl.java:396)
at com.hazelcast.internal.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:217)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
at com.hazelcast.internal.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:64)
at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:80)
Sep 08, 2020 7:09:34 AM com.hazelcast.internal.ascii.rest.HttpPostCommandProcessor
WARNING: [10.42.128.11]:5701 [dev] [4.0.2] An error occurred while handling request HttpCommand [HTTP_GET]{uri='/hazelcast/health/node-state'}AbstractTextCommand[HTTP_GET]{requestId=0}
java.lang.NullPointerException
at com.hazelcast.internal.ascii.rest.HttpGetCommandProcessor.handleHealthcheck(HttpGetCommandProcessor.java:137)
at com.hazelcast.internal.ascii.rest.HttpGetCommandProcessor.handle(HttpGetCommandProcessor.java:79)
at com.hazelcast.internal.ascii.rest.HttpGetCommandProcessor.handle(HttpGetCommandProcessor.java:47)
at com.hazelcast.internal.ascii.TextCommandServiceImpl$CommandExecutor.run(TextCommandServiceImpl.java:396)
at com.hazelcast.internal.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:217)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
at com.hazelcast.internal.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:64)
at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:80)
This is our custom dockerfile for Hazelcast that because we need some changes in it:
FROM alpine:3.11
# Versions of Hazelcast and Hazelcast plugins
ARG HZ_VERSION=4.0.2
ARG CACHE_API_VERSION=1.1.1
ARG JMX_PROMETHEUS_AGENT_VERSION=0.13.0
ARG BUCKET4J_VERSION=4.10.0
# Build constants
ARG HZ_HOME="/opt/hazelcast"
# JARs to download
# for lib directory:
ARG HAZELCAST_ALL_URL="https://repo1.maven.org/maven2/com/hazelcast/hazelcast-all/${HZ_VERSION}/hazelcast-all-${HZ_VERSION}.jar"
# for user-lib directory:
ARG JCACHE_API_URL="https://repo1.maven.org/maven2/javax/cache/cache-api/${CACHE_API_VERSION}/cache-api-${CACHE_API_VERSION}.jar"
ARG PROMETHEUS_AGENT_URL="https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/${JMX_PROMETHEUS_AGENT_VERSION}/jmx_prometheus_javaagent-${JMX_PROMETHEUS_AGENT_VERSION}.jar"
ARG BUCKET4J_CORE_URL="https://repo1.maven.org/maven2/com/github/vladimir-bukhtoyarov/bucket4j-core/${BUCKET4J_VERSION}/bucket4j-core-${BUCKET4J_VERSION}.jar"
ARG BUCKET4J_HAZELCAST_URL="https://repo1.maven.org/maven2/com/github/vladimir-bukhtoyarov/bucket4j-hazelcast/${BUCKET4J_VERSION}/bucket4j-hazelcast-${BUCKET4J_VERSION}.jar"
ARG BUCKET4J_JCACHE_URL="https://repo1.maven.org/maven2/com/github/vladimir-bukhtoyarov/bucket4j-jcache/${BUCKET4J_VERSION}/bucket4j-jcache-${BUCKET4J_VERSION}.jar"
# Runtime constants / variables
ENV HZ_HOME="${HZ_HOME}" \
CLASSPATH_DEFAULT="${HZ_HOME}/*:${HZ_HOME}/lib/*:${HZ_HOME}/user-lib/*" \
JAVA_OPTS_DEFAULT="-Djava.net.preferIPv4Stack=true -Djava.util.logging.config.file=${HZ_HOME}/logging.properties -XX:MaxRAMPercentage=80.0 -XX:+UseParallelGC --add-modules java.se --add-exports java.base/jdk.internal.ref=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.nio=ALL-UNNAMED --add-opens java.base/sun.nio.ch=ALL-UNNAMED --add-opens java.management/sun.management=ALL-UNNAMED --add-opens jdk.management/com.sun.management.internal=ALL-UNNAMED" \
PROMETHEUS_PORT="" \
PROMETHEUS_CONFIG="${HZ_HOME}/jmx_agent_config.yaml" \
LOGGING_LEVEL="" \
CLASSPATH="" \
JAVA_OPTS=""
# Expose port
EXPOSE 5701
COPY *.sh *.yaml *.jar *.properties ${HZ_HOME}/
RUN echo "Updating Alpine system" \
&& apk upgrade --update-cache --available \
&& echo "Installing new APK packages" \
&& apk add openjdk11-jre bash curl procps nss
RUN mkdir "${HZ_HOME}/user-lib"\
&& cd "${HZ_HOME}/user-lib" \
&& for USER_JAR_URL in ${JCACHE_API_URL} ${PROMETHEUS_AGENT_URL} ${BUCKET4J_CORE_URL} ${BUCKET4J_HAZELCAST_URL} ${BUCKET4J_JCACHE_URL}; do curl -sf -O -L ${USER_JAR_URL}; done
# Install
RUN echo "Downloading Hazelcast and related JARs" \
&& mkdir "${HZ_HOME}/lib" \
&& cd "${HZ_HOME}/lib" \
&& for JAR_URL in ${HAZELCAST_ALL_URL}; do curl -sf -O -L ${JAR_URL}; done \
&& echo "Granting read permission to ${HZ_HOME}" \
&& chmod 755 -R ${HZ_HOME} \
&& echo "Setting Pardot ID to 'docker'" \
&& echo 'hazelcastDownloadId=docker' > "${HZ_HOME}/hazelcast-download.properties" \
&& echo "Cleaning APK packages" \
&& rm -rf /var/cache/apk/*
WORKDIR ${HZ_HOME}
# Start Hazelcast server
CMD ["/opt/hazelcast/start-hazelcast.sh"]

Answer 1

Hazelcast Kubernetes discovery plugin does not allow you to specify the custom location of certificates. They are always read from the default location: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt.
The parameter ca-certificate lets you inline the certificate like presented here, but not specify the path of the certificate.
If you think such a feature would be useful, feel to create a GH issue at https://github.com/hazelcast/hazelcast-kubernetes (you can also send a PR with the change).