opensearch blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized] - kubernetes

Question

I'm trying to setup an opensearch cluster on kubernetes.
when settings my nodes nothing fails but I get an error at a certain point and
this is my stateful set:
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: {{ .Values.global.name }} --> opensearch
namespace: {{ .Values.global.namespace }}
clusterName: {{ .Values.global.clusterName }}
labels:
app: {{ .Values.global.name }}
annotations:
majorVersion: "{{ include "opensearch.majorVersion" . }}"
spec:
serviceName: "opensearch"
selector:
matchLabels:
app: {{ .Values.global.name }}
replicas: {{ .Values.replicas }} ---> 3
template:
metadata:
name: {{ .Values.global.name }}
labels:
app: {{ .Values.global.name }}
role: master
spec:
initContainers:
- name: init-sysctl
image: busybox
imagePullPolicy: IfNotPresent
command:
- sysctl
- -w
- vm.max_map_count=262144
securityContext:
privileged: true
- name: increase-fd-ulimit
image: busybox
imagePullPolicy: IfNotPresent
command: [ "sh", "-c", "ulimit -n 65536" ]
containers:
- name: "{{.Values.global.name }}-master"
image: opensearchproject/opensearch
imagePullPolicy: IfNotPresent
resources:
limits:
memory: '8Gi'
cpu: "1"
requests:
memory: '8Gi'
cpu: "1"
ports:
- containerPort: 9200
name: rest
protocol: TCP
- containerPort: 9300
name: inter-node
protocol: TCP
env:
- name: node.name
value: opensearch
- name: cluster.name
value: "{{ .Values.global.clusterName }}"
- name: node.master
value: "true"
- name: node.data
value: "true"
- name: node.ingest
value: "true"
- name: cluster.initial_master_nodes
value: "opensearch-0"
- name: discovery.seed_hosts
value: "opensearch-0"
- name: ES_JAVA_OPTS
value: "-Xms4g -Xmx4g"
volumeMounts:
- name: {{ .Values.global.name }}
mountPath: /etc/opensearch/data
- name: config
mountPath: /usr/share/opensearch/config/opensearch.yml
subPath: opensearch.yml
- name: node-key
mountPath: {{ .Values.privateKeyPathOnMachine }}
subPath: node-key.pem
readOnly: true
- name: node
mountPath: {{ .Values.certPathOnMachine }}
subPath: node.pem
readOnly: true
- name: root-ca
mountPath: {{ .Values.rootCertPathOnMachine }}
subPath: root-ca.pem
- name: admin-key
mountPath: {{ .Values.adminKeyCertPathOnMachine }}
subPath: admin-key.pem
readOnly: true
- name: admin
mountPath: {{ .Values.adminCertPathOnMachine }}
subPath: admin.pem
readOnly: true
- name: client
mountPath: {{ .Values.clientCertPathOnMachine }}
subPath: client.pem
readOnly: true
- name: client-key
mountPath: {{ .Values.clientKeyCertPathOnMachine }}
subPath: client-key.pem
readOnly: true
volumes:
- name: config
configMap:
name: opensearch-config
- name: config-opensearch
configMap:
name: config
- name: node
secret:
secretName: node
items:
- key: node.pem
path: node.pem
- name: node-key
secret:
secretName: node-key
items:
- key: node-key.pem
path: node-key.pem
- name: root-ca
secret:
secretName: root-ca
items:
- key: root-ca.pem
path: root-ca.pem
- name: admin-key
secret:
secretName: admin-key
items:
- key: admin-key.pem
path: admin-key.pem
- name: admin
secret:
secretName: admin
items:
- key: admin.pem
path: admin.pem
- name: client-key
secret:
secretName: client-key
items:
- key: client-key.pem
path: client-key.pem
- name: client
secret:
secretName: client
items:
- key: client.pem
path: client.pem
volumeClaimTemplates:
- metadata:
name: {{ .Values.global.name }}
labels:
app: {{ .Values.global.name }}
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: "20Gi"
when I set use this definition, at some point I get this error:
[ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch] Exception while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, AUDIT] (index=.opendistro_security)
org.opensearch.cluster.block.ClusterBlockException: bl
ocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
now if I'm trying to set the nodes:
- name: cluster.initial_master_nodes
value: "opensearch-0.opensearch.search.svc.cluster.local,opensearch-1.opensearch.search.svc.cluster.local,opensearch-2.opensearch.search.svc.cluster.local"
- name: discovery.seed_hosts
value: "opensearch-0.opensearch.search.svc.cluster.local,opensearch-1.opensearch.search.svc.cluster.local,opensearch-2.opensearch.search.svc.cluster.local"
It fails on the same error, only this time this warning comes before.
[opensearch] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [opensearch-0.opensearch.search.svc.cluster.local, opensearch-1.opensearch.search.svc.cluster.local, opensearch-2.opensearch.search.svc.cluster.local] to bootstrap a cluster: have discovered [{opensearch}{SKON7g98RnyQsz6SAYqWRg}{GkUCV8mISZqITHiU0LDEzQ}{10.20.1.103}{10.20.1.103:9300}{dimr}{shard_indexing_pressure_enabled=true}, {opensearch}{qRuv6YgYQjGVatLGRGfPtQ}{62EmR4a_Sb-nhV9_7F05aA}{10.20.2.137}{10.20.2.137:9300}{dimr}{shard_indexing_pressure_enabled=true}, {opensearch}{8flMQsmxQEGN4LeBMemHsQ}{6zNV_pTZRnO6YneCzvOA4Q}{10.20.3.204}{10.20.3.204:9300}{dimr}{shard_indexing_pressure_enabled=true}]; discovery will continue using [10.20.2.137:9300, 10.20.3.204:9300] from hosts providers and [{opensearch}{SKON7g98RnyQsz6SAYqWRg}{GkUCV8mISZqITHiU0LDEzQ}{10.20.1.103}{10.20.1.103:9300}{dimr}{shard_indexing_pressure_enabled=true}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
When I try to run in the pod the security setup script
/usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh -cd ../securityconfig/ -icl -nhnv -cacert /usr/share/opensearch/config/certificates/root-ca.pem -cert /usr/share/opensearch/config/certificates/admin.pem -key /usr/share/opensearch/config/certificates/admin-key.pem
it fails too, output:
Cannot retrieve cluster state due to: null. This is not an error, will keep on trying ...
Root cause: MasterNotDiscoveredException[null] (org.opensearch.discovery.MasterNotDiscoveredException/org.opensearch.discovery.MasterNotDiscoveredException)
kubectl get svc opensearch -o yaml
apiVersion: v1
kind: Service
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"clusterName":"gloat-dev","labels":{"app.kubernetes.io/instance":"opensearch-gloat-dev-search"},"name":"opensearch","namespace":"search"},"spec":{"clusterIP":"None","ports":[{"name":"http","port":9200},{"name":"transport","port":9300}],"publishNotReadyAddresses":true,"selector":{"app":"opensearch"},"type":"ClusterIP"}}
creationTimestamp: "2022-01-17T12:21:56Z"
labels:
app.kubernetes.io/instance: opensearch-gloat-dev-search
managedFields:
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.: {}
f:kubectl.kubernetes.io/last-applied-configuration: {}
f:labels:
.: {}
f:app.kubernetes.io/instance: {}
f:spec:
f:clusterIP: {}
f:ports:
.: {}
k:{"port":9200,"protocol":"TCP"}:
.: {}
f:name: {}
f:port: {}
f:protocol: {}
f:targetPort: {}
k:{"port":9300,"protocol":"TCP"}:
.: {}
f:name: {}
f:port: {}
f:protocol: {}
f:targetPort: {}
f:publishNotReadyAddresses: {}
f:selector:
.: {}
f:app: {}
f:sessionAffinity: {}
f:type: {}
manager: argocd-application-controller
operation: Update
time: "2022-01-17T12:21:56Z"
name: opensearch
namespace: search
resourceVersion: "173096782"
selfLink: /api/v1/namespaces/search/services/opensearch
uid: ec2a49a1-f4e8-4419-9324-1761b892aeca
spec:
clusterIP: None
ports:
- name: http
port: 9200
protocol: TCP
targetPort: 9200
- name: transport
port: 9300
protocol: TCP
targetPort: 9300
publishNotReadyAddresses: true
selector:
app: opensearch
sessionAffinity: None
type: ClusterIP
status:
loadBalancer: {}
error log trace:
https://pastebin.com/MtJp9iwf (loops)

Answer 1

Try with:
- name: cluster.initial_master_nodes
value: "opensearch-0,opensearch-1,opensearch-2" # opensearch master node names
- name: discovery.seed_hosts
value: "opensearch" # headless service dns which points to master nodes, in your case it's "opensearch".