I have installed Solr helm and deployed it in k8s.
When I port-forwarded and checked at http://localhost:7000/solr/, solr ui is loaded and works fine
kubectl port-forward solrcloud-dev-solrcloud-0 7000:8983
But when I mapped it to the domain solr-dev.aws.domain.local and when I hit the url, it shows
HTTP ERROR 404 Not Found
URI: /
STATUS: 404
MESSAGE: Not Found
SERVLET: -
I have also added annotations whereever necessary as we are using traefik
annotations:
kubernetes.io/ingress.class: traefik-internal
Following is my values.yaml file. Please suggest If I have done anything wrong
nameOverride: ''
fullnameOverride: ''
replicas: null
global:
imagePullSecrets: []
clusterDomain: ''
serviceAccount:
create: false
name: ''
image:
repository: solr
tag: ''
pullPolicy: ''
imagePullSecret: ''
busyBoxImage: {}
solrOptions:
javaMemory: ''
javaOpts: ''
zkJavaOpts: ''
logLevel: ''
gcTune: ''
solrModules: []
additionalLibs: []
security: {}
addressability:
podPort: 8983
commonServicePort: null
kubeDomain: ''
external:
method: Ingress
domainName: solr-dev.aws.domain.local
additionalDomainNames: []
useExternalAddress: true
hideNodes: false
hideCommon: false
nodePortOverride: null
updateStrategy:
method: Managed
managed: {}
restartSchedule: ''
dataStorage:
type: ephemeral
capacity: 20Gi
ephemeral: {}
persistent:
reclaimPolicy: Retain
pvc:
name: ''
labels: {}
annotations: {}
storageClassName: ''
backupRepositories: []
zk:
chroot: ''
uniqueChroot: false
address: ''
externalAddress: ''
provided:
replicas: 3
image: {}
zookeeperPodPolicy: {}
storageType: ''
persistence: {}
ephemeral: {}
config: {}
acl: {}
readOnlyAcl: {}
solrTLS: {}
solrClientTLS: {}
podOptions:
annotations:
kubernetes.io/ingress.class: traefik-internal
labels: {}
sidecarContainers: []
initContainers: []
priorityClassName: ''
envVars: []
podSecurityContext: {}
terminationGracePeriodSeconds: null
serviceAccountName: ''
affinity: null
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: pe-stateless-dev-amd
operator: In
values:
- 'true'
tolerations: []
nodeSelector: {}
topologySpreadConstraints: []
livenessProbe: {}
readinessProbe: {}
startupProbe: {}
lifecycle: {}
imagePullSecrets: []
resources: {}
defaultInitContainerResources: {}
volumes: []
statefulSetOptions:
annotations: {}
labels: {}
podManagementPolicy: ''
commonServiceOptions:
annotations: {}
labels: {}
headlessServiceOptions:
annotations: {}
labels: {}
nodeServiceOptions:
annotations: {}
labels: {}
ingressOptions:
annotations:
kubernetes.io/ingress.class: traefik-internal
hosts:
- host: solr-dev.aws.domain.local
paths:
- path: /
labels: {}
ingressClassName: ''
configMapOptions:
annotations: {}
labels: {}
providedConfigMap: ''
Related
I'm trying to have SSO in opensearch-dashboards via openid to AzureAD.
Overally - there is no need to have an encrypted communication between opensearch and nodes, there is no need to have encrypted communication between dashboards and master pod. All I need is to have working SSO to Azure AD to see dashboards.
I got errors in dashboards pod like: "res":{"statusCode":302,"responseTime":746,"contentLength":9} and tags":["error","plugins","securityDashboards"],"pid":1,"message":"OpenId authentication failed: Error: [index_not_found_exception] no such index [_plugins], with { index=\"_plugins\" │ │ & resource.id=\"_plugins\" & resource.type=\"index_expression\" & index_uuid=\"_na_\" }"} and the browser tells me The page isn’t redirecting properly
With last try I got from the ingress pod the error: Service "default/opensearch-values-opensearch-dashboards" does not have any active Endpoint.
I really appreciate any advice what am I missing...
I use helm installation of opensearch to AWS EKS (with nginx-controller ingress to publish the adress)
In AD I have an app registered like https://<some_address>/auth/openid/login
Here are my actual helm values:
opensearch.yaml
---
clusterName: "opensearch-cluster"
nodeGroup: "master"
masterService: "opensearch-cluster-master"
roles:
- master
- ingest
- data
- remote_cluster_client
replicas: 3
minimumMasterNodes: 1
majorVersion: ""
global:
dockerRegistry: "<registry>"
opensearchHome: /usr/share/opensearch
config:
log4j2.properties: |
rootLogger.level = debug
opensearch.yml: |
cluster.name: opensearch-cluster
network.host: 0.0.0.0
plugins.security.disabled: true
plugins:
security:
ssl:
transport:
pemcert_filepath: esnode.pem
pemkey_filepath: esnode-key.pem
pemtrustedcas_filepath: root-ca.pem
enforce_hostname_verification: false
http:
enabled: false
pemcert_filepath: esnode.pem
pemkey_filepath: esnode-key.pem
pemtrustedcas_filepath: root-ca.pem
allow_unsafe_democertificates: true
allow_default_init_securityindex: true
authcz:
admin_dn:
- CN=kirk,OU=client,O=client,L=test,C=de
audit.type: internal_opensearch
enable_snapshot_restore_privilege: true
check_snapshot_restore_write_privileges: true
restapi:
roles_enabled: ["all_access", "security_rest_api_access"]
system_indices:
enabled: true
indices:
[
".opendistro-alerting-config",
".opendistro-alerting-alert*",
".opendistro-anomaly-results*",
".opendistro-anomaly-detector*",
".opendistro-anomaly-checkpoints",
".opendistro-anomaly-detection-state",
".opendistro-reports-*",
".opendistro-notifications-*",
".opendistro-notebooks",
".opendistro-asynchronous-search-response*",
]
extraEnvs: []
envFrom: []
secretMounts: []
hostAliases: []
image:
repository: "opensearchproject/opensearch"
tag: ""
pullPolicy: "IfNotPresent"
podAnnotations: {}
labels: {}
opensearchJavaOpts: "-Xmx512M -Xms512M"
resources:
requests:
cpu: "1000m"
memory: "100Mi"
initResources: {}
sidecarResources: {}
networkHost: "0.0.0.0"
rbac:
create: false
serviceAccountAnnotations: {}
serviceAccountName: ""
podSecurityPolicy:
create: false
name: ""
spec:
privileged: true
fsGroup:
rule: RunAsAny
runAsUser:
rule: RunAsAny
seLinux:
rule: RunAsAny
supplementalGroups:
rule: RunAsAny
volumes:
- secret
- configMap
- persistentVolumeClaim
- emptyDir
persistence:
enabled: true
enableInitChown: true
labels:
enabled: false
accessModes:
- ReadWriteOnce
size: 8Gi
annotations: {}
extraVolumes: []
extraVolumeMounts: []
extraContainers: []
extraInitContainers:
- name: sysctl
image: docker.io/bitnami/bitnami-shell:10-debian-10-r199
imagePullPolicy: "IfNotPresent"
command:
- /bin/bash
- -ec
- |
CURRENT=`sysctl -n vm.max_map_count`;
DESIRED="262144";
if [ "$DESIRED" -gt "$CURRENT" ]; then
sysctl -w vm.max_map_count=262144;
fi;
CURRENT=`sysctl -n fs.file-max`;
DESIRED="65536";
if [ "$DESIRED" -gt "$CURRENT" ]; then
sysctl -w fs.file-max=65536;
fi;
securityContext:
privileged: true
priorityClassName: ""
antiAffinityTopologyKey: "kubernetes.io/hostname"
antiAffinity: "soft"
nodeAffinity: {}
topologySpreadConstraints: []
podManagementPolicy: "Parallel"
enableServiceLinks: true
protocol: http
httpPort: 9200
transportPort: 9300
service:
labels: {}
labelsHeadless: {}
headless:
annotations: {}
type: ClusterIP
nodePort: ""
annotations: {}
httpPortName: http
transportPortName: transport
loadBalancerIP: ""
loadBalancerSourceRanges: []
externalTrafficPolicy: ""
updateStrategy: RollingUpdate
maxUnavailable: 1
podSecurityContext:
fsGroup: 1000
runAsUser: 1000
securityContext:
capabilities:
drop:
- ALL
runAsNonRoot: true
runAsUser: 1000
securityConfig:
enabled: true
path: "/usr/share/opensearch/plugins/opensearch-security/securityconfig"
actionGroupsSecret:
configSecret:
internalUsersSecret:
rolesSecret:
rolesMappingSecret:
tenantsSecret:
config:
securityConfigSecret: ""
dataComplete: true
data:
config.yml: |-
config:
dynamic:
authc:
basic_internal_auth_domain:
description: "Authenticate via HTTP Basic"
http_enabled: true
transport_enabled: true
order: 1
http_authenticator:
type: "basic"
challenge: false
authentication_backend:
type: "internal"
openid_auth_domain:
order: 0
http_enabled: true
transport_enabled: true
http_authenticator:
type: openid
challenge: false
config:
enable_ssl: true
verify_hostnames: false
subject_key: preferred_username
roles_key: roles
openid_connect_url: https://login.microsoftonline.com/<ms_id>/v2.0/.well-known/openid-configuration
authentication_backend:
type: noop
roles_mapping.yml: |-
all_access
reserved: false
backend_roles:
- "admin"
description: "Maps admin to all_access"
terminationGracePeriod: 120
sysctlVmMaxMapCount: 262144
readinessProbe:
failureThreshold: 3
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 3
timeoutSeconds: 2000
schedulerName: ""
imagePullSecrets:
- name: regcred
nodeSelector: {}
tolerations: []
ingress:
enabled: false
annotations: {}
path: /
hosts:
- chart-example.local
tls: []
nameOverride: ""
fullnameOverride: ""
masterTerminationFix: false
lifecycle: {}
keystore: []
networkPolicy:
create: false
http:
enabled: false
fsGroup: ""
sysctl:
enabled: false
plugins:
enabled: false
installList: []
extraObjects: []
opensearch-dashboards.yaml
---
opensearchHosts: "http://opensearch-cluster-master:9200"
replicaCount: 1
image:
repository: "<registry>"
tag: "1.3.1"
pullPolicy: "IfNotPresent"
imagePullSecrets:
- name: regcred
nameOverride: ""
fullnameOverride: ""
serviceAccount:
create: true
annotations: {}
name: ""
rbac:
create: true
secretMounts: []
podAnnotations: {}
extraEnvs: []
envFrom: []
extraVolumes: []
extraVolumeMounts: []
extraInitContainers: ""
extraContainers: ""
podSecurityContext: {}
securityContext:
capabilities:
drop:
- ALL
runAsNonRoot: true
runAsUser: 1000
config:
opensearch_dashboards.yml: |
opensearch_security.cookie.secure: false
opensearch_security.auth.type: openid
opensearch_security.openid.client_id: <client_id>
opensearch_security.openid.client_secret: <client_secret>
opensearch_security.openid.base_redirect_url: https://<some_aws_id>.elb.amazonaws.com
opensearch_security.openid.connect_url: https://login.microsoftonline.com/<MS id>/v2.0/.well-known/openid-configuration
priorityClassName: ""
opensearchAccount:
secret: ""
keyPassphrase:
enabled: false
labels: {}
hostAliases: []
serverHost: "0.0.0.0"
service:
type: ClusterIP
port: 5601
loadBalancerIP: ""
nodePort: ""
labels: {}
annotations: {}
loadBalancerSourceRanges: []
httpPortName: http
ingress:
enabled: false
annotations: {}
hosts:
- host: chart-example.local
paths:
- path: /
backend:
serviceName: chart-example.local
servicePort: 80
tls: []
resources:
requests:
cpu: "100m"
memory: "512M"
limits:
cpu: "100m"
memory: "512M"
autoscaling:
enabled: false
minReplicas: 1
maxReplicas: 10
targetCPUUtilizationPercentage: 80
updateStrategy:
type: "Recreate"
nodeSelector: {}
tolerations: []
affinity: {}
extraObjects: []
I'm trying to setup an opensearch cluster on kubernetes.
when settings my nodes nothing fails but I get an error at a certain point and
this is my stateful set:
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: {{ .Values.global.name }} --> opensearch
namespace: {{ .Values.global.namespace }}
clusterName: {{ .Values.global.clusterName }}
labels:
app: {{ .Values.global.name }}
annotations:
majorVersion: "{{ include "opensearch.majorVersion" . }}"
spec:
serviceName: "opensearch"
selector:
matchLabels:
app: {{ .Values.global.name }}
replicas: {{ .Values.replicas }} ---> 3
template:
metadata:
name: {{ .Values.global.name }}
labels:
app: {{ .Values.global.name }}
role: master
spec:
initContainers:
- name: init-sysctl
image: busybox
imagePullPolicy: IfNotPresent
command:
- sysctl
- -w
- vm.max_map_count=262144
securityContext:
privileged: true
- name: increase-fd-ulimit
image: busybox
imagePullPolicy: IfNotPresent
command: [ "sh", "-c", "ulimit -n 65536" ]
containers:
- name: "{{.Values.global.name }}-master"
image: opensearchproject/opensearch
imagePullPolicy: IfNotPresent
resources:
limits:
memory: '8Gi'
cpu: "1"
requests:
memory: '8Gi'
cpu: "1"
ports:
- containerPort: 9200
name: rest
protocol: TCP
- containerPort: 9300
name: inter-node
protocol: TCP
env:
- name: node.name
value: opensearch
- name: cluster.name
value: "{{ .Values.global.clusterName }}"
- name: node.master
value: "true"
- name: node.data
value: "true"
- name: node.ingest
value: "true"
- name: cluster.initial_master_nodes
value: "opensearch-0"
- name: discovery.seed_hosts
value: "opensearch-0"
- name: ES_JAVA_OPTS
value: "-Xms4g -Xmx4g"
volumeMounts:
- name: {{ .Values.global.name }}
mountPath: /etc/opensearch/data
- name: config
mountPath: /usr/share/opensearch/config/opensearch.yml
subPath: opensearch.yml
- name: node-key
mountPath: {{ .Values.privateKeyPathOnMachine }}
subPath: node-key.pem
readOnly: true
- name: node
mountPath: {{ .Values.certPathOnMachine }}
subPath: node.pem
readOnly: true
- name: root-ca
mountPath: {{ .Values.rootCertPathOnMachine }}
subPath: root-ca.pem
- name: admin-key
mountPath: {{ .Values.adminKeyCertPathOnMachine }}
subPath: admin-key.pem
readOnly: true
- name: admin
mountPath: {{ .Values.adminCertPathOnMachine }}
subPath: admin.pem
readOnly: true
- name: client
mountPath: {{ .Values.clientCertPathOnMachine }}
subPath: client.pem
readOnly: true
- name: client-key
mountPath: {{ .Values.clientKeyCertPathOnMachine }}
subPath: client-key.pem
readOnly: true
volumes:
- name: config
configMap:
name: opensearch-config
- name: config-opensearch
configMap:
name: config
- name: node
secret:
secretName: node
items:
- key: node.pem
path: node.pem
- name: node-key
secret:
secretName: node-key
items:
- key: node-key.pem
path: node-key.pem
- name: root-ca
secret:
secretName: root-ca
items:
- key: root-ca.pem
path: root-ca.pem
- name: admin-key
secret:
secretName: admin-key
items:
- key: admin-key.pem
path: admin-key.pem
- name: admin
secret:
secretName: admin
items:
- key: admin.pem
path: admin.pem
- name: client-key
secret:
secretName: client-key
items:
- key: client-key.pem
path: client-key.pem
- name: client
secret:
secretName: client
items:
- key: client.pem
path: client.pem
volumeClaimTemplates:
- metadata:
name: {{ .Values.global.name }}
labels:
app: {{ .Values.global.name }}
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: "20Gi"
when I set use this definition, at some point I get this error:
[ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch] Exception while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, AUDIT] (index=.opendistro_security)
org.opensearch.cluster.block.ClusterBlockException: bl
ocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
now if I'm trying to set the nodes:
- name: cluster.initial_master_nodes
value: "opensearch-0.opensearch.search.svc.cluster.local,opensearch-1.opensearch.search.svc.cluster.local,opensearch-2.opensearch.search.svc.cluster.local"
- name: discovery.seed_hosts
value: "opensearch-0.opensearch.search.svc.cluster.local,opensearch-1.opensearch.search.svc.cluster.local,opensearch-2.opensearch.search.svc.cluster.local"
It fails on the same error, only this time this warning comes before.
[opensearch] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [opensearch-0.opensearch.search.svc.cluster.local, opensearch-1.opensearch.search.svc.cluster.local, opensearch-2.opensearch.search.svc.cluster.local] to bootstrap a cluster: have discovered [{opensearch}{SKON7g98RnyQsz6SAYqWRg}{GkUCV8mISZqITHiU0LDEzQ}{10.20.1.103}{10.20.1.103:9300}{dimr}{shard_indexing_pressure_enabled=true}, {opensearch}{qRuv6YgYQjGVatLGRGfPtQ}{62EmR4a_Sb-nhV9_7F05aA}{10.20.2.137}{10.20.2.137:9300}{dimr}{shard_indexing_pressure_enabled=true}, {opensearch}{8flMQsmxQEGN4LeBMemHsQ}{6zNV_pTZRnO6YneCzvOA4Q}{10.20.3.204}{10.20.3.204:9300}{dimr}{shard_indexing_pressure_enabled=true}]; discovery will continue using [10.20.2.137:9300, 10.20.3.204:9300] from hosts providers and [{opensearch}{SKON7g98RnyQsz6SAYqWRg}{GkUCV8mISZqITHiU0LDEzQ}{10.20.1.103}{10.20.1.103:9300}{dimr}{shard_indexing_pressure_enabled=true}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
When I try to run in the pod the security setup script
/usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh -cd ../securityconfig/ -icl -nhnv -cacert /usr/share/opensearch/config/certificates/root-ca.pem -cert /usr/share/opensearch/config/certificates/admin.pem -key /usr/share/opensearch/config/certificates/admin-key.pem
it fails too, output:
Cannot retrieve cluster state due to: null. This is not an error, will keep on trying ...
Root cause: MasterNotDiscoveredException[null] (org.opensearch.discovery.MasterNotDiscoveredException/org.opensearch.discovery.MasterNotDiscoveredException)
kubectl get svc opensearch -o yaml
apiVersion: v1
kind: Service
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"clusterName":"gloat-dev","labels":{"app.kubernetes.io/instance":"opensearch-gloat-dev-search"},"name":"opensearch","namespace":"search"},"spec":{"clusterIP":"None","ports":[{"name":"http","port":9200},{"name":"transport","port":9300}],"publishNotReadyAddresses":true,"selector":{"app":"opensearch"},"type":"ClusterIP"}}
creationTimestamp: "2022-01-17T12:21:56Z"
labels:
app.kubernetes.io/instance: opensearch-gloat-dev-search
managedFields:
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.: {}
f:kubectl.kubernetes.io/last-applied-configuration: {}
f:labels:
.: {}
f:app.kubernetes.io/instance: {}
f:spec:
f:clusterIP: {}
f:ports:
.: {}
k:{"port":9200,"protocol":"TCP"}:
.: {}
f:name: {}
f:port: {}
f:protocol: {}
f:targetPort: {}
k:{"port":9300,"protocol":"TCP"}:
.: {}
f:name: {}
f:port: {}
f:protocol: {}
f:targetPort: {}
f:publishNotReadyAddresses: {}
f:selector:
.: {}
f:app: {}
f:sessionAffinity: {}
f:type: {}
manager: argocd-application-controller
operation: Update
time: "2022-01-17T12:21:56Z"
name: opensearch
namespace: search
resourceVersion: "173096782"
selfLink: /api/v1/namespaces/search/services/opensearch
uid: ec2a49a1-f4e8-4419-9324-1761b892aeca
spec:
clusterIP: None
ports:
- name: http
port: 9200
protocol: TCP
targetPort: 9200
- name: transport
port: 9300
protocol: TCP
targetPort: 9300
publishNotReadyAddresses: true
selector:
app: opensearch
sessionAffinity: None
type: ClusterIP
status:
loadBalancer: {}
error log trace:
https://pastebin.com/MtJp9iwf (loops)
Try with:
- name: cluster.initial_master_nodes
value: "opensearch-0,opensearch-1,opensearch-2" # opensearch master node names
- name: discovery.seed_hosts
value: "opensearch" # headless service dns which points to master nodes, in your case it's "opensearch".
How do you change the cockroachdb YAML configuration from Rancher catalog that it will just
Use local disk (testing on a local computer)
Just use 1GB of disk space (should be enough for testing)
Here's the complete YAML
clusterDomain: cluster.local
conf:
attrs: []
cache: 25%
cluster-name: ''
disable-cluster-name-verification: false
http-port: 8080
join: []
locality: ''
logtostderr: INFO
max-disk-temp-storage: 0
max-offset: 500ms
max-sql-memory: 25%
port: 26257
single-node: false
sql-audit-dir: ''
image:
credentials: {}
pullPolicy: IfNotPresent
repository: cockroachdb/cockroach
tag: v20.1.3
ingress:
annotations: {}
enabled: false
hosts: []
labels: {}
paths:
- /
tls: []
init:
affinity: {}
annotations: {}
labels:
app.kubernetes.io/component: init
nodeSelector: {}
resources: {}
tolerations: []
labels: {}
networkPolicy:
enabled: false
ingress:
grpc: []
http: []
service:
discovery:
annotations: {}
labels:
app.kubernetes.io/component: cockroachdb
ports:
grpc:
external:
name: grpc
port: 26257
internal:
name: grpc-internal
port: 26257
http:
name: http
port: 8080
public:
annotations: {}
labels:
app.kubernetes.io/component: cockroachdb
type: ClusterIP
statefulset:
annotations: {}
args: []
budget:
maxUnavailable: 1
env: []
labels:
app.kubernetes.io/component: cockroachdb
nodeAffinity: {}
nodeSelector: {}
podAffinity: {}
podAntiAffinity:
type: soft
weight: 100
podManagementPolicy: Parallel
priorityClassName: ''
replicas: 3
resources: {}
secretMounts: []
tolerations: []
updateStrategy:
type: RollingUpdate
storage:
hostPath: ''
persistentVolume:
annotations: {}
enabled: true
labels: {}
size: 100Gi
storageClass: ''
tls:
certs:
clientRootSecret: cockroachdb-root
nodeSecret: cockroachdb-node
provided: false
tlsSecret: false
enabled: false
init:
image:
credentials: {}
pullPolicy: IfNotPresent
repository: cockroachdb/cockroach-k8s-request-cert
tag: '0.4'
serviceAccount:
create: true
name: ''
Storage: 100Gi
Storage Size
It looks like Storage and store.persistentVolume.size can both be set to 1Gi if you are looking for one gigabyte of storage.
Local Storage
Then, I would check if you have a storageClass by running kubectl get storageClass. Many times, cluters come with the local-path-provisioner storageClass. If you have that, I would try setting store.persistentVolume.storageClass to the name of the local path proversion storageClass you have installed on your system. If you don't have that or an alternative, I would consider installing it.
More Info
I'm not certain, but it's possible that it is using this helm chart to install the database. This section of the chart deals with the volume claim managment, so I would look here if you need to do more troubleshooting: https://github.com/cockroachdb/helm-charts/blob/master/cockroachdb/templates/statefulset.yaml#L376
I'm trying to install CockroachDB with Rancher and getting some problems, showing:
FailedBinding (5) 14 sec ago no persistent volumes available for this claim and no storage class is set
How can this be solved?
Here are the configurations in my local machine:
PersistentVolumeClaim: datadir-cockroachdb-0
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
creationTimestamp: "2021-01-07T23:50:42Z"
finalizers:
- kubernetes.io/pvc-protection
labels:
app.kubernetes.io/component: cockroachdb
app.kubernetes.io/instance: cockroachdb
app.kubernetes.io/name: cockroachdb
managedFields:
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:labels:
.: {}
f:app.kubernetes.io/component: {}
f:app.kubernetes.io/instance: {}
f:app.kubernetes.io/name: {}
f:spec:
f:accessModes: {}
f:resources:
f:requests:
.: {}
f:storage: {}
f:volumeMode: {}
f:status:
f:phase: {}
manager: k3s
operation: Update
time: "2021-01-07T23:50:41Z"
name: datadir-cockroachdb-0
namespace: default
resourceVersion: "188922"
selfLink: /api/v1/namespaces/default/persistentvolumeclaims/datadir-cockroachdb-0
uid: ef83d3c7-0309-44a8-b379-0134835d97a9
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 100Gi
volumeMode: Filesystem
status:
phase: Pending
CockroachDB
clusterDomain: cluster.local
conf:
attrs: []
cache: 25%
cluster-name: ''
disable-cluster-name-verification: false
http-port: 8080
join: []
locality: ''
logtostderr: INFO
max-disk-temp-storage: 0
max-offset: 500ms
max-sql-memory: 25%
port: 26257
single-node: false
sql-audit-dir: ''
image:
credentials: {}
pullPolicy: IfNotPresent
repository: cockroachdb/cockroach
tag: v20.1.3
ingress:
annotations: {}
enabled: false
hosts: []
labels: {}
paths:
- /
tls: []
init:
affinity: {}
annotations: {}
labels:
app.kubernetes.io/component: init
nodeSelector: {}
resources: {}
tolerations: []
labels: {}
networkPolicy:
enabled: false
ingress:
grpc: []
http: []
service:
discovery:
annotations: {}
labels:
app.kubernetes.io/component: cockroachdb
ports:
grpc:
external:
name: grpc
port: 26257
internal:
name: grpc-internal
port: 26257
http:
name: http
port: 8080
public:
annotations: {}
labels:
app.kubernetes.io/component: cockroachdb
type: ClusterIP
statefulset:
annotations: {}
args: []
budget:
maxUnavailable: 1
env: []
labels:
app.kubernetes.io/component: cockroachdb
nodeAffinity: {}
nodeSelector: {}
podAffinity: {}
podAntiAffinity:
type: soft
weight: 100
podManagementPolicy: Parallel
priorityClassName: ''
replicas: 3
resources: {}
secretMounts: []
tolerations: []
updateStrategy:
type: RollingUpdate
storage:
hostPath: ''
persistentVolume: volume1
annotations: {}
enabled: true
labels: {}
size: 1Gi
storageClass: local-storage ''
tls:
certs:
clientRootSecret: cockroachdb-root
nodeSecret: cockroachdb-node
provided: false
tlsSecret: false
enabled: false
init:
image:
credentials: {}
pullPolicy: IfNotPresent
repository: cockroachdb/cockroach-k8s-request-cert
tag: '0.4'
serviceAccount:
create: true
name: ''
Storage: 1Gi
PersistentVolume
apiVersion: v1
kind: PersistentVolume
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: '{"apiVersion":"v1","kind":"PersistentVolume","metadata":{"annotations":{},"labels":{"type":"local"},"name":"volume1"},"spec":{"accessModes":["ReadWriteOnce"],"capacity":{"storage":"10Gi"},"hostPath":{"path":"/data/volume1"}}}'
creationTimestamp: "2021-01-07T23:11:43Z"
finalizers:
- kubernetes.io/pv-protection
labels:
type: local
managedFields:
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:status:
f:phase: {}
manager: k3s
operation: Update
time: "2021-01-07T23:11:43Z"
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations: {}
f:labels:
.: {}
f:type: {}
f:spec:
f:accessModes: {}
f:capacity: {}
f:hostPath:
.: {}
f:path: {}
f:type: {}
f:persistentVolumeReclaimPolicy: {}
f:volumeMode: {}
manager: kubectl
operation: Update
time: "2021-01-07T23:11:43Z"
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
f:kubectl.kubernetes.io/last-applied-configuration: {}
f:spec:
f:capacity:
f:storage: {}
manager: Go-http-client
operation: Update
time: "2021-01-07T23:12:11Z"
name: volume1
resourceVersion: "173783"
selfLink: /api/v1/persistentvolumes/volume1
uid: 6e76984c-22cd-4219-9ff6-ba7f67c1ca72
spec:
accessModes:
- ReadWriteOnce
capacity:
storage: 4Gi
hostPath:
path: /data/volume1
type: ""
persistentVolumeReclaimPolicy: Retain
volumeMode: Filesystem
status:
phase: Available
StorageClass
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
creationTimestamp: "2021-01-07T23:29:17Z"
managedFields:
- apiVersion: storage.k8s.io/v1
fieldsType: FieldsV1
fieldsV1:
f:provisioner: {}
f:reclaimPolicy: {}
f:volumeBindingMode: {}
manager: rancher
operation: Update
time: "2021-01-07T23:29:17Z"
name: local-storage
resourceVersion: "180190"
selfLink: /apis/storage.k8s.io/v1/storageclasses/local-storage
uid: 0a5f8b75-7fb5-4965-91ee-91b0a087339a
provisioner: kubernetes.io/no-provisioner
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
With provided details looks like your storage class is missing on rancher.
Without storage class respective PVC won't get created so it's giving an error. Storage classes may change with cloud providers and also based on the requirement of the type of disk SSD, HDD.
You can get more idea : https://rancher.com/docs/rancher/v2.x/en/cluster-admin/volumes-and-storage/provisioning-new-storage/
check first your PV is available and after that check for storage class and PVC.
It looks like the issue was with Rancher this time (Thank you #Harsh Manvar for answering). If you have more questions about CockroachDB you can also join the CockroachDB community slack channel where you will find loads of experts who can answer your questions in a timely manner. (And be sure to join the #community channel also to have some FUN!) :) https://go.crdb.dev/p/slack
I am running Airflow on Kubernetes from the stable helm chart. I'm running this in an AWS environment. This error exists with and without mounting any external volumes for log storage. I tried to set the configuration of the [logs] section to point to an EFS volume that I created. The PV gets mounted through a PVC but my containers are crashing (scheduler and web) due to the following error:
*** executing Airflow initdb...
Unable to load the config, contains a configuration error.
Traceback (most recent call last):
File "/usr/local/lib/python3.6/logging/config.py", line 565, in configure
handler = self.configure_handler(handlers[name])
File "/usr/local/lib/python3.6/logging/config.py", line 738, in configure_handler
result = factory(**kwargs)
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/utils/log/file_processor_handler.py", line 50, in __init__
os.makedirs(self._get_log_directory())
File "/usr/local/lib/python3.6/os.py", line 220, in makedirs
mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/opt/airflow/logs/scheduler/2020-08-20'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/airflow/.local/bin/airflow", line 25, in <module>
from airflow.configuration import conf
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/__init__.py", line 47, in <module>
settings.initialize()
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/settings.py", line 374, in initialize
LOGGING_CLASS_PATH = configure_logging()
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/logging_config.py", line 68, in configure_logging
raise e
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/logging_config.py", line 63, in configure_logging
dictConfig(logging_config)
File "/usr/local/lib/python3.6/logging/config.py", line 802, in dictConfig
dictConfigClass(config).configure()
File "/usr/local/lib/python3.6/logging/config.py", line 573, in configure
'%r: %s' % (name, e))
ValueError: Unable to configure handler 'processor': [Errno 13] Permission denied: '/opt/airflow/logs/scheduler/2020-08-20'
Persistent volume (created manually not from the stable/airflow chart)
apiVersion: v1
kind: PersistentVolume
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","kind":"PersistentVolume","metadata":{"annotations":{},"name":"efs-pv"},"spec":{"accessModes":["ReadWriteMany"],"capacity":{"storage":"5Gi"},"csi":{"driver":"efs.csi.aws.com","volumeHandle":"fs-e476a166"},"persistentVolumeReclaimPolicy":"Retain","storageClassName":"efs-sc","volumeMode":"Filesystem"}}
pv.kubernetes.io/bound-by-controller: "yes"
creationTimestamp: "2020-08-20T15:47:21Z"
finalizers:
- kubernetes.io/pv-protection
name: efs-pv
resourceVersion: "49476860"
selfLink: /api/v1/persistentvolumes/efs-pv
uid: 45d9f5ea-66c1-493e-a2f5-03e17f397747
spec:
accessModes:
- ReadWriteMany
capacity:
storage: 5Gi
claimRef:
apiVersion: v1
kind: PersistentVolumeClaim
name: efs-claim
namespace: airflow
resourceVersion: "49476857"
uid: 354103ea-f8a9-47f1-a7cf-8f449f9a2e8b
csi:
driver: efs.csi.aws.com
volumeHandle: fs-e476a166
persistentVolumeReclaimPolicy: Retain
storageClassName: efs-sc
volumeMode: Filesystem
status:
phase: Bound
Persistent Volume Claim for logs (created manually not from the stable/airflow chart):
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{},"name":"efs-claim","namespace":"airflow"},"spec":{"accessModes":["ReadWriteMany"],"resources":{"requests":{"storage":"5Gi"}},"storageClassName":"efs-sc"}}
pv.kubernetes.io/bind-completed: "yes"
pv.kubernetes.io/bound-by-controller: "yes"
creationTimestamp: "2020-08-20T15:47:46Z"
finalizers:
- kubernetes.io/pvc-protection
name: efs-claim
namespace: airflow
resourceVersion: "49476866"
selfLink: /api/v1/namespaces/airflow/persistentvolumeclaims/efs-claim
uid: 354103ea-f8a9-47f1-a7cf-8f449f9a2e8b
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 5Gi
storageClassName: efs-sc
volumeMode: Filesystem
volumeName: efs-pv
status:
accessModes:
- ReadWriteMany
capacity:
storage: 5Gi
phase: Bound
My values.yaml below:
airflow:
image:
repository: apache/airflow
tag: 1.10.10-python3.6
## values: Always or IfNotPresent
pullPolicy: IfNotPresent
pullSecret: ""
executor: KubernetesExecutor
fernetKey: "XXXXXXXXXHIVb8jK6lfmSAvx4mO6Arehnc="
config:
AIRFLOW__CORE__REMOTE_LOGGING: "True"
AIRFLOW__CORE__REMOTE_BASE_LOG_FOLDER: "s3://mybucket/airflow/logs"
AIRFLOW__CORE__REMOTE_LOG_CONN_ID: "MyS3Conn"
AIRFLOW__KUBERNETES__WORKER_CONTAINER_REPOSITORY: "apache/airflow"
AIRFLOW__KUBERNETES__WORKER_CONTAINER_TAG: "1.10.10-python3.6"
AIRFLOW__KUBERNETES__WORKER_CONTAINER_IMAGE_PULL_POLICY: "IfNotPresent"
AIRFLOW__KUBERNETES__WORKER_PODS_CREATION_BATCH_SIZE: "10"
AIRFLOW__KUBERNETES__LOGS_VOLUME_CLAIM: "efs-claim"
AIRFLOW__KUBERNETES__GIT_REPO: "git#github.com:org/myrepo.git"
AIRFLOW__KUBERNETES__GIT_BRANCH: "develop"
AIRFLOW__KUBERNETES__GIT_DAGS_FOLDER_MOUNT_POINT: "/opt/airflow/dags"
AIRFLOW__KUBERNETES__DAGS_VOLUME_SUBPATH: "repo/"
AIRFLOW__KUBERNETES__GIT_SSH_KEY_SECRET_NAME: "airflow-git-keys"
AIRFLOW__KUBERNETES__NAMESPACE: "airflow"
AIRFLOW__KUBERNETES__DELETE_WORKER_PODS: "True"
AIRFLOW__KUBERNETES__RUN_AS_USER: "50000"
AIRFLOW__CORE__LOAD_EXAMPLES: "False"
AIRFLOW__SCHEDULER__DAG_DIR_LIST_INTERVAL: "60"
AIRFLOW__KUBERNETES__WORKER_SERVICE_ACCOUNT_NAME: "airflow"
podAnnotations: {}
extraEnv: []
extraConfigmapMounts: []
extraContainers: []
extraPipPackages: []
extraVolumeMounts: []
extraVolumes: []
scheduler:
resources: {}
nodeSelector: {}
affinity: {}
tolerations: []
labels: {}
podLabels: {}
annotations: {}
podAnnotations: {}
podDisruptionBudget:
enabled: true
maxUnavailable: "100%"
minAvailable: ""
connections:
- id: MyS3Conn
type: aws
extra: |
{
"aws_access_key_id": "XXXXXXXXX",
"aws_secret_access_key": "XXXXXXXX",
"region_name":"us-west-1"
}
refreshConnections: true
variables: |
{}
pools: |
{}
numRuns: -1
initdb: true
preinitdb: false
initialStartupDelay: 0
extraInitContainers: []
web:
resources: {}
replicas: 1
nodeSelector: {}
affinity: {}
tolerations: []
labels: {}
podLabels: {}
annotations: {}
podAnnotations: {}
service:
annotations: {}
sessionAffinity: "None"
sessionAffinityConfig: {}
type: ClusterIP
externalPort: 8080
loadBalancerIP: ""
loadBalancerSourceRanges: []
nodePort:
http: ""
baseUrl: "http://localhost:8080"
serializeDAGs: false
extraPipPackages: []
initialStartupDelay: 0
minReadySeconds: 5
readinessProbe:
enabled: false
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 1
successThreshold: 1
failureThreshold: 3
livenessProbe:
enabled: true
scheme: HTTP
initialDelaySeconds: 300
periodSeconds: 30
timeoutSeconds: 3
successThreshold: 1
failureThreshold: 2
secretsDir: /var/airflow/secrets
secrets: []
secretsMap:
workers:
enabled: false
resources: {}
replicas: 1
nodeSelector: {}
affinity: {}
tolerations: []
labels: {}
podLabels: {}
annotations: {}
podAnnotations: {}
autoscaling:
enabled: false
maxReplicas: 2
metrics: []
initialStartupDelay: 0
celery:
instances: 1
gracefullTermination: false
gracefullTerminationPeriod: 600
terminationPeriod: 60
secretsDir: /var/airflow/secrets
secrets: []
secretsMap:
flower:
enabled: false
resources: {}
nodeSelector: {}
affinity: {}
tolerations: []
labels: {}
podLabels: {}
annotations: {}
podAnnotations: {}
basicAuthSecret: ""
basicAuthSecretKey: ""
urlPrefix: ""
service:
annotations: {}
type: ClusterIP
externalPort: 5555
loadBalancerIP: ""
loadBalancerSourceRanges: []
nodePort:
http: ""
initialStartupDelay: 0
extraConfigmapMounts: []
logs:
path: /opt/airflow/logs
persistence:
enabled: true
existingClaim: efs-claim
subPath: ""
storageClass: efs-sc
accessMode: ReadWriteMany
size: 1Gi
dags:
path: /opt/airflow/dags
doNotPickle: false
installRequirements: false
persistence:
enabled: false
existingClaim: ""
subPath: ""
storageClass: ""
accessMode: ReadOnlyMany
size: 1Gi
git:
url: git#github.com:org/myrepo.git
ref: develop
secret: airflow-git-keys
sshKeyscan: false
privateKeyName: id_rsa
repoHost: github.com
repoPort: 22
gitSync:
enabled: true
resources: {}
image:
repository: alpine/git
tag: latest
pullPolicy: Always
refreshTime: 60
initContainer:
enabled: false
resources: {}
image:
repository: alpine/git
tag: latest
pullPolicy: Always
mountPath: "/dags"
syncSubPath: ""
ingress:
enabled: false
web:
annotations: {}
path: ""
host: ""
livenessPath: ""
tls:
enabled: false
secretName: ""
precedingPaths: []
succeedingPaths: []
flower:
annotations: {}
path: ""
host: ""
livenessPath: ""
tls:
enabled: false
secretName: ""
rbac:
create: true
serviceAccount:
create: true
name: ""
annotations: {}
extraManifests: []
postgresql:
enabled: true
postgresqlDatabase: airflow
postgresqlUsername: postgres
postgresqlPassword: airflow
existingSecret: ""
existingSecretKey: "postgresql-password"
persistence:
enabled: true
storageClass: ""
accessModes:
- ReadWriteOnce
size: 5Gi
externalDatabase:
type: postgres
host: localhost
port: 5432
database: airflow
user: airflow
passwordSecret: ""
passwordSecretKey: "postgresql-password"
redis:
enabled: false
password: airflow
existingSecret: ""
existingSecretKey: "redis-password"
cluster:
enabled: false
slaveCount: 1
master:
resources: {}
persistence:
enabled: false
storageClass: ""
accessModes:
- ReadWriteOnce
size: 8Gi
slave:
resources: {}
persistence:
enabled: false
storageClass: ""
accessModes:
- ReadWriteOnce
size: 8Gi
externalRedis:
host: localhost
port: 6379
databaseNumber: 1
passwordSecret: ""
passwordSecretKey: "redis-password"
serviceMonitor:
enabled: false
selector:
prometheus: kube-prometheus
path: /admin/metrics
interval: "30s"
prometheusRule:
enabled: false
additionalLabels: {}
groups: []
I'm not really sure what to do here if anyone knows how to fix the permission error.
I have had this issue with the Google Cloud Plateform and the helm airflow 1.2.0 chart (which uses airflow 2).
What ended up working was:
extraInitContainers:
- name: fix-volume-logs-permissions
image: busybox
command: [ "sh", "-c", "chown -R 50000:0 /opt/airflow/logs/" ]
securityContext:
runAsUser: 0
volumeMounts:
- mountPath: /opt/airflow/logs/
name: logs
by tweaking based on Ajay's answer. Please note that:
the values 50000:0 are based on uid and gid setup in your values.yaml
you need to use extraInitContainers under scheduler and not worker
"logs" seems to be the volume name automatically used by the helm logging config when enabled
Security context was necessary for me or else the chown failed due to unprivileged rights
You can use extraInitContainers with scheduler to change the permission, something like this.
extraInitContainers:
- name: volume-logs
image: busybox
command: ["sh", "-c", "chown -R 50000:50000 /opt/airflow/logs/"]
volumeMounts:
- mountPath: /opt/airflow/logs/
name: logs-data
This will change permission of the mount point.
you can try modify workers.persistence.fixPermissions: true in values.yaml, it is OK.