I have installed Solr helm and deployed it in k8s.
When I port-forwarded and checked at http://localhost:7000/solr/, solr ui is loaded and works fine
kubectl port-forward solrcloud-dev-solrcloud-0 7000:8983
But when I mapped it to the domain solr-dev.aws.domain.local and when I hit the url, it shows
HTTP ERROR 404 Not Found
URI: /
STATUS: 404
MESSAGE: Not Found
SERVLET: -
I have also added annotations whereever necessary as we are using traefik
annotations:
kubernetes.io/ingress.class: traefik-internal
Following is my values.yaml file. Please suggest If I have done anything wrong
nameOverride: ''
fullnameOverride: ''
replicas: null
global:
imagePullSecrets: []
clusterDomain: ''
serviceAccount:
create: false
name: ''
image:
repository: solr
tag: ''
pullPolicy: ''
imagePullSecret: ''
busyBoxImage: {}
solrOptions:
javaMemory: ''
javaOpts: ''
zkJavaOpts: ''
logLevel: ''
gcTune: ''
solrModules: []
additionalLibs: []
security: {}
addressability:
podPort: 8983
commonServicePort: null
kubeDomain: ''
external:
method: Ingress
domainName: solr-dev.aws.domain.local
additionalDomainNames: []
useExternalAddress: true
hideNodes: false
hideCommon: false
nodePortOverride: null
updateStrategy:
method: Managed
managed: {}
restartSchedule: ''
dataStorage:
type: ephemeral
capacity: 20Gi
ephemeral: {}
persistent:
reclaimPolicy: Retain
pvc:
name: ''
labels: {}
annotations: {}
storageClassName: ''
backupRepositories: []
zk:
chroot: ''
uniqueChroot: false
address: ''
externalAddress: ''
provided:
replicas: 3
image: {}
zookeeperPodPolicy: {}
storageType: ''
persistence: {}
ephemeral: {}
config: {}
acl: {}
readOnlyAcl: {}
solrTLS: {}
solrClientTLS: {}
podOptions:
annotations:
kubernetes.io/ingress.class: traefik-internal
labels: {}
sidecarContainers: []
initContainers: []
priorityClassName: ''
envVars: []
podSecurityContext: {}
terminationGracePeriodSeconds: null
serviceAccountName: ''
affinity: null
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: pe-stateless-dev-amd
operator: In
values:
- 'true'
tolerations: []
nodeSelector: {}
topologySpreadConstraints: []
livenessProbe: {}
readinessProbe: {}
startupProbe: {}
lifecycle: {}
imagePullSecrets: []
resources: {}
defaultInitContainerResources: {}
volumes: []
statefulSetOptions:
annotations: {}
labels: {}
podManagementPolicy: ''
commonServiceOptions:
annotations: {}
labels: {}
headlessServiceOptions:
annotations: {}
labels: {}
nodeServiceOptions:
annotations: {}
labels: {}
ingressOptions:
annotations:
kubernetes.io/ingress.class: traefik-internal
hosts:
- host: solr-dev.aws.domain.local
paths:
- path: /
labels: {}
ingressClassName: ''
configMapOptions:
annotations: {}
labels: {}
providedConfigMap: ''
Im trying to deploy telegraf helm chart on kubernetes.
helm upgrade --install telegraf-instance -f values.yaml influxdata/telegraf
When I add modbus input plugin with holding_register i get error
[telegraf] Error running agent: Error loading config file /etc/telegraf/telegraf.conf: Error parsing data: line 49: key `name’ is in conflict with line 2fd
my values.yaml like below
## Default values.yaml for Telegraf
## This is a YAML-formatted file.
## ref: https://hub.docker.com/r/library/telegraf/tags/
replicaCount: 1
image:
repo: "telegraf"
tag: "1.21.4"
pullPolicy: IfNotPresent
podAnnotations: {}
podLabels: {}
imagePullSecrets: []
args: []
env:
- name: HOSTNAME
value: "telegraf-polling-service"
resources: {}
nodeSelector: {}
affinity: {}
tolerations: []
service:
enabled: true
type: ClusterIP
annotations: {}
rbac:
create: true
clusterWide: false
rules: []
serviceAccount:
create: false
name:
annotations: {}
config:
agent:
interval: 60s
round_interval: true
metric_batch_size: 1000000
metric_buffer_limit: 100000000
collection_jitter: 0s
flush_interval: 60s
flush_jitter: 0s
precision: ''
hostname: '9825128'
omit_hostname: false
processors:
- enum:
mapping:
field: "status"
dest: "status_code"
value_mappings:
healthy: 1
problem: 2
critical: 3
inputs:
- modbus:
name: "PS MAIN ENGINE"
controller: 'tcp://192.168.0.101:502'
slave_id: 1
holding_registers:
- name: "Coolant Level"
byte_order: CDAB
data_type: FLOAT32
scale: 0.001
address: [51410, 51411]
- modbus:
name: "SB MAIN ENGINE"
controller: 'tcp://192.168.0.102:502'
slave_id: 1
holding_registers:
- name: "Coolant Level"
byte_order: CDAB
data_type: FLOAT32
scale: 0.001
address: [51410, 51411]
outputs:
- influxdb_v2:
token: token
organization: organisation
bucket: bucket
urls:
- "url"
metrics:
health:
enabled: true
service_address: "http://:8888"
threshold: 5000.0
internal:
enabled: true
collect_memstats: false
pdb:
create: true
minAvailable: 1
Problem resolved by doing the following steps
deleted config section of my values.yaml
added my telegraf.conf to /additional_config path
added configmap to kubernetes with the following command
kubectl create configmap external-config --from-file=/additional_config
added the following command to values.yaml
volumes:
- name: my-config
configMap:
name: external-config
volumeMounts:
- name: my-config
mountPath: /additional_config
args:
- "--config=/etc/telegraf/telegraf.conf"
- "--config-directory=/additional_config"
I am working on a use-case where I need to add a new container in jupyterhub pod , This new container (sidecontainer) monitors jupyterhub directories.
Jupyterhub container while comming up creates a dynamic PV , see the section below
singleuser:
podNameTemplate:
extraTolerations: []
nodeSelector: {}
extraNodeAffinity:
required: []
preferred: []
extraPodAffinity:
required: []
preferred: []
extraPodAntiAffinity:
required: []
preferred: []
networkTools:
image:
name: jupyterhub/k8s-network-tools
tag: "set-by-chartpress"
pullPolicy:
pullSecrets: []
resources: {}
cloudMetadata:
# block set to true will append a privileged initContainer using the
# iptables to block the sensitive metadata server at the provided ip.
blockWithIptables: true
ip: 169.254.169.254
networkPolicy:
enabled: true
ingress: []
egress:
# Required egress to communicate with the hub and DNS servers will be
# augmented to these egress rules.
#
# This default rule explicitly allows all outbound traffic from singleuser
# pods, except to a typical IP used to return metadata that can be used by
# someone with malicious intent.
- to:
- ipBlock:
cidr: 0.0.0.0/0
except:
- 169.254.169.254/32
interNamespaceAccessLabels: ignore
allowedIngressPorts: []
events: true
extraAnnotations: {}
extraLabels:
hub.jupyter.org/network-access-hub: "true"
extraFiles: {}
extraEnv: {}
lifecycleHooks: {}
initContainers: []
extraContainers: []
uid: 1000
fsGid: 100
serviceAccountName:
storage:
type: dynamic
extraLabels: {}
extraVolumes: []
extraVolumeMounts: []
static:
pvcName:
subPath: "{username}"
capacity: 10Gi
homeMountPath: /home/jovyan
dynamic:
storageClass:
pvcNameTemplate: claim-{username}{servername}
volumeNameTemplate: volume-{username}{servername}
storageAccessModes: [ReadWriteOnce]
image:
name: jupyterhub/k8s-singleuser-sample
tag: "set-by-chartpress"
pullPolicy:
pullSecrets: []
startTimeout: 300
cpu:
limit:
guarantee:
memory:
limit:
guarantee: 1G
extraResource:
limits: {}
guarantees: {}
cmd:
defaultUrl:
extraPodConfig: {}
profileList: []
I have added my new container in the extraContainer section of deployment file , my container does start but the dynamic PV is not mounted on that container.
Is the use-case I am trying to achieve technically possible at kubernetes level.
full yaml file is present here for reference
https://github.com/jupyterhub/zero-to-jupyterhub-k8s/blob/main/jupyterhub/values.yaml
Config Map for reference
singleuser:
baseUrl: /
cloudMetadata:
enabled: true
ip: xx.xx.xx.xx
cpu: {}
events: true
extraAnnotations: {}
extraConfigFiles:
config_files:
- cm_key: ''
content: ''
file_path: ''
- cm_key: ''
content: ''
file_path: ''
enabled: false
extraContainers:
- image: 'docker:19.03-rc-dind'
lifecycle:
postStart:
exec:
command:
- sh
- '-c'
- update-ca-certificates; echo Certificates Updated
name: dind
securityContext:
privileged: true
volumeMounts:
- mountPath: /var/lib/docker
name: dind-storage
- mountPath: /usr/local/share/ca-certificates/
name: docker-cert
extraEnv:
ACTUAL_HADOOP_CONF_DIR: ''
ACTUAL_HIVE_CONF_DIR: ''
ACTUAL_SPARK_CONF_DIR: ''
CDH_PARCEL_DIR: ''
DOCKER_HOST: ''
JAVA_HOME:
LIVY_URL: ''
SPARK2_PARCEL_DIR: ''
extraLabels:
hub.jupyter.org/network-access-hub: 'true'
extraNodeAffinity:
preferred: []
required: []
extraPodAffinity:
preferred: []
required: []
extraPodAntiAffinity:
preferred: []
required: []
extraPodConfig: {}
extraResource:
guarantees: {}
limits: {}
extraTolerations: []
fsGid: 0
image:
name: >-
/jupyterhub/jpt-spark-magic
pullPolicy: IfNotPresent
tag: xxx
imagePullSecret:
email: null
enabled: false
registry: null
username: null
initContainers: []
lifecycleHooks: {}
memory:
guarantee: 1G
networkPolicy:
egress:
- to:
- ipBlock:
cidr: 0.0.0.0/0
except:
- 169.254.169.254/32
enabled: false
ingress: []
networkTools:
image:
name: >-
/k8s-hub-multispawn
pullPolicy: IfNotPresent
tag: '12345'
nodeSelector: {}
profileList:
- description: Python for data enthusiasts
display_name: 0
kubespawner_override:
cmd:
- jpt-entry-cmd.sh
cpu_limit: 1
environment:
XYZ_SERVICE_URL: 'http://XYZ:8080'
CURL_CA_BUNDLE: /etc/ssl/certs/ca-certificates.crt
DOCKER_HOST: 'tcp://localhost:2375'
HADOOP_CONF_DIR: /etc/hadoop/conf
HADOOP_HOME: /usr/hdp/3.1.5.6091-7/hadoop/
HDP_DIR: /usr/hdp/3.1.5.6091-7
HDP_HOME_DIR: /usr/hdp/3.1.5.6091-7
HDP_VERSION: 3.1.5.6091-7
HIVE_CONF_DIR: /usr/hdp/3.1.5.6091-7/hive
HIVE_HOME: /usr/hdp/3.1.5.6091-7/hive
INTEGRATION_ENV: HDP3
JAVA_HOME: /usr/jdk64/jdk1.8.0_112
LD_LIBRARY_PATH: >-
/usr/hdp/3.1.5.6091-7/hadoop/lib/native:/usr/jdk64/jdk1.8.0_112/jre:/usr/hdp/3.1.5.6091-7/usr/lib/:/usr/hdp/3.1.5.6091-7/usr/lib/
LIVY_URL: 'http://ammaster01.fake.org:8999'
MLFLOW_TRACKING_URI: 'http://mlflow:5100'
NO_PROXY: mlflow
SPARK_CONF_DIR: /etc/spark2/conf
SPARK_HOME: /usr/hdp/3.1.5.6091-7/spark2
SPARK2_PARCEL_DIR: /usr/hdp/3.1.5.6091-7/spark2
TOOLS_BASE_PATH: /usr/local/bin
image: >-
/jupyterhub/jpt-spark-magic:1.1.2
mem_limit: 4096M
uid: 0
- description: R for data enthusiasts
display_name: 1
kubespawner_override:
cmd:
- start-all.sh
environment:
XYZ_SERVICE_URL: 'http://XYZ-service:8080'
DISABLE_AUTH: 'true'
XYZ: /home/rstudio/kitematic
image: '/jupyterhub/rstudio:364094'
uid: 0
- description: Python for data enthusiasts test2
display_name: 2
kubespawner_override:
cmd:
- jpt-entry-cmd.sh
cpu_limit: 4
environment:
XYZ_SERVICE_URL: 'http://XYZ-service:8080'
CURL_CA_BUNDLE: /etc/ssl/certs/ca-certificates.crt
DOCKER_HOST: 'tcp://localhost:2375'
HADOOP_CONF_DIR: /etc/hadoop/conf
HADOOP_HOME: /usr/hdp/3.1.5.6091-7/hadoop/
HDP_DIR: /usr/hdp/3.1.5.6091-7
HDP_HOME_DIR: /usr/hdp/3.1.5.6091-7
HDP_VERSION: 3.1.5.6091-7
HIVE_CONF_DIR: /usr/hdp/3.1.5.6091-7/hive
HIVE_HOME: /usr/hdp/3.1.5.6091-7/hive
INTEGRATION_ENV: HDP3
JAVA_HOME: /usr/jdk64/jdk1.8.0_112
LD_LIBRARY_PATH: >-
/usr/hdp/3.1.5.6091-7/hadoop/lib/native:/usr/jdk64/jdk1.8.0_112/jre:/usr/hdp/3.1.5.6091-7/usr/lib/:/usr/hdp/3.1.5.6091-7/usr/lib/
LIVY_URL: 'http://xyz:8999'
MLFLOW_TRACKING_URI: 'http://mlflow:5100'
NO_PROXY: mlflow
SPARK_CONF_DIR: /etc/spark2/conf
SPARK_HOME: /usr/hdp/3.1.5.6091-7/spark2
SPARK2_PARCEL_DIR: /usr/hdp/3.1.5.6091-7/spark2
TOOLS_BASE_PATH: /usr/local/bin
image: >-
/jupyterhub/jpt-spark-magic:1.1.2
mem_limit: 8192M
uid: 0
- description: Python for data enthusiasts test3
display_name: 3
kubespawner_override:
cmd:
- jpt-entry-cmd.sh
cpu_limit: 8
environment:
XYZ_SERVICE_URL: 'http://XYZ-service:8080'
CURL_CA_BUNDLE: /etc/ssl/certs/ca-certificates.crt
DOCKER_HOST: 'tcp://localhost:2375'
HADOOP_CONF_DIR: /etc/hadoop/conf
HADOOP_HOME: /usr/hdp/3.1.5.6091-7/hadoop/
HDP_DIR: /usr/hdp/3.1.5.6091-7
HDP_HOME_DIR: /usr/hdp/3.1.5.6091-7
HDP_VERSION: 3.1.5.6091-7
HIVE_CONF_DIR: /usr/hdp/3.1.5.6091-7/hive
HIVE_HOME: /usr/hdp/3.1.5.6091-7/hive
INTEGRATION_ENV: HDP3
JAVA_HOME: /usr/jdk64/jdk1.8.0_112
LD_LIBRARY_PATH: >-
/usr/hdp/3.1.5.6091-7/hadoop/lib/native:/usr/jdk64/jdk1.8.0_112/jre:/usr/hdp/3.1.5.6091-7/usr/lib/:/usr/hdp/3.1.5.6091-7/usr/lib/
LIVY_URL: 'http://fake.org:8999'
MLFLOW_TRACKING_URI: 'http://mlflow:5100'
NO_PROXY: mlflow
SPARK_CONF_DIR: /etc/spark2/conf
SPARK_HOME: /usr/hdp/3.1.5.6091-7/spark2
SPARK2_PARCEL_DIR: /usr/hdp/3.1.5.6091-7/spark2
TOOLS_BASE_PATH: /usr/local/bin
image: >-
/jupyterhub/jpt-spark-magic:1.1.2
mem_limit: 16384M
uid: 0
startTimeout: 300
storage:
capacity: 10Gi
dynamic:
pvcNameTemplate: 'claim-{username}{servername}'
storageAccessModes:
- ReadWriteOnce
storageClass: nfs-client
volumeNameTemplate: 'volume-{username}{servername}'
extraLabels: {}
extraVolumeMounts:
- mountPath: /etc/krb5.conf
name: krb
readOnly: true
- mountPath: /usr/jdk64/jdk1.8.0_112
name: java-home
readOnly: true
- mountPath: /xyz/conda/envs
name: xyz-conda-envs
readOnly: false
- mountPath: /usr/hdp/
name: bigdata
readOnly: true
subPath: usr-hdp
- mountPath: /etc/hadoop/
name: bigdata
readOnly: true
subPath: HDP
- mountPath: /etc/hive/
name: bigdata
readOnly: true
subPath: hdp-hive
- mountPath: /etc/spark2/
name: bigdata
readOnly: true
subPath: hdp-spark2
extraVolumes:
- emptyDir: {}
name: dind-storage
- name: docker-cert
secret:
secretName: docker-cert
- hostPath:
path: /var/lib/ut_xyz_ts/jdk1.8.0_112
type: Directory
name: java-home
- hostPath:
path: /xyz/conda/envs
type: Directory
name: xyz-conda-envs
- hostPath:
path: /etc/krb5.conf
type: File
name: krb
- name: bigdata
persistentVolumeClaim:
claimName: bigdata
homeMountPath: '/home/{username}'
static:
subPath: '{username}'
type: dynamic
uid: 0
Thanks in advance ..
For you question, whether technically two or more containers of same pod share same volume, the answer in Yes. Refer here - https://youtu.be/GQJP9QdHHs8?t=82 .
But you should have a volumeMount (refer example in the video as well) defined in your extra containers spec as well. If you can check that, or share output of kubectl describe deployment <your-deployment> I can confirm it.
How do you change the cockroachdb YAML configuration from Rancher catalog that it will just
Use local disk (testing on a local computer)
Just use 1GB of disk space (should be enough for testing)
Here's the complete YAML
clusterDomain: cluster.local
conf:
attrs: []
cache: 25%
cluster-name: ''
disable-cluster-name-verification: false
http-port: 8080
join: []
locality: ''
logtostderr: INFO
max-disk-temp-storage: 0
max-offset: 500ms
max-sql-memory: 25%
port: 26257
single-node: false
sql-audit-dir: ''
image:
credentials: {}
pullPolicy: IfNotPresent
repository: cockroachdb/cockroach
tag: v20.1.3
ingress:
annotations: {}
enabled: false
hosts: []
labels: {}
paths:
- /
tls: []
init:
affinity: {}
annotations: {}
labels:
app.kubernetes.io/component: init
nodeSelector: {}
resources: {}
tolerations: []
labels: {}
networkPolicy:
enabled: false
ingress:
grpc: []
http: []
service:
discovery:
annotations: {}
labels:
app.kubernetes.io/component: cockroachdb
ports:
grpc:
external:
name: grpc
port: 26257
internal:
name: grpc-internal
port: 26257
http:
name: http
port: 8080
public:
annotations: {}
labels:
app.kubernetes.io/component: cockroachdb
type: ClusterIP
statefulset:
annotations: {}
args: []
budget:
maxUnavailable: 1
env: []
labels:
app.kubernetes.io/component: cockroachdb
nodeAffinity: {}
nodeSelector: {}
podAffinity: {}
podAntiAffinity:
type: soft
weight: 100
podManagementPolicy: Parallel
priorityClassName: ''
replicas: 3
resources: {}
secretMounts: []
tolerations: []
updateStrategy:
type: RollingUpdate
storage:
hostPath: ''
persistentVolume:
annotations: {}
enabled: true
labels: {}
size: 100Gi
storageClass: ''
tls:
certs:
clientRootSecret: cockroachdb-root
nodeSecret: cockroachdb-node
provided: false
tlsSecret: false
enabled: false
init:
image:
credentials: {}
pullPolicy: IfNotPresent
repository: cockroachdb/cockroach-k8s-request-cert
tag: '0.4'
serviceAccount:
create: true
name: ''
Storage: 100Gi
Storage Size
It looks like Storage and store.persistentVolume.size can both be set to 1Gi if you are looking for one gigabyte of storage.
Local Storage
Then, I would check if you have a storageClass by running kubectl get storageClass. Many times, cluters come with the local-path-provisioner storageClass. If you have that, I would try setting store.persistentVolume.storageClass to the name of the local path proversion storageClass you have installed on your system. If you don't have that or an alternative, I would consider installing it.
More Info
I'm not certain, but it's possible that it is using this helm chart to install the database. This section of the chart deals with the volume claim managment, so I would look here if you need to do more troubleshooting: https://github.com/cockroachdb/helm-charts/blob/master/cockroachdb/templates/statefulset.yaml#L376
I am running Airflow on Kubernetes from the stable helm chart. I'm running this in an AWS environment. This error exists with and without mounting any external volumes for log storage. I tried to set the configuration of the [logs] section to point to an EFS volume that I created. The PV gets mounted through a PVC but my containers are crashing (scheduler and web) due to the following error:
*** executing Airflow initdb...
Unable to load the config, contains a configuration error.
Traceback (most recent call last):
File "/usr/local/lib/python3.6/logging/config.py", line 565, in configure
handler = self.configure_handler(handlers[name])
File "/usr/local/lib/python3.6/logging/config.py", line 738, in configure_handler
result = factory(**kwargs)
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/utils/log/file_processor_handler.py", line 50, in __init__
os.makedirs(self._get_log_directory())
File "/usr/local/lib/python3.6/os.py", line 220, in makedirs
mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/opt/airflow/logs/scheduler/2020-08-20'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/airflow/.local/bin/airflow", line 25, in <module>
from airflow.configuration import conf
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/__init__.py", line 47, in <module>
settings.initialize()
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/settings.py", line 374, in initialize
LOGGING_CLASS_PATH = configure_logging()
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/logging_config.py", line 68, in configure_logging
raise e
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/logging_config.py", line 63, in configure_logging
dictConfig(logging_config)
File "/usr/local/lib/python3.6/logging/config.py", line 802, in dictConfig
dictConfigClass(config).configure()
File "/usr/local/lib/python3.6/logging/config.py", line 573, in configure
'%r: %s' % (name, e))
ValueError: Unable to configure handler 'processor': [Errno 13] Permission denied: '/opt/airflow/logs/scheduler/2020-08-20'
Persistent volume (created manually not from the stable/airflow chart)
apiVersion: v1
kind: PersistentVolume
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","kind":"PersistentVolume","metadata":{"annotations":{},"name":"efs-pv"},"spec":{"accessModes":["ReadWriteMany"],"capacity":{"storage":"5Gi"},"csi":{"driver":"efs.csi.aws.com","volumeHandle":"fs-e476a166"},"persistentVolumeReclaimPolicy":"Retain","storageClassName":"efs-sc","volumeMode":"Filesystem"}}
pv.kubernetes.io/bound-by-controller: "yes"
creationTimestamp: "2020-08-20T15:47:21Z"
finalizers:
- kubernetes.io/pv-protection
name: efs-pv
resourceVersion: "49476860"
selfLink: /api/v1/persistentvolumes/efs-pv
uid: 45d9f5ea-66c1-493e-a2f5-03e17f397747
spec:
accessModes:
- ReadWriteMany
capacity:
storage: 5Gi
claimRef:
apiVersion: v1
kind: PersistentVolumeClaim
name: efs-claim
namespace: airflow
resourceVersion: "49476857"
uid: 354103ea-f8a9-47f1-a7cf-8f449f9a2e8b
csi:
driver: efs.csi.aws.com
volumeHandle: fs-e476a166
persistentVolumeReclaimPolicy: Retain
storageClassName: efs-sc
volumeMode: Filesystem
status:
phase: Bound
Persistent Volume Claim for logs (created manually not from the stable/airflow chart):
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{},"name":"efs-claim","namespace":"airflow"},"spec":{"accessModes":["ReadWriteMany"],"resources":{"requests":{"storage":"5Gi"}},"storageClassName":"efs-sc"}}
pv.kubernetes.io/bind-completed: "yes"
pv.kubernetes.io/bound-by-controller: "yes"
creationTimestamp: "2020-08-20T15:47:46Z"
finalizers:
- kubernetes.io/pvc-protection
name: efs-claim
namespace: airflow
resourceVersion: "49476866"
selfLink: /api/v1/namespaces/airflow/persistentvolumeclaims/efs-claim
uid: 354103ea-f8a9-47f1-a7cf-8f449f9a2e8b
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 5Gi
storageClassName: efs-sc
volumeMode: Filesystem
volumeName: efs-pv
status:
accessModes:
- ReadWriteMany
capacity:
storage: 5Gi
phase: Bound
My values.yaml below:
airflow:
image:
repository: apache/airflow
tag: 1.10.10-python3.6
## values: Always or IfNotPresent
pullPolicy: IfNotPresent
pullSecret: ""
executor: KubernetesExecutor
fernetKey: "XXXXXXXXXHIVb8jK6lfmSAvx4mO6Arehnc="
config:
AIRFLOW__CORE__REMOTE_LOGGING: "True"
AIRFLOW__CORE__REMOTE_BASE_LOG_FOLDER: "s3://mybucket/airflow/logs"
AIRFLOW__CORE__REMOTE_LOG_CONN_ID: "MyS3Conn"
AIRFLOW__KUBERNETES__WORKER_CONTAINER_REPOSITORY: "apache/airflow"
AIRFLOW__KUBERNETES__WORKER_CONTAINER_TAG: "1.10.10-python3.6"
AIRFLOW__KUBERNETES__WORKER_CONTAINER_IMAGE_PULL_POLICY: "IfNotPresent"
AIRFLOW__KUBERNETES__WORKER_PODS_CREATION_BATCH_SIZE: "10"
AIRFLOW__KUBERNETES__LOGS_VOLUME_CLAIM: "efs-claim"
AIRFLOW__KUBERNETES__GIT_REPO: "git#github.com:org/myrepo.git"
AIRFLOW__KUBERNETES__GIT_BRANCH: "develop"
AIRFLOW__KUBERNETES__GIT_DAGS_FOLDER_MOUNT_POINT: "/opt/airflow/dags"
AIRFLOW__KUBERNETES__DAGS_VOLUME_SUBPATH: "repo/"
AIRFLOW__KUBERNETES__GIT_SSH_KEY_SECRET_NAME: "airflow-git-keys"
AIRFLOW__KUBERNETES__NAMESPACE: "airflow"
AIRFLOW__KUBERNETES__DELETE_WORKER_PODS: "True"
AIRFLOW__KUBERNETES__RUN_AS_USER: "50000"
AIRFLOW__CORE__LOAD_EXAMPLES: "False"
AIRFLOW__SCHEDULER__DAG_DIR_LIST_INTERVAL: "60"
AIRFLOW__KUBERNETES__WORKER_SERVICE_ACCOUNT_NAME: "airflow"
podAnnotations: {}
extraEnv: []
extraConfigmapMounts: []
extraContainers: []
extraPipPackages: []
extraVolumeMounts: []
extraVolumes: []
scheduler:
resources: {}
nodeSelector: {}
affinity: {}
tolerations: []
labels: {}
podLabels: {}
annotations: {}
podAnnotations: {}
podDisruptionBudget:
enabled: true
maxUnavailable: "100%"
minAvailable: ""
connections:
- id: MyS3Conn
type: aws
extra: |
{
"aws_access_key_id": "XXXXXXXXX",
"aws_secret_access_key": "XXXXXXXX",
"region_name":"us-west-1"
}
refreshConnections: true
variables: |
{}
pools: |
{}
numRuns: -1
initdb: true
preinitdb: false
initialStartupDelay: 0
extraInitContainers: []
web:
resources: {}
replicas: 1
nodeSelector: {}
affinity: {}
tolerations: []
labels: {}
podLabels: {}
annotations: {}
podAnnotations: {}
service:
annotations: {}
sessionAffinity: "None"
sessionAffinityConfig: {}
type: ClusterIP
externalPort: 8080
loadBalancerIP: ""
loadBalancerSourceRanges: []
nodePort:
http: ""
baseUrl: "http://localhost:8080"
serializeDAGs: false
extraPipPackages: []
initialStartupDelay: 0
minReadySeconds: 5
readinessProbe:
enabled: false
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 1
successThreshold: 1
failureThreshold: 3
livenessProbe:
enabled: true
scheme: HTTP
initialDelaySeconds: 300
periodSeconds: 30
timeoutSeconds: 3
successThreshold: 1
failureThreshold: 2
secretsDir: /var/airflow/secrets
secrets: []
secretsMap:
workers:
enabled: false
resources: {}
replicas: 1
nodeSelector: {}
affinity: {}
tolerations: []
labels: {}
podLabels: {}
annotations: {}
podAnnotations: {}
autoscaling:
enabled: false
maxReplicas: 2
metrics: []
initialStartupDelay: 0
celery:
instances: 1
gracefullTermination: false
gracefullTerminationPeriod: 600
terminationPeriod: 60
secretsDir: /var/airflow/secrets
secrets: []
secretsMap:
flower:
enabled: false
resources: {}
nodeSelector: {}
affinity: {}
tolerations: []
labels: {}
podLabels: {}
annotations: {}
podAnnotations: {}
basicAuthSecret: ""
basicAuthSecretKey: ""
urlPrefix: ""
service:
annotations: {}
type: ClusterIP
externalPort: 5555
loadBalancerIP: ""
loadBalancerSourceRanges: []
nodePort:
http: ""
initialStartupDelay: 0
extraConfigmapMounts: []
logs:
path: /opt/airflow/logs
persistence:
enabled: true
existingClaim: efs-claim
subPath: ""
storageClass: efs-sc
accessMode: ReadWriteMany
size: 1Gi
dags:
path: /opt/airflow/dags
doNotPickle: false
installRequirements: false
persistence:
enabled: false
existingClaim: ""
subPath: ""
storageClass: ""
accessMode: ReadOnlyMany
size: 1Gi
git:
url: git#github.com:org/myrepo.git
ref: develop
secret: airflow-git-keys
sshKeyscan: false
privateKeyName: id_rsa
repoHost: github.com
repoPort: 22
gitSync:
enabled: true
resources: {}
image:
repository: alpine/git
tag: latest
pullPolicy: Always
refreshTime: 60
initContainer:
enabled: false
resources: {}
image:
repository: alpine/git
tag: latest
pullPolicy: Always
mountPath: "/dags"
syncSubPath: ""
ingress:
enabled: false
web:
annotations: {}
path: ""
host: ""
livenessPath: ""
tls:
enabled: false
secretName: ""
precedingPaths: []
succeedingPaths: []
flower:
annotations: {}
path: ""
host: ""
livenessPath: ""
tls:
enabled: false
secretName: ""
rbac:
create: true
serviceAccount:
create: true
name: ""
annotations: {}
extraManifests: []
postgresql:
enabled: true
postgresqlDatabase: airflow
postgresqlUsername: postgres
postgresqlPassword: airflow
existingSecret: ""
existingSecretKey: "postgresql-password"
persistence:
enabled: true
storageClass: ""
accessModes:
- ReadWriteOnce
size: 5Gi
externalDatabase:
type: postgres
host: localhost
port: 5432
database: airflow
user: airflow
passwordSecret: ""
passwordSecretKey: "postgresql-password"
redis:
enabled: false
password: airflow
existingSecret: ""
existingSecretKey: "redis-password"
cluster:
enabled: false
slaveCount: 1
master:
resources: {}
persistence:
enabled: false
storageClass: ""
accessModes:
- ReadWriteOnce
size: 8Gi
slave:
resources: {}
persistence:
enabled: false
storageClass: ""
accessModes:
- ReadWriteOnce
size: 8Gi
externalRedis:
host: localhost
port: 6379
databaseNumber: 1
passwordSecret: ""
passwordSecretKey: "redis-password"
serviceMonitor:
enabled: false
selector:
prometheus: kube-prometheus
path: /admin/metrics
interval: "30s"
prometheusRule:
enabled: false
additionalLabels: {}
groups: []
I'm not really sure what to do here if anyone knows how to fix the permission error.
I have had this issue with the Google Cloud Plateform and the helm airflow 1.2.0 chart (which uses airflow 2).
What ended up working was:
extraInitContainers:
- name: fix-volume-logs-permissions
image: busybox
command: [ "sh", "-c", "chown -R 50000:0 /opt/airflow/logs/" ]
securityContext:
runAsUser: 0
volumeMounts:
- mountPath: /opt/airflow/logs/
name: logs
by tweaking based on Ajay's answer. Please note that:
the values 50000:0 are based on uid and gid setup in your values.yaml
you need to use extraInitContainers under scheduler and not worker
"logs" seems to be the volume name automatically used by the helm logging config when enabled
Security context was necessary for me or else the chown failed due to unprivileged rights
You can use extraInitContainers with scheduler to change the permission, something like this.
extraInitContainers:
- name: volume-logs
image: busybox
command: ["sh", "-c", "chown -R 50000:50000 /opt/airflow/logs/"]
volumeMounts:
- mountPath: /opt/airflow/logs/
name: logs-data
This will change permission of the mount point.
you can try modify workers.persistence.fixPermissions: true in values.yaml, it is OK.