Volume is not mounted even when pvc is created - kubernetes

I am using helm chart for the installation of the application, the volume is not mounted. I am doing something wrong but not sure what is it. I am new to devops
values.yaml
persistence:
enabled: true
existingClaim: grafana-persistent-storage
mountPath: "/dev/grafana/"
pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: grafana-block-pvc
spec:
accessModes:
- ReadWriteOnce
volumeMode: Block
storageClassName: grafana-persistent-storage
resources:
requests:
storage: 10Gi
storageClass.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: grafana-persistent-storage
provisioner: kubernetes.io/aws-ebs
parameters:
type: gp2
iopsPerGB: "10"
reclaimPolicy: Retain
allowVolumeExpansion: true
mountOptions:
- debug
volumeBindingMode: Immediate
PVC is creaed
kubectl --kubeconfig=<configfile> get pvc -n grafana
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
grafana-block-pvc Bound pvc-6dc39e0d-471e-11ea-b432-0a505018290a 10Gi RWO grafana-persistent-storage 10m
PV created too
pvc-6dc39e0d-471e-11ea-b432-0a505018290a 10Gi RWO Retain Bound grafana/grafana-block-pvc grafana-persistent-storage 10m
Kubectl describe pod - the description of the pod created.
Name: grafana1-v1-79fb988995-lnnl6
Namespace: grafana
Priority: 0
Node: ip-10-10-108-165.ap-southeast-1.compute.internal/10.10.108.165
Start Time: Tue, 04 Feb 2020 13:15:17 +0530
Labels: app.kubernetes.io/instance=grafana1
app.kubernetes.io/name=grafana1
pod-template-hash=79fb988995
Annotations: kubernetes.io/psp: eks.privileged
sidecar.istio.io/status:
{"version":"761ebc53976754715f22fcf548f05270fb4b8db07324894aebdb31fa81d960","initContainers":["istio-init"],"containers":["istio-proxy"]...
Status: Running
IP: 10.10.127.38
IPs: <none>
Controlled By: ReplicaSet/grafana1-v1-79fb988995
Init Containers:
istio-init:
Container ID: docker://a95db52c5b45c8147fb6c6d0ce4013bef6d495752dc820565188032bc36926
Image: docker.io/istio/proxy_init:1.2.5
Image ID: docker-pullable://istio/proxy_init#sha256:c9964a8c28b85cc631bbc90390eac238c90f82c8f929495d1e9f9a9135b724
Port: <none>
Host Port: <none>
Args:
-p
15001
-u
1337
-m
REDIRECT
-i
*
-x
-b
3000
-d
15020
State: Terminated
Reason: Completed
Exit Code: 0
Started: Tue, 04 Feb 2020 13:15:18 +0530
Finished: Tue, 04 Feb 2020 13:15:19 +0530
Ready: True
Restart Count: 0
Limits:
cpu: 100m
memory: 50Mi
Requests:
cpu: 10m
memory: 10Mi
Environment: <none>
Mounts: <none>
Containers:
grafana1:
Container ID: docker://92338e43bbf69a2c0919e81f5ae16948e6f7966353a3db52274a5a14902599
Image: grafana/grafana:latest
Image ID: docker-pullable://grafana/grafana#sha256:4319ca3e5592ee408f5842ce5b5955312549d89dc1572d2543f2f6d67ca619
Port: 3000/TCP
Host Port: 0/TCP
State: Running
Started: Tue, 04 Feb 2020 13:15:23 +0530
Ready: True
Restart Count: 0
Requests:
cpu: 100m
memory: 200Mi
Environment:
GF_SECURITY_ADMIN_PASSWORD: deskera#reports
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-99rfk (ro)
istio-proxy:
Container ID: docker://21b965ec954474b3bcb941a20782f99642f002bb0e9a212aed20e19838c2f0
Image: docker.io/istio/proxyv2:1.2.5
Image ID: docker-pullable://istio/proxyv2#sha256:8f210c3d09beb6b8658a55d9ac30e25549295834a44083ed67d652ad7453e4
Port: 15090/TCP
Host Port: 0/TCP
Args:
proxy
sidecar
--domain
$(POD_NAMESPACE).svc.cluster.local
--configPath
/etc/istio/proxy
--binaryPath
/usr/local/bin/envoy
--serviceCluster
istio-proxy.grafana
--drainDuration
45s
--parentShutdownDuration
1m0s
--discoveryAddress
istio-pilot.istio-system:15010
--zipkinAddress
zipkin.istio-system:9411
--dnsRefreshRate
300s
--connectTimeout
10s
--proxyAdminPort
15000
--concurrency
2
--controlPlaneAuthPolicy
NONE
--statusPort
15020
--applicationPorts
3000
State: Running
Started: Tue, 04 Feb 2020 13:15:23 +0530
Ready: True
Restart Count: 0
Limits:
cpu: 2
memory: 1Gi
Requests:
cpu: 100m
memory: 128Mi
Readiness: http-get http://:15020/healthz/ready delay=1s timeout=1s period=2s #success=1 #failure=30
Environment:
POD_NAME: grafana1-v1-79fb988995-lnnl6 (v1:metadata.name)
POD_NAMESPACE: grafana (v1:metadata.namespace)
INSTANCE_IP: (v1:status.podIP)
ISTIO_META_POD_NAME: grafana1-v1-79fb988995-lnnl6 (v1:metadata.name)
ISTIO_META_CONFIG_NAMESPACE: grafana (v1:metadata.namespace)
ISTIO_META_INTERCEPTION_MODE: REDIRECT
ISTIO_META_INCLUDE_INBOUND_PORTS: 3000
ISTIO_METAJSON_ANNOTATIONS: {"kubernetes.io/psp":"eks.privileged"}
ISTIO_METAJSON_LABELS: {"app.kubernetes.io/instance":"grafana1","app.kubernetes.io/name":"grafana1","pod-template-hash":"79fb988995"}
Mounts:
/etc/certs/ from istio-certs (ro)
/etc/istio/proxy from istio-envoy (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-99rfk (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
default-token-99rfk:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-99rfk
Optional: false
istio-envoy:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium: Memory
SizeLimit: <unset>
istio-certs:
Type: Secret (a volume populated by a Secret)
SecretName: istio.default
Optional: true
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 13m default-scheduler Successfully assigned grafana/grafana1-v1-79fb988995-lnnl6 to ip-10-10-108-165.ap-southeast-1.compute.internal
Normal Pulled 13m kubelet, ip-10-10-108-165.ap-southeast-1.compute.internal Container image "docker.io/istio/proxy_init:1.2.5" already present on machine
Normal Created 13m kubelet, ip-10-10-108-165.ap-southeast-1.compute.internal Created container istio-init
Normal Started 13m kubelet, ip-10-10-108-165.ap-southeast-1.compute.internal Started container istio-init
Normal Pulling 13m kubelet, ip-10-10-108-165.ap-southeast-1.compute.internal Pulling image "grafana/grafana:latest"
Normal Pulled 13m kubelet, ip-10-10-108-165.ap-southeast-1.compute.internal Successfully pulled image "grafana/grafana:latest"
Normal Created 13m kubelet, ip-10-10-108-165.ap-southeast-1.compute.internal Created container grafana1
Normal Started 13m kubelet, ip-10-10-108-165.ap-southeast-1.compute.internal Started container grafana1
Normal Pulled 13m kubelet, ip-10-10-108-165.ap-southeast-1.compute.internal Container image "docker.io/istio/proxyv2:1.2.5" already present on machine
Normal Created 13m kubelet, ip-10-10-108-165.ap-southeast-1.compute.internal Created container istio-proxy
Normal Started 13m kubelet, ip-10-10-108-165.ap-southeast-1.compute.internal Started container istio-proxy
Please refer the describe part of the pod. The volume is still not mounted even after changing the existing claim to pvc

persistence:
enabled: true
existingClaim: grafana-block-pvc
mountPath: "/dev/grafana/"
Claim name should be grafana-block-pvc rather than grafana-persistent-storage in your values.yaml

Related

k8s readiness probes working in GKE, not in Microk8s (on MacOS)

I have a Kong deployment.
apiVersion: apps/v1
kind: Deployment
metadata:
name: local-test-kong
labels:
app: local-test-kong
spec:
replicas: 1
selector:
matchLabels:
app: local-test-kong
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 0
type: RollingUpdate
template:
metadata:
labels:
app: local-test-kong
spec:
automountServiceAccountToken: false
containers:
- envFrom:
- configMapRef:
name: kong-env-vars
image: kong:2.6
imagePullPolicy: IfNotPresent
lifecycle:
preStop:
exec:
command:
- /bin/sh
- -c
- /bin/sleep 15 && kong quit
livenessProbe:
failureThreshold: 3
httpGet:
path: /status
port: status
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 5
name: proxy
ports:
- containerPort: 8000
name: proxy
protocol: TCP
- containerPort: 8100
name: status
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: /status
port: status
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 5
resources: # ToDo
limits:
cpu: 256m
memory: 256Mi
requests:
cpu: 256m
memory: 256Mi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /kong_prefix/
name: kong-prefix-dir
- mountPath: /tmp
name: tmp-dir
- mountPath: /kong_dbless/
name: kong-custom-dbless-config-volume
terminationGracePeriodSeconds: 30
volumes:
- name: kong-prefix-dir
- name: tmp-dir
- configMap:
defaultMode: 0555
name: kong-declarative
name: kong-custom-dbless-config-volume
I applied this YAML in GKE. Then i ran kubectl describe on its pod.
➜ kubectl get pods
NAME READY STATUS RESTARTS AGE
local-test-kong-678598ffc6-ll9s8 1/1 Running 0 25m
➜ kubectl describe pod/local-test-kong-678598ffc6-ll9s8
Name: local-test-kong-678598ffc6-ll9s8
Namespace: local-test-kong
Priority: 0
Node: gke-paas-cluster-prd-tf9-default-pool-e7cb502a-ggxl/10.128.64.95
Start Time: Wed, 23 Nov 2022 00:12:56 +0800
Labels: app=local-test-kong
pod-template-hash=678598ffc6
Annotations: kubectl.kubernetes.io/restartedAt: 2022-11-23T00:12:56+08:00
Status: Running
IP: 10.128.96.104
IPs:
IP: 10.128.96.104
Controlled By: ReplicaSet/local-test-kong-678598ffc6
Containers:
proxy:
Container ID: containerd://1bd392488cfe33dcc62f717b3b8831349e8cf573326add846c9c843c7bf15e2a
Image: kong:2.6
Image ID: docker.io/library/kong#sha256:62eb6d17133b007cbf5831b39197c669b8700c55283270395b876d1ecfd69a70
Ports: 8000/TCP, 8100/TCP
Host Ports: 0/TCP, 0/TCP
State: Running
Started: Wed, 23 Nov 2022 00:12:58 +0800
Ready: True
Restart Count: 0
Limits:
cpu: 256m
memory: 256Mi
Requests:
cpu: 256m
memory: 256Mi
Liveness: http-get http://:status/status delay=10s timeout=5s period=10s #success=1 #failure=3
Readiness: http-get http://:status/status delay=10s timeout=5s period=10s #success=1 #failure=3
Environment Variables from:
kong-env-vars ConfigMap Optional: false
Environment: <none>
Mounts:
/kong_dbless/ from kong-custom-dbless-config-volume (rw)
/kong_prefix/ from kong-prefix-dir (rw)
/tmp from tmp-dir (rw)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
kong-prefix-dir:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
tmp-dir:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
kong-custom-dbless-config-volume:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: kong-declarative
Optional: false
QoS Class: Guaranteed
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 25m default-scheduler Successfully assigned local-test-kong/local-test-kong-678598ffc6-ll9s8 to gke-paas-cluster-prd-tf9-default-pool-e7cb502a-ggxl
Normal Pulled 25m kubelet Container image "kong:2.6" already present on machine
Normal Created 25m kubelet Created container proxy
Normal Started 25m kubelet Started container proxy
➜
I applied the same YAML in my localhost's MicroK8S (on MacOS) and then I ran kubectl describe on its pod.
➜ kubectl get pods
NAME READY STATUS RESTARTS AGE
local-test-kong-54cfc585cb-7grj8 1/1 Running 0 86s
➜ kubectl describe pod/local-test-kong-54cfc585cb-7grj8
Name: local-test-kong-54cfc585cb-7grj8
Namespace: local-test-kong
Priority: 0
Node: microk8s-vm/192.168.64.5
Start Time: Wed, 23 Nov 2022 00:39:33 +0800
Labels: app=local-test-kong
pod-template-hash=54cfc585cb
Annotations: cni.projectcalico.org/podIP: 10.1.254.79/32
cni.projectcalico.org/podIPs: 10.1.254.79/32
kubectl.kubernetes.io/restartedAt: 2022-11-23T00:39:33+08:00
Status: Running
IP: 10.1.254.79
IPs:
IP: 10.1.254.79
Controlled By: ReplicaSet/local-test-kong-54cfc585cb
Containers:
proxy:
Container ID: containerd://d60d09ca8b77ee59c80ea060dcb651c3e346c3a5f0147b0d061790c52193d93d
Image: kong:2.6
Image ID: docker.io/library/kong#sha256:62eb6d17133b007cbf5831b39197c669b8700c55283270395b876d1ecfd69a70
Ports: 8000/TCP, 8100/TCP
Host Ports: 0/TCP, 0/TCP
State: Running
Started: Wed, 23 Nov 2022 00:39:37 +0800
Ready: True
Restart Count: 0
Limits:
cpu: 256m
memory: 256Mi
Requests:
cpu: 256m
memory: 256Mi
Liveness: http-get http://:status/status delay=10s timeout=5s period=10s #success=1 #failure=3
Readiness: http-get http://:status/status delay=10s timeout=5s period=10s #success=1 #failure=3
Environment Variables from:
kong-env-vars ConfigMap Optional: false
Environment: <none>
Mounts:
/kong_dbless/ from kong-custom-dbless-config-volume (rw)
/kong_prefix/ from kong-prefix-dir (rw)
/tmp from tmp-dir (rw)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
kong-prefix-dir:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
tmp-dir:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
kong-custom-dbless-config-volume:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: kong-declarative
Optional: false
QoS Class: Guaranteed
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 92s default-scheduler Successfully assigned local-test-kong/local-test-kong-54cfc585cb-7grj8 to microk8s-vm
Normal Pulled 90s kubelet Container image "kong:2.6" already present on machine
Normal Created 90s kubelet Created container proxy
Normal Started 89s kubelet Started container proxy
Warning Unhealthy 68s kubelet Readiness probe failed: Get "http://10.1.254.79:8100/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Warning Unhealthy 68s kubelet Liveness probe failed: Get "http://10.1.254.79:8100/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
➜
It's the exact same deployment YAML. However, the deployment created inside GKE cluster are running all fine with no complaints. But, the deployment created inside my localhost microk8s (on MacOS) is showing probe failures.
What could i be missing here while deploying to microk8s (on MacOS)?
Your readiness probes are failing on the local pod on port 8100. It looks like you have a firewall(s) rule preventing internal pod and/or pod to pod communication.
As per the docs:
You may need to configure your firewall to allow pod-to-pod and pod-to-internet communication:
sudo ufw allow in on cni0 && sudo ufw allow out on cni0
sudo ufw default allow routed

Pod restart very often after adding resources

Pod restart very often after adding resources.
Before that the resources were not added.
The pod doesn't restart at all, or maybe it only happens once or twice a day.
I'm not sure if resources affect health-check or not, so pod restarts very often.
apiVersion: apps/v1
kind: Deployment
metadata:
name: testservice-dpm
labels:
app: testservice-api
spec:
replicas: 1
selector:
matchLabels:
app: testservice-api
template:
metadata:
labels:
app: testservice-api
spec:
containers:
- name: testservice
image: testservice:v6.0.0
env:
- name: MSSQL_PORT
value: "1433"
resources:
limits:
cpu: 500m
memory: 1000Mi
requests:
cpu: 100m
memory: 100Mi
ports:
- containerPort: 80
name: test-p
volumeMounts:
- name: test-v
mountPath: /app/appsettings.json
subPath: appsettings.json
livenessProbe:
httpGet:
path: /api/ServiceHealth/CheckLiveness
port: 80
scheme: HTTP
initialDelaySeconds: 3
timeoutSeconds: 1
periodSeconds: 3
successThreshold: 1
failureThreshold: 1
readinessProbe:
httpGet:
path: /api/ServiceHealth/CheckReadiness
port: 80
scheme: HTTP
initialDelaySeconds: 3
timeoutSeconds: 1
periodSeconds: 3
successThreshold: 1
failureThreshold: 1
volumes:
- name: test-v
configMap:
name: testservice-config
Below are the results describe all of the testservice pods.
testservice-dpm-d7979cc69-rwxr4
(restart 7 times in 10 minutes and still Back-off restarting failed container now)
Name: testservice-dpm-d7979cc69-rwxr4
Namespace: testapi
Priority: 0
Node: workernode3/yyy.yyy.yy.yy
Start Time: Thu, 30 Dec 2021 12:48:50 +0700
Labels: app=testservice-api
pod-template-hash=d7979cc69
Annotations: kubectl.kubernetes.io/restartedAt: 2021-12-29T20:02:45Z
Status: Running
IP: xx.xxx.x.xxx
IPs:
IP: xx.xxx.x.xxx
Controlled By: ReplicaSet/testservice-dpm-d7979cc69
Containers:
testservice:
Container ID: docker://86a50f98b48bcf8bfa209a478c1127e998e36c1c7bcece71599f50feabb89834
Image: testservice:v6.0.0
Image ID: docker-pullable://testservice#sha256:57a3955d07febf4636eeda1bc6a18468aacf66e883d7f6d8d3fdcb5163724a84
Port: 80/TCP
Host Port: 0/TCP
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Completed
Exit Code: 0
Started: Thu, 30 Dec 2021 12:55:13 +0700
Finished: Thu, 30 Dec 2021 12:55:19 +0700
Ready: False
Restart Count: 7
Limits:
cpu: 500m
memory: 1000Mi
Requests:
cpu: 100m
memory: 100Mi
Liveness: http-get http://:80/api/ServiceHealth/CheckLiveness delay=3s timeout=1s period=3s #success=1 #failure=1
Readiness: http-get http://:80/api/ServiceHealth/CheckReadiness delay=3s timeout=1s period=3s #success=1 #failure=1
Environment:
MSSQL_PORT: 1433
Mounts:
/app/appsettings.json from authen-v (rw,path="appsettings.json")
/etc/localtime from tz-config (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-fd9bt (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
authen-v:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: testservice-config
Optional: false
tz-config:
Type: HostPath (bare host directory volume)
Path: /usr/share/zoneinfo/Asia/Bangkok
HostPathType: File
kube-api-access-fd9bt:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 11m default-scheduler Successfully assigned testapi/testservice-dpm-d7979cc69-rwxr4 to workernode3
Warning Unhealthy 11m (x2 over 11m) kubelet Readiness probe failed: Get "http://xx.xxx.x.xxx:80/api/ServiceHealth/CheckReadiness": dial tcp xx.xxx.x.xxx:80: connect: connection refused
Warning Unhealthy 11m (x3 over 11m) kubelet Readiness probe failed: Get "http://xx.xxx.x.xxx:80/api/ServiceHealth/CheckReadiness": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Warning Unhealthy 11m (x3 over 11m) kubelet Liveness probe failed: Get "http://xx.xxx.x.xxx:80/api/ServiceHealth/CheckLiveness": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Normal Killing 11m (x3 over 11m) kubelet Container testservice failed liveness probe, will be restarted
Normal Created 10m (x4 over 11m) kubelet Created container testservice
Normal Started 10m (x4 over 11m) kubelet Started container testservice
Normal Pulled 10m (x4 over 11m) kubelet Container image "testservice:v6.0.0" already present on machine
Warning BackOff 80s (x51 over 11m) kubelet Back-off restarting failed container
testservice-dpm-d7979cc69-7nq28
(restart 4 times in 10 minutes and running now)
Name: testservice-dpm-d7979cc69-7nq28
Namespace: testapi
Priority: 0
Node: workernode3/yyy.yyy.yy.yy
Start Time: Thu, 30 Dec 2021 12:47:37 +0700
Labels: app=testservice-api
pod-template-hash=d7979cc69
Annotations: kubectl.kubernetes.io/restartedAt: 2021-12-29T20:02:45Z
Status: Running
IP: xx.xxx.x.xxx
IPs:
IP: xx.xxx.x.xxx
Controlled By: ReplicaSet/testservice-dpm-d7979cc69
Containers:
testservice:
Container ID: docker://03739fc1694370abda202ba56928b46fb5f3ef7545f527c2dd73764e55f725cd
Image: testservice:v6.0.0
Image ID: docker-pullable://testservice#sha256:57a3955d07febf4636eeda1bc6a18468aacf66e883d7f6d8d3fdcb5163724a84
Port: 80/TCP
Host Port: 0/TCP
State: Running
Started: Thu, 30 Dec 2021 12:48:44 +0700
Last State: Terminated
Reason: Completed
Exit Code: 0
Started: Thu, 30 Dec 2021 12:48:10 +0700
Finished: Thu, 30 Dec 2021 12:48:14 +0700
Ready: True
Restart Count: 4
Limits:
cpu: 500m
memory: 1000Mi
Requests:
cpu: 100m
memory: 100Mi
Liveness: http-get http://:80/api/ServiceHealth/CheckLiveness delay=3s timeout=1s period=3s #success=1 #failure=1
Readiness: http-get http://:80/api/ServiceHealth/CheckReadiness delay=3s timeout=1s period=3s #success=1 #failure=1
Environment:
MSSQL_PORT: 1433
Mounts:
/app/appsettings.json from authen-v (rw,path="appsettings.json")
/etc/localtime from tz-config (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-slz4b (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
authen-v:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: testservice-config
Optional: false
tz-config:
Type: HostPath (bare host directory volume)
Path: /usr/share/zoneinfo/Asia/Bangkok
HostPathType: File
kube-api-access-slz4b:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 14m default-scheduler Successfully assigned testapi/testservice-dpm-d7979cc69-7nq28 to workernode3
Warning Unhealthy 14m (x2 over 14m) kubelet Readiness probe failed: Get "http://xx.xxx.x.xxx:80/api/ServiceHealth/CheckReadiness": dial tcp xx.xxx.x.xxx:80: connect: connection refused
Warning Unhealthy 14m (x3 over 14m) kubelet Readiness probe failed: Get "http://xx.xxx.x.xxx:80/api/ServiceHealth/CheckReadiness": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Warning Unhealthy 14m (x3 over 14m) kubelet Liveness probe failed: Get "http://xx.xxx.x.xxx:80/api/ServiceHealth/CheckLiveness": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Normal Killing 14m (x3 over 14m) kubelet Container testservice failed liveness probe, will be restarted
Warning BackOff 14m (x2 over 14m) kubelet Back-off restarting failed container
Normal Started 14m (x4 over 14m) kubelet Started container testservice
Normal Pulled 14m (x4 over 14m) kubelet Container image "testservice:v6.0.0" already present on machine
Normal Created 14m (x4 over 14m) kubelet Created container testservice
testservice-dpm-d7979cc69-z566c
(no restart in 10 minutes and running now)
Name: testservice-dpm-d7979cc69-z566c
Namespace: testapi
Priority: 0
Node: workernode3/yyy.yyy.yy.yy
Start Time: Thu, 30 Dec 2021 12:47:30 +0700
Labels: app=testservice-api
pod-template-hash=d7979cc69
Annotations: kubectl.kubernetes.io/restartedAt: 2021-12-29T20:02:45Z
Status: Running
IP: xx.xxx.x.xxx
IPs:
IP: xx.xxx.x.xxx
Controlled By: ReplicaSet/testservice-dpm-d7979cc69
Containers:
testservice:
Container ID: docker://19c3a672cd8453e1c5526454ffb0fbdec67fa5b17d6d8166fae38930319ed247
Image: testservice:v6.0.0
Image ID: docker-pullable://testservice#sha256:57a3955d07febf4636eeda1bc6a18468aacf66e883d7f6d8d3fdcb5163724a84
Port: 80/TCP
Host Port: 0/TCP
State: Running
Started: Thu, 30 Dec 2021 12:47:31 +0700
Ready: True
Restart Count: 0
Limits:
cpu: 500m
memory: 1000Mi
Requests:
cpu: 100m
memory: 100Mi
Liveness: http-get http://:80/api/ServiceHealth/CheckLiveness delay=3s timeout=1s period=3s #success=1 #failure=1
Readiness: http-get http://:80/api/ServiceHealth/CheckReadiness delay=3s timeout=1s period=3s #success=1 #failure=1
Environment:
MSSQL_PORT: 1433
Mounts:
/app/appsettings.json from authen-v (rw,path="appsettings.json")
/etc/localtime from tz-config (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-cpdnc (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
authen-v:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: testservice-config
Optional: false
tz-config:
Type: HostPath (bare host directory volume)
Path: /usr/share/zoneinfo/Asia/Bangkok
HostPathType: File
kube-api-access-cpdnc:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 16m default-scheduler Successfully assigned testapi/testservice-dpm-d7979cc69-z566c to workernode3
Normal Pulled 16m kubelet Container image "testservice:v6.0.0" already present on machine
Normal Created 16m kubelet Created container testservice
Normal Started 16m kubelet Started container testservice

How to use fdb-document-layer with kubernetes?

I have a fdb cluster using https://github.com/FoundationDB/fdb-kubernetes-operator and now trying to raise a pod with https://github.com/FoundationDB/fdb-document-layer
The result is CrashLoopBackOff of the pod
The description of the pod:
Name: fdb-doc-layer-84c4b84595-9rv8c
Namespace: default
Priority: 0
Node: faraday/5.188.158.233
Start Time: Sat, 21 Nov 2020 03:10:06 +0300
Labels: app=fdb-doc-layer
pod-template-hash=84c4b84595
Annotations: cni.projectcalico.org/podIP: 10.1.80.235/32
cni.projectcalico.org/podIPs: 10.1.80.235/32
Status: Running
IP: 10.1.80.235
IPs:
IP: 10.1.80.235
Controlled By: ReplicaSet/fdb-doc-layer-84c4b84595
Containers:
fdb-doc-layer:
Container ID: containerd://86f599ef8bd0684023a093f0e725fde02ac60f3899681053857e411b7c8c4b3b
Image: foundationdb/fdb-document-layer-build:latest
Image ID: docker.io/foundationdb/fdb-document-layer-build#sha256:5d1e84c5954141ce67be3fa28a428f572c3d8bbff1541ec8588fe82da600cb97
Port: 27017/TCP
Host Port: 0/TCP
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Completed
Exit Code: 0
Started: Sat, 21 Nov 2020 03:36:23 +0300
Finished: Sat, 21 Nov 2020 03:36:23 +0300
Ready: False
Restart Count: 10
Limits:
cpu: 200m
memory: 128Mi
Requests:
cpu: 200m
memory: 128Mi
Environment:
FDB_CLUSTER_FILE: /etc/foundationdb/fdb.cluster
Mounts:
/etc/foundationdb from config-volume (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-mf8pp (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
config-volume:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: sample-cluster-config
Optional: false
default-token-mf8pp:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-mf8pp
Optional: false
QoS Class: Guaranteed
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 28m default-scheduler Successfully assigned default/fdb-doc-layer-84c4b84595-9rv8c to faraday
Normal Pulled 28m kubelet Successfully pulled image "foundationdb/fdb-document-layer-build:latest" in 1.421941982s
Normal Pulled 28m kubelet Successfully pulled image "foundationdb/fdb-document-layer-build:latest" in 1.223503462s
Normal Pulled 27m kubelet Successfully pulled image "foundationdb/fdb-document-layer-build:latest" in 1.253710381s
Normal Pulled 27m kubelet Successfully pulled image "foundationdb/fdb-document-layer-build:latest" in 1.672481437s
Normal Created 27m (x4 over 28m) kubelet Created container fdb-doc-layer
Normal Started 27m (x4 over 28m) kubelet Started container fdb-doc-layer
Normal Pulling 26m (x5 over 28m) kubelet Pulling image "foundationdb/fdb-document-layer-build:latest"
Normal Pulled 26m kubelet Successfully pulled image "foundationdb/fdb-document-layer-build:latest" in 1.270867366s
Warning BackOff 3m2s (x116 over 28m) kubelet Back-off restarting failed container
My deployment.yaml:
apiVersion: apps/v1
kind: Deployment
metadata:
name: fdb-doc-layer
spec:
replicas: 1
selector:
matchLabels:
app: fdb-doc-layer
template:
metadata:
labels:
app: fdb-doc-layer
spec:
containers:
- name: fdb-doc-layer
image: foundationdb/fdb-document-layer-build:latest
env:
- name: FDB_CLUSTER_FILE
value: /etc/foundationdb/fdb.cluster
volumeMounts:
- name: config-volume
mountPath: /etc/foundationdb
resources:
limits:
memory: "128Mi"
cpu: "200m"
ports:
- containerPort: 27017
volumes:
- name: config-volume
configMap:
name: sample-cluster-config
How to make fdb-document-layer work with kubernetes?
There are two issues:
You are using the wrong image instead of foundationdb/fdb-document-layer-build:latest you should use foundationdb/fdb-document-layer:latest. The first image is only used to build the document layer.
The ConfigMap contains the fdb.cluster file under the key cluster-file so you need to remap this or adjust the env variable.
The following config works:
apiVersion: apps/v1
kind: Deployment
metadata:
name: fdb-doc-layer
spec:
replicas: 1
selector:
matchLabels:
app: fdb-doc-layer
template:
metadata:
labels:
app: fdb-doc-layer
spec:
containers:
- name: fdb-doc-layer
image: foundationdb/fdb-document-layer:latest
env:
- name: FDB_CLUSTER_FILE
value: /etc/foundationdb/fdb.cluster
volumeMounts:
- name: config-volume
mountPath: /etc/foundationdb
resources:
limits:
memory: "128Mi"
cpu: "200m"
ports:
- containerPort: 27017
volumes:
- name: config-volume
configMap:
name: sample-cluster-config
items:
- key: cluster-file
path: fdb.cluster

Kubernetes with helm on local persistent volume with docker for Windows

I tried to use helm on docker for windows on the local machine. When I used a storage class as local storage, persistent volume, and persistent volume claim without helm, it works fine. But when I used this setting with helm, CrashLoopBackOff happened.
localStrageClass.yaml
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: local-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: local-pv002
labels:
type: local
spec:
capacity:
storage: 1Gi
accessModes:
- ReadWriteMany
#storageClassName: hostpath
mountOptions:
- hard
persistentVolumeReclaimPolicy: Delete
storageClassName: local-storage
local:
path: /c/k/share/mysql
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
#- key: docker.io/hostname
- key: kubernetes.io/hostname
operator: In
values:
- docker-desktop
pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: local-mysql-claim
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 1Gi
storageClassName: local-storage
mysqlConf.yaml
persistence:
enabled: true
storageClass: local-storage
existingClaim: local-mysql-claim
accessMode: ReadWriteOnce
size: 1Gi
annotations: {}
$ helm install --name mysql stable/mysql -f mysqlConf.yaml
$ kubectl describe pod mysql
Containers:
mysql:
Container ID: docker://533e4569603b05fac83a0a701da97898b3190503618796678ac5db6340c4dce6
Image: mysql:5.7.14
Image ID: docker-pullable://mysql#sha256:c8f03238ca1783d25af320877f063a36dbfce0daa56a7b4955e6c6e05ab5c70b
Port: 3306/TCP
Host Port: 0/TCP
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Thu, 28 Mar 2019 13:24:25 +0900
Finished: Thu, 28 Mar 2019 13:24:25 +0900
Ready: False
Restart Count: 2
Requests:
cpu: 100m
memory: 256Mi
Liveness: exec [sh -c mysqladmin ping -u root -p${MYSQL_ROOT_PASSWORD}] delay=30s timeout=5s period=10s #success=1 #failure=3
Readiness: exec [sh -c mysqladmin ping -u root -p${MYSQL_ROOT_PASSWORD}] delay=5s timeout=1s period=10s #success=1 #failure=3
Environment:
MYSQL_ROOT_PASSWORD: <set to the key 'mysql-root-password' in secret 'mysql'> Optional: false
MYSQL_PASSWORD: <set to the key 'mysql-password' in secret 'mysql'> Optional: true
MYSQL_USER:
MYSQL_DATABASE:
Mounts:
/var/lib/mysql from data (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-dccpv (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
data:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: local-mysql-claim
ReadOnly: false
default-token-dccpv:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-dccpv
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 39s default-scheduler Successfully assigned default/mysql-698897ff79-n768k to docker-desktop
Normal Pulled 38s kubelet, docker-desktop Container image "busybox:1.29.3" already present on machine
Normal Created 38s kubelet, docker-desktop Created container
Normal Started 38s kubelet, docker-desktop Started container
Normal Pulled 18s (x3 over 37s) kubelet, docker-desktop Container image "mysql:5.7.14" already present on machine
Normal Created 17s (x3 over 37s) kubelet, docker-desktop Created container
Normal Started 17s (x3 over 37s) kubelet, docker-desktop Started container
Warning BackOff 13s (x5 over 35s) kubelet, docker-desktop Back-off restarting failed container
When storageClassName was hostpath or did not used the configuration file as
$ helm install --name mysql stable/mysql
it worked fine.
Please tell me how to fix this problem.
I think you are having a mismatch of accessModes between what you claim in PVC definition (ReadWriteOnce) and what your Storage Class offers (ReadWriteMany).
Please mind also that PersistentVolume(s) of HostPath type does not support ReadWriteMany mode (see spec here).
I would propose you to create PV similar to this one:
# Create PV of manual StorageClass
kind: PersistentVolume
apiVersion: v1
metadata:
name: task-pv-volume
labels:
type: local
spec:
storageClassName: manual
capacity:
storage: 10Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/C/Users/K8S/mysql"
and override default PVC storageClassName configuration during helm install like this:
helm install --name my-sql stable/mysql --set persistence.storageClass=manual

Istio allowing all outbound traffic

So putting everything in detail here for better clarification. My service consist of following attributes in dedicated namespace (Not using ServiceEntry)
Deployment (1 deployment)
Configmaps (1 configmap)
Service
VirtualService
GW
Istio is enabled in namespace and when I create / run deployment it create 2 pods as it should. Now as stated in issues subject I want to allow all outgoing traffic for deployment because my serives needs to connect with 2 service discovery server:
vault running on port 8200
spring config server running on http
download dependencies and communicate with other services (which are not part of vpc/ k8)
Using following deployment file will not open outgoing connections. Only thing works is simple https request on port 443 like when i run curl https://google.com its success but no response on curl http://google.com Also logs showing connection with vault is not establishing as well.
I have used almost all combinations in deployment but non of them seems to work. Anything I am missing or doing this in wrong way? would really appreciate contributions in this :)
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
labels:
app: my-application-service
name: my-application-service-deployment
namespace: temp-nampesapce
annotations:
traffic.sidecar.istio.io/excludeOutboundIPRanges: 0.0.0.0/0
spec:
replicas: 1
template:
metadata:
labels:
app: my-application-service-deployment
spec:
containers:
- envFrom:
- configMapRef:
name: my-application-service-env-variables
image: image.from.dockerhub:latest
name: my-application-service-pod
ports:
- containerPort: 8080
name: myappsvc
resources:
limits:
cpu: 700m
memory: 1.8Gi
requests:
cpu: 500m
memory: 1.7Gi
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: my-application-service-ingress
namespace: temp-namespace
spec:
hosts:
- my-application.mydomain.com
gateways:
- http-gateway
http:
- route:
- destination:
host: my-application-service
port:
number: 80
kind: Service
apiVersion: v1
metadata:
name: my-application-service
namespace: temp-namespace
spec:
selector:
app: api-my-application-service-deployment
ports:
- port: 80
targetPort: myappsvc
protocol: TCP
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: http-gateway
namespace: temp-namespace
spec:
selector:
istio: ingressgateway # use Istio default gateway implementation
servers:
- port:
number: 80
name: http
protocol: HTTP
hosts:
- "*.mydomain.com"
Namespace with istio enabled:
Name: temp-namespace
Labels: istio-injection=enabled
Annotations: <none>
Status: Active
No resource quota.
No resource limits.
Describe pods showing that istio and sidecare is working.
Name: my-application-service-deployment-fb897c6d6-9ztnx
Namespace: temp-namepsace
Node: ip-172-31-231-93.eu-west-1.compute.internal/172.31.231.93
Start Time: Sun, 21 Oct 2018 14:40:26 +0500
Labels: app=my-application-service-deployment
pod-template-hash=964537282
Annotations: sidecar.istio.io/status={"version":"2e0c897425ef3bd2729ec5f9aead7c0566c10ab326454e8e9e2b451404aee9a5","initContainers":["istio-init"],"containers":["istio-proxy"],"volumes":["istio-envoy","istio-certs...
Status: Running
IP: 100.115.0.4
Controlled By: ReplicaSet/my-application-service-deployment-fb897c6d6
Init Containers:
istio-init:
Container ID: docker://a47003a092ec7d3dc3b1d155bca0ec53f00e545ad1b70e1809ad812e6f9aad47
Image: docker.io/istio/proxy_init:1.0.2
Image ID: docker-pullable://istio/proxy_init#sha256:e16a0746f46cd45a9f63c27b9e09daff5432e33a2d80c8cc0956d7d63e2f9185
Port: <none>
Host Port: <none>
Args:
-p
15001
-u
1337
-m
REDIRECT
-i
*
-x
-b
8080,
-d
State: Terminated
Reason: Completed
Exit Code: 0
Started: Sun, 21 Oct 2018 14:40:26 +0500
Finished: Sun, 21 Oct 2018 14:40:26 +0500
Ready: True
Restart Count: 0
Environment: <none>
Mounts: <none>
Containers:
my-application-service-pod:
Container ID: docker://1a30a837f359d8790fb72e6b8fda040e121fe5f7b1f5ca47a5f3732810fd4f39
Image: image.from.dockerhub:latest
Image ID: docker-pullable://848569320300.dkr.ecr.eu-west-1.amazonaws.com/k8_api_env#sha256:98abee8d955cb981636fe7a81843312e6d364a6eabd0c3dd6b3ff66373a61359
Port: 8080/TCP
Host Port: 0/TCP
State: Running
Started: Sun, 21 Oct 2018 14:40:28 +0500
Ready: True
Restart Count: 0
Limits:
cpu: 700m
memory: 1932735283200m
Requests:
cpu: 500m
memory: 1825361100800m
Environment Variables from:
my-application-service-env-variables ConfigMap Optional: false
Environment:
vault.token: <set to the key 'vault_token' in secret 'vault.token'> Optional: false
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-rc8kc (ro)
istio-proxy:
Container ID: docker://3ae851e8ded8496893e5b70fc4f2671155af41c43e64814779935ea6354a8225
Image: docker.io/istio/proxyv2:1.0.2
Image ID: docker-pullable://istio/proxyv2#sha256:54e206530ba6ca9b3820254454e01b7592e9f986d27a5640b6c03704b3b68332
Port: <none>
Host Port: <none>
Args:
proxy
sidecar
--configPath
/etc/istio/proxy
--binaryPath
/usr/local/bin/envoy
--serviceCluster
my-application-service-deployment
--drainDuration
45s
--parentShutdownDuration
1m0s
--discoveryAddress
istio-pilot.istio-system:15007
--discoveryRefreshDelay
1s
--zipkinAddress
zipkin.istio-system:9411
--connectTimeout
10s
--statsdUdpAddress
istio-statsd-prom-bridge.istio-system:9125
--proxyAdminPort
15000
--controlPlaneAuthPolicy
NONE
State: Running
Started: Sun, 21 Oct 2018 14:40:28 +0500
Ready: True
Restart Count: 0
Requests:
cpu: 10m
Environment:
POD_NAME: my-application-service-deployment-fb897c6d6-9ztnx (v1:metadata.name)
POD_NAMESPACE: temp-namepsace (v1:metadata.namespace)
INSTANCE_IP: (v1:status.podIP)
ISTIO_META_POD_NAME: my-application-service-deployment-fb897c6d6-9ztnx (v1:metadata.name)
ISTIO_META_INTERCEPTION_MODE: REDIRECT
Mounts:
/etc/certs/ from istio-certs (ro)
/etc/istio/proxy from istio-envoy (rw)
Conditions:
Type Status
Initialized True
Ready True
PodScheduled True
Volumes:
default-token-rc8kc:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-rc8kc
Optional: false
istio-envoy:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium: Memory
istio-certs:
Type: Secret (a volume populated by a Secret)
SecretName: istio.default
Optional: true
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Started 3m kubelet, ip-172-31-231-93.eu-west-1.compute.internal Started container
Normal SuccessfulMountVolume 3m kubelet, ip-172-31-231-93.eu-west-1.compute.internal MountVolume.SetUp succeeded for volume "istio-certs"
Normal SuccessfulMountVolume 3m kubelet, ip-172-31-231-93.eu-west-1.compute.internal MountVolume.SetUp succeeded for volume "default-token-rc8kc"
Normal SuccessfulMountVolume 3m kubelet, ip-172-31-231-93.eu-west-1.compute.internal MountVolume.SetUp succeeded for volume "istio-envoy"
Normal Pulled 3m kubelet, ip-172-31-231-93.eu-west-1.compute.internal Container image "docker.io/istio/proxy_init:1.0.2" already present on machine
Normal Created 3m kubelet, ip-172-31-231-93.eu-west-1.compute.internal Created container
Normal Scheduled 3m default-scheduler Successfully assigned my-application-service-deployment-fb897c6d6-9ztnx to ip-172-42-231-93.eu-west-1.compute.internal
Normal Pulled 3m kubelet, ip-172-31-231-93.eu-west-1.compute.internal Container image "image.from.dockerhub:latest" already present on machine
Normal Created 3m kubelet, ip-172-31-231-93.eu-west-1.compute.internal Created container
Normal Started 3m kubelet, ip-172-31-231-93.eu-west-1.compute.internal Started container
Normal Pulled 3m kubelet, ip-172-31-231-93.eu-west-1.compute.internal Container image "docker.io/istio/proxyv2:1.0.2" already present on machine
Normal Created 3m kubelet, ip-172-31-231-93.eu-west-1.compute.internal Created container
Normal Started 3m kubelet, ip-172-31-231-93.eu-west-1.compute.internal Started container
Issue was that I tried to adding sidecar in deployment not in pod by adding in pod resolved the issue. Got help from here:
https://github.com/istio/istio/issues/9304