Taints and Toleration - kubernetes

im practicing with kubernetes taints , i have tainted my node and than make a deploy like this:
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx
labels:
app: nginx
spec:
replicas: 3
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.15.4
ports:
- containerPort: 80
tolerations:
- key: "test"
operator: "Equal"
value: "blue"
effect: "NoSchedule"
kubectl describe nodes knode2 :
Name: knode2
Roles: <none>
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
kubernetes.io/arch=amd64
kubernetes.io/hostname=knode2
kubernetes.io/os=linux
testing=test
Annotations: kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock
node.alpha.kubernetes.io/ttl: 0
projectcalico.org/IPv4Address: **********
projectcalico.org/IPv4IPIPTunnelAddr: ********
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Tue, 27 Oct 2020 17:23:47 +0200
Taints: test=blue:NoSchedule
but when i deploy this yaml file the pods are not going only to that tainted node.
Why is that?

Taints and tolerations work together to ensure that pods are not scheduled onto inappropriate nodes. That's exactly opposite of what you intend to do.
You can constrain a Pod to only be able to run on particular Node(s), or to prefer to run on particular nodes using NodeSelector or NodeAffinity.
NodeSelector example
apiVersion: v1
kind: Pod
metadata:
name: nginx
labels:
env: test
spec:
containers:
- name: nginx
image: nginx
imagePullPolicy: IfNotPresent
nodeSelector:
disktype: ssd
Node affinity is conceptually similar to nodeSelector -- it allows you to constrain which nodes your pod is eligible to be scheduled on, based on labels on the node.

Related

Recreate job or delete old job on every helm install

I have created Kubernetes job and job got created, but it is failing on deployment to the Kubernetes cluster. When I try to redeploy it using Helm the job is not redeploying (deleting old job and recreating new one, unlike a microservice deployment).
How can I achieve this redeploying job without manually deleting it in Kubernetes? Could I tell it to recreate the container?
job.yaml contains:
apiVersion: batch/v1
kind: Job
metadata:
name: "{{ .Release.Name }}-init-job"
namespace: {{ .Release.Namespace }}
spec:
template:
metadata:
annotations:
linkerd.io/inject: disabled
"helm.sh/hook-delete-policy": before-hook-creation
"helm.sh/hook": pre-install,pre-upgrade,pre-delete
"helm.sh/hook-weight": "-5"
spec:
serviceAccountName: {{ .Release.Name }}-init-service-account
containers:
- name: app-installer
image: some image
command:
- /bin/bash
- -c
- echo Hello executing k8s init-container
securityContext:
readOnlyRootFilesystem: true
restartPolicy: OnFailure
Job is not getting redeployed
kubectl get jobs -n namespace
NAME COMPLETIONS DURATION AGE
test-init-job 0/1 13h 13h
kubectl describe job test-init-job -n test
Name: test-init-job
Namespace: test
Selector: controller-uid=86370470-0964-42d5-a9c1-00c8a462239f
Labels: app.kubernetes.io/managed-by=Helm
Annotations: meta.helm.sh/release-name: test
meta.helm.sh/release-namespace: test
Parallelism: 1
Completions: 1
Start Time: Fri, 14 Oct 2022 18:03:31 +0530
Pods Statuses: 0 Running / 0 Succeeded / 1 Failed
Pod Template:
Labels: controller-uid=86370470-0964-42d5-a9c1-00c8a462239f
job-name=test-init-job
Annotations: helm.sh/hook: pre-install,pre-upgrade
helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
helm.sh/hook-weight: -5
linkerd.io/inject: disabled
Service Account: test-init-service-account
Containers:
test-app-installer:
Image: artifactory/test-init-container:1.0.0
Port: <none>
Host Port: <none>
Environment:
test.baseUrl: baser
test.authType: basic
test.basic.username: jack
test.basic.password: password
Mounts:
/etc/test/config from test-installer-config-vol (ro)
Volumes:
test-installer-config-vol:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: test-installer-config-files
Optional: false
Events: <none>

StatefulSet: Longer rolling update lead Version mismatching

Application is deployed on K8s using StatefulSet because of stateful in nature. There is around 250+ pods are running and HPA has been implemented on it too that can scale upto 400 pods.
When new deployment occurs, it takes longer time (~ 10-15m) to update all pods in Rolling Update fashion.
Problem: End user get response from 2 version of pods until all pods are replaced with new revision.
I am googling for an architecture where overall deployment time can be reduced and getting the best possible solutions to use BLUE/GREEN strategy but it has bunch of impact with integrated services like monitoring, logging, telemetry etc because of 2 naming conventions.
Ideally I am looking for a solutions like maxSurge for Deployment in which firstly new pods are created and then traffic are shifted to it at a time but in case of StatefulSet, it won't support maxSurge with RollingUpdate strategy & controller will delete and recreate each Pod in the StatefulSet based on ordinal index from bigger to smaller.
The solution is to do a partitioning rolling update along with a canary deployment.
Let’s suppose we have the statefulset workload defined by the following yaml file:
apiVersion: v1
kind: Service
metadata:
name: nginx
labels:
app: nginx
version: "1.20"
spec:
ports:
- port: 80
name: web
clusterIP: None
selector:
app: nginx
version: "1.20"
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: web
spec:
selector:
matchLabels:
app: nginx # Label selector that determines which Pods belong to the StatefulSet
# Must match spec: template: metadata: labels
serviceName: "nginx"
replicas: 3
template:
metadata:
labels:
app: nginx # Pod template's label selector
version: "1.20"
spec:
terminationGracePeriodSeconds: 10
containers:
- name: nginx
image: nginx:1.20
ports:
- containerPort: 80
name: web
volumeMounts:
- name: www
mountPath: /usr/share/nginx/html
volumeClaimTemplates:
- metadata:
name: www
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 1Gi
You could patch the statefulset to create a partition, and change the image and version label for the remaining pods: (In this case, since there are only 3 pods, the last one will be the one that will change its image.)
$ kubectl patch statefulset web -p '{"spec":{"updateStrategy":{"type":"RollingUpdate","rollingUpdate":{"partition":2}}}}'
$ kubectl patch statefulset web --type='json' -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/image", "value":"nginx:1.21"}]'
$ kubectl patch statefulset web --type='json' -p='[{"op": "replace", "path": "/spec/template/metadata/labels/version", "value":"1.21"}]'
At this point, you have a pod with the new image and version label ready to use, but since the version label is different, the traffic is still going to the other two pods. If you change the version in the yaml file and apply the new configuration, the rollout will be transparent, since there is already a pod ready to migrate the traffic:
apiVersion: v1
kind: Service
metadata:
name: nginx
labels:
app: nginx
version: "1.21"
spec:
ports:
- port: 80
name: web
clusterIP: None
selector:
app: nginx
version: "1.21"
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: web
spec:
selector:
matchLabels:
app: nginx # Label selector that determines which Pods belong to the StatefulSet
# Must match spec: template: metadata: labels
serviceName: "nginx"
replicas: 3
template:
metadata:
labels:
app: nginx # Pod template's label selector
version: "1.21"
spec:
terminationGracePeriodSeconds: 10
containers:
- name: nginx
image: nginx:1.21
ports:
- containerPort: 80
name: web
volumeMounts:
- name: www
mountPath: /usr/share/nginx/html
volumeClaimTemplates:
- metadata:
name: www
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 1Gi
$ kubectl apply -f file-name.yaml
Once traffic is migrated to the pod containing the new image and version label, you should patch again the statefulset and remove the partition with the command kubectl patch statefulset web -p '{"spec":{"updateStrategy":{"type":"RollingUpdate","rollingUpdate":{"partition":0}}}}'
Note: You will need to be very careful with the size of the partition, since the remaining pods will handle the whole traffic for some time.

Kubernetes status hang on Pending status

I tried multiple times and searched a lot but could not figure out why my pod is still in Pending status.
I have a very simple docker-compose.yml file as below:
version: '3'
services:
nginx:
build: .
container_name: "something_cool"
ports:
- '80:80'
and converted it to Kubernetes syntax with kompose command, so it created the two deployment and service files as below.
Deployment:
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
kompose.cmd: kompose convert
kompose.version: 1.22.0 (955b78124)
creationTimestamp: null
labels:
io.kompose.service: nginx
name: nginx
spec:
replicas: 1
selector:
matchLabels:
io.kompose.service: nginx
strategy: {}
template:
metadata:
annotations:
kompose.cmd: kompose convert
kompose.version: 1.22.0 (955b78124)
creationTimestamp: null
labels:
io.kompose.service: nginx
spec:
containers:
- image: nginx
name: something-cool
ports:
- containerPort: 80
resources: {}
restartPolicy: Always
status: {}
Service:
apiVersion: v1
kind: Service
metadata:
annotations:
kompose.cmd: kompose convert
kompose.version: 1.22.0 (955b78124)
creationTimestamp: null
labels:
io.kompose.service: nginx
name: nginx
spec:
ports:
- name: "80"
port: 80
targetPort: 80
selector:
io.kompose.service: nginx
status:
loadBalancer: {}
Now that when I run kubectl apply -k ., I see service/nginx configured and deployment.apps/nginx configured, but kubectl get pods shows it in Pending status.
This is the result of grepping pod name in events:
6s Warning FailedScheduling pod/nginx-77546f7866-j5gmd 0/3 nodes are available: 1 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate, 2 node(s) had taint {node.kubernetes.io/unreachable: }, that the pod didn't tolerate.
4m4s Warning FailedScheduling pod/nginx-77546f7866-j5gmd 0/3 nodes are available: 1 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate, 2 node(s) had taint {node.kubernetes.io/unreachable: }, that the pod didn't tolerate.
5m6s Normal SuccessfulCreate replicaset/nginx-77546f7866 Created pod: nginx-77546f7866-j5gmd
If I'm correct, I see that I can ping and nslookup kubernetes.io fine, but I'm not sure why I'm getting this error.
I solve the issue first by resetting the Kubernetes by the below ways:
kubeadm reset
rm ~/.kube/config
kubeadm init --pod-network-cidr=10.244.0.0/16
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
kubectl apply -k .

istio side car is not created

we have istio installed without the side car enabled gloablly , and I want to enable it to specific service in a new namespace
I’ve added to my deployment the following:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: gow
labels:
app: gowspec:
replicas: 2
template:
metadata:
labels:
app: gow
tier: service
annotations:
sidecar.istio.io/inject: "true"
while running
get namespace -L istio-injection I don’t see anything enabled , everything is empty…
How can I verify that the side car is created ? I dont see anything new ...
You can use istioctl kube-inject to make that
kubectl create namespace asdd
istioctl kube-inject -f nginx.yaml | kubectl apply -f -
nginx.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
namespace: asdd
labels:
app: nginx
spec:
replicas: 2
selector:
matchLabels:
app: nginx
template:
metadata:
annotations:
sidecar.istio.io/inject: "True"
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.7.9
ports:
- containerPort: 80
Result:
nginx-deployment-55b6fb474b-77788 2/2 Running 0 5m36s
nginx-deployment-55b6fb474b-jrkqk 2/2 Running 0 5m36s
Let me know if You have any more questions.
You can describe your pod to see list of containers and one of those should be sidecar container. Look for something called istio-proxy.
kubectl describe pod pod name
It should look something like below
$ kubectl describe pod demo-red-pod-8b5df99cc-pgnl7
SNIPPET from the output:
Name: demo-red-pod-8b5df99cc-pgnl7
Namespace: default
.....
Labels: app=demo-red
pod-template-hash=8b5df99cc
version=version-red
Annotations: sidecar.istio.io/status={"version":"3c0b8d11844e85232bc77ad85365487638ee3134c91edda28def191c086dc23e","initContainers":["istio-init"],"containers":["istio-proxy"],"volumes":["istio-envoy","istio-certs...
Status: Running
IP: 10.32.0.6
Controlled By: ReplicaSet/demo-red-pod-8b5df99cc
Init Containers:
istio-init:
Container ID: docker://bef731eae1eb3b6c9d926cacb497bb39a7d9796db49cd14a63014fc1a177d95b
Image: docker.io/istio/proxy_init:1.0.2
Image ID: docker-pullable://docker.io/istio/proxy_init#sha256:e16a0746f46cd45a9f63c27b9e09daff5432e33a2d80c8cc0956d7d63e2f9185
.....
State: Terminated
Reason: Completed
.....
Ready: True
Containers:
demo-red:
Container ID: docker://8cd9957955ff7e534376eb6f28b56462099af6dfb8b9bc37aaf06e516175495e
Image: chugtum/blue-green-image:v3
Image ID: docker-pullable://docker.io/chugtum/blue-green-image#sha256:274756dbc215a6b2bd089c10de24fcece296f4c940067ac1a9b4aea67cf815db
State: Running
Started: Sun, 09 Dec 2018 18:12:31 -0800
Ready: True
istio-proxy:
Container ID: docker://ca5d690be8cd6557419cc19ec4e76163c14aed2336eaad7ebf17dd46ca188b4a
Image: docker.io/istio/proxyv2:1.0.2
Image ID: docker-pullable://docker.io/istio/proxyv2#sha256:54e206530ba6ca9b3820254454e01b7592e9f986d27a5640b6c03704b3b68332
Args:
proxy
sidecar
.....
State: Running
Started: Sun, 09 Dec 2018 18:12:31 -0800
Ready: True
.....
You need to have the admission webhook for automatic sidecar injection.
kubectl get mutatingwebhookconfiguration istio-sidecar-injector -o yaml | grep "namespaceSelector:" -A5
There may be many reasons for sidecar injection failures as described here
Here is a table which shows final injection status based on different scenarios.
Based on above table its mandatory to label the namespace with a label istio-injection=enabled
kubectl label namespace default istio-injection=enabled --overwrite

Pod naming in google cluster

I created a deployment like this:
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: scs-db-sink
spec:
selector:
matchLabels:
app: scs-db-sink
replicas: 1
template:
metadata:
labels:
app: scs-db-sink
spec:
nodeSelector:
cloud.google.com/gke-nodepool: service-pool
containers:
- name: scs-db-sink
image: 'IMAGE_NAME'
imagePullPolicy: Always
ports:
- containerPort: 1068
kubectl get pods shows me that the pod is running:
scs-db-sink-74c4b6cd6b-tchm9 1/1 Running 0 16m
Question:
How can I setup the pod name to be scs-db-sink-0 and increase to scs-db-sink-1 when I scale up?
Thanks
Deployments pods are named as <replicaset-name>-<random-suffix> where replicaset name is <deployment-name>-<random-suffix>. Here, replicaset is created automatically by deployment. So, you can't achieve your expected name with deployment.
However, you can use Statefulset in this case. Statefulset's pods are named as you specified. Check about Statefulset here.