I have tried defining LoadBalancer in my service type and creating a deployment for it with 3 replicas:
kind: Service
apiVersion: v1
metadata:
name: springboot-postgres-k8s
labels:
name: springboot-postgres-k8s
spec:
ports: # ...
selector: # type: ...
type: LoadBalancer # <=====
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: springboot-postgres-k8s
spec:
selector:
matchLabels:
app: springboot-postgres-k8s
replicas: 3 # <=====
template: # ...
This starts up three pod and a load balancer which successfully load balances request among these three pods.
I want to know if k8s allows to manually autoscale the pods. That is if my cluster with 3 replicas + a load balancer is up and running, how can I manually increase replicas and still make the existing load balancer to scale across all 4 replicas (3 old and one newly created)?
Do I have to run (ref1 ref2):
kubectl scale --current-replicas=3 --replicas=4 deployment/springboot-postgres-k8s
Q1. Will above command notify existing load balancer of newly created pod?
or I have to run following (as specified in ref2)
kubectl scale --replicas=4 -f foo.yaml
Q2. Will above command notify existing load balancer of newly created pod?
Q3. What if my foo.yaml contains both service and deployment definition?
Yes its allows manual autoscale.
When you create a service in Kubernetes, k8s automatically creates kind: Endpoints type resource(s) for your pods with a matching label selector. This resource referenced by a Service to define which Pods the traffic can be sent to and periodically updated by the k8s when pods are created or deleted.
So regardless of the creation time of resources, k8s will handle the update and pods will be able to receive traffic from loadbalancer.
Related
I have a Kubernetes cluster with 3 nodes. On this cluster, there is one Kubernetes StatefulSet that is responsible for receiving messages, persisting the messages, and scheduling the received messages and an application processor of type Deployment that has 3 instances (replicas), Here the App Processor POD is receiving multiple kinds of messages from Message Processor POD using REST calls.
So finally I have
message-processor-0
and
app-processor-12345
app-processor-23456
app-processor-45567
Here my requirement is, I want to send particular types of messages to only one dedicated instance of the App Processor.
Since you are using a deployment, that specific POD has no reliable network identity. A POD's name and IP address will change whenever it is killed and recreated.
You have two options. One is to create a brand-new deployment and service for that particular pod, so you can communicate with that pod using the newly created service.
Otherwise StatefulSets are another choice, as they have a persistent network identity regardless of whether the POD is killed or restarted.
The StatefulSet pods get fixed ordered names, which will be in the format $(statefulset name)-$(ordinal).
For example if your yaml looks like this:
---
apiVersion: v1
kind: Service
metadata:
name: testservice
namespace: testnamespace
labels:
app: testapp
spec:
ports:
- port: 80
name: http
type: ClusterIP
selector:
app: myapp
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: testapp
namespace: testnamespace
spec:
serviceName: testservice
replicas: 5
selector:
matchLabels:
app: testapp
The names of Pods will be testapp-0, testapp-1 and so on.
Then you can use:
testapp-0.testservice.testnamespace.svc.cluster.local for accessing Pod 0 of testapp StatetefulSet
For more information refer to: StatefulSet Docs
What you could use here is a Statefulset for your app-processor.
Statefulsets have the capability of having unique names for each of its replicas, persisted on the restarts in the form of:
app-processor-0
app-processor-1
app-processor-2
So, they keep a sticky identity that you could use to send specific messages to one dedicated instance.
I have this Deployment object:
apiVersion: apps/v1
kind: Deployment
metadata:
name: deployment-webserver-nginx
annotations:
description: This is a demo deployment for nginx webserver
labels:
app: deployment-webserver-nginx
spec:
replicas: 3
selector:
matchLabels:
app: deployment-webserver-pods
template:
metadata:
labels:
app: deployment-webserver-pods
spec:
containers:
- name: nginx
image: nginx:alpine
ports:
- containerPort: 80
My understanding on this Deployment object is that any Pod with app:deployment-webserver-pods label will be selected. Of course, this Deployment object is creating 3 replicas, but I wanted to add one more Pod explicitly like this, so I created a Pod object and had its label as app:deployment-webserver-pods, below is its Pod definition:
apiVersion: v1
kind: Pod
metadata:
name: deployment-webserver-nginx-extra-pod
labels:
app: deployment-webserver-pods
spec:
containers:
- name: nginx-alpine-container-1
image: nginx:alpine
ports:
- containerPort: 81
My expectation was that continuously running Deployment Controller will pick this new Pod, and when I do kubectl get deploy then I will see 4 pods running. But that didn't happen.
I even tried to first create this pod with this label, and then created my Deployment and thought that maybe now this explicit Pod will be picked but still that didn't happen.
Doesn't Labels and Selectors work like this?
I know I can scale by deployment to 4 Replicas, but I am trying to understand how Pods / other Kubernetes objects are selected using Labels and Selectors.
From the official docs:
Note: You should not create other Pods whose labels match this
selector, either directly, by creating another Deployment, or by
creating another controller such as a ReplicaSet or a
ReplicationController. If you do so, the first Deployment thinks that
it created these other Pods. Kubernetes does not stop you from doing
this.
As described further in docs, it is not recommended to scale replicas of the deployments using the above approach.
Another important point to note from same section of docs:
If you have multiple controllers that have overlapping selectors, the
controllers will fight with each other and won't behave correctly.
My expectation was that continuously running Deployment Controller will pick this new Pod, and when I do kubectl get deploy then I will see 4 pods running. But that didn't happen.
The Deployment Controller does not work like that, it listen for Deployment-resources and "drive" them to desired state. That typically means, if any change in the template:-part, then a new ReplicaSet is created with the number of replicas. You cannot add a Pod to a Deployment in another way than changing replicas: - each instance is created from the same Pod-template and is identical.
Doesn't Labels and Selectors work like this?
... but I am trying to understand how Pods / other Kubernetes objects are selected using Labels and Selectors.
Yes, Labels and Selectors are used for many things in Kubernetes, but not for everything. When you create a Deployment with a label, and a Pod with the same label and finally a Service with a selector - then the traffic addressed to that Service will distribute traffic to your instances of your Deployment as well as to your extra Pod.
Example:
apiVersion: v1
kind: Service
metadata:
name: my-service
spec:
selector:
app: deployment-webserver-pods
ports:
- protocol: TCP
port: 80
targetPort: 8080
Labels and Selector are also useful for management when using e.g. kubectl. You can add labels for Teams or e.g. App, then you can select all Deployments or Pods belonging to that Team or App (e.g. if the app consist of App-deployment and a cache-deployment), e.g:
kubectl get pods -l team=myteam,app=customerservice
My expectation was that continuously running Deployment Controller
will pick this new Pod, and when I do kubectl get deploy then I will
see 4 pods running. But that didn't happen.
Kubernetes is a system that operates "Declaratively" and not "Imperatively" which means you write down the desired state of the application in the cluster typically through a YAML file, and these declared desired states define all of the pieces of your application.
If a cluster were to configured imperatively like the way you are expecting it to be, it would have been very difficult to understand and replicate how the cluster came to be in that state.
Just to add in the above explanations that if we are trying to manually create pod and manage then what is the purpose of having controllers in K8s.
My expectation was that continuously running Deployment Controller
will pick this new Pod, and when I do kubectl get deploy then I will
see 4 pods running. But that didn't happen.
As per your yaml replicas:3 was already set so deployment would not take a new pod as the 4th replica.
I was wondering if there is a way to create a service for a pod on a specific node.
For example:
Lets say I have a cluster with 4 worker nodes [compute-0 ... compute-3].
Nodes "compute-0" and "compute-1" have a label "app=firstApplication"
Nodes "compute-2" and "compute-3" have a different label "app=secondApplication"
I have a single Daemonset running across all 4 nodes.
I want to create 2 services, one for each couple of nodes.
Is this possible somehow?
Thanks!
EDIT
The reason for what we are trying to do is that we have an Openshift4.6 cluster, and for security reasons we have VXLAN port blocked off between 2 groups of nodes. When pods try to resolve DNS queries using the default dns service (172.30.0.10), sometimes they access the dns pods on the blocked off nodes.
No - this is not possible! Since services are referencing their Pods by Labels and all Pods in a DaemonSet are labelled the same, you can't do that. Of course, you could label your Pods after creation, but since this would be lost after recreation of the DaemonSet, I would not go down that route.
You could split your DaemonSet into parts and use Node Selectors or Affinity to control the distribution of Pods over Nodes.
If you specify a .spec.template.spec.nodeSelector, then the DaemonSet controller will create Pods on nodes which match that node selector. Likewise if you specify a .spec.template.spec.affinity, then DaemonSet controller will create Pods on nodes which match that node affinity.
That way, each DaemonSet can have its own Service.
You just need to patch existing pods. Add those label in your pods. May be you need to handle another operator. The job of the operator is to get the pods first. Then check if the desire label exist or not . If not exist patch the label of the pod. this is just like kubectl patch. With the help of kubeclient just update the label if the label is not exist in the pods. do some research about kubeclient. There are also an example sample-controller in kubernetes. Here is the link :
ref
if there are some extra label in pod just add them in selector.
---
kind: Service
apiVersion: v1
metadata:
name: first-svc
labels:
app: firstApplication
spec:
selector:
app: firstApplication
ports:
- name: http
port: 8080
targetPort: 8080
---
kind: Service
apiVersion: v1
metadata:
name: second-svc
labels:
app: secondApplication
spec:
selector:
app: secondApplication
ports:
- name: http
port: 8080
targetPort: 8080
---
I have a deployment with a defined number of replicas. I use readiness probe to communicate if my Pod is ready/ not ready to handle new connections – my Pods toggle between ready/ not ready state during their lifetime.
I want Kubernetes to scale the deployment up/ down to ensure that there is always the desired number of pods in a ready state.
Example:
If replicas is 4 and there are 4 Pods in ready state, then Kubernetes should keep the current replica count.
If replicas is 4 and there are 2 ready pods and 2 not ready pods, then Kubernetes should add 2 more pods.
How do I make Kubernetes scale my deployment based on the "ready"/ "not ready" status of my Pods?
I don't think this is possible. If pod is not ready, k8 will not make it ready as It is something which releated to your application.Even if it create new pod, how readiness will be guaranted. So you have to resolve the reasons behind non ready status and then k8. Only thing k8 does it keep them away from taking world load to avoid request failure
Ensuring you always have 4 pods running can be done by specifying the replicas property in your deployment definition:
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
labels:
app: nginx
spec:
replicas: 4 #here we define a requirement for 4 replicas
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.7.9
ports:
- containerPort: 80
Kubernetes will ensure that if any pods crash, replacement pods will be created so that a total of 4 are always available.
You cannot schedule deployments on unhealthy nodes in the cluster. The master api will only create pods on nodes which are healthy and meet the quota criteria to create any additional pods on the nodes which are schedulable.
Moreover, what you define is called an auto-heal concept of k8s which in basic terms will be taken care of.
I'm trying to understand why kubernetes docs recommend to specify service before deployment in one configuration file:
The resources will be created in the order they appear in the file. Therefore, it’s best to specify the service first, since that will ensure the scheduler can spread the pods associated with the service as they are created by the controller(s), such as Deployment.
Does it mean spread pods between kubernetes cluster nodes?
I tested with the following configuration where a deployment is located before a service and pods are distributed between nods without any issues.
apiVersion: apps/v1
kind: Deployment
metadata:
name: incorrect-order
namespace: test
spec:
selector:
matchLabels:
app: incorrect-order
replicas: 2
template:
metadata:
labels:
app: incorrect-order
spec:
containers:
- name: incorrect-order
image: nginx
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: incorrect-order
namespace: test
labels:
app: incorrect-order
spec:
type: NodePort
ports:
- port: 80
selector:
app: incorrect-order
Another explanation is that some environment variables with service URL will not be set for pods in this case. However it also works ok in case a configuration is inside one file like the example above.
Could you please explain why it is better to specify service before the deployment in case of one configuration file? Or may be it is some outdated recommendation.
If you use DNS as service discovery, the order of creation doesn't matter.
In case of Environment Vars (the second way K8S offers service discovery) the order matters, because once that vars are passed to the starting pod, they cannot be modified later if the service definition changes.
So if your service is deployed before you start your pod, the service envvars are injected inside the linked pod.
If you create a Pod/Deployment resource with labels, this resource will be exposed through a service once this last is created (with proper selector to indicate what resource to expose).
You are correct in that it effects the spread among the worker nodes.
Deployments without a Service will simply be scheduled onto the nodes with the least cpu/memory allocation. For instance, a brand new and empty node will get all new pods from a new deployment.
With a Deployment that also has a service the Scheduler tries to spread the pods between nodes, disregarding the cpu/memory load (within limits), to help the Service survive better.
It puzzles me that a Deployment on it's own doesn't cause a optimal spread but it doesn't, not yet at least.
This is the answer from the official documentation:
The resources will be created in the order they appear in the file.
Therefore, it's best to specify the service first, since that will
ensure the scheduler can spread the pods associated with the service
as they are created by the controller(s), such as Deployment.
Kubernetes Documentation/Concepts/Cluster/Administration/Managing Resources