I have hpa configured for one of my deployment in Kubernetes.
Is there any way to check if HPA scaling happened to the deployment and when it happened?
I don't have prometheus or any monitoring solutions deployed.
If you created HPA you can check current status using command
$ kubectl get hpa
You can also use "watch" flag to refresh view each 30 seconds
$ kubectl get hpa -w
To check if HPA worked you have to describe it
$ kubectl describe hpa <yourHpaName>
Information will be in Events: section.
Also your deployment will contain some information about scaling
$ kubectl describe deploy <yourDeploymentName>
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ScalingReplicaSet 11m deployment-controller Scaled up replica set php-apache-b5f58cc5f to 1
Normal ScalingReplicaSet 9m45s deployment-controller Scaled up replica set php-apache-b5f58cc5f to 4
Normal ScalingReplicaSet 9m30s deployment-controller Scaled up replica set php-apache-b5f58cc5f to 8
Normal ScalingReplicaSet 9m15s deployment-controller Scaled up replica set php-apache-b5f58cc5f to 10
Another way is use events
$ kubectl get events | grep HorizontalPodAutoscaler
5m20s Normal SuccessfulRescale HorizontalPodAutoscaler New size: 4; reason: cpu resource utilization (percentage of request) above target
5m5s Normal SuccessfulRescale HorizontalPodAutoscaler New size: 8; reason: cpu resource utilization (percentage of request) above target
4m50s Normal SuccessfulRescale HorizontalPodAutoscaler New size: 10; reason:
Related
I want to update k8s deployment image from 22.41.70 to 22.41.73，as follow:
NewReplicaSet: hiroir-deployment-5b9f574565 (3/3 replicas created)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ScalingReplicaSet 13m deployment-controller Scaled up replica set hiroir-deployment-7ff8845548 to 3
Normal ScalingReplicaSet 8m56s deployment-controller Scaled up replica set hiroir-deployment-5b9f574565 to 1
Normal ScalingReplicaSet 8m56s deployment-controller Scaled down replica set hiroir-deployment-7ff8845548 to 2
Normal ScalingReplicaSet 8m56s deployment-controller Scaled up replica set hiroir-deployment-5b9f574565 to 2
Normal ScalingReplicaSet 8m52s deployment-controller Scaled down replica set hiroir-deployment-7ff8845548 to 1
Normal ScalingReplicaSet 8m52s deployment-controller Scaled up replica set hiroir-deployment-5b9f574565 to 3
Normal ScalingReplicaSet 8m52s deployment-controller Scaled down replica set hiroir-deployment-7ff8845548 to 0
I want to know how to ensure Scaled down pod replica is success?
You can check using kubectl get deployment <name, eg. hiroir> --namespace <namespace if not default> -o wide. Look at the "AVAILABLE" column and check the count if it aligns to last scaled replicas count, "IMAGES" column for the image that you have updated.
We are using AKS version 1.19.11.
It is noticed that whenever a new rollout is in placed for our deployments or a new pod got created as part of the hpa settings or pod got restarted, We are getting high cpu usage alerts.
For example, -if a new pod got created as part of any of the above activities, will this take up more CPU than the allowed Threshold ? [ the “Maximum limit” of 1 core specified in the deployment spec and the apps are light weight and doesnt need thatmuch cpu anuyways ] ? its in turn makes sudden spike in the AzureMonitor for a short time and then it became normal.
Why the pods are taking more cpu during its startup or creation time?
if the pods are not using thatmuch cpu, what will be the reason for this repeating issues?
hpa settings as below
Name: myapp
Namespace: myapp
Labels: app.kubernetes.io/managed-by=Helm
Annotations: meta.helm.sh/release-name: myapp
meta.helm.sh/release-namespace: myapp
CreationTimestamp: Mon, 26 Apr 2021 07:02:32 +0000
Reference: Deployment/myapp
Metrics: ( current / target )
resource cpu on pods (as a percentage of request): 5% (17m) / 75%
Min replicas: 5
Max replicas: 12
Deployment pods: 1 current / 1 desired
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale True ReadyForNewScale recommended size matches current size
ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
ScalingLimited False DesiredWithinRange the desired count is within the acceptable range
ading the events when a new rollout placed.
as per the events captured from the “myapp” Namespace , there were new deployment rolled out for myapp as below.
During the new pods creation its showing more CPU spikes as we are getting alert from the Azuremonitor that its exceeds the threshold of 80%.[the “Maximum limit” of 1 core specified in the deployment spec]
30m Normal SuccessfulDelete replicaset/myapp-1a2b3c4d5e Deleted pod: myapp-1a2b3c4d5e-9fmrk
30m Normal SuccessfulDelete replicaset/myapp-1a2b3c4d5e Deleted pod: myapp-1a2b3c4d5e-hfr8w
29m Normal SuccessfulDelete replicaset/myapp-1a2b3c4d5e Deleted pod: myapp-1a2b3c4d5e-l2pnd
31m Normal ScalingReplicaSet deployment/myapp Scaled up replica set myapp-5ddc98fb69 to 1
30m Normal ScalingReplicaSet deployment/myapp Scaled down replica set myapp-1a2b3c4d5e to 2
30m Normal ScalingReplicaSet deployment/myapp Scaled up replica set myapp-5ddc98fb69 to 2
30m Normal ScalingReplicaSet deployment/myapp Scaled down replica set myapp-1a2b3c4d5e to 1
30m Normal ScalingReplicaSet deployment/myapp Scaled up replica set myapp-5ddc98fb69 to 3
29m Normal ScalingReplicaSet deployment/myapp Scaled down replica set myapp-1a2b3c4d5e to 0
Alert settings
Period Over the last 15 mins
Value 100.274747
Operator GreaterThan
Threshold 80
i am not sure what metrics you are looking at in AKS monitoring specifically as you have not mentioned it but it could be possible,
when you are deploying the POD or HPA scaling the replicas your AKS showing the total resource of all replicas.
During the deployment, it's possible at a certain stage all PODs are in the running phase and taking & consuming the resources.
Are you checking specific resources of one single POD and it's going
above the threshold ?
As you have mentioned application is lightweight however it is possible initially it taking resources to start the process, in that case, you might have to check resources using profiling.
I know about several ways to perform rolling update of deployment. But do either kubectl apply -f deployment.yaml or kubectl apply -k ... update deployment according to the rolling update policy of a new version of deployment or an old one?
Yes it will, with one note :
Note: A Deployment's rollout is triggered if and only if the
Deployment's Pod template (that is, .spec.template) is changed, for
example if the labels or container images of the template are updated.
Other updates, such as scaling the Deployment, do not trigger a
rollout.
Reference : https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#updating-a-deployment
For example, you can see the events section of a deployment update after updating the nginx image and running kubectl apply -f nginx-deploy.yml :
...
NewReplicaSet: nginx-deployment-559d658b74 (3/3 replicas created)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ScalingReplicaSet 112s deployment-controller Scaled up replica set nginx-deployment-66b6c48dd5 to 3
Normal ScalingReplicaSet 44s deployment-controller Scaled up replica set nginx-deployment-559d658b74 to 1
Normal ScalingReplicaSet 20s deployment-controller Scaled down replica set nginx-deployment-66b6c48dd5 to 2
Normal ScalingReplicaSet 20s deployment-controller Scaled up replica set nginx-deployment-559d658b74 to 2
Normal ScalingReplicaSet 19s deployment-controller Scaled down replica set nginx-deployment-66b6c48dd5 to 1
Normal ScalingReplicaSet 19s deployment-controller Scaled up replica set nginx-deployment-559d658b74 to 3
Normal ScalingReplicaSet 18s deployment-controller Scaled down replica set nginx-deployment-66b6c48dd5 to 0
$ kubectl get deploy
NAME READY UP-TO-DATE AVAILABLE AGE
nginx-deployment 3/3 3 3 114s
When running this command for resources ( deployment, ReplicaSet ...) other than Pod
$ kubectl describe deployment xxx-deployment
---- ------ ------
Events: <none>
I have deployed several resources, but I haven't seen the event yet except for Pod.
What type of event will occur if events occur in other resources?
Could you recommend any materials to refer to?
Good explanation what is event in Kubernetes you can find in Types of Kubernetes Events article. Author also mentioned about types of events.
Kubernetes events are a resource type in Kubernetes that are automatically created when other resources have state changes, errors, or other messages that should be broadcast to the system. While there is not a lot of documentation available for events, they are an invaluable resource when debugging issues in your Kubernetes cluster.
You can describe not only pod, deployment or replicaset but almost all resources in kubernetes.
Examples:
kubectl describe job pi -n test
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulCreate 12s job-controller Created pod: pi-5rgbz
kubectl describe node ubuntu
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning MissingClusterDNS 22h (x98 over 23h) kubelet, ubuntu-18 kubelet does not have ClusterDNS IP configured and cannot create Pod using "ClusterFirst" policy. Falling back to "Default" policy.
Normal Starting 22h kubelet, ubuntu-18 Starting kubelet.
Warning InvalidDiskCapacity 22h kubelet, ubuntu-18 invalid capacity 0 on image filesystem
Normal NodeHasSufficientMemory 22h kubelet, ubuntu-18 Node ubuntu-18 status is now: NodeHasSufficientMemory
Normal NodeHasSufficientPID 22h
To list all resources events you can use
$ kubectl get events --all-namespaces
$ kubectl get events --all-namespaces
NAMESPACE LAST SEEN TYPE REASON OBJECT MESSAGE
default 50m Normal Starting node/gke-cluster-1-default-pool-XXXXXXXXXXXXX Starting kubelet.
default 50m Normal NodeHasSufficientMemory node/gke-cluster-1-default-pool-XXXXXXXXXXXXX Node gke-cluster-1-default-pool-XXXXXXXXXXXXX status is now: NodeHasSufficientMemory
default 2m47s Normal SuccessfulCreate job/pi Created pod: pi-5rgbz
kube-system 50m Normal ScalingReplicaSet deployment/fluentd-gcp-scaler Scaled up replica set fluentd-gcp-scaler-6855f55bcc to 1
In Object column you resource type.
If you would like more detailed information you can use -o wide flag - $ kubectl get events --all-namespaces -o wide
$ kubectl get events -o wide
LAST SEEN TYPE REASON OBJECT SUBOBJECT SOURCE MESSAGE
FIRST SEEN COUNT NAME
20m Normal Scheduled pod/hello-world-86d6c6f84d-8qz9d default-scheduler Successfully assigned default/hello-world-86d
6c6f84d-8qz9d to ubuntu-18
Possibly root cause.
I wasn't able to create deployment without any event at the beginning I would guess that you have set --event-ttl which is described in Kube-apiserver docs.
--event-ttl duration Default: 1h0m0s
Amount of time to retain events.
It was also mentioned in Github thread.
In short, all events will disappear after 1 hour if you have this flag set.
To check if you have this flag set in kube-apiserver you can check this StackOverflow thread.
If this didn't help you please edit your question with informations like your configuration YAMLs, what version of K8s are you using, steps to reproduce etc.
Well yes deployment do have events. But keep that in mind events only available for around 1 hr.
you can also filter by labels with --labelsfor describe all resources
We are using Kubernetes 1.8 to deploy our software in a cloud provider. Frequently, when deploying a specific pod-template, the deployment fails to create a successful replicaset and no instance is created. I am not able to find a better description than kubectl describe deploy.
Type Status Reason
---- ------ ------
Available False MinimumReplicasUnavailable
Progressing False ProgressDeadlineExceeded
OldReplicaSets: <none>
NewReplicaSet: <none>
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ScalingReplicaSet 21m (x3 over 2d) deployment-controller Scaled up replica set cbase-d-6bbfbdb5dc to 1
Normal ScalingReplicaSet 19m (x3 over 2d) deployment-controller Scaled down replica set cbase-d-6bbfbdb5dc to 0
also you can check the status of the replicaet:
kubectl describe replicaset cbase-d-6bbfbdb5dc
hope you will find the conditions and the reason why the replicaset could not be scaled up
While this might not be always true but a likely reason could be the unavailability of resources. Try increasing the resources (cpu+memory) allocated to the cluster.
This was exactly the error I got and increasing allocated resources fixed the issue (on GKE).
I got a similar error like yours yesterday and finally figured out that I could get error message from the pod corresponds with the deployment by using command kubectl get pod YOUR_POD_NAME -o yaml. You can check the status and error message there.