Is there a way to prevent a Pod from deploying onto Kubernetes if it does not have memory resource requests & limits set?
Yes, you can apply Limit Ranges. See e.g. Configure Minimum and Maximum CPU Constraints for a Namespace for an example for CPU resources, but it can be applied for e.g. memory and storage as well.
For this you could enable the Policy addon for AKS:
az aks enable-addons --addons azure-policy --name MyAKSCluster --resource-group MyResourceGroup
This installs a managed Gatekeeper instance to your cluster. With this enabled you can apply Azure build-in policies or apply your own Gatekeeper policies to the AKS cluster. Here is a list of built-in polices from Azure specially for Kubernetes.
Here is the built-in policy to enforce limits. Here you will find a sample ConstraintTemplate for your use case described above. As those templates are CRDs your need to activate those with a Constraint. You may need to tweak them to also enforce memory & cpu requests.
Another Policy tool is Kyverno. The downside is that it is not Azure manage so you have to to update it yourself and you have no built-in polices from Microsoft. Here are some examples policies:
Require Limits and Requests
Memory Requests Equal Limits
Hope that helped in addition to the LimitRange hint from Jonas :)
Related
I am trying to use VPA for autoscaling my deployed services. Due to limitation in resources in my cluster I set the min_replica option to 1. The workflow of VPA that have seen so far is that it first deletes the existing pod and then re-create the pod. This approach will cause a downtime to my services. What I want is that the VPA first create the new pod and then deletes the old pod, completely similar to the rolling updates for deployments. Is there an option or hack to reverse the flow to the desired order in my case?
This can be achieved by using python script or by using an IAC pipeline, you can get the metrics of the kubernetes cluster and whenever these metrics exceed a certain threshold, trigger this python code for creating new pod with the required resources and shutdown the old pod. Follow this github link for more info on python plugin for kubernetes.
Ansible can also be used for performing this operation. This can be achieved by triggering your ansible playbook whenever the threshold breaches a certain limit and you can specify the new sizes of the pods that need to be created. Follow this official ansible document for more information. However both these procedures involve manual analysis for selecting the desired pod size for scaling. So if you don’t want to use vertical scaling you can go for horizontal scaling.
Note: The information is gathered from official Ansible and github pages and the urls are referred to in the post.
In ECS with Fargate, we can manage service isolation via security group. However that is no longer the case with EKS on Fargate.
Is there a way where pods on the same cluster can be isolated from each other like a Network Policy? I know this is possible with kubernetes but it needs to be implemented by the network plugin. Tried to install the network provider listed here without success as it needs daemonset (limitation of eks fargate: Cannot run Daemonsets, Privileged pods, or pods that use HostNetwork or HostPort.)
This is something we are tracking in this roadmap item. There isn't a viable workaround for now. As you pointed out when using EC2 we'd suggest to use the Calico network policy engine but with Fargate there is no DaemonSet support and it can't be used.
Given the SG associated to a pod is defined at the cluster level, one way to try to mitigate this would be to spread like-pods across different clusters where the pod SG is configured for that specific type of workload BUT this will mean more work and higher control plane costs.
I'm using Istio at the moment combined with the cert-manager. Because I need to have multiple certificates I'm using SDS instead of the volume mount approach.
But the hardware requirements for this stuff are really high. For GKE it is recommended to use a node-pool of 4x n1-standard-2 machines. This sums up to 200$ per month just for Istio. The recommendation for EKS is 2x m5.large machines. So it is a little bit cheaper but still around 150$. What confuses me is, that Minikube "just" needs 4vCPUs and 16GB memory in total which is round about the half of the requirements for GKE and EKS.
You'll see the resource hungry components by looking at the istio-system namespace, I mean especially the limits. For me it is:
istio-telemetry > 1100m / 6800m (requested / limits)
istio-policys (I have 5 of them) > 110m / 2000m
My question is:
Did you manage to reduce the limits without facing issues in production?
What node-pool size / machine type are your running your Istio plane?
Did someone tried auto-scaling for this node-pool? Did it reduce the costs?
Kind regards from Berlin.
Managed Istio for GKE is offered by Google as a pre-configured bundle. 4x n1-standard-2 is recommended to provide enough resources for all Istio components being installed.
Downsizing a cluster below the recommended size does not make sense.
Installation of managed Istio onto a standard GKE cluster (3x n1-standard-1)
will fail due to lack of resources. Besides that you wouldn't have
free computing capacity for your workloads. Recommended cluster size
seems reasonable.
Apart from recommended hardware configuration (4x n1-standard-2),
managed Istio can be installed and running on a cluster with configuration
8x n1-standard-1.
Taking into account mentioned in the point ##1, autoscaling could be beneficial
mostly for volatile workloads, but won't help that much for saving resources
allocated for Istio.
If the managed Istio for GKE seemed too resource consuming, you could install original version of Istio and select an installation profile with the components you actually need, as described here:
Customizable Install with Helm
Pardon me for my limited knowledge of k8s. As per k8s best practices we need to run pods in non default namespace. few reasons for this approach is to.
create logical isolation and creating uat, sit,dev environment on
same k8s cluster
default namespace is ok when we are having less than
10 micro services running in same PODs.
do we have any other benefits in terms of security, performance and maintenance point of view?
I would say the best practice is to think about how you will use your cluster and take namespaces into account. So thinking about what you'll run in the cluster, how much resource you want to dedicate to it and who can do what. Namespaces can help with controlling all of these things.
In terms of what you run, it's important that kubernetes object names have to be unique within a namespace. So if you want to run two instances of the same app, then you either install them in different namespaces or distinguish the resource names - helm charts for example default to adding prefixes to ensure uniqueness.
Also role-based access control permissions can be set as namespace-specific and resource usage quotas can be applied to namespaces. So if you had adev namespace on the same cluster as UAT then you could ensure that permissions are more restricted on UAT and that it has more resource availability guaranteed for it.
For more on these points see https://dzone.com/articles/kubernetes-namespaces-explained and https://kubernetes.io/blog/2016/08/kubernetes-namespaces-use-cases-insights/
I am exploring about implementation of Kubernetes cluster and deployment into Kubernetes cluster using Jenkins via CI/CD pipeline. When exploring I found that we don't need to define the worker machine node where we need to deploy our pods. Kubernetes master will take care for where to deploy / free pod in worker machine for deployment. We only need to define how much memory need to that pod in definition.
Here my confusion is that, Already we assigned and configured Kubernetes cluster for deployment. That all nodes containing its own memory according to creation of AWS EC2 (since I am planning to use AWS Ec2 - Ubuntu 16.04 LTS).
So why we again need to define memory in pod ? Is that proper way of pod deployment ?
I am only started in CI/CD pipeline world.
Specifying memory and cpu in the pod specification is completely optional. Still there are a couple of aspects to specifying memory and CPU at pod level:
As explained here, if you don't specify CPU/memory - the pod/container can consume all resources on that node and potentially affect other pod/containers running on that node.
Each application should specify the memory and CPU they need for running the application. This information is used by Kubernetes during scheduling the pod on one of the nodes in the cluster where enough resources are available. This information ensures better scheduling decisions.
It enables the Horizontal Pod Autoscaler (HPA) to scale the pods when the resource consumption beyond a certain limit. The details are explained in this doc. Unless there is a memory/cpu limit specified, you can not calculate that the pod is running 80% of that metric and it should be scaled into two replicas.
You can also enable a certain default at namespace level and then only override for specific applications, details here