we have a scenario where we want to stop the cluster (worker nodes) in Night when it is not being used and start it again in the morning when people start using the application running on AWS EKS. Any suggestions will be helpful.
I think you can achieve it by changing the desired capacity of auto scaling group using aws cli. You can run it as a CRON job:
aws autoscaling update-auto-scaling-group --auto-scaling-group-name <my-auto-scaling-group> --desired-capacity 0 --min-size 0
Related
I am using Terraform to create infrastructure on AWS environment. Out of many services, we are also creating AWS EKS using terraform-aws-modules/eks/aws module. The EKS is primarily used for spinning dynamic containers to handle asynchronous job execution. Once a given task is completed the container releases resources and terminates.
What I have noticed is that, the dead containers lying on the EKS cluster forever. This is causing too many dead containers just sitting on EKS and consuming storage. I came across few blogs which mention that Kubernetes has garbage collection process, but none describes how it can be specified using Terraform or explicitly for AWS EKS.
Hence I am looking for a solution, which will help to specify garbage collection policy for dead containers on AWS EKS. If not achievable via Terraform, I am ok with using kubectl with AWS EKS.
These two kubelet flags will cause the node to clean up docker images when the filesystem reaches those percentages. https://kubernetes.io/docs/concepts/architecture/garbage-collection/#container-image-lifecycle
--image-gc-high-threshold="85"
--image-gc-low-threshold="80"
But you also probably want to set --maximum-dead-containers 1 so that running multiple (same) images doesn't leave dead containers around.
In EKS you can add these flags to the UserData section of your EC2 instance/Autoscaling group.
#!/bin/bash
set -o xtrace
/etc/eks/bootstrap.sh --apiserver-endpoint ..... --kubelet-extra-args '<here>'
Might be a strange one to ask, but found myself with:
An AWS Simple Queue that holds messages to process.
A deployment in a Kubernetes cluster on Google Cloud (GKE) that processes this queue.
I want to scale the deployment according to the queue size. A simple logic for example:
Queue size = 0 => deploy 3 pods
Queue size 0 > 10 > 1000 => deploy 20 pods
Queue size < 1000 => deploy 100 pods
Turns out that this isn't such a simple task, and I'm looking for ideas.
I tried to achieve this via the Horizontal pod autoscaler, but it looks like an impossible task.
My best idea is an AWS Lambda that monitors the queue (by messages or a cron schedule), and updates the Kubernetes deployment via API.
The easy part was monitoring the queue size and getting the desired scale for the deployment, but I'm not managing to physically control the deployment size via the AWS Lambda.
TL:DR, I would like to achieve kubectl functionality (scale deployment), but via an external lambda running node.js code, while authenticating to my google cloud platform, And it seems really tricky as well. There are a few client libraries, but none of them really documents how to authenticate and connect to my cluster.
I even thought about running the bash script from my deployment system - but running that through a lambda function using node.js 'exec' seems very very wrong.
Am I missing an easier way?
There's a project called Keda: https://keda.sh/docs/2.0/scalers/aws-sqs/. It supports horizontal scaling basing on a bunch of queue types. SQS is supported.
To securely access SQS/CloudWatch from a GKE one can use https://github.com/doitintl/gtoken which lets you assume AWS role from a GKE. Or in a simpler and less secure way - dedicated AWS user with periodic keys rotation. Also look at https://cloud.google.com/pubsub/docs/overview, perhaps you can replace your SQS to stay in one stack.
You can use WPA: https://github.com/practo/k8s-worker-pod-autoscaler to scale a GKE deployment based on SQS queue. The project scales based on combination of SQS metrics. https://medium.com/practo-engineering/launching-worker-pod-autoscaler-3f6079728e8b
I'm trying to deploy high available flink cluster on kubernetes. In the below examples worker nodes are replicated but we have only one master pod.
https://github.com/apache/flink-statefun
As far as I understand there are 2 approaches to make job manager HA.
https://ci.apache.org/projects/flink/flink-docs-stable/ops/jobmanager_high_availability.html
https://medium.com/hepsiburadatech/high-available-flink-cluster-on-kubernetes-setup-73b2baf9200e
In the first example we deploy another job manager to switch between them in case of failure
In the second example kubernetes redeploy the job manager pod in case of failure
So I have few questions
For both examples what happens to the running jobs when the active job manager fails?
Can the first scenario be applied on kubernetes?
For the second scenario in case of job manager failure flink UI will be unavailable until the pod recover but in the second first scenario it will be available am I right?
What is the pros/cons of the both scenarios?
There is one approach to make job manager HA, both of your link is using the JM HA using zookeeper cluster to make active/standby arhitecture of the JM.
When JobManager fails there is a "Failover" such as describe in apache flink documentation(first link), the standby JM become to be Active.
Ofcouse, kubernetes is just the deployment of the whole Flink cluster, you can still use the HA cluster mode using zk.
No, both will make the "failover" and a standby JM will become active.
You are not understand that kubernetes is only the deploy cluster of flink, Same as you can deploy it on phsical/virtual servers, than u can deploy it on kubernetes, but things like High Aviability will stay the same.
EDIT:
You can make 2 or more pods in kubernetes of JobManager and then it`ll be equal to the first solution.
We have a bunch of pods running in dev environment. The pods are auto-provisioned by an application on every business action. The problem is that across various namespaces they are accumulating and eating available resources in EKS.
Is there a way without jenkins/k8s jobs to simply put some parameter on the pod manifest to tell it to self destruct say in 24 hours?
Add to your pod.spec:
activeDeadlineSeconds: 86400
After deadline your Pod will be stopped for good with the status DeadlineExceeded
If I understood your situation properly, you would like to scale your cluster down in order to save resources.
Kubernetes is featured with the ability to autoscale your application in a cluster. Literally, it means that Kubernetes can start additional pods when the load is increasing and terminate excessive pods when the load is decreasing.
It is possible to downscale the application to zero pods, but, in this case, you will have a delay serving the first request while the pod is starting.
This functionality relies on performance metrics. From the practical side, it means that autoscaling doesn't happen instantly, because it takes some time to performance metrics reach the configured threshold.
The mentioned Kubernetes feature called HPA(horizontal pod autoscale) is described in this document.
In case you are running your cluster on GCP or GKE, you are able to go further and automatically start additional nodes for your cluster when you need more computing capacity and shut down nodes when they are not running application pods anymore.
More information about this functionality can be found following the link.
Last, but not least, you can use tool like Ansible to manage all your kubernetes assets (it can create/manage deployments via playbooks).
If you decide to give it a try, you might find this information useful:
Creating a Container cluster in GKE
70% cheaper Kubernetes cluster on AWS
How to build a Kubernetes Horizontal Pod Autoscaler using custom metrics
I have kubernetes cluster that contains two node pools. I have a task to automate resizing node pools to 0 nodes on weekends to save the money.
I know that I can stop the compute instances by standard schedule.
But I can't stop the instances that are members of instance pools. I can only resize the pool to 0. How can I do that by gcloud schedule?
Cloud scheduler won't allow you to resize the node pool. You can instead use Cloud scheduler along with Cloud Functions to call the container API to resize the node pool. There is an example on the Google public docs to do something like this for a compute instance, you'll have to convert the function call to use the container API instead.
Here are some possible solutions:
Use GKE to manage your cluster, so you can resizing-a-cluster or migration to
different size machine.
Manage your own kubernetes cluster, uses a Compute Engine instance group for the nodes in your cluster, you can actually update it without needing GKE's help
If you want automation, you can use Jenkins or Airflow to schedule resizing jobs.
Hope this can help you.