I am using terraform to provision gke clusters. Currently if we have to modify the node pool properties we do it manually (create new nodepool, cordon old nodepool, drain nodes).
Is there a way i can automate it using terraform itself? currently if i change any properties it destroys and recreates the entire node.
You can use the
lifecycle {
create_before_destroy = true
}
in terraform so it will create the Node pool first and then stop the old instance or node pool.
Read more about it : https://www.terraform.io/language/meta-arguments/lifecycle
Related
I'm trying to upgrade the VM size of my AKS cluster using this approach with Terraform. Basically I create a new nodepool with the required amount of nodes, then I cordon the old node to disallow scheduling of new pods. Next, I drain the old node to reschedule all the pods in the newly created node. Then, I proceed to upgrade the VM size.
The problem I am facing is that azurerm_kubernetes_cluster resource allow for the creation of the default_node_pool and another resource, azurerm_kuberentes_cluster_node_pool allows me to create new nodepools with extra nodes.
Everything works until I create the new nodepool, cordon and drain the old one. However when I change the default_nod_pool.vm_size and run terraform apply, it tells me that the whole resource has to be recreated, including the new nodepool I just created, because it's linked to the cluster id, which will be replaced.
How should I manage this upgrade from the documentation with Terraform if upgrading the default node pool always forces replacement even if a new nodepool is in place?
terraform version
terraform v1.1.7
on linux_amd64
+ provider registry.terraform.io/hashicorp/azurerm v2.82.0
+ provider registry.terraform.io/hashicorp/local v2.2.2
Lets assume I have an existing GKE cluster that contains all my applications. They were all deployed using different methods. Now I want to deploy some resources to that cluster using Terraform. The trouble here is that terraform doesn't see it in his state file so it can't interact with it. Another problem is that even if I get that cluster to my state file, terraform doesn't see all of the created resources in that cluster. This could lead to some conflicts e.g. I'm trying to deploy two resources with the same name. Is there a way to solve this problem or do I just have to deal with the reality of my existence and create a new cluster for every new project that I deploy with terraform?
You can use terraform import command to import your existing GKE cluster to terraform state. Prior to run it, you need to have the adequate terraform configuration for your cluster.
example of import command :
terraform import google_container_cluster.<TF_RESOURCE_NAME> projects/<PROJECT_ID>/locations/<YOUR-CLUSTER-ZONE>/clusters/<CLUSTER_NAME>
for a terraform configuration :
resource "google_container_cluster" "<TF_RESOURCE_NAME>" {
name = "<CLUSTER_NAME>"
location = "<YOUR-CLUSTER-ZONE>"
}
The CLUSTER_NAME is the name displayed in your GKE clusters list on Google Cloud Console.
Then you need also to import the cluster node pool(s) in the same way using terraform google_container_node_pool resource.
I successfully maintain a kubernetes cluster in digital ocean throught terraform. The core cluster configuration is the following:
resource "digitalocean_kubernetes_cluster" "cluster" {
name = var.name
region = var.region
version = var.k8s_version
vpc_uuid = digitalocean_vpc.network.id
node_pool {
name = "workers"
size = var.k8s_worker_size
node_count = var.k8s_worker_count
}
}
The problem is, I now need to increase the node size (stored in the variable k8s_worker_size).
If I simply change the variable to a new string, the terraform plan results in a full replace of the kubernetes cluster:
digitalocean_kubernetes_cluster.cluster must be replaced
This is not doable in our production environment.
The correct procedure to perform this operation inside digital ocean is to:
Create a new node pool, with the required size
Use kubectl drain to remove our pods from the 'old' nodes
Remove the previous node pool.
Of course, by doing this manually inside the digital ocean console, the terraform state is completely out-of-sync and is therefore unusable.
Is there a way to perform that operation through terraform?
As an alternative options, is it possible to "manually" update the terraform state in order to sync it with the real cluster state after I perform the migration manually?
Is there a way to perform that operation through terraform?
There might be some edge cases where there is a solution to this. Since I am not familiar with kubernetes inside DigitalOcean I can't share a specific solution.
As an alternative options, is it possible to "manually" update the terraform state in order to sync it with the real cluster state after I perform the migration manually?
Yes! Do as you proposed manually and then remove the out-of-sync cluster with
terraform state rm digitalocean_kubernetes_cluster.cluster
from the state. Please visit the corresponding documentation for state rm and update the address if your cluster is in a module etc. Then use
terraform import digitalocean_kubernetes_cluster.cluster <id of your cluster>
to reimport the cluster. Please consult the documentation for importing the cluster for the details. The documentations mentions something around tagging the default node pool.
I am using Cloud Composer (Apache Airflow) on Google Cloud. Some of our processes require more resources than what's available on Composer's default node pool, so I've created an additional node pool within our cluster. The resource-intensive DAG's use the KubernetesPodOperator and specifically target the special node pool through the affinity={ nodeAffinity...} attribute.
My issue is that since creating the new node pool, I've noticed that ALL of my workloads are being scheduled on this new pool. How can I keep my normal workloads running on the default pool, while reserving the new node pool for special use cases?
Here is an example of KubernetesPodOperator definition that targets the special pool. The regular KubernetesPodOperator don't have the affinity attribute filled out:
KubernetesPodOperator(namespace='default',
image="image_name",
image_pull_policy='Always',
name="example_name",
task_id="example_name",
get_logs=True,
affinity={
'nodeAffinity': {
'requiredDuringSchedulingIgnoredDuringExecution': {
'nodeSelectorTerms': [{
'matchExpressions': [{
'key': 'cloud.google.com/gke-nodepool',
'operator': 'In',
'values': ['datascience-pool']
}]
}]
}
}
},
is_delete_operator_pod=True,
dag=dag)
The KubernetesPodOperator does not have any default affinity preferences, so the scheduling decision for your normal workloads to have ended up in the new node pool were made by the Kubernetes scheduler. To avoid this, you will now have to set affinity on all instances of KubernetesPodOperator (which you can make somewhat less painful by using default_args and the apply_defaults Airflow decorator).
At least as of versions of Cloud Composer up to v1.8.3, the Composer system pods will always run in the node pool default-pool. Therefore, you can use this to ensure the pods run in the Composer node pool instead of a custom one.
I don't know if it is a work around but I have solved this issue by assigning affinity to all of the tasks. Tasks requiring high-cpu or high-memory are assigned to respective node pool and default tasks are assigned to default-pool. This resolves the issue, I have tested in many flows.
In my terraform infrastructure, I spin up several Kubernetes clusters based on parameters, then install some standard contents to those Kubernetes clusters using the kubernetes provider.
When I change the parameters and one of the clusters is no longer needed, terraform is unable to tear it down because the provider and resources are both in the module. I don't see an alternative, however, because I create the kubernetes cluster in that same module, and the kubernetes object are all per kubernetes cluster.
All solutions I can think of involve adding a bunch of boilerplate to my terraform config. Should I consider generating my terraform config from a script?
I made a git repo that shows exactly the problems I'm having:
https://github.com/bukzor/terraform-gke-k8s-demo
TL;DR
Two solutions:
Create two separate modules with Terraform
Use interpolations and depends_on between the code that creates your Kubernetes cluster and the kubernetes resources:
resource "kubernetes_service" "example" {
metadata {
name = "my-service"
}
depends_on = ["aws_vpc.kubernetes"]
}
resource "aws_vpc" "kubernetes" {
...
}
When destroying resources
You are encountering a dependency lifecycle issue
PS: I don't know the code you've used to create / provision your Kubernetes cluster but I guess it looks like this
Write code for the Kubernetes cluster (creates a VPC)
Apply it
Write code for provisionning Kubernetes (create an Service that creates an ELB)
Apply it
Try to destroy everything => Error
What is happenning is that by creating a LoadBalancer Service, Kubernetes will provision an ELB on AWS. But Terraform doesn't know that and there is no link between the ELB created and any other resources managed by Terraform.
So when terraform tries to destroy the resources in the code, it will try to destroy the VPC. But it can't because there is an ELB inside that VPC that terraform doesn't know about.
The first thing would be to make sure that Terraform "deprovision" the Kubernetes cluster and then destroy the cluster itself.
Two solutions here:
Use different modules so there is no dependency lifecycle. For example the first module could be k8s-infra and the other could be k8s-resources. The first one manages all the squeleton of Kubernetes and is apply first / destroy last. The second one manages what is inside the cluster and is apply last / destroy first.
Use the depends_on parameter to write the dependency lifecycle explicitly
When creating resources
You might also ran into a dependency issue when terraform apply cannot create resources even if nothing is applied yet. I'll give an other example with a postgres
Write code to create an RDS PostgreSQL server
Apply it with Terraform
Write code, in the same module, to provision that RDS instance with the postgres terraform provider
Apply it with Terraform
Destroy everything
Try to apply everything => ERROR
By debugging Terraform a bit I've learned that all the providers are initialized at the beggining of the plan / apply so if one has an invalid config (wrong API keys / unreachable endpoint) then Terraform will fail.
The solution here is to use the target parameter of a plan / apply command.
Terraform will only initialize providers that are related to the resources that are applied.
Apply the RDS code with the AWS provider: terraform apply -target=aws_db_instance
Apply everything terraform apply. Because the RDS instance is already reachable, the PostgreSQL provider can also initiate itself