Goal: Deploy a helm chart Terraform targeting a Azure kubernetes cluster. The chart has to be pulled from Azure Container registry.
Steps followed to push the helm chart to ACR:
helm registry login command:
echo $spPassword | helm registry login .azurecr.io --username --password-stdin
helm save chart to local
From within the directory that has the helm chart.yml, values.yaml and other dirs:
helm chart save . .azurecr.io/:
helm push chart to Azure container Repository
helm push .azurecr.io/:
I was able to pull the chart from registry, export in local again and could successfully install manually.
Following which proceeded with Terraform based deployment approach. Below is the code snippet used:
provider "azurerm" {
version = "~>2.0"
features {}
}
data "azurerm_kubernetes_cluster" "credentials" {
name = "k8stest"
resource_group_name = "azure-k8stest"
}
provider "helm" {
kubernetes {
host = data.azurerm_kubernetes_cluster.credentials.kube_config.0.host
client_certificate = base64decode(data.azurerm_kubernetes_cluster.credentials.kube_config.0.client_certificate)
client_key = base64decode(data.azurerm_kubernetes_cluster.credentials.kube_config.0.client_key)
cluster_ca_certificate = base64decode(data.azurerm_kubernetes_cluster.credentials.kube_config.0.cluster_ca_certificate)
}
}
resource "helm_release" "test-tf" {
name = "test-twinkle"
repository = "https://<myacrname>.azurecr.io/helm/v1/repo"
repository_username = <serviceprincipal appid>
repository_password = <serviceprincipal password>
chart = <chart name with which it was pushed to ACR>
version = <chart version with which it was pushed to ACR>
namespace = "test-dev"
create_namespace = "true"
}
Error:
Chart version not found in repository.
I've had the same issue and the documentation isn't helpful at all currently. I did find another answer on stack which somewhat sorts the issue out in a hacky sort of way by using a null_resource. I think the OCI for ACR might not be supported properly yet so you have to choose between using the deprecated API or helm3 which is currently experimental for ACR. I have not found another way to add the Repo URL :(
I am currently having issues updating the Vault server HA (high-availability) storage to use PostgreSQL upon Vault installation via Helm 3.
Things I have tried:
Setting the values needed for HA (high-availability) manually, using the --set= Helm flag, by running the following command:
helm install vault hashicorp/vault \
--set='server.ha.enabled=true' \
--set='server.ha.replicas=4' \
--set='server.ha.raft.config= |
ui = true
listener "tcp" {
address = "[::]:8200"
cluster_address = "[::]:8201"
}
storage "postgresql" {
connection_url = "postgres://<pg_user>:<pg_pw>#<pg_host>:5432/<pg_db>"
}
service_registration "kubernetes" {}'
This would be great if it worked, but the storageconfig.hcl was not updated on installation.
I have tried creating a Helm override config file, and replaced the storage section from raft to postgresql. As mentioned here: Vault on Kubernetes Deployment Guide | Vault - HashiCorp Learn
Tried editing the storageconfig.hcl running directly in the pod. I can delete the file, but I can not use vim to edit/replace with a config on my machine – plus, I think this is bad practice since it is not linked with the Helm installation.
Looking for general information about what I might be doing wrong, or maybe some other ideas of what I could try to get this working as intended.
I'm new to Terraform and Helm world! I need to set up Istio on the AWS EKS cluster. I was able to set up the EKS cluster using Terraform. I'm thinking of installing ISTIO on top of the EKS cluster using Terraform by writing terraform modules. However, I found that we can set up Istio on top of eks using the helm chart.
Can someone help me to answer my few queries:
Should I install Istio using Terraform? If yes, Is there any terraform module available or How can I write one?
Should I install Istio using Helm Chart? If yes, what are the pros and cons of it?
I need to write a pipeline to install Istio on EKS cluster. Should I use a combination of both terraform and Helm as provider?
Thank you very much for your time. Appreciate all your help!
To extend #Chris 3rd option of terraform + helm provider,
as for version 1.12.0+ of istio they officially have a working helm repo:
istio helm install
and that with terraform's helm provider
Terraform helm provider
allows an easy setup that is configured only by terraform:
provider "helm" {
kubernetes {
// enter the relevant authentication
}
}
locals {
istio_charts_url = "https://istio-release.storage.googleapis.com/charts"
}
resource "helm_release" "istio-base" {
repository = local.istio_charts_url
chart = "base"
name = "istio-base"
namespace = var.istio-namespace
version = "1.12.1"
create_namespace = true
}
resource "helm_release" "istiod" {
repository = local.istio_charts_url
chart = "istiod"
name = "istiod"
namespace = var.istio-namespace
create_namespace = true
version = "1.12.1"
depends_on = [helm_release.istio-base]
}
resource "kubernetes_namespace" "istio-ingress" {
metadata {
labels = {
istio-injection = "enabled"
}
name = "istio-ingress"
}
}
resource "helm_release" "istio-ingress" {
repository = local.istio_charts_url
chart = "gateway"
name = "istio-ingress"
namespace = kubernetes_namespace.istio-ingress-label.id
version = "1.12.1"
depends_on = [helm_release.istiod]
}
This is the last step that was missing to make this production ready
It is no longer needed to locally keep the helm charts with the null_resource
If you wish to override the default helm values it is nicely shown in here:
Artifact hub, choose the relevant chart and see the values
As #Matt Schuchard mentioned, this is a bit opinion based question, that's why I will answer that based on my understanding.
Question number 1.
To answer your question, Should I install Istio using Terraform?, yes, if you follow Devops practices then you should write everything in a code, so I would recommend to do that.
As per the second part of your question, If yes, Is there any Terraform module available, no, from what I see currently there is no Istio module for Terraform, there is only a helm one.
As for the last part of the first question, How can I write Terraform module? I would recommend to start with the Terraform documentation. There is also a tutorial for creating a module.
Question number 2.
To answer your question, Should I install Istio using Helm Chart?, depends on your use case, you can do it either with helm or istioctl/istio operator.
As for the following question, If yes, what are the pros and cons of it? I'm not sure if the current helm chart is production ready, according to Istio documentation, Providing the full configuration in an IstioOperator CR is considered an Istio best practice for production environments, so from what I understand, you should rather use operator than helm. Also worth to note that the helm chart was not used by several versions, if was broughts back to life in version 1.8.
Question number 3.
As per the last question, I need to write a pipeline to install Istio on EKS cluster. Should I use a combination of both terraform and Helm as provider?, depends, it could be either Terraform and Helm, but from what I see it's also possible to do that with an Terraform and Istio Operator, there is an example. So it's rather up to you to decide which path will you take.
I would also recommend to take a look at this reddit thread. You might find few useful comments from the prod environment here, about installing Istio with Terraform.
I have been researching this in the last months and want to add my findings to #Jakob's answer:
First, there is an answer to the pros/cons of the different installation method, so I will not say anything about that:
https://istio.io/latest/faq/setup/#install-method-selection
Basically all of them can be done with terraform in a certain way.
terraform + istioctl with terraform null_resource provider
This is basically the istioctl install -f <file> command. You can create a template file and to the istictl install command with the null_resource provider.
resource "local_file" "setup_istio_config" {
content = templatefile("${path.module}/istio-operator.tmpl", {
enableHoldAppUntilProxyStarts = var.hold_app_until_proxy_starts
})
filename = "istio-operator.yaml"
}
resource "null_resource" "install_istio" {
provisioner "local-exec" {
command = "istioctl install -f \"istio-operator.yaml\" --kubeconfig ../${var.kubeconfig}"
}
depends_on = [local_file.setup_istio_config]
}
Pros:
Very easy setup
Cons:
How to upgrade using istioctl upgrade -f <file has to be solved somehow
istioctl must be installed in different versions when handling multiple clusters with different istio versions
Right istioctl version must be choosen on setup
I guess you can solve the upgrade process somehow, but the hole process is not really "infrastructure as code" enough. I didn't look into it further, because it doesn't seam to be good practice.
terraform + istio operator with terraform null_resource provider and kubectl provider
Similar the istio operator setup initializes the operator pod and takes a istio-operator.yml to setup istio for you.
resource "null_resource" "init_operator" {
provisioner "local-exec" {
command = "istioctl operator init --kubeconfig ../${var.kubeconfig}"
}
}
resource "kubectl_manifest" "setup_istio" {
yaml_body = <<YAML
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
name: istio-setup
namespace: istio-system
spec:
profile: default
hub: gcr.io/istio-release
tag: 1.9.2
components:
ingressGateways:
- name: istio-ingressgateway
enabled: true
meshConfig:
defaultConfig:
holdApplicationUntilProxyStarts: ${var.hold_app_until_proxy_starts}"
YAML
depends_on = [null_resource.init_operator]
}
It would be a good idea to wait for some seconds between the init and applying the config.
Here is a good article about doing this with Azure's aks:
https://medium.com/#vipinagarwal18/install-istio-on-azure-kubernetes-cluster-using-terraform-214f6d3f611
Pros:
Easy to setup
Easy to upgrade istio using the kubectl provider
As long as helm is in alpha, this might be the best approach.
terraform + helm with terraform helm provider
Istio provides some charts for the different componentes, when downloading istioctl. Those can be used for installing it with helm.
resource "helm_release" "istio_base" {
name = "istio-base"
chart = "./manifests/charts/base"
namespace = "istio-system"
}
Cons:
Not ready for production
Bonus
istio manifest + helm
Some time ago I've read an article on how to use istio manifest from istioctl manifest generate in combination with helm to install and mange istio. This approach needs some custom code, but it could be done with terraform and the helm provider as well.
Please read: https://karlstoney.com/2021/03/04/ci-for-istio-mesh/index.html
Conclusion
Installing istio with terraform works but seams to be a bit dirty at the moment. Once the helm setup is stable, I guess this would be the best approach. And with the helm provider it can be composed with terraform creation of other resources. Terraform certainly misses an istio provider, but I don't think they will create one in the foreseeable future.
For all those who found #Benda's solution to the point. Here is the working template for the same. Since I faced a couple of issues with that template, I compiled it for my own use case. I hope its helpful.
provider "helm" {
kubernetes {
config_path = "~/.kube/config"
}
}
provider "kubernetes" {
config_path = "~/.kube/config"
}
locals {
istio_charts_url = "https://istio-release.storage.googleapis.com/charts"
}
resource "kubernetes_namespace" "istio_system" {
metadata {
name = "istio-system"
labels = {
istio-injection = "enabled"
}
}
}
resource "helm_release" "istio-base" {
repository = local.istio_charts_url
chart = "base"
name = "istio-base"
namespace = kubernetes_namespace.istio_system.metadata.0.name
version = ">= 1.12.1"
timeout = 120
cleanup_on_fail = true
force_update = false
}
resource "helm_release" "istiod" {
repository = local.istio_charts_url
chart = "istiod"
name = "istiod"
namespace = kubernetes_namespace.istio_system.metadata.0.name
version = ">= 1.12.1"
timeout = 120
cleanup_on_fail = true
force_update = false
set {
name = "meshConfig.accessLogFile"
value = "/dev/stdout"
}
depends_on = [helm_release.istio-base]
}
resource "helm_release" "istio-ingress" {
repository = local.istio_charts_url
chart = "gateway"
name = "istio-ingress"
namespace = kubernetes_namespace.istio_system.metadata.0.name
version = ">= 1.12.1"
timeout = 500
cleanup_on_fail = true
force_update = false
depends_on = [helm_release.istiod]
}
PS: Please make sure you enable ports 15017 and 15021 in the master firewall rule for the istio ingress pod to properly start.
I want to create a secret in several k8s clusters in the Google Kubernetes Engine using the Terraform.
I know that I can use "host", "token" and some else parameters in "kubernetes" provider, but I can describe these parameters only once, and I don’t know how to connect to another cluster during the file of terraform.
My question is how to create a secret (or do other operations) in multiple k8s cluster via Terraform. Maybe you know some tools on github or other tips for doing via single terraform file?
You can use alias for provider in terraform like described in documentation
So you can define multiple providers for multiple k8s clusters and then refer them by alias.
e.g.
provider "kubernetes" {
config_context_auth_info = "ops1"
config_context_cluster = "mycluster1"
alias = "cluster1"
}
provider "kubernetes" {
config_context_auth_info = "ops2"
config_context_cluster = "mycluster2"
alias = "cluster2"
}
resource "kubernetes_secret" "example" {
...
provider = kubernetes.cluster1
}
If you're using terraform submodules, the setup is a bit more involved. See this terraform github issue comment.
In my terraform config files I create a Kubernetes cluster on GKE and when created, set up a Kubernetes provider to access said cluster and perform various actions such as setting up namespaces.
The problem is that some new namespaces were created in the cluster without terraform and now my attempts to import these namespaces into my state seem fail due to inability to connect to the cluster, which I believe is due to the following (taken from Terraform's official documentation of the import command):
The only limitation Terraform has when reading the configuration files is that the import provider configurations must not depend on non-variable inputs. For example, a provider configuration cannot depend on a data source.
The command I used to import the namespaces is pretty straightforward:
terraform import kubernetes_namespace.my_new_namespace my_new_namespace
I also tried using the -provdier="" and -config="" but to no avail.
My Kubernetes provider configuration is this:
provider "kubernetes" {
version = "~> 1.8"
host = module.gke.endpoint
token = data.google_client_config.current.access_token
cluster_ca_certificate = base64decode(module.gke.cluster_ca_certificate)
}
An example for a namespace resource I am trying to import is this:
resource "kubernetes_namespace" "my_new_namespace" {
metadata {
name = "my_new_namespace"
}
}
The import command results in the following:
Error: Get http://localhost/api/v1/namespaces/my_new_namespace: dial tcp [::1]:80: connect: connection refused
It's obvious it's doomed to fail since it's trying to reach localhost instead of the actual cluster IP and configurations.
Is there any workaround for this use case?
Thanks in advance.
the issue lies with the dynamic data provider. The import statement doesn't have access to it.
For the process of importing, you have to hardcode the provider values.
Change this:
provider "kubernetes" {
version = "~> 1.8"
host = module.gke.endpoint
token = data.google_client_config.current.access_token
cluster_ca_certificate = base64decode(module.gke.cluster_ca_certificate)
}
to:
provider "kubernetes" {
version = "~> 1.8"
host = "https://<ip-of-cluster>"
token = "<token>"
cluster_ca_certificate = base64decode(<cert>)
load_config_file = false
}
The token can be retrieved from gcloud auth print-access-token.
The IP and cert can be retrieved by inspecting the created container resource using terraform state show module.gke.google_container_cluster.your_cluster_resource_name_here
For provider version 2+ you have to drop load_config_file.
Once in place, import and revert the changes on the provider.
(1) Create an entry in your kubeconfig file for your GKE cluster.
gcloud container clusters get-credentials cluster-name
see: https://cloud.google.com/kubernetes-engine/docs/how-to/cluster-access-for-kubectl#generate_kubeconfig_entry
(2) Point terraform Kubernetes provider to your kubeconfig:
provider "kubernetes" {
config_path = "~/.kube/config"
}