I am trying to deploy Kubernetes resource(Secrets) in AWS EKS using Terraform. Here is my resource looks like
terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 4.34"
}
kubernetes = {
source = "hashicorp/kubernetes"
version = "2.15.0"
}
}
}
data "aws_eks_cluster" "this" {
name = "shared_eks01"
}
data "aws_eks_cluster_auth" "this" {
name = "shared_eks01"
}
provider "kubernetes" {
host = data.aws_eks_cluster.this.endpoint
cluster_ca_certificate = base64decode(data.aws_eks_cluster.this.certificate_authority.0.data)
token = data.aws_eks_cluster_auth.this.token
}
provider "aws" {
region = "us-west-2"
}
resource "kubernetes_secret" "spacelift" {
metadata {
name = "spacelift123"
namespace = "spacelift"
}
data = {
"token" = "123"
}
}
I am unable to deploy the resource and I am getting this below error
kubernetes_secret.spacelift: Creating...
╷
│ Error: Post "https://6FC8A63F36709AA...........gr7.us-west-2.eks.amazonaws.com/api/v1/namespaces/spacelift/secrets": dial tcp 127.0.0.1:443: connectex: No connection could be made because the target machine actively refused it.
│
│ with kubernetes_secret.spacelift,
│ on main.tf line 63, in resource "kubernetes_secret" "spacelift":
│ 63: resource "kubernetes_secret" "spacelift" {
Also tried Adding load_config_file = false in provider results in
An argument named "load_config_file" is not expected here.
Can you pls tell me what am I missing?
Related
I have build a kubernetes cluster on google cloud and I am now trying to use the kubernetes_secret resource to create a secret. Here is my configuration:
resource "google_service_account" "default" {
account_id = "gke-service-account"
display_name = "GKE Service Account"
}
resource "google_container_cluster" "cluster" {
name = "${var.cluster-name}-${terraform.workspace}"
location = var.region
initial_node_count = 1
project = var.project-id
remove_default_node_pool = true
}
resource "google_container_node_pool" "cluster_node_pool" {
name = "${var.cluster-name}-${terraform.workspace}-node-pool"
location = var.region
cluster = google_container_cluster.cluster.name
node_count = 1
node_config {
preemptible = true
machine_type = "e2-medium"
service_account = google_service_account.default.email
oauth_scopes = [
"https://www.googleapis.com/auth/cloud-platform"
]
}
}
provider "kubernetes" {
host = "https://${google_container_cluster.cluster.endpoint}"
client_certificate = base64decode(google_container_cluster.cluster.master_auth.0.client_certificate)
client_key = base64decode(google_container_cluster.cluster.master_auth.0.client_key)
cluster_ca_certificate = base64decode(google_container_cluster.cluster.master_auth.0.cluster_ca_certificate)
}
resource "kubernetes_secret" "cloudsql-credentials" {
metadata {
name = "database-credentials" # The name of the secret
}
data = {
connection-name = var.database-connection-name
username = var.database-user
password = var.database-password
}
type = "kubernetes.io/basic-auth"
However I get the following error when creating the kubernetes_secret resource:
╷
│ Error: secrets is forbidden: User "system:anonymous" cannot create resource "secrets" in API group "" in the namespace "default"
│
│ with module.kubernetes-cluster.kubernetes_secret.cloudsql-credentials,
│ on gke/main.tf line 58, in resource "kubernetes_secret" "cloudsql-credentials":
│ 58: resource "kubernetes_secret" "cloudsql-credentials" {
│
╵
What am I missing here? I really don't understand. In the documentation I have found the following that could maybe help:
Depending on whether you have a current context set this may require `config_context_auth_info` and/or `config_context_cluster` and/or `config_context`
But it is not clear at all how this should be set and there are no examples provided. Any help will be appreciated. Thank you.
well, I'm new in terraforming and also in Kubernetes, I faced an issue in deploying images after creating a Kubernetes cluster.
I have created a module that creates a Kubernetes cluster and provides output for the Kube config data.
now I'm using the code below but i need to run terraform apply 2 times because first time the local file is not created and terraform could not connect to kubernetes or helm or kubectl but if i run the command twice it works as expected.
any solution?
Note: i also applied the solution as on the comment section of the
code and that comment did now work either.
File : main.tf
module "deploy_lke" {
source = "./modules/linode/kubernetes"
token = var.token
k8s_version = var.k8s_version
label = var.label
region = var.region
tags = var.tags
instance_type = var.instance_type
number_of_instance = var.number_of_instance
min = var.min
max = var.max
}
module "deploy_image" {
source = "./modules/kubernetes"
kube_config_path = module.deploy_lke.kubeconfig
dockerconfigjson = file("./secret/docker-sec.json")
deploy_name = var.deploy_name
desire_replicas = var.desire_replicas
image_link = var.image_link
image_name = var.image_name
image_port = var.image_port
ip_type = var.ip_type
max_replicas_val = var.max_replicas_val
min_replicas_val = var.min_replicas_val
service_name = var.service_name
}
File : ./module/linode/kubernetes
terraform {
required_providers {
linode = {
source = "linode/linode"
version = "1.29.4"
}
kubernetes = {
source = "hashicorp/kubernetes"
version = ">= 2.0.0"
}
}
}
provider "linode" {
token = var.token
}
resource "linode_lke_cluster" "gaintplay-web-lke" {
k8s_version = var.k8s_version
label = var.label
region = var.region
tags = var.tags
pool {
type = var.instance_type
count = var.number_of_instance
autoscaler {
min = var.min
max = var.max
}
}
lifecycle {
ignore_changes = [
pool.0.count
]
}
}
output "kubeconfig" {
value = linode_lke_cluster.gaintplay-web-lke.kubeconfig
}
output "api_endpoints" {
value = linode_lke_cluster.gaintplay-web-lke.api_endpoints
}
File : ./module/kubernetes
terraform {
required_providers {
kubernetes = {
source = "hashicorp/kubernetes"
version = ">= 2.0.0"
}
}
}
resource "local_file" "kube_config_file" {
content = var.kube_config_path
filename = "${path.module}/config"
}
provider "kubernetes" {
config_path = var.kube_config_path
}
provider "helm" {
kubernetes {
config_path = var.kube_config_path
}
}
resource "kubernetes_secret" "docker_secret" {
metadata {
name = "docker-cfg"
}
data = {
".dockerconfigjson" = var.dockerconfigjson
}
type = "kubernetes.io/dockerconfigjson"
}
resource "kubernetes_deployment" "beta" {
depends_on = [
kubernetes_secret.docker_secret
]
metadata {
name = var.deploy_name
namespace = "default"
}
spec {
replicas = var.desire_replicas
selector {
match_labels = {
app = var.deploy_name
}
}
template {
metadata {
labels = {
app = var.deploy_name
}
}
spec {
image_pull_secrets {
name = kubernetes_secret.docker_secret.metadata[0].name
}
container {
image_pull_policy = "Always"
image = var.image_link
name = var.image_name
port {
container_port = var.image_port
}
}
}
}
}
}
# provider "kubernetes" {
# host = "${yamldecode(var.kube_config_path).clusters.0.cluster.server}"
# client_certificate = "${base64decode(yamldecode(var.kube_config_path).users.0.user.client-certificate-data)}"
# client_key = "${base64decode(yamldecode(var.kube_config_path).users.0.user.client-key-data)}"
# cluster_ca_certificate = "${base64decode(yamldecode(var.kube_config_path).clusters.0.cluster.certificate-authority-data)}"
# }
# provider "helm" {
# kubernetes {
# host = "${yamldecode(var.kube_config_path).clusters.0.cluster.server}"
# client_certificate = "${base64decode(yamldecode(var.kube_config_path).users.0.user.client-certificate-data)}"
# client_key = "${base64decode(yamldecode(var.kube_config_path).users.0.user.client-key-data)}"
# cluster_ca_certificate = "${base64decode(yamldecode(var.kube_config_path).clusters.0.cluster.certificate-authority-data)}"
# }
# }
If i use the command as it is i got this error in terraform plan that the file is not found and i need to run it twice.
Invalid attribute in provider configuration
with module.deploy_image.provider["registry.terraform.io/hashicorp/kubernetes"],
on modules/kubernetes/main.tf line 13, in provider "kubernetes":
13: provider "kubernetes" {
'config_path' refers to an invalid path: "modules/kubernetes/config": stat modules/kubernetes/config: no such file or directory
and
If I use commented code i get error like this:
│ Error: Unsupported attribute
│
│ on main.tf line 35, in provider "kubernetes":
│ 35: host = "${yamldecode(linode_lke_cluster.gaintplay-web-lke.kubeconfig).clusters.0.cluster.server}"
│
│ Can't access attributes on a primitive-typed value (string).
╵
╷
│ Error: Unsupported attribute
│
│ on main.tf line 36, in provider "kubernetes":
│ 36: client_certificate = "${base64decode(yamldecode(linode_lke_cluster.gaintplay-web-lke.kubeconfig).users.0.user.client-certificate-data)}"
│
│ Can't access attributes on a primitive-typed value (string).
Terraform version
terraform {
required_providers {
azurerm = {
source = "hashicorp/azurerm"
version = "=3.16.0"
}
kubernetes = {
source = "hashicorp/kubernetes"
version = "=2.11.0"
}
helm = {
source = "hashicorp/helm"
version = "=2.6.0"
}
}
required_version = "=1.2.6"
}
Terraform Code
resource "azurerm_kubernetes_cluster" "my_cluster" {
name = local.cluster_name
location = azurerm_resource_group.rg.location
resource_group_name = azurerm_resource_group.rg.name
dns_prefix = local.dns_prefix
node_resource_group = local.resource_group_node_name
kubernetes_version = "1.24.3"
automatic_channel_upgrade = "patch"
sku_tier = var.sku_tier
default_node_pool {
name = "default"
type = "VirtualMachineScaleSets"
vm_size = var.default_pool_vm_size
enable_auto_scaling = true
max_count = var.default_pool_max_count
min_count = var.default_pool_min_count
os_disk_type = "Ephemeral"
os_disk_size_gb = var.default_pool_os_disk_size_gb
}
identity {
type = "SystemAssigned"
}
network_profile {
network_plugin = "kubenet"
}
}
provider "helm" {
kubernetes {
host = azurerm_kubernetes_cluster.my_cluster.kube_admin_config.0.host
client_certificate = base64decode(azurerm_kubernetes_cluster.my_cluster.kube_admin_config.0.client_certificate)
client_key = base64decode(azurerm_kubernetes_cluster.my_cluster.kube_admin_config.0.client_key)
cluster_ca_certificate = base64decode(azurerm_kubernetes_cluster.my_cluster.kube_admin_config.0.cluster_ca_certificate)
}
}
resource "helm_release" "argocd" {
name = "argocd"
repository = "https://argoproj.github.io/argo-helm"
chart = "argo-cd"
version = "4.10.5"
create_namespace = true
namespace = "argocd"
}
Steps to Reproduce
All resources were created successfully when I executed the terraform code at first time creation.
But it was failed on terraform plan when I changed the vm_size in default node pool.
$ terraform plan
Error: Kubernetes cluster unreachable: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable
with helm_release.argocd,
│ on argocd.tf line 1, in resource "helm_release" "argocd":
│ 1: resource "helm_release" "argocd" {
Expected Behavior
The cluster should be reachable even the vm_size was changed.
Actual Behavior
Kubernetes cluster is unreachable for other terraform providers (ex: kubernetes, helm)
Test
I removed resource argocd to prevent the above situation, then terraform could plan and apply successfully.
I get the cluster config data from azure portal, the azurePortalFQDN is different between first time creation.
Question
Will The cluster be recreated if I change default node pool config which is commented “Changing this forces a new resource to be created” on terraform documents? Or only the default node pool will be deleted then create new one however the aks cluster doesn't changed?
Why provider helm could connect to the cluster at first time creation, but it was failed to connect when the resource recreation?
Thanks for your reply.
This was working perfectly fine before but for some reason it no longer is, would appreciate if someone can help fix this:
My terraform code as follows, have replaced key info. with "<>" just for sharing publicly here:
Outer main.tf has this:
module "<name>_service_account" {
source = "../modules/kubernetes/service-account"
name = "<name>-deployer"
}
# Create <name> platform namespace
resource "kubernetes_namespace" "<name>-platform" {
metadata {
name = "<name>-platform"
}
}
The service account main.tf module:
resource "kubernetes_service_account" "serviceaccount" {
metadata {
name = var.name
namespace = "kube-system"
}
}
resource "kubernetes_cluster_role_binding" "serviceaccount" {
metadata {
name = var.name
}
subject {
kind = "User"
name = "system:serviceaccount:kube-system:${var.name}"
}
role_ref {
kind = "ClusterRole"
name = "cluster-admin"
api_group = "rbac.authorization.k8s.io"
}
}
data "kubernetes_service_account" "serviceaccount" {
metadata {
name = var.name
namespace = "kube-system"
}
depends_on = [
resource.kubernetes_service_account.serviceaccount
]
}
data "kubernetes_secret" "serviceaccount" {
metadata {
name = data.kubernetes_service_account.serviceaccount.default_secret_name
namespace = "kube-system"
}
binary_data = {
"token": ""
}
depends_on = [
resource.kubernetes_service_account.serviceaccount
]
}
My outputs.tf for the above module:
output "secret_token" {
sensitive = true
value = lookup(data.kubernetes_secret.serviceaccount.binary_data, "token")
}
The error that I get in my terraform pipeline:
│ Error: Unable to fetch service account from Kubernetes: serviceaccounts "<name>-deployer" not found
│
│ with module.<name>_service_account.data.kubernetes_service_account.serviceaccount,
│ on ../modules/kubernetes/service-account/main.tf line 27, in data "kubernetes_service_account" "serviceaccount":
│ 27: data "kubernetes_service_account" "serviceaccount" {
Figured it out, this is a new environment/project and I had the terraform refresh stage still in the pipeline hence why it couldnt find the service account, removing that and just letting the plan and apply run first solved it.
I am attempting to run this module: https://registry.terraform.io/modules/azavea/postgresql-rds/aws/latest Here is the main.tf file created based on the information found there:
provider "aws" {
region = "us-east-2"
access_key = "key_here"
secret_key = "key_here"
}
module postgresql_rds {
source = "github.com/azavea/terraform-aws-postgresql-rds"
vpc_id = "vpc-2470994d"
instance_type = "db.t3.micro"
database_name = "tyler"
database_username = "admin"
database_password = "admin1234"
subnet_group = "tyler-subnet-1"
project = "Postgres-ts"
environment = "Staging"
alarm_actions = ["arn:aws:sns:us-east-2:304831784377:tyler-test"]
ok_actions = ["arn:aws:sns:us-east-2:304831784377:tyler-test"]
insufficient_data_actions = ["arn:aws:sns:us-east-2:304831784377:tyler-test"]
database_identifier = "jl23kj32sdf"
}
I am getting an error:
Error: Error creating DB Instance: DBSubnetGroupNotFoundFault: DBSubnetGroup 'tyler-subnet-1' not found.
│ status code: 404, request id: a95975dd-5456-444a-8f64-440fc4c1782f
│
│ with module.postgresql_rds.aws_db_instance.postgresql,
│ on .terraform/modules/postgresql_rds/main.tf line 46, in resource "aws_db_instance" "postgresql":
│ 46: resource "aws_db_instance" "postgresql" {
I have tried the example from the page:
subnet_group = aws_db_subnet_group.default.name
I used the default example from the page, under "Usage" ie subnet_group = aws_db_subnet_group.default.name. I have also used the subnet ID from AWS. I also assigned a name to the subnet, and used the name "tyler-subnet-1 in the above main.tf). I am getting the same basic error, with all three attempted inputs. Is there something I'm not understanding about the information that is being requested here?
Assuming you have a default subnet group, you can just use it:
subnet_group = "default"
If not you have to create a custom subnet group using aws_db_subnet_group:
resource "aws_db_subnet_group" "default" {
name = "my-subnet-group"
subnet_ids = [<subnet-id-1>, <subnet-id-2>]
tags = {
Name = "My DB subnet group"
}
}
and use the custom group:
subnet_group = aws_db_subnet_group.default.name