How does one use Terraform to deploy a Postgres database to RDS in Production?
I am currently running Terraform run locally, deploying my RDS instance to an AWS environment. However, in-order to create the Postgres resources (using the provider from cyrilgdn), I have to make my RDS instance publicly available. This is frowned upon in PROD environments, for obvious reasons. So how do people get around this for PROD Terraform deployed environments? I'm thinking the only way would be to run Terraform from within the RDS private subnet of the VPC... but how is that even possible?? Or am I missing something important :S
provider "aws" {
access_key = var.aws_access_key
secret_key = var.aws_secret_key
region = var.aws_region
default_tags {
tags = {
Owner = var.owner
Environment = var.env
}
}
}
resource "aws_security_group" "db-rules" {
name = "rds-sec-grp"
vpc_id = var.rds_vpc_id
tags = {
Name = "rds-sec-grp"
}
ingress {
from_port = "5432"
to_port = "5432"
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}
egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
}
resource "aws_db_instance" "default" {
identifier = "db-instance"
allocated_storage = 10
engine = "postgres"
engine_version = "13.2"
port = 5432
instance_class = "db.t3.medium"
multi_az = false
name = "db_system_${var.env}"
username = var.db_master_username
password = var.db_master_password
publicly_accessible = true
skip_final_snapshot = true
db_subnet_group_name = var.rds_public_subnet_group
vpc_security_group_ids = [aws_security_group.db-rules.id]
tags = {
Name = "db-instance"
}
}
provider "postgresql" {
host = aws_db_instance.default.address
port = aws_db_instance.default.port
database = aws_db_instance.default.name
username = var.db_master_username
password = var.db_master_password
superuser = false
sslmode = "require"
connect_timeout = 15
expected_version = aws_db_instance.default.engine_version
}
resource "postgresql_database" "default" {
name = "db_${var.env}"
owner = var.db_master_username
}
resource "postgresql_role" "readonly" {
name = var.db_readonly_username
password = var.db_readonly_password
login = true
connection_limit = 5
}
resource "postgresql_role" "readwrite" {
name = var.db_readwrite_username
password = var.db_readwrite_password
login = true
connection_limit = 5
}
resource "postgresql_role" "migrator" {
name = var.db_migrator_username
password = var.db_migrator_password
login = true
connection_limit = 5
}
resource "postgresql_schema" "ps" {
name = "ps"
database = postgresql_database.default.name
owner = var.db_master_username
}
resource "postgresql_grant" "revoke_public_schema" {
database = postgresql_database.default.name
role = "public"
schema = "public"
object_type = "schema"
privileges = []
}
resource "postgresql_grant" "readonly_schema" {
database = postgresql_database.default.name
role = postgresql_role.readonly.name
schema = postgresql_schema.ps.name
object_type = "schema"
privileges = ["USAGE"]
depends_on = [
postgresql_grant.revoke_public_schema
]
}
resource "postgresql_grant" "readwrite_schema" {
database = postgresql_database.default.name
role = postgresql_role.readwrite.name
schema = postgresql_schema.ps.name
object_type = "schema"
privileges = ["USAGE"]
depends_on = [
postgresql_grant.readonly_schema
]
}
resource "postgresql_grant" "migrator_schema" {
database = postgresql_database.default.name
role = postgresql_role.migrator.name
schema = postgresql_schema.ps.name
object_type = "schema"
privileges = ["CREATE", "USAGE"]
depends_on = [
postgresql_grant.readwrite_schema
]
}
resource "postgresql_grant" "readonly" {
database = postgresql_database.default.name
role = postgresql_role.readonly.name
schema = postgresql_schema.ps.name
object_type = "table"
privileges = ["SELECT"]
depends_on = [
postgresql_grant.migrator_schema
]
}
resource "postgresql_grant" "readwrite" {
database = postgresql_database.default.name
role = postgresql_role.readwrite.name
schema = postgresql_schema.ps.name
object_type = "table"
privileges = ["SELECT", "INSERT", "UPDATE", "DELETE"]
depends_on = [
postgresql_grant.migrator_schema
]
}
resource "postgresql_grant" "migrator" {
database = postgresql_database.default.name
role = postgresql_role.migrator.name
schema = postgresql_schema.ps.name
object_type = "table"
privileges = ["INSERT"]
depends_on = [
postgresql_grant.migrator_schema
]
}
resource "aws_ecr_repository" "webapi" {
name = "webapi"
image_scanning_configuration {
scan_on_push = true
}
}
resource "aws_ecr_repository" "service" {
name = "pollingservice"
image_scanning_configuration {
scan_on_push = true
}
}
resource "aws_ecr_repository" "migration" {
name = "datamigrations"
image_scanning_configuration {
scan_on_push = true
}
}
Related
I am getting error while creating AKS Cluster Using Terraform
Error:
Error: Failed to query available provider packages
Could not retrieve the list of available versions for provider hashicorp/file: provider registry registry.terraform.io does not have a provider named
registry.terraform.io/hashicorp/file
All modules should specify their required_providers so that external consumers will get the correct providers when using a module. To see which modules are currently depending on hashicorp/file, run the following command:
terraform providers
Above is the error i am facing. I have written Terraform code as shown below.
provider.tf:
============
provider "azurerm" {
features {}
}
terraform {
required_providers {
azurerm = {
source = "hashicorp/azurerm"
version = "2.39.0"
}
}
}
terraform.tfvars:
=================
resource_group_name = "a0474899701"
location = "CentralUS"
cluster_name = "aks01"
kubernetes_version = "1.24.4"
system_node_count = 2
user_node_count = 1
spot_node_count = 2
acr_name = "devops_acr_tf"
aks_network_plugin = "kubenet"
client_id = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
client_secret = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
main.tf:
========
# Create an Resource Group
resource "azurerm_resource_group" "aks-rg" {
name = var.resource_group_name
location = var.location
}
# Create an ACR instance
resource "azurerm_container_registry" "acr" {
name = var.acr_name
resource_group_name = azurerm_resource_group.aks-rg.name
location = var.location
sku = "Standard"
admin_enabled = false
}
# Create a role assignment to allow AKS to access ACR
resource "azurerm_role_assignment" "role_acrpull" {
scope = azurerm_container_registry.acr.id
role_definition_name = "AcrPull"
# principal_id = azurerm_kubernetes_cluster.aks.kubelet_identity.0.object_id
principal_id = azurerm_kubernetes_cluster.aks.kubelet_identity.0.client_id
skip_service_principal_aad_check = true
}
# Create a Kubernetes secret to hold the ACR credentials
# It holds the ACR credentials in a Docker config JSON format
resource "kubernetes_secret" "acr_credentials" {
metadata {
name = "acr-credentials"
}
data = {
".dockerconfigjson" = azurerm_container_registry.acr.docker_config_json
}
}
# Private Key Creation
resource "tls_private_key" "aks_ssh_key" {
algorithm = "RSA"
}
resource "tls_public_key" "aks_ssh_key" {
private_key_pem = tls_private_key.aks_ssh_key.private_key_pem
}
resource "file" "private_key" {
content = tls_private_key.aks_ssh_key.private_key_pem
filename = "aks_private_key.pem"
}
# virtual network (aks_vnet) is created in the same resource group
resource "azurerm_virtual_network" "aks_vnet" {
name = "${var.resource_group_name}-vnet01"
# address_space = ["10.0.0.0/16"]
address_space = ["10.172.144.0/26"]
location = azurerm_resource_group.aks_rg.location
resource_group_name = azurerm_resource_group.aks_rg.name
}
# subnet (aks_subnet) is created within the virtual network
resource "azurerm_subnet" "aks_subnet" {
name = "${var.resource_group_name}-vnet01-subnet01"
resource_group_name = azurerm_resource_group.aks_rg.name
virtual_network_name = azurerm_virtual_network.aks_vnet.name
# address_prefix = "10.0.1.0/24"
address_prefix = "10.172.144.0/27"
}
resource "azurerm_network_security_group" "azure-sg" {
name = "${var.resource_group_name}-nsg01"
location = azurerm_resource_group.aks_rg.location
resource_group_name = azurerm_resource_group.aks_rg.name
security_rule {
name = "allow-ssh"
priority = 100
direction = "Inbound"
access = "Allow"
protocol = "Tcp"
source_port_range = "*"
destination_port_range = "22"
source_address_prefix = "*"
destination_address_prefix = "*"
}
}
resource "azurerm_kubernetes_cluster" "aks" {
name = var.cluster_name
kubernetes_version = var.kubernetes_version
location = var.location
resource_group_name = azurerm_resource_group.aks-rg.name
security_group_name = azurerm_network_security_group.azure-sg.name
dns_prefix = var.cluster_name
default_node_pool {
name = "system"
node_count = var.system_node_count
vm_size = "Standard_E4as_v4"
os_disk_size_gb = 20
os_disk_type = "Ephemeral"
vnet_subnet_id = azurerm_subnet.aks_subnet.id
os_type = "Linux"
node_image_version = "AKSUbuntu-1804gen2containerd-2023.01.10"
enable_node_public_ip = false
enable_auto_scaling = false
}
additional_node_pools {
name = "user"
node_count = var.user_node_count
vm_size = "Standard_E8as_v4"
os_disk_size_gb = 20
os_disk_type = "Ephemeral"
vnet_subnet_id = azurerm_subnet.aks_subnet.id
type = "User"
# os_type = "RedHat"
os_type = "Linux"
node_image_version = "AKSUbuntu-1804gen2containerd-2023.01.10"
enable_node_public_ip = false
enable_auto_scaling = false
}
additional_node_pools {
name = "spot"
node_count = var.spot_node_count
vm_size = "Standard_D2s_v3"
os_disk_size_gb = 20
os_disk_type = "Ephemeral"
vnet_subnet_id = azurerm_subnet.aks_subnet.id
type = "User"
# os_type = "RedHat"
os_type = "Linux"
node_image_version = "AKSUbuntu-1804gen2containerd-2023.01.10"
max_price = 0.5
enable_node_public_ip = false
enable_auto_scaling = false
eviction_policy = "Spot"
taints = ["kubernetes.azure.com/scalesetpriority=spot:NoSchedule"]
labels = {
"kubernetes.azure.com/scalesetpriority" = "spot"
}
}
kubernetes_cluster_config {
max_pods_per_node = "110"
}
identity {
type = "SystemAssigned"
}
linux_profile {
admin_username = "azureuser"
ssh_key {
key_data = tls_public_key.aks_ssh_key.public_key_openssh
}
}
network_profile {
pod_cidr = "172.32.0.0/19"
service_cidr = "172.32.0.0/19"
load_balancer_sku = "Standard"
network_plugin = var.aks_network_plugin
dns_service_ip = "172.32.0.10"
docker_bridge_cidr = "172.34.0.1/16"
}
service_principal {
client_id = var.client_id
client_secret = var.client_secret
}
tags = {
Environment = "Development"
}
}
# ACR can be attached to the AKS cluster using the "azurerm_kubernetes_cluster_container_registry_config" resource type
resource "azurerm_kubernetes_cluster_container_registry_config" "acr_config" {
cluster_name = azurerm_kubernetes_cluster.aks.name
registry_id = azurerm_container_registry.acr.id
namespace = "aks"
default_action = "Allow"
}
Above is my Code I am facing above error. even i have changed my provider.tf still facing same issue. Can anyone please tell me How to solve this error
Thanks
I tried to reproduce the same in my environment to create AKS Cluster using Terraform:
Kindly use the below Terraform code to create AKS Cluster.
Terraform Code:
provider "azurerm" {
features {}
}
resource "azurerm_resource_group" "venkatesh" {
name = "venkat-resources"
location = "West Europe"
}
resource "azurerm_container_registry" "venkatreg" {
name = "Testcontainerregistery"
resource_group_name = azurerm_resource_group.venkatesh.name
location = azurerm_resource_group.venkatesh.location
sku = "Premium"
}
resource "azurerm_kubernetes_cluster" "venkatcluster" {
name = "example-aks1"
location = azurerm_resource_group.venkatesh.location
resource_group_name = azurerm_resource_group.venkatesh.name
dns_prefix = "exampleaks1"
default_node_pool {
name = "default"
node_count = 1
vm_size = "Standard_D2_v2"
}
identity {
type = "SystemAssigned"
}
tags = {
Environment = "Production"
}
}
resource "azurerm_role_assignment" "example" {
principal_id = azurerm_kubernetes_cluster.venkatcluster.kubelet_identity[0].object_id
role_definition_name = "AcrPull"
scope = azurerm_container_registry.venkatreg.id
skip_service_principal_aad_check = true
}
Terraform apply:
Once ran the code resources are created successfully.
Reference:
Create a Kubernetes cluster with Azure Kubernetes Service using Terraform.
I have some terraform code that creates a 3vm & vnet. but I cannot find work way to add 2 SGN and 3 subnets.
I have already read a lot of Terraform and Microsoft Documentation.
They have a lot of good solutions, but they not working on my terraform code. I will be very glad full for some help.
The main.tf
resource "random_pet" "rg-name" {
prefix = var.resource_group_name_prefix
}
resource "azurerm_resource_group" "rg" {
name = random_pet.rg-name.id
location = var.resource_group_location
}
# Create virtual network
resource "azurerm_virtual_network" "Vnet" {
name = "Vnet"
address_space = ["10.0.0.0/16"]
location = azurerm_resource_group.rg.location
resource_group_name = azurerm_resource_group.rg.name
}
# Create subnet
resource "azurerm_subnet" "Subnet" {
name = "subnet0"
resource_group_name = azurerm_resource_group.rg.name
virtual_network_name = azurerm_virtual_network.Vnet.name
address_prefixes = ["10.0.1.0/24"]
}
resource "azurerm_subnet" "Subnet2" {
name = "subnet1"
resource_group_name = azurerm_resource_group.rg.name
virtual_network_name = azurerm_virtual_network.Vnet.name
address_prefixes = ["10.0.2.0/24"]
}
#Associate subnet to subnet_network_security_group
resource "azurerm_subnet_network_security_group_association" "public" {
subnet_id = azurerm_subnet.Subnet.id
network_security_group_id = azurerm_network_security_group.NSG.id
}
# Create public IPs
resource "azurerm_public_ip" "Pub_IP" {
name = "PublicIP"
location = azurerm_resource_group.rg.location
resource_group_name = azurerm_resource_group.rg.name
allocation_method = "Static"
}
# Create Network Security Group and rule
resource "azurerm_network_security_group" "NSG" {
name = "app_nsg_rule"
location = azurerm_resource_group.rg.location
resource_group_name = azurerm_resource_group.rg.name
}
#Create an Inbound rule for app nsg
resource "azurerm_network_security_rule" "app_nsg_rule" {
name = "port8080"
priority = 110
direction = "inbound"
access = "Allow"
protocol = "Tcp"
source_port_range = "*"
destination_port_range = "8080"
source_address_prefix = "*"
destination_address_prefix = "*"
resource_group_name = azurerm_resource_group.rg.name
network_security_group_name = azurerm_network_security_group.NSG.name
}
#Create an Inbound rule for db nsg
resource "azurerm_network_security_rule" "db_nsg_rule" {
name = "port5432"
priority = 100
direction = "inbound"
access = "Allow"
protocol = "Tcp"
source_port_range = "*"
destination_port_range = "5432"
source_address_prefix = "*"
destination_address_prefix = "*"
resource_group_name = azurerm_resource_group.rg.name
network_security_group_name = azurerm_network_security_group.NSG.name
}
# Create network interface
resource "azurerm_network_interface" "nic1" {
name = "NIC1"
location = azurerm_resource_group.rg.location
resource_group_name = azurerm_resource_group.rg.name
ip_configuration {
name = "NicConf1"
subnet_id = azurerm_subnet.Subnet.id
private_ip_address_allocation = "Dynamic"
public_ip_address_id = azurerm_public_ip.Pub_IP.id
}
# ip_configuration {
# name = "NicConf2"
# subnet_id = azurerm_subnet.Subnet2.id
# private_ip_address_allocation = "Dynamic"
# public_ip_address_id = azurerm_public_ip.Pub_IP.id
# }
}
# Connect the security group to the network interface
resource "azurerm_network_interface_security_group_association" "NSG_Connect" {
network_interface_id = azurerm_network_interface.nic1.id
network_security_group_id = azurerm_network_security_group.NSG.id
}
# Create virtual machine
resource "azurerm_virtual_machine" "vm_app" {
count = 3
name = "app_vm${count.index}"
location = azurerm_resource_group.rg.location
resource_group_name = azurerm_resource_group.rg.name
network_interface_ids = [azurerm_network_interface.nic1.id,count.index]
vm_size = "Standard_B2s"
# Uncomment this line to delete the OS disk automatically when deleting the VM
# delete_os_disk_on_termination = true
# Uncomment this line to delete the data disks automatically when deleting the VM
# delete_data_disks_on_termination = true
storage_image_reference {
publisher = "Canonical"
offer = "0001-com-ubuntu-server-focal"
sku = "20_04-lts-gen2"
version = "latest"
}
storage_os_disk {
name = "disk${count.index}"
caching = "ReadWrite"
create_option = "FromImage"
managed_disk_type = "Standard_LRS"
}
}
# Import image from Image gallery
resource "azurerm_resource_group" "Image_App" {
name = "Weight_Tracker_application"
location = "EastUS"
}
#Create a Loadbalancer
resource "azurerm_lb" "LB" {
name = "loadBalancer"
location = azurerm_resource_group.rg.location
resource_group_name = azurerm_resource_group.rg.name
frontend_ip_configuration {
name = "PublicIPAddress"
public_ip_address_id = azurerm_public_ip.Pub_IP.id
}
}
#Create a Postgress Flexible Server
resource "azurerm_postgresql_flexible_server" "PS_SQL" {
name = "sqlflexserver"
resource_group_name = azurerm_resource_group.rg.name
location = azurerm_resource_group.rg.location
version = "12"
administrator_login = "postgres"
administrator_password = "p#ssw0rd42"
storage_mb = 32768
sku_name = "GP_Standard_D4s_v3"
}
resource "azurerm_postgresql_flexible_server_database" "PS_FLEX" {
name = "db-server"
server_id = azurerm_postgresql_flexible_server.PS_SQL.id
collation = "en_US.utf8"
charset = "utf8"
}
I have a K8S cluster on GCP where I have to run Data Science workload.
Some of they are in status "Evicted" because
The node was low on resource: memory. Container base was using 5417924Ki, which exceeds its request of 0.
I manage my architecture with Terraform and know how to manage cluster auto-scaling but I have no idea, even after reading the doc, how to manage this at a Pod level
resource "google_container_cluster" "k8s_cluster" {
name = "my-cluster-name
description = ""
location = var.default_region
network = var.network
subnetwork = var.subnetwork
initial_node_count = 1
remove_default_node_pool = true
ip_allocation_policy {
# VPC-native cluster using alias IP addresses
cluster_secondary_range_name = "gke-pods"
services_secondary_range_name = "gke-services"
}
maintenance_policy {
daily_maintenance_window {
start_time = "03:00"
}
}
master_authorized_networks_config {
cidr_blocks {
display_name = var.airflow.display_name
cidr_block = var.airflow.cidr_block
}
cidr_blocks {
display_name = var.gitlab.display_name
cidr_block = var.gitlab.cidr_block
}
}
network_policy {
enabled = false
}
private_cluster_config {
enable_private_endpoint = true
enable_private_nodes = true
master_ipv4_cidr_block = var.vpc_range_k8s_master
}
resource_labels = {
zone = var.zone
role = var.role
env = var.environment
}
# Disable basic auth and client certificate
master_auth {
username = ""
password = ""
client_certificate_config {
issue_client_certificate = false
}
}
cluster_autoscaling {
enabled = true
resource_limits {
resource_type = "cpu"
minimum = 1
maximum = 4
}
resource_limits {
resource_type = "memory"
minimum = 1
maximum = 2
}
}
}
I am trying to create a secret within my kubernetes cluster and terraform cloud.
I can create the cluster with no problems, but problems arise when I try to inject a secret in the cluster.
Here is a simplified version of my terraform manifest:
terraform {
backend "remote" {
organization = "my-org"
// Workspaces separate deployment envs (like prod, stage, or UK, Italy)
workspaces {
name = "my-workspace-name"
}
}
}
resource "google_container_cluster" "demo-k8s-cluster" {
name = "demo-cluster"
location = var.region
initial_node_count = 1
project = var.project-id
master_auth {
username = ""
password = ""
client_certificate_config {
issue_client_certificate = false
}
}
node_config {
oauth_scopes = [
"https://www.googleapis.com/auth/logging.write",
"https://www.googleapis.com/auth/monitoring",
]
// service_account = var.service-account
metadata = {
disable-legacy-endpoints = "true"
}
}
timeouts {
create = "30m"
update = "40m"
}
}
provider "kubernetes" {
host = google_container_cluster.demo-k8s-cluster.endpoint
username = google_container_cluster.demo-k8s-cluster.master_auth.0.username
password = google_container_cluster.demo-k8s-cluster.master_auth.0.password
client_certificate = base64decode(google_container_cluster.demo-k8s-cluster.master_auth.0.client_certificate)
client_key = base64decode(google_container_cluster.demo-k8s-cluster.master_auth.0.client_key)
cluster_ca_certificate = base64decode(google_container_cluster.demo-k8s-cluster.master_auth.0.cluster_ca_certificate)
load_config_file = "false"
}
resource "kubernetes_secret" "cloudsql-db-credentials" {
metadata {
name = "cloudsql-instance-credentials-test"
}
data = {
"stack-creds.json" = var.service-account
}
}
The plan works fine, I get the following error at Apply stage:
Error: secrets is forbidden: User "system:anonymous" cannot create resource "secrets" in API group "" in the namespace "default"
on infrastructure.tf line 149, in resource "kubernetes_secret" "cloudsql-db-credentials":
149: resource "kubernetes_secret" "cloudsql-db-credentials" {
As per #mario comment, it turns out terraform cloud can't get the right identity and can't connect to the cluster to inject the secret. Instead of using terraform cloud I have instead opted to use GCS backend and managed to get it working. The following configuration works:
terraform {
backend "gcs" {
bucket = "infrastructure-state-bucket"
prefix = "test/so_simple2"
}
}
// The project-id variable contains project id to use.
variable "project-id" {
type = string
}
variable "region" {
type = string
}
variable "cluster-name" {
type = string
}
provider "google" {
project = var.project-id
region = var.region
}
provider "random" {}
resource "random_id" "id" {
byte_length = 4
prefix = "${var.cluster-name}-"
}
resource "google_container_cluster" "cluster" {
name = random_id.id.hex
location = var.region
initial_node_count = 1
project = var.project-id
}
provider "kubernetes" {
host = google_container_cluster.cluster.endpoint
username = google_container_cluster.cluster.master_auth.0.username
password = google_container_cluster.cluster.master_auth.0.password
client_certificate = base64decode(google_container_cluster.cluster.master_auth.0.client_certificate)
client_key = base64decode(google_container_cluster.cluster.master_auth.0.client_key)
cluster_ca_certificate = base64decode(google_container_cluster.cluster.master_auth.0.cluster_ca_certificate)
// This is a deal breaker, if is set to false I get same error.
// load_config_file = "false"
}
resource "kubernetes_secret" "example" {
metadata {
name = "basic-auth"
}
data = {
username = "admin"
password = "P4ssw0rd"
}
type = "kubernetes.io/basic-auth"
}
New to terraform so i'm hoping this is an easy issue. I'm creating some resources in azure and deploying a simple flask application to AKS. Creating works fine using terraform plan. I can see that azure is provisioned correctly and I can hit the flask app.
When I try to run terraform destroy I get the error - "StatusCode=400...In order to delete the public IP, disassociate/detach the Public IP address from the resource.
Main.tf
variable "subscription_id" {}
variable "client_id" {}
variable "client_secret" {}
variable "tenant_id" {}
provider "azurerm" {
version = "=1.28.0"
tenant_id = "${var.tenant_id}"
subscription_id = "${var.subscription_id}"
}
resource "azurerm_resource_group" "aks" {
name = "${var.name_prefix}"
location = "${var.location}"
}
resource "azurerm_kubernetes_cluster" "k8s" {
name = "${var.name_prefix}-aks"
kubernetes_version = "${var.kubernetes_version}"
location = "${azurerm_resource_group.aks.location}"
resource_group_name = "${azurerm_resource_group.aks.name}"
dns_prefix = "AKS-${var.dns_prefix}"
agent_pool_profile {
name = "${var.node_pool_name}"
count = "${var.node_pool_size}"
vm_size = "${var.node_pool_vmsize}"
os_type = "${var.node_pool_os}"
os_disk_size_gb = 30
}
service_principal {
client_id = "${var.client_id}"
client_secret = "${var.client_secret}"
}
tags = {
environment = "${var.env_tag}"
}
}
provider "helm" {
install_tiller = true
kubernetes {
host = "${azurerm_kubernetes_cluster.k8s.kube_config.0.host}"
client_certificate = "${base64decode(azurerm_kubernetes_cluster.k8s.kube_config.0.client_certificate)}"
client_key = "${base64decode(azurerm_kubernetes_cluster.k8s.kube_config.0.client_key)}"
cluster_ca_certificate = "${base64decode(azurerm_kubernetes_cluster.k8s.kube_config.0.cluster_ca_certificate)}"
}
}
# Create Static Public IP Address to be used by Nginx Ingress
resource "azurerm_public_ip" "nginx_ingress" {
name = "nginx-ingress-public-ip"
location = "${azurerm_kubernetes_cluster.k8s.location}"
resource_group_name = "${azurerm_kubernetes_cluster.k8s.node_resource_group}"
allocation_method = "Static"
domain_name_label = "${var.name_prefix}"
}
# Add Kubernetes Stable Helm charts repo
data "helm_repository" "stable" {
name = "stable"
url = "https://kubernetes-charts.storage.googleapis.com"
}
# Install Nginx Ingress using Helm Chart
resource "helm_release" "nginx_ingress" {
name = "nginx-ingress"
repository = "${data.helm_repository.stable.metadata.0.name}"
chart = "nginx-ingress"
set {
name = "rbac.create"
value = "false"
}
set {
name = "controller.service.externalTrafficPolicy"
value = "Local"
}
set {
name = "controller.service.loadBalancerIP"
value = "${azurerm_public_ip.nginx_ingress.ip_address}"
}
}
Also deploying my kubernetes stuff in this file k8s.tf
provider "kubernetes" {
host = "${azurerm_kubernetes_cluster.k8s.kube_config.0.host}"
username = "${azurerm_kubernetes_cluster.k8s.kube_config.0.username}"
password = "${azurerm_kubernetes_cluster.k8s.kube_config.0.password}"
client_certificate = "${base64decode(azurerm_kubernetes_cluster.k8s.kube_config.0.client_certificate)}"
client_key = "${base64decode(azurerm_kubernetes_cluster.k8s.kube_config.0.client_key)}"
cluster_ca_certificate = "${base64decode(azurerm_kubernetes_cluster.k8s.kube_config.0.cluster_ca_certificate)}"
}
resource "kubernetes_deployment" "flask-api-deployment" {
metadata {
name = "flask-api-deployment"
}
spec {
replicas = 2
selector {
match_labels {
component = "api"
}
}
template {
metadata {
labels = {
component = "api"
}
}
spec {
container {
image = "xxx.azurecr.io/sampleflask:0.1.0"
name = "flask-api"
port {
container_port = 5000
}
}
}
}
}
}
resource "kubernetes_service" "api-cluster-ip-service" {
metadata {
name = "flask-api-cluster-ip-service"
}
spec {
selector {
component = "api"
}
port {
port = 5000
target_port = 5000
}
}
}
resource "kubernetes_ingress" "flask-ingress-service" {
metadata {
name = "flask-ingress-service"
}
spec {
backend {
service_name = "flask-api-cluster-ip-service"
service_port = 5000
}
}
}
For your issue, this is a problem about the sequence of the resources. When you create the nginx ingress with the public IP, the public IP should be created first. But when you delete the public IP, it's still in use by the nginx ingress. So It causes the error.
The solution is that you can detach the public IP from the resource which uses it. Then use the destroy the resource from the Terraform. You can take a look at the explanation in the issue.
The user #4c74356b41 is right, but to give more information assuming a config like this:
resource "azurerm_kubernetes_cluster" "k8s" {
name = "aks-e2x-nuffield-uat"
resource_group_name = azurerm_resource_group.core_rg.name
location = azurerm_resource_group.core_rg.location
dns_prefix = "aks-e2x-nuffield-uat-dns"
kubernetes_version = var.k8s_version
# NOTE currently only a single node pool, default, is configured
private_cluster_enabled = true
...
network_profile {
network_plugin = "kubenet"
load_balancer_sku = "standard"
service_cidr = var.k8s_service_subnet
pod_cidr = var.k8s_pod_subnet
docker_bridge_cidr = "172.17.0.1/16"
dns_service_ip = "40.0.8.10" # within the service subnet
}
}
Where the load_balancer_sku is set to standard, you can access the public IP to be used elsewhere like this:
data "azurerm_public_ip" "k8s_load_balancer_ip" {
name = reverse(split("/", tolist(azurerm_kubernetes_cluster.k8s.network_profile.0.load_balancer_profile.0.effective_outbound_ips)[0]))[0]
resource_group_name = azurerm_kubernetes_cluster.k8s.node_resource_group
}
output "ingress_public_ip" {
# value = azurerm_public_ip.ingress.ip_address
value = data.azurerm_public_ip.k8s_load_balancer_ip.ip_address
}