Terraform: Create multiple Kubernetes manifests using templates - kubernetes

I have a requirement to execute multiple Kubernetes manifests of the same kind. Just names, action, and port changes
my networkpolicy.yaml.tpl
${yamlencode(
apiVersion: projectcalico.org/v3
kind: GlobalNetworkPolicy
metadata:
name: ${name}-policy
spec:
action: ${action}
rules:
- to:
- operation:
ports: [for port in ports : "${port}"]
)}
values that need to be populated for each microservice
| -------- | -------- | --------------| --------------|
| name | frontend | backend | middleware |
| action | allow | allow | allow |
| ports | 8080,443 | 4731 | 8751,7542 |
Example networkpolicy.yaml after generation
apiVersion: projectcalico.org/v3
kind: GlobalNetworkPolicy
metadata:
name: frontend-policy
spec:
action: allow
rules:
- to:
- operation:
ports: ["8080", "443"]
how can I achieve this? I am not clear as how-to write main.tf
resource "kubernetes_manifest" "istio-config" {
manifest = yamldecode(templatefile("${path.module}/networkpolicy.yaml.tpl", {
name =
action =
port =
}))
}

What you are looking for is for_each. You can declare a local values block and then loop over that block, replacing what you need. f.e.:
locals {
services = {
frontend = {
action = "allow"
ports = [8080,433]
}
backend = {
...
}
middleware = {
...
}
}
}
resource "kubernetes_manifest" "istio-config" {
for_each = local.services
manifest = yamldecode(templatefile("${path.module}/networkpolicy.yaml.tpl", each.value)
}

You can use the loop
Example
menifest = yamldecode(templatefile("${path.module}/networkpolicy.yaml.tpl", {
name = var.name
})
update the networkpolicy.yaml.tpl file
%{ for s in nameservers ~}
name ${s}
%{ endfor ~}
If you don't want to edit the tpl file you can edit the main.tf directly
name = <<-EOT
%{ for s in var.name ~}
name ${s}
%{ endfor ~}
EOT
Ref : https://www.terraform.io/language/functions/templatefile
Read more about simple loop : https://blog.gruntwork.io/terraform-tips-tricks-loops-if-statements-and-gotchas-f739bbae55f9
You can use the count or for_each loop to simple make it possible.

Related

ERROR controller.provisioning Could not schedule pod, incompatible with provisioner "default", incompatible requirements, key karpenter.sh/provisioner

I read through the karpenter document at https://karpenter.sh/v0.16.1/getting-started/getting-started-with-terraform/#install-karpenter-helm-chart. I followed instructions step by step. I got errors at the end.
kubectl logs -f -n karpenter -l app.kubernetes.io/name=karpenter -c controller
DEBUG controller.provisioning Relaxing soft constraints for pod since it previously failed to schedule, removing: spec.topologySpreadConstraints = {"maxSkew":1,"topologyKey":"topology.kubernetes.io/zone","whenUnsatisfiable":"ScheduleAnyway","labelSelector":{"matchLabels":{"app.kubernetes.io/instance":"karpenter","app.kubernetes.io/name":"karpenter"}}} {"commit": "b157d45", "pod": "karpenter/karpenter-5755bb5b54-rh65t"}
2022-09-10T00:13:13.122Z
ERROR controller.provisioning Could not schedule pod, incompatible with provisioner "default", incompatible requirements, key karpenter.sh/provisioner-name, karpenter.sh/provisioner-name DoesNotExist not in karpenter.sh/provisioner-name In [default] {"commit": "b157d45", "pod": "karpenter/karpenter-5755bb5b54-rh65t"}
Below is the source code:
cat main.tf
terraform {
required_version = "~> 1.0"
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 4.0"
}
helm = {
source = "hashicorp/helm"
version = "~> 2.5"
}
kubectl = {
source = "gavinbunney/kubectl"
version = "~> 1.14"
}
}
}
provider "aws" {
region = "us-east-1"
}
locals {
cluster_name = "karpenter-demo"
# Used to determine correct partition (i.e. - `aws`, `aws-gov`, `aws-cn`, etc.)
partition = data.aws_partition.current.partition
}
data "aws_partition" "current" {}
module "vpc" {
# https://registry.terraform.io/modules/terraform-aws-modules/vpc/aws/latest
source = "terraform-aws-modules/vpc/aws"
version = "3.14.4"
name = local.cluster_name
cidr = "10.0.0.0/16"
azs = ["us-east-1a", "us-east-1b", "us-east-1c"]
private_subnets = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]
public_subnets = ["10.0.101.0/24", "10.0.102.0/24", "10.0.103.0/24"]
enable_nat_gateway = true
single_nat_gateway = true
one_nat_gateway_per_az = false
public_subnet_tags = {
"kubernetes.io/cluster/${local.cluster_name}" = "shared"
"kubernetes.io/role/elb" = 1
}
private_subnet_tags = {
"kubernetes.io/cluster/${local.cluster_name}" = "shared"
"kubernetes.io/role/internal-elb" = 1
}
}
module "eks" {
# https://registry.terraform.io/modules/terraform-aws-modules/eks/aws/latest
source = "terraform-aws-modules/eks/aws"
version = "18.29.0"
cluster_name = local.cluster_name
cluster_version = "1.22"
vpc_id = module.vpc.vpc_id
subnet_ids = module.vpc.private_subnets
# Required for Karpenter role below
enable_irsa = true
node_security_group_additional_rules = {
ingress_nodes_karpenter_port = {
description = "Cluster API to Node group for Karpenter webhook"
protocol = "tcp"
from_port = 8443
to_port = 8443
type = "ingress"
source_cluster_security_group = true
}
}
node_security_group_tags = {
# NOTE - if creating multiple security groups with this module, only tag the
# security group that Karpenter should utilize with the following tag
# (i.e. - at most, only one security group should have this tag in your account)
"karpenter.sh/discovery/${local.cluster_name}" = local.cluster_name
}
# Only need one node to get Karpenter up and running.
# This ensures core services such as VPC CNI, CoreDNS, etc. are up and running
# so that Karpenter can be deployed and start managing compute capacity as required
eks_managed_node_groups = {
initial = {
instance_types = ["m5.large"]
# Not required nor used - avoid tagging two security groups with same tag as well
create_security_group = false
min_size = 1
max_size = 1
desired_size = 1
iam_role_additional_policies = [
"arn:${local.partition}:iam::aws:policy/AmazonSSMManagedInstanceCore", # Required by Karpenter
"arn:${local.partition}:iam::aws:policy/AmazonEKSWorkerNodePolicy",
"arn:${local.partition}:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly", #for access to ECR images
"arn:${local.partition}:iam::aws:policy/CloudWatchAgentServerPolicy"
]
tags = {
# This will tag the launch template created for use by Karpenter
"karpenter.sh/discovery/${local.cluster_name}" = local.cluster_name
}
}
}
}
#The EKS module creates an IAM role for the EKS managed node group nodes. We’ll use that for Karpenter.
#We need to create an instance profile we can reference.
#Karpenter can use this instance profile to launch new EC2 instances and those instances will be able to connect to your cluster.
resource "aws_iam_instance_profile" "karpenter" {
name = "KarpenterNodeInstanceProfile-${local.cluster_name}"
role = module.eks.eks_managed_node_groups["initial"].iam_role_name
}
#Create the KarpenterController IAM Role
#Karpenter requires permissions like launching instances, which means it needs an IAM role that grants it access. The config
#below will create an AWS IAM Role, attach a policy, and authorize the Service Account to assume the role using IRSA. We will
#create the ServiceAccount and connect it to this role during the Helm chart install.
module "karpenter_irsa" {
source = "terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks"
version = "5.3.3"
role_name = "karpenter-controller-${local.cluster_name}"
attach_karpenter_controller_policy = true
karpenter_tag_key = "karpenter.sh/discovery/${local.cluster_name}"
karpenter_controller_cluster_id = module.eks.cluster_id
karpenter_controller_node_iam_role_arns = [
module.eks.eks_managed_node_groups["initial"].iam_role_arn
]
oidc_providers = {
ex = {
provider_arn = module.eks.oidc_provider_arn
namespace_service_accounts = ["karpenter:karpenter"]
}
}
}
#Install Karpenter Helm Chart
#Use helm to deploy Karpenter to the cluster. We are going to use the helm_release Terraform resource to do the deploy and pass in the
#cluster details and IAM role Karpenter needs to assume.
provider "helm" {
kubernetes {
host = module.eks.cluster_endpoint
cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data)
exec {
api_version = "client.authentication.k8s.io/v1beta1"
command = "aws"
args = ["eks", "get-token", "--cluster-name", local.cluster_name]
}
}
}
resource "helm_release" "karpenter" {
namespace = "karpenter"
create_namespace = true
name = "karpenter"
repository = "https://charts.karpenter.sh"
chart = "karpenter"
version = "v0.16.1"
set {
name = "serviceAccount.annotations.eks\\.amazonaws\\.com/role-arn"
value = module.karpenter_irsa.iam_role_arn
}
set {
name = "clusterName"
value = module.eks.cluster_id
}
set {
name = "clusterEndpoint"
value = module.eks.cluster_endpoint
}
set {
name = "aws.defaultInstanceProfile"
value = aws_iam_instance_profile.karpenter.name
}
}
#Provisioner
#Create a default provisioner using the command below. This provisioner configures instances to connect to your cluster’s endpoint and
#discovers resources like subnets and security groups using the cluster’s name.
#This provisioner will create capacity as long as the sum of all created capacity is less than the specified limit.
provider "kubectl" {
apply_retry_count = 5
host = module.eks.cluster_endpoint
cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data)
load_config_file = false
exec {
api_version = "client.authentication.k8s.io/v1beta1"
command = "aws"
args = ["eks", "get-token", "--cluster-name", module.eks.cluster_id]
}
}
resource "kubectl_manifest" "karpenter_provisioner" {
yaml_body = <<-YAML
apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
metadata:
name: default
spec:
requirements:
- key: karpenter.sh/capacity-type
operator: In
values: ["spot"]
limits:
resources:
cpu: 1000
provider:
subnetSelector:
Name: "*private*"
securityGroupSelector:
karpenter.sh/discovery/${module.eks.cluster_id}: ${module.eks.cluster_id}
tags:
karpenter.sh/discovery/${module.eks.cluster_id}: ${module.eks.cluster_id}
ttlSecondsAfterEmpty: 30
YAML
depends_on = [
helm_release.karpenter
]
}
cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
name: inflate
spec:
replicas: 0
selector:
matchLabels:
app: inflate
template:
metadata:
labels:
app: inflate
spec:
terminationGracePeriodSeconds: 0
containers:
- name: inflate
image: public.ecr.aws/eks-distro/kubernetes/pause:3.2
resources:
requests:
cpu: 1
EOF
kubectl scale deployment inflate --replicas 5
kubectl logs -f -n karpenter -l app.kubernetes.io/name=karpenter -c controller
DEBUG controller.provisioning Relaxing soft constraints for pod since it previously failed to schedule, removing: spec.topologySpreadConstraints = {"maxSkew":1,"topologyKey":"topology.kubernetes.io/zone","whenUnsatisfiable":"ScheduleAnyway","labelSelector":{"matchLabels":{"app.kubernetes.io/instance":"karpenter","app.kubernetes.io/name":"karpenter"}}} {"commit": "b157d45", "pod": "karpenter/karpenter-5755bb5b54-rh65t"}
2022-09-10T00:13:13.122Z
ERROR controller.provisioning Could not schedule pod, incompatible with provisioner "default", incompatible requirements, key karpenter.sh/provisioner-name, karpenter.sh/provisioner-name DoesNotExist not in karpenter.sh/provisioner-name In [default] {"commit": "b157d45", "pod": "karpenter/karpenter-5755bb5b54-rh65t"}
I belive this is due to the pod topology defined in the Karpenter deployment here:
https://github.com/aws/karpenter/blob/main/charts/karpenter/values.yaml#L73-L77
, you can read further on what pod topologySpreadConstraints does here:
https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/
If you increase the desired_size to 2 which matches the default deployment replicas above, that should resove the error.

No known RoleID Vault Agent

I'm using vault approle auth method to fetch secrets from vault. Below is my vault agent configmap.
---
apiVersion: v1
kind: ConfigMap
metadata:
name: my-configmap
data:
config-init.hcl: |
"auto_auth" = {
"method" = {
"config" = {
"role" = "test"
"role_id_file_path" = "roleid"
"secret_id_file_path" = "secretid"
"remove_secret_id_file_after_reading" = false
}
"type" = "approle"
}
"sink" = {
"config" = {
"path" = "/home/vault/.token"
}
"type" = "file"
"wrap_ttl" = "30m"
}
}
"vault" = {
"address" = "https://myvault.com"
}
"exit_after_auth" = true
"pid_file" = "/home/vault/.pid"
Then I'm referencing the above configmap in the deployment file.
annotations:
vault.hashicorp.com/agent-inject: 'true'
vault.hashicorp.com/agent-configmap: 'my-configmap'
But I get below error
vault-agent-init 2022-07-20T10:43:13.306Z [ERROR] auth.handler: error getting path or data from method: error="no known role ID" backoff=4m51.03s
Create a secret file in Kubernetes by using the RoleID and SecretID and pass the below annotation
vault.hashicorp.com/extra-secret: "secret-file"

Terraform: retrieve the nginx ingress controller Load Balancer IP

I'm trying to get the nginx ingress controller load balancer ip in Azure AKS. I figured I would use the kubernetes provider via:
data "kubernetes_service" "nginx_service" {
metadata {
name = "${local.ingress_name}-ingress-nginx-controller"
namespace = local.ingress_ns
}
depends_on = [helm_release.ingress]
}
However, i'm not seeing the IP address, this is what i get back:
nginx_service = [
+ {
+ cluster_ip = "10.0.165.249"
+ external_ips = []
+ external_name = ""
+ external_traffic_policy = "Local"
+ health_check_node_port = 31089
+ load_balancer_ip = ""
+ load_balancer_source_ranges = []
+ port = [
+ {
+ name = "http"
+ node_port = 30784
+ port = 80
+ protocol = "TCP"
+ target_port = "http"
},
+ {
+ name = "https"
+ node_port = 32337
+ port = 443
+ protocol = "TCP"
+ target_port = "https"
},
]
+ publish_not_ready_addresses = false
+ selector = {
+ "app.kubernetes.io/component" = "controller"
+ "app.kubernetes.io/instance" = "nginx-ingress-internal"
+ "app.kubernetes.io/name" = "ingress-nginx"
}
+ session_affinity = "None"
+ type = "LoadBalancer"
},
]
However when I pull down the service via kubectl I can get the IP address via:
kubectl get svc nginx-ingress-internal-ingress-nginx-controller -n nginx-ingress -o json | jq -r '.status.loadBalancer.ingress[].ip'
10.141.100.158
Is this a limitation of kubernetes provider for AKS? If so, what is a workaround other people have used? My end goals is to use the IP to configure the application gateway backend.
I guess I can use local-exec, but that seem hacky. Howerver, this might be my only option at the moment.
Thanks,
Jerry
although i strongly advise against creating resources inside Kubernetes with Terraform, you can do that:
Create a Public IP with Terraform -> Create the ingress-nginx inside Kubernetes with Terraform and pass annotations and loadBalancerIPwith data from your Terraform resources. The final manifest should look like this:
apiVersion: v1
kind: Service
metadata:
annotations:
service.beta.kubernetes.io/azure-load-balancer-resource-group: myResourceGroup
name: ingress-nginx-controller
spec:
loadBalancerIP: <YOUR_STATIC_IP>
type: LoadBalancer
Terraform could look like this:
resource "kubernetes_service" "ingress_nginx" {
metadata {
name = "tingress-nginx-controller"
annotations {
"service.beta.kubernetes.io/azure-load-balancer-resource-group" = "${azurerm_resource_group.YOUR_RG.name}"
}
spec {
selector = {
app = <PLACEHOLDER>
}
port {
port = <PLACEHOLDER>
target_port = <PLACEHOLDER>
}
type = "LoadBalancer"
load_balancer_ip = "${azurerm_public_ip.YOUR_IP.ip_address}"
}
}
Unfortunately, this is for internal ingress and not public facing and the IP is allocated dynamically. We currently dont want to use static ips
This is what I came up with:
module "load_balancer_ip" {
count = local.create_ingress ? 1 : 0
source = "github.com/matti/terraform-shell-resource?ref=v1.5.0"
command = "./scripts/get_load_balancer_ip.sh"
environment = {
KUBECONFIG = base64encode(module.aks.kube_admin_config_raw)
}
depends_on = [local_file.load_balancer_ip_script]
}
resource "local_file" "load_balancer_ip_script" {
count = local.create_ingress ? 1 : 0
filename = "./scripts/get_load_balancer_ip.sh"
content = <<-EOT
#!/bin/bash
echo $KUBECONFIG | base64 --decode > kubeconfig
kubectl get svc -n ${local.ingress_ns} ${local.ingress_name}-ingress-nginx-controller --kubeconfig kubeconfig -o=jsonpath='{.status.loadBalancer.ingress[0].ip}'
rm -f kubeconfig 2>&1 >/dev/null
EOT
}
output nginx_ip {
description = "IP address of the internal nginx controller"
value = local.create_ingress ? module.load_balancer_ip[0].content : null
}

How to wait for StatefulSet using Ansible's community.kubernetes.k8s_info module

Ansible has a great community.kubernetes module.
One of the useful flags of k8s_info is wait that is implemented for Deployment, DaemonSet and Pod.
For other k8s kinds it will return instantly unless wait_condition is provided.
What wait_condition should be provided to wait for StatefulSet?
I would do this using an "helper" tasks file. Say I have some roles/commes/tasks/helpers/wait-for.yaml, that goes something like this:
- name: "Waits for {{ obj_name }} to startup"
block:
- name: "Checks latest {{ obj_name }} status"
debug:
msg: |
Object Kind {{ check_with.kind | default('nothing returned') }}
delay: "{{ wait_for | default(10) }}"
ignore_errors: True
retries: "{{ retries_for | default(10) }}"
until:
- >
check_with.status is defined
and check_with.kind is defined
and check_with.status is defined
and ((check_with.kind == 'Pod'
and (check_with.status.containerStatuses[0].ready | default(False)))
or (check_with.kind == 'DataVolume'
and (check_with.status.phase | default(False)) == 'Succeeded')
or (check_with.kind in [ 'Deployment', 'DeploymentConfig' ]
and (check_with.status.availableReplicas | default(0)) >= 1)
or (check_with.kind == 'Job'
and check_with.status.completionTime is defined
and check_with.status.succeeded is defined)
or (check_with.kind == 'PersistentVolumeClaim'
and (check_with.status.phase | default(False)) == 'Bound')
or (check_with.kind == 'StatefulSet'
and (check_with.status.readyReplicas | default(0)) >= 1))
Then, whenever I need to wait for a Kubernetes resource, I would include that tasks file, using:
- include_role:
name: commons
tasks_from: helpers/wait-for.yaml
vars:
check_with: "{{ lookup('k8s', api_version='apps/v1',
kind='StatefulSet', namespace='default',
resource_name='my-statefulset') }}"
obj_name: "Statefulset default/my-statefulset"
retries_for: 30
wait_for: 10

Why istio's mixer logs are working differently for my case?

I have a DB in namespace ns-restriction-demo-2 and a nodeJs application that is using that DB from namespace ns-restriction-demo-1.
So I was trying to add authorization i.e. DB can only be access by one namespace (ns-restriction-demo-1), but when I enable the mixer logs, I realise that the source attributes are different.
{
"level": "warn",
"time": "0001-01-01T00:00:00.000000Z",
"instance": "newlog.instance.ns-restriction-demo-2",
"destination": "db-demo-2",
"destinationName": "db-demo-2-868c4bb6c7-f5l7c",
"destinationNamespace": "ns-restriction-demo-2",
"destinationPort": 3306,
"destinationPrinciple": "unkown",
"latency": "0s",
"mtls": false,
"requestHost": "unkown",
"responseCode": 0,
"responseSize": 0,
"source": "calico-node",
"sourceName": "calico-node-ljzxl",
"sourceNamespace": "kube-system",
"sourcePrinicple": "unkown",
"sourceServiceAccount": "calico-node",
"sourceservices": "unkown",
"user": "unknown"
}
Configuration I have used
apiVersion: config.istio.io/v1alpha2
kind: instance
metadata:
name: newlog
namespace: ns-restriction-demo-2
spec:
compiledTemplate: logentry
params:
severity: '"warning"'
variables:
sourceName: source.name | "unkown"
sourceNamespace: source.namespace | "unkown"
sourcePrinicple: source.principal | "unkown"
sourceServiceAccount: source.serviceAccount | "unkown"
sourceservices: source.services | "unkown"
destinationPort: destination.port | 0
destinationName: destination.name | "unkown"
destinationNamespace: destination.namespace | "unkown"
destinationPrinciple: destination.principal | "unkown"
requestHost: request.host | "unkown"
source: source.labels["app"] | source.workload.name | "unknown"
user: source.user | "unknown"
mtls: connection.mtls | false
destination: destination.labels["app"] | destination.workload.name | "unknown"
responseCode: response.code | 0
responseSize: response.size | 0
latency: response.duration | "0ms"
monitored_resource_type: '"UNSPECIFIED"'
---
# Configuration for a stdio handler
apiVersion: config.istio.io/v1alpha2
kind: handler
metadata:
name: newloghandler
namespace: ns-restriction-demo-2
spec:
compiledAdapter: stdio
params:
severity_levels:
warning: 1 # Params.Level.WARNING
outputAsJson: true
---
# Rule to send logentry instances to a stdio handler
apiVersion: config.istio.io/v1alpha2
kind: rule
metadata:
name: newlogstdio
namespace: ns-restriction-demo-2
spec:
match: "true" # match for all requests
actions:
- handler: newloghandler
instances:
- newlog
---
So I have a question for the istio's expert here.
Why source or sourceName or sourceNamespace is related to calico? why is it not about that nodeJs app that is using this DB?
Please help me understand what have I done wrong or missing something?
Calico Version
Client Version: v3.5.8
Git commit: 107e128
Cluster Version: v3.6.2
Cluster Type: k8s,bgp,kdd
Other Versions
Istio: 1.1.10
Kubernetes: 1.13.6
Collecting metrics with Mixer for TCP Services is different from the HTTP one.
The access to your database is an example of TCP workload inside Istio Mesh.
Therefore please use a dedicated 'tcpaccesslog' template instead of 'logentry' one, which includes in attributes mapping a valid protocol configuration:
'protocol: context.protocol | "tcp"'
This helps Istio to properly derive source and sourceName attributes.
Note:
in case a requests to DB are sent over mTLS enabled TCP connection, you should have available some extra attributes:
source.workload.name
source.workload.namespace