I created a pvc, which dynamically creates a persistenvolume (using k3s with local-path) that gets used by a deployment. I am provisioning everything using terraform but encountered an error. The terraform apply enters a infinite loop while creating the pvc and pod. The pvc is in this state:
Name: grafana-pvc
Namespace: default
StorageClass: local-path
Status: Pending
Volume:
Labels: io.kompose.service=grafana-data
Annotations: <none>
Finalizers: [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode: Filesystem
Used By: grafana-778c7f77c7-w7x9f
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal WaitForFirstConsumer 79s persistentvolume-controller waiting for first consumer to be created before binding
Normal WaitForPodScheduled 7s (x5 over 67s) persistentvolume-controller waiting for pod grafana-778c7f77c7-w7x9f to be scheduled
and the pod is in this state:
Name: grafana-778c7f77c7-w7x9f
Namespace: default
Priority: 0
Service Account: default
Node: <none>
Labels: io.kompose.service=grafana
pod-template-hash=778c7f77c7
Annotations: <none>
Status: Pending
IP:
IPs: <none>
Controlled By: ReplicaSet/grafana-778c7f77c7
Containers:
grafana:
Image: grafana/grafana:9.2.4
Port: 3000/TCP
Host Port: 0/TCP
Environment: <none>
Mounts:
/etc/grafana from grafana-configuration (rw)
/var/lib/grafana from grafana-data (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-n7cmt (ro)
Conditions:
Type Status
PodScheduled False
Volumes:
grafana-configuration:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: grafana-configuration
ReadOnly: false
grafana-data:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: grafana-pvc
ReadOnly: false
kube-api-access-n7cmt:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 90s default-scheduler 0/1 nodes are available: 1 persistentvolumeclaim "grafana-configuration" not found. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.
Warning FailedScheduling 89s default-scheduler 0/1 nodes are available: 1 persistentvolumeclaim "grafana-configuration" not found. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.
At this stage, nothing works anymore, terraform times out and I am not able to restore the state anymore.
My terraform files:
grafana.tf:
resource "kubernetes_persistent_volume_claim" "grafana-configuration" {
metadata {
name = "grafana-configuration"
labels = {
"io.kompose.service" = "grafana-configuration"
}
}
spec {
access_modes = ["ReadWriteOnce"]
storage_class_name = "local-path"
resources {
requests = {
storage = "1Gi"
}
}
volume_name = "grafana-configuration"
}
}
resource "kubernetes_persistent_volume" "grafana-configuration" {
metadata {
name = "grafana-configuration"
}
spec {
storage_class_name = "local-path"
access_modes = ["ReadWriteOnce"]
capacity = {
storage = "1Gi"
}
node_affinity {
required {
node_selector_term {
match_expressions {
key = "node-role.kubernetes.io/master"
operator = "In"
values = ["true"]
}
}
}
}
persistent_volume_source {
local {
path = "/home/administrator/Metrics.Infrastructure/grafana/"
}
}
}
}
resource "kubernetes_persistent_volume_claim" "grafana-pvc" {
metadata {
name = "grafana-pvc"
labels = {
"io.kompose.service" = "grafana-data"
}
}
spec {
access_modes = ["ReadWriteOnce"]
storage_class_name = "local-path"
resources {
requests = {
storage = "5Gi"
}
}
}
}
resource "kubernetes_deployment" "grafana" {
metadata {
name = "grafana"
labels = {
"io.kompose.service" = "grafana"
}
}
spec {
replicas = 1
selector {
match_labels = {
"io.kompose.service" = "grafana"
}
}
template {
metadata {
labels = {
"io.kompose.service" = "grafana"
}
}
spec {
volume {
name = "grafana-configuration"
persistent_volume_claim {
claim_name = "grafana-configuration"
}
}
volume {
name = "grafana-data"
persistent_volume_claim {
claim_name = "grafana-pvc"
}
}
container {
name = "grafana"
image = "grafana/grafana:9.2.4"
port {
container_port = 3000
}
volume_mount {
name = "grafana-configuration"
mount_path = "/etc/grafana"
}
volume_mount {
name = "grafana-data"
mount_path = "/var/lib/grafana"
}
}
restart_policy = "Always"
}
}
strategy {
type = "Recreate"
}
}
}
resource "kubernetes_service" "grafana" {
metadata {
name = "grafana"
labels = {
"io.kompose.service" = "grafana"
}
}
spec {
port {
port = 3000
target_port = 3000
node_port = 30001
}
type = "NodePort"
selector = {
"io.kompose.service" = "grafana"
}
}
}
prometheus.tf:
# We need these resources so that prometheus can fetch kubernetes metrics
resource "kubernetes_cluster_role" "prometheus-clusterrole" {
metadata {
name = "prometheus-clusterrole"
}
rule {
api_groups = [""]
resources = ["nodes", "nodes/proxy", "services", "endpoints", "pods"]
verbs = ["get", "list", "watch"]
}
rule {
api_groups = ["extensions"]
resources = ["ingresses"]
verbs = ["get", "list", "watch"]
}
rule {
non_resource_urls = ["/metrics"]
verbs = ["get"]
}
}
resource "kubernetes_cluster_role_binding" "prometheus_clusterrolebinding" {
metadata {
name = "prometheus-clusterrolebinding"
}
role_ref {
api_group = "rbac.authorization.k8s.io"
kind = "ClusterRole"
name = "prometheus-clusterrole"
}
subject {
kind = "ServiceAccount"
name = "default"
namespace = "default"
}
}
resource "kubernetes_config_map" "prometheus-config" {
metadata {
name = "prometheus-config"
}
data = {
"prometheus.yml" = "${file("${path.module}/prometheus/prometheus.yml")}"
}
}
resource "kubernetes_persistent_volume_claim" "prometheus_data_claim" {
metadata {
name = "prometheus-data-claim"
labels = {
"io.kompose.service" = "prometheus-data"
}
}
spec {
access_modes = ["ReadWriteOnce"]
storage_class_name = "local-path"
resources {
requests = {
storage = "20Gi"
}
}
}
}
resource "kubernetes_deployment" "prometheus" {
metadata {
name = "prometheus"
labels = {
"io.kompose.service" = "prometheus"
}
}
spec {
replicas = 1
selector {
match_labels = {
"io.kompose.service" = "prometheus"
}
}
template {
metadata {
labels = {
"io.kompose.service" = "prometheus"
}
}
spec {
volume {
name = "prometheus-data"
persistent_volume_claim {
claim_name = "prometheus-data-claim"
}
}
volume {
name = "prometheus-config"
config_map {
name = "prometheus-config"
}
}
container {
name = "prometheus"
image = "prom/prometheus:v2.40.0"
args = [
"--config.file=/config/prometheus.yml",
"--storage.tsdb.path=/prometheus",
"--web.enable-lifecycle"
]
port {
container_port = 9090
}
volume_mount {
name = "prometheus-config"
mount_path = "/config"
}
volume_mount {
name = "prometheus-data"
mount_path = "/prometheus"
}
}
restart_policy = "Always"
}
}
strategy {
type = "Recreate"
}
}
}
resource "kubernetes_service" "prometheus" {
metadata {
name = "prometheus"
labels = {
"io.kompose.service" = "prometheus"
}
}
spec {
port {
port = 80
target_port = 9090
node_port = 30000
}
type = "NodePort"
selector = {
"io.kompose.service" = "prometheus"
}
}
}
Instead of hardcoding the PVC names, you can use exported attributes to create implicit dependencies in terraform. That way terraform will know in which order to create resources and it will help you avoid needing to type the names all the time. For brevity:
volume {
name = "grafana-configuration"
persistent_volume_claim {
claim_name = kubernetes_persistent_volume_claim.grafana-configuration.metadata.0.name
}
}
volume {
name = "grafana-data"
persistent_volume_claim {
claim_name = kubernetes_persistent_volume_claim.grafana-pvc.metadata.0.name
}
}
Terraform plan output for reference:
+ volume {
+ name = "grafana-configuration"
+ persistent_volume_claim {
+ claim_name = "grafana-configuration"
+ read_only = false
}
}
+ volume {
+ name = "grafana-data"
+ persistent_volume_claim {
+ claim_name = "grafana-configuration"
+ read_only = false
}
}
Additionally, I would strongly suggest moving to a newer version of the resources which are denoted with _v1 at the end of the resource name, e.g., for PVC.
EDIT: As per my comment, in order to make sure the volume is created before the PVC, a similar piece of code can be used to create implicit dependency between the grafana-configuration PV and PVC (shortened for brevity):
spec {
access_modes = ["ReadWriteOnce"]
storage_class_name = "local-path"
resources {
requests = {
storage = "1Gi"
}
}
volume_name = kubernetes_persistent_volume.grafana-configuration.metadata.0.name # <--- This has changed
}
In case you haven't solved your problem. I had the same issue and the fix was just a config change in the TF resource
resource "kubernetes_persistent_volume_claim_v1" "mypvc" {
metadata {
...
}
spec {
...
}
wait_until_bound = false
}
https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/persistent_volume_claim#wait_until_bound
With that TF will not wait for the PVC to be consumed, and it will be able to create the next resource which does consume the PVC
Related
On eks default storage class is called gp2 and configured with:
allow_volume_expansion = false
parameters = {
"encrypted" = "false"
"fsType" = "ext4"
"type" = "gp2"
}
and I would like to change the default storage like so:
allow_volume_expansion = true
parameters = {
"encrypted" = "true"
"fsType" = "ext4"
"type" = "gp3"
}
How can it be done using terraform?
Following this kubrnetes guide I created the following config:
# Remove non encrypted default storage class
resource "kubernetes_annotations" "default-storageclass" {
api_version = "storage.k8s.io/v1"
kind = "StorageClass"
force = "true"
metadata {
name = "gp2"
}
annotations = {
"storageclass.kubernetes.io/is-default-class" = "false"
}
}
# Create the new wanted StorageClass and make it default
resource "kubernetes_storage_class" "gp3-enc" {
metadata {
name = "gp3-enc"
annotations = {
"storageclass.kubernetes.io/is-default-class" = "true"
}
}
storage_provisioner = "ebs.csi.aws.com"
volume_binding_mode = "WaitForFirstConsumer"
allow_volume_expansion = true
parameters = {
"encrypted" = "true"
"fsType" = "ext4"
"type" = "gp3"
}
}
I have 2 storage class deployed using terraform(storage and expandable-storage). Below is the code for both.
Before
resource "kubernetes_storage_class" "expandable-storage" {
metadata {
name = "expandable-storage"
}
storage_provisioner = "kubernetes.io/aws-ebs"
reclaim_policy = "Retain"
parameters = {
type = "gp2"
fsType: "ext4"
encrypted: "true"
}
allow_volume_expansion = true
volume_binding_mode = "Immediate"
}
resource "kubernetes_storage_class" "storage" {
metadata {
name = "storage"
}
storage_provisioner = "ebs.csi.aws.com"
reclaim_policy = "Retain"
parameters = {
type = "gp2"
fsType: "ext4"
encrypted: "true"
}
allow_volume_expansion = true
volume_binding_mode = "Immediate"
}
resource "null_resource" "k8s_storage_class_patch" {
depends_on = [kubernetes_storage_class.expandable-storage]
provisioner "local-exec" {
command = "/bin/bash scripts/storage_class_patch.sh"
}
}
After this I tried to update parameter for both storage class from type gp2 to type gp3.
After
resource "kubernetes_storage_class" "expandable-storage" {
metadata {
name = "expandable-storage"
}
storage_provisioner = "kubernetes.io/aws-ebs"
reclaim_policy = "Retain"
parameters = {
type = "gp3"
fsType: "ext4"
encrypted: "true"
}
allow_volume_expansion = true
volume_binding_mode = "Immediate"
}
resource "kubernetes_storage_class" "storage" {
metadata {
name = "storage"
}
storage_provisioner = "ebs.csi.aws.com"
reclaim_policy = "Retain"
parameters = {
type = "gp3"
fsType: "ext4"
encrypted: "true"
}
allow_volume_expansion = true
volume_binding_mode = "Immediate"
}
resource "null_resource" "k8s_storage_class_patch" {
depends_on = [kubernetes_storage_class.expandable-storage]
provisioner "local-exec" {
command = "/bin/bash scripts/storage_class_patch.sh"
}
}
After applying the resource module "storage" updated to gp3 but for "expandable-storage" module I am getting error
Error: storageclasses.storage.k8s.io "expandable-storage" already exists
I am not sure what is causing this as same changes worked for other storage class.
I'm getting this error when creating the persistent volume through terraform for EKS.
Created the EBS volume through terraform only.. It successfully created. But when try to create the Persistent volume getting the error
Please check the code below
resource "kubernetes_persistent_volume" "api-application-pv" {
metadata {
name = "api-application-pv"
}
spec {
capacity = {
storage = "2Gi"
}
access_modes = ["ReadWriteMany"]
persistent_volume_source {
aws_elastic_block_store {
volume_id = aws_launch_template.default.arn
}
}
}
}
resource "kubernetes_persistent_volume_claim" "api-application-pvc" {
metadata {
name = "api-application-pvc"
}
spec {
resources {
requests = {
storage = "2Gi"
}
}
access_modes = ["ReadWriteMany"]
storage_class_name = "gp2"
volume_name = "${kubernetes_persistent_volume.api-application-pv.metadata.0.name}"
}
wait_until_bound = false
depends_on = [kubernetes_persistent_volume.api-application-pv]
}
resource "aws_launch_template" "default" {
name_prefix = "eks-stage-template"
description = "eks-stage-template"
update_default_version = true
block_device_mappings {
device_name = "/dev/xvda"
ebs {
volume_size = 50
volume_type = "gp2"
delete_on_termination = true
encrypted = true
}
}
I'm using minikube locally.
The following is the .tf file I use to create my kubernetes cluster:
provider "kubernetes" {
config_path = "~/.kube/config"
}
resource "kubernetes_namespace" "tfs" {
metadata {
name = "tfs" # terraform-sandbox
}
}
resource "kubernetes_deployment" "golang_webapp" {
metadata {
name = "golang-webapp"
namespace = "tfs"
labels = {
app = "webapp"
}
}
spec {
replicas = 3
selector {
match_labels = {
app = "webapp"
}
}
template {
metadata {
labels = {
app = "webapp"
}
}
spec {
container {
image = "golang-docker-example"
name = "golang-webapp"
image_pull_policy = "Never" # this is set so that kuberenetes wont try to download the image but use the localy built one
liveness_probe {
http_get {
path = "/"
port = 8080
}
initial_delay_seconds = 15
period_seconds = 15
}
readiness_probe {
http_get {
path = "/"
port = 8080
}
initial_delay_seconds = 3
period_seconds = 3
}
}
}
}
}
}
resource "kubernetes_service" "golang_webapp" {
metadata {
name = "golang-webapp"
namespace = "tfs"
labels = {
app = "webapp_ingress"
}
}
spec {
selector = {
app = kubernetes_deployment.golang_webapp.metadata.0.labels.app
}
port {
port = 8080
target_port = 8080
protocol = "TCP"
}
# type = "ClusterIP"
type = "NodePort"
}
}
resource "kubernetes_ingress" "main_ingress" {
metadata {
name = "main-ingress"
namespace = "tfs"
}
spec {
rule {
http {
path {
backend {
service_name = "golang-webapp"
service_port = 8080
}
path = "/golang-webapp"
}
}
}
}
}
When executing terraform apply, I am successfully able to create all of the resources except for the ingress.
The error is:
Error: Failed to create Ingress 'tfs/main-ingress' because: the server could not find the requested resource (post ingresses.extensions)
with kubernetes_ingress.main_ingress,
on main.tf line 86, in resource "kubernetes_ingress" "main_ingress":
86: resource "kubernetes_ingress" "main_ingress" {
When I try to create an ingress service with kubectl using the same configuration as the one above (only in .yaml and using the kubectl apply command) it works, so it seems that kubectl & minikube are able to create this type of ingress, but terraform cant for some reason...
Thanks in advance for any help!
Edit 1:
adding the .yaml that I'm able to create the ingress with
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: example-ingress
namespace: tfs
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /$1
spec:
rules:
- http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: golang-webapp
port:
number: 8080
The kubernetes_ingress resource generate an ingress with an apiVersion which is not supported by your kubernetes cluster. You have to use [kubernetes_ingress_v1][1] resource which looks similar to kubernetes_ingress resource with some diferences. For your example, it will be like this :
resource "kubernetes_ingress_v1" "jenkins-ingress" {
metadata {
name = "example-ingress"
namespace = "tfs"
annotations = {
"nginx.ingress.kubernetes.io/rewrite-target" = "/$1"
}
}
spec {
rule {
http {
path {
path = "/"
backend {
service {
name = "golang-webapp"
port {
number = 8080
}
}
}
}
}
}
}
}
I think the issue can be related to the ingress classname. May be you need to explicitely provide it in your .tf:
metadata {
name = "example"
annotations = {
"kubernetes.io/ingress.class" = "nginx or your classname"
}
Or may be it's ingresses.extensions that does not exist in your cluster. Can you provide the .yaml that executed correctly ?
Something like this should help using kubernetes_ingress_v1
locals{
ingress_rules = [
{
service_path = "/"
service_name = "golang-webapp"
service_port = 8080
}
}
resource "kubernetes_ingress_v1" "jenkins-ingress" {
metadata {
annotations = var.ingress_annotations
name = "example-ingress"
namespace = "tfs"
labels = var.labels
}
spec {
ingress_class_name = var.ingress_class_name
rule {
http {
dynamic "path" {
for_each = local.ingress_rules
content {
backend {
service {
name = path.value.service_name
port {
number = path.value.service_port
}
}
}
path = path.value.service_path
}
}
}
}
tls {
secret_name = "tls-secret"
}
}
}
I'm experimenting with terraform to deploy k8s resources.
I created a mongodb deployment
provider "kubernetes" {
config_context = "kubernetes-admin#kubernetes"
}
resource "kubernetes_namespace" "demo-namespace" {
metadata {
name = "my-demo-namespace"
}
}
// mongodb
resource "kubernetes_deployment" "mongodb" {
metadata {
name = "mongodb"
namespace = kubernetes_namespace.demo-namespace.metadata[0].name
labels = {
app = "mongodb"
}
}
spec {
replicas = 1
selector {
match_labels = {
app = "mongodb"
}
}
template {
metadata {
labels = {
app = "mongodb"
}
}
spec {
container {
image = "mongo"
name = "mongodb"
env_from {
secret_ref {
name = kubernetes_secret.scrt-mongodb.metadata[0].name
}
config_map_ref {
name = kubernetes_config_map.cm-mongodb.metadata[0].name
}
}
resources {
limits {
cpu = "500m"
memory = "1Gi"
}
requests {
cpu = "150m"
memory = "256Mi"
}
}
liveness_probe {
exec {
command = ["bash", "-c", "mongo -u $MONGO_INITDB_ROOT_USERNAME -p $MONGO_INITDB_ROOT_PASSWORD --eval db.adminCommand(\"ping\")"]
}
initial_delay_seconds = 3
period_seconds = 1
}
}
}
}
}
}
// mongodb configmap
resource "kubernetes_config_map" "cm-mongodb" {
metadata {
name = "cm-mongodb"
namespace = kubernetes_namespace.demo-namespace.metadata.0.name
}
// improve creds with secret
data = {
MONGO_INITDB_DATABASE = "movies"
}
}
// monbodb secret
resource "kubernetes_secret" "scrt-mongodb" {
metadata {
name = "mongodb-creds"
}
data = {
MONGO_INITDB_ROOT_USERNAME = "root-user"
MONGO_INITDB_ROOT_PASSWORD = "secret"
}
type = "opaque"
}
This fails with:
kubernetes_config_map.cm-mongodb: Creation complete after 0s [id=my-demo-namespace/cm-mongodb]
kubernetes_deployment.mongodb: Creating...
Error: Failed to create deployment: Deployment.apps "mongodb" is invalid: spec.template.spec.containers[0].envFrom: Invalid value: "": may not have more than one field specified at a time
on template.tf line 12, in resource "kubernetes_deployment" "mongodb":
12: resource "kubernetes_deployment" "mongodb" {
What is wrong here?
You missed this line:
namespace = kubernetes_namespace.demo-namespace.metadata.0.name
You did not define the resource in the desired namespace so terraform failed to "find" the desired value.
// monbodb secret
resource "kubernetes_secret" "scrt-mongodb" {
metadata {
name = "mongodb-creds"
# -------------------------------------------------------------
# -------------------------------------------------------------
# Add the namespace here
namespace = kubernetes_namespace.demo-namespace.metadata.0.name
# -------------------------------------------------------------
# -------------------------------------------------------------
}
data = {
MONGO_INITDB_ROOT_USERNAME = "root-user"
MONGO_INITDB_ROOT_PASSWORD = "secret"
}
type = "opaque"
}