I have 2 storage class deployed using terraform(storage and expandable-storage). Below is the code for both.
Before
resource "kubernetes_storage_class" "expandable-storage" {
metadata {
name = "expandable-storage"
}
storage_provisioner = "kubernetes.io/aws-ebs"
reclaim_policy = "Retain"
parameters = {
type = "gp2"
fsType: "ext4"
encrypted: "true"
}
allow_volume_expansion = true
volume_binding_mode = "Immediate"
}
resource "kubernetes_storage_class" "storage" {
metadata {
name = "storage"
}
storage_provisioner = "ebs.csi.aws.com"
reclaim_policy = "Retain"
parameters = {
type = "gp2"
fsType: "ext4"
encrypted: "true"
}
allow_volume_expansion = true
volume_binding_mode = "Immediate"
}
resource "null_resource" "k8s_storage_class_patch" {
depends_on = [kubernetes_storage_class.expandable-storage]
provisioner "local-exec" {
command = "/bin/bash scripts/storage_class_patch.sh"
}
}
After this I tried to update parameter for both storage class from type gp2 to type gp3.
After
resource "kubernetes_storage_class" "expandable-storage" {
metadata {
name = "expandable-storage"
}
storage_provisioner = "kubernetes.io/aws-ebs"
reclaim_policy = "Retain"
parameters = {
type = "gp3"
fsType: "ext4"
encrypted: "true"
}
allow_volume_expansion = true
volume_binding_mode = "Immediate"
}
resource "kubernetes_storage_class" "storage" {
metadata {
name = "storage"
}
storage_provisioner = "ebs.csi.aws.com"
reclaim_policy = "Retain"
parameters = {
type = "gp3"
fsType: "ext4"
encrypted: "true"
}
allow_volume_expansion = true
volume_binding_mode = "Immediate"
}
resource "null_resource" "k8s_storage_class_patch" {
depends_on = [kubernetes_storage_class.expandable-storage]
provisioner "local-exec" {
command = "/bin/bash scripts/storage_class_patch.sh"
}
}
After applying the resource module "storage" updated to gp3 but for "expandable-storage" module I am getting error
Error: storageclasses.storage.k8s.io "expandable-storage" already exists
I am not sure what is causing this as same changes worked for other storage class.
Related
I have an image to pull from a private registry. I did all the configs and added the secret to the pod config under pod.spec.image_pull_secrets. But I am getting an error like
An argument named "image_pull_secrets" is not expected here. Did you mean to define a block of type "image_pull_secrets"?
As per documentation this should be ok.
https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/pod#nested-blocks
resource "kubernetes_pod" "main" {
count = data.coder_workspace.me.start_count
metadata {
name = "coder-${lower(data.coder_workspace.me.owner)}-${lower(data.coder_workspace.me.name)}"
namespace = var.workspaces_namespace
}
spec {
image_pull_secrets = {
name = ["coder-ocir-secret"]
}
security_context {
# run_as_user = "1000"
fs_group = "1000"
}
init_container {
name = "init-eclipse"
image = "busybox:latest"
command = [ "chown","-R","1000:1000","/data"]
security_context {
run_as_user = "0"
privileged = "true"
allow_privilege_escalation = "true"
read_only_root_filesystem = "false"
run_as_non_root = "false"
capabilities {
add = ["CAP_SYS_ADMIN","CHOWN",
"FOWNER",
"DAC_OVERRIDE"]
drop = [
"ALL"]
}
}
volume_mount {
mount_path = "/data"
name = "home-coder-vol-${data.coder_workspace.me.owner}-${lower(data.coder_workspace.me.name)}"
}
}
container {
name = "eclipse"
image = "docker.io/manumaan/eclipsevncv2.2:latest"
command = ["sh", "-c", coder_agent.coder.init_script]
image_pull_policy = "Always"
security_context {
run_as_user = "1000"
# fs_group = "1000"
}
env {
name = "CODER_AGENT_TOKEN"
value = coder_agent.coder.token
}
resources {
requests = {
cpu = "${var.cpu}"
memory = "${var.memory}G"
ephemeral-storage = "2Gi"
}
limits = {
cpu = "${var.cpu}"
memory = "${var.memory}G"
ephemeral-storage = "4Gi"
}
}
volume_mount {
mount_path = "/home/coder"
name = "home-coder-vol-${data.coder_workspace.me.owner}-${lower(data.coder_workspace.me.name)}"
}
}
I also tried giving it inside container, after all containers etc inside spec but it does not accept it.I am going crazy!
Also made it not a list: No difference.
image_pull_secrets = {
name = "coder-ocir-secret"
}
This might be caused by a typo, image_pull_secrets is a block, so you don't need the =, neither the square brackets ([]) here:
image_pull_secrets = {
name = ["coder-ocir-secret"]
}
It should be instead:
image_pull_secrets {
name = "coder-ocir-secret"
}
If you need to define multiple pull_secrets you can define multiple ones, or use dynamic blocks
Make sure your block is perfect and indentation also, this one is working for me
resource "kubernetes_pod" "main" {
metadata {
name = "coder-name"
namespace = "default"
}
spec {
image_pull_secrets {
name = "coder-ocir-secret"
}
security_context {
# run_as_user = "1000"
fs_group = "1000"
}
init_container {
name = "init-eclipse"
image = "busybox:latest"
command = [ "chown","-R","1000:1000","/data"]
security_context {
run_as_user = "0"
privileged = "true"
allow_privilege_escalation = "true"
read_only_root_filesystem = "false"
run_as_non_root = "false"
capabilities {
add = ["CAP_SYS_ADMIN","CHOWN",
"FOWNER",
"DAC_OVERRIDE"]
drop = [
"ALL"]
}
}
volume_mount {
mount_path = "/data"
name = "home-coder-vol-fake-name"
}
}
container {
name = "eclipse"
image = "docker.io/manumaan/eclipsevncv2.2:latest"
command = ["sh", "-c", "command"]
image_pull_policy = "Always"
security_context {
run_as_user = "1000"
# fs_group = "1000"
}
env {
name = "CODER_AGENT_TOKEN"
value = "value"
}
resources {
requests = {
cpu = "1"
memory = "1G"
ephemeral-storage = "2Gi"
}
limits = {
cpu = "1"
memory = "2G"
ephemeral-storage = "4Gi"
}
}
volume_mount {
mount_path = "/home/coder"
name = "home-coder-vol-fake-name"
}
}
}
}
I created a pvc, which dynamically creates a persistenvolume (using k3s with local-path) that gets used by a deployment. I am provisioning everything using terraform but encountered an error. The terraform apply enters a infinite loop while creating the pvc and pod. The pvc is in this state:
Name: grafana-pvc
Namespace: default
StorageClass: local-path
Status: Pending
Volume:
Labels: io.kompose.service=grafana-data
Annotations: <none>
Finalizers: [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode: Filesystem
Used By: grafana-778c7f77c7-w7x9f
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal WaitForFirstConsumer 79s persistentvolume-controller waiting for first consumer to be created before binding
Normal WaitForPodScheduled 7s (x5 over 67s) persistentvolume-controller waiting for pod grafana-778c7f77c7-w7x9f to be scheduled
and the pod is in this state:
Name: grafana-778c7f77c7-w7x9f
Namespace: default
Priority: 0
Service Account: default
Node: <none>
Labels: io.kompose.service=grafana
pod-template-hash=778c7f77c7
Annotations: <none>
Status: Pending
IP:
IPs: <none>
Controlled By: ReplicaSet/grafana-778c7f77c7
Containers:
grafana:
Image: grafana/grafana:9.2.4
Port: 3000/TCP
Host Port: 0/TCP
Environment: <none>
Mounts:
/etc/grafana from grafana-configuration (rw)
/var/lib/grafana from grafana-data (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-n7cmt (ro)
Conditions:
Type Status
PodScheduled False
Volumes:
grafana-configuration:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: grafana-configuration
ReadOnly: false
grafana-data:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: grafana-pvc
ReadOnly: false
kube-api-access-n7cmt:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 90s default-scheduler 0/1 nodes are available: 1 persistentvolumeclaim "grafana-configuration" not found. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.
Warning FailedScheduling 89s default-scheduler 0/1 nodes are available: 1 persistentvolumeclaim "grafana-configuration" not found. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.
At this stage, nothing works anymore, terraform times out and I am not able to restore the state anymore.
My terraform files:
grafana.tf:
resource "kubernetes_persistent_volume_claim" "grafana-configuration" {
metadata {
name = "grafana-configuration"
labels = {
"io.kompose.service" = "grafana-configuration"
}
}
spec {
access_modes = ["ReadWriteOnce"]
storage_class_name = "local-path"
resources {
requests = {
storage = "1Gi"
}
}
volume_name = "grafana-configuration"
}
}
resource "kubernetes_persistent_volume" "grafana-configuration" {
metadata {
name = "grafana-configuration"
}
spec {
storage_class_name = "local-path"
access_modes = ["ReadWriteOnce"]
capacity = {
storage = "1Gi"
}
node_affinity {
required {
node_selector_term {
match_expressions {
key = "node-role.kubernetes.io/master"
operator = "In"
values = ["true"]
}
}
}
}
persistent_volume_source {
local {
path = "/home/administrator/Metrics.Infrastructure/grafana/"
}
}
}
}
resource "kubernetes_persistent_volume_claim" "grafana-pvc" {
metadata {
name = "grafana-pvc"
labels = {
"io.kompose.service" = "grafana-data"
}
}
spec {
access_modes = ["ReadWriteOnce"]
storage_class_name = "local-path"
resources {
requests = {
storage = "5Gi"
}
}
}
}
resource "kubernetes_deployment" "grafana" {
metadata {
name = "grafana"
labels = {
"io.kompose.service" = "grafana"
}
}
spec {
replicas = 1
selector {
match_labels = {
"io.kompose.service" = "grafana"
}
}
template {
metadata {
labels = {
"io.kompose.service" = "grafana"
}
}
spec {
volume {
name = "grafana-configuration"
persistent_volume_claim {
claim_name = "grafana-configuration"
}
}
volume {
name = "grafana-data"
persistent_volume_claim {
claim_name = "grafana-pvc"
}
}
container {
name = "grafana"
image = "grafana/grafana:9.2.4"
port {
container_port = 3000
}
volume_mount {
name = "grafana-configuration"
mount_path = "/etc/grafana"
}
volume_mount {
name = "grafana-data"
mount_path = "/var/lib/grafana"
}
}
restart_policy = "Always"
}
}
strategy {
type = "Recreate"
}
}
}
resource "kubernetes_service" "grafana" {
metadata {
name = "grafana"
labels = {
"io.kompose.service" = "grafana"
}
}
spec {
port {
port = 3000
target_port = 3000
node_port = 30001
}
type = "NodePort"
selector = {
"io.kompose.service" = "grafana"
}
}
}
prometheus.tf:
# We need these resources so that prometheus can fetch kubernetes metrics
resource "kubernetes_cluster_role" "prometheus-clusterrole" {
metadata {
name = "prometheus-clusterrole"
}
rule {
api_groups = [""]
resources = ["nodes", "nodes/proxy", "services", "endpoints", "pods"]
verbs = ["get", "list", "watch"]
}
rule {
api_groups = ["extensions"]
resources = ["ingresses"]
verbs = ["get", "list", "watch"]
}
rule {
non_resource_urls = ["/metrics"]
verbs = ["get"]
}
}
resource "kubernetes_cluster_role_binding" "prometheus_clusterrolebinding" {
metadata {
name = "prometheus-clusterrolebinding"
}
role_ref {
api_group = "rbac.authorization.k8s.io"
kind = "ClusterRole"
name = "prometheus-clusterrole"
}
subject {
kind = "ServiceAccount"
name = "default"
namespace = "default"
}
}
resource "kubernetes_config_map" "prometheus-config" {
metadata {
name = "prometheus-config"
}
data = {
"prometheus.yml" = "${file("${path.module}/prometheus/prometheus.yml")}"
}
}
resource "kubernetes_persistent_volume_claim" "prometheus_data_claim" {
metadata {
name = "prometheus-data-claim"
labels = {
"io.kompose.service" = "prometheus-data"
}
}
spec {
access_modes = ["ReadWriteOnce"]
storage_class_name = "local-path"
resources {
requests = {
storage = "20Gi"
}
}
}
}
resource "kubernetes_deployment" "prometheus" {
metadata {
name = "prometheus"
labels = {
"io.kompose.service" = "prometheus"
}
}
spec {
replicas = 1
selector {
match_labels = {
"io.kompose.service" = "prometheus"
}
}
template {
metadata {
labels = {
"io.kompose.service" = "prometheus"
}
}
spec {
volume {
name = "prometheus-data"
persistent_volume_claim {
claim_name = "prometheus-data-claim"
}
}
volume {
name = "prometheus-config"
config_map {
name = "prometheus-config"
}
}
container {
name = "prometheus"
image = "prom/prometheus:v2.40.0"
args = [
"--config.file=/config/prometheus.yml",
"--storage.tsdb.path=/prometheus",
"--web.enable-lifecycle"
]
port {
container_port = 9090
}
volume_mount {
name = "prometheus-config"
mount_path = "/config"
}
volume_mount {
name = "prometheus-data"
mount_path = "/prometheus"
}
}
restart_policy = "Always"
}
}
strategy {
type = "Recreate"
}
}
}
resource "kubernetes_service" "prometheus" {
metadata {
name = "prometheus"
labels = {
"io.kompose.service" = "prometheus"
}
}
spec {
port {
port = 80
target_port = 9090
node_port = 30000
}
type = "NodePort"
selector = {
"io.kompose.service" = "prometheus"
}
}
}
Instead of hardcoding the PVC names, you can use exported attributes to create implicit dependencies in terraform. That way terraform will know in which order to create resources and it will help you avoid needing to type the names all the time. For brevity:
volume {
name = "grafana-configuration"
persistent_volume_claim {
claim_name = kubernetes_persistent_volume_claim.grafana-configuration.metadata.0.name
}
}
volume {
name = "grafana-data"
persistent_volume_claim {
claim_name = kubernetes_persistent_volume_claim.grafana-pvc.metadata.0.name
}
}
Terraform plan output for reference:
+ volume {
+ name = "grafana-configuration"
+ persistent_volume_claim {
+ claim_name = "grafana-configuration"
+ read_only = false
}
}
+ volume {
+ name = "grafana-data"
+ persistent_volume_claim {
+ claim_name = "grafana-configuration"
+ read_only = false
}
}
Additionally, I would strongly suggest moving to a newer version of the resources which are denoted with _v1 at the end of the resource name, e.g., for PVC.
EDIT: As per my comment, in order to make sure the volume is created before the PVC, a similar piece of code can be used to create implicit dependency between the grafana-configuration PV and PVC (shortened for brevity):
spec {
access_modes = ["ReadWriteOnce"]
storage_class_name = "local-path"
resources {
requests = {
storage = "1Gi"
}
}
volume_name = kubernetes_persistent_volume.grafana-configuration.metadata.0.name # <--- This has changed
}
In case you haven't solved your problem. I had the same issue and the fix was just a config change in the TF resource
resource "kubernetes_persistent_volume_claim_v1" "mypvc" {
metadata {
...
}
spec {
...
}
wait_until_bound = false
}
https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/persistent_volume_claim#wait_until_bound
With that TF will not wait for the PVC to be consumed, and it will be able to create the next resource which does consume the PVC
On eks default storage class is called gp2 and configured with:
allow_volume_expansion = false
parameters = {
"encrypted" = "false"
"fsType" = "ext4"
"type" = "gp2"
}
and I would like to change the default storage like so:
allow_volume_expansion = true
parameters = {
"encrypted" = "true"
"fsType" = "ext4"
"type" = "gp3"
}
How can it be done using terraform?
Following this kubrnetes guide I created the following config:
# Remove non encrypted default storage class
resource "kubernetes_annotations" "default-storageclass" {
api_version = "storage.k8s.io/v1"
kind = "StorageClass"
force = "true"
metadata {
name = "gp2"
}
annotations = {
"storageclass.kubernetes.io/is-default-class" = "false"
}
}
# Create the new wanted StorageClass and make it default
resource "kubernetes_storage_class" "gp3-enc" {
metadata {
name = "gp3-enc"
annotations = {
"storageclass.kubernetes.io/is-default-class" = "true"
}
}
storage_provisioner = "ebs.csi.aws.com"
volume_binding_mode = "WaitForFirstConsumer"
allow_volume_expansion = true
parameters = {
"encrypted" = "true"
"fsType" = "ext4"
"type" = "gp3"
}
}
I'm getting this error when creating the persistent volume through terraform for EKS.
Created the EBS volume through terraform only.. It successfully created. But when try to create the Persistent volume getting the error
Please check the code below
resource "kubernetes_persistent_volume" "api-application-pv" {
metadata {
name = "api-application-pv"
}
spec {
capacity = {
storage = "2Gi"
}
access_modes = ["ReadWriteMany"]
persistent_volume_source {
aws_elastic_block_store {
volume_id = aws_launch_template.default.arn
}
}
}
}
resource "kubernetes_persistent_volume_claim" "api-application-pvc" {
metadata {
name = "api-application-pvc"
}
spec {
resources {
requests = {
storage = "2Gi"
}
}
access_modes = ["ReadWriteMany"]
storage_class_name = "gp2"
volume_name = "${kubernetes_persistent_volume.api-application-pv.metadata.0.name}"
}
wait_until_bound = false
depends_on = [kubernetes_persistent_volume.api-application-pv]
}
resource "aws_launch_template" "default" {
name_prefix = "eks-stage-template"
description = "eks-stage-template"
update_default_version = true
block_device_mappings {
device_name = "/dev/xvda"
ebs {
volume_size = 50
volume_type = "gp2"
delete_on_termination = true
encrypted = true
}
}
I'm experimenting with terraform to deploy k8s resources.
I created a mongodb deployment
provider "kubernetes" {
config_context = "kubernetes-admin#kubernetes"
}
resource "kubernetes_namespace" "demo-namespace" {
metadata {
name = "my-demo-namespace"
}
}
// mongodb
resource "kubernetes_deployment" "mongodb" {
metadata {
name = "mongodb"
namespace = kubernetes_namespace.demo-namespace.metadata[0].name
labels = {
app = "mongodb"
}
}
spec {
replicas = 1
selector {
match_labels = {
app = "mongodb"
}
}
template {
metadata {
labels = {
app = "mongodb"
}
}
spec {
container {
image = "mongo"
name = "mongodb"
env_from {
secret_ref {
name = kubernetes_secret.scrt-mongodb.metadata[0].name
}
config_map_ref {
name = kubernetes_config_map.cm-mongodb.metadata[0].name
}
}
resources {
limits {
cpu = "500m"
memory = "1Gi"
}
requests {
cpu = "150m"
memory = "256Mi"
}
}
liveness_probe {
exec {
command = ["bash", "-c", "mongo -u $MONGO_INITDB_ROOT_USERNAME -p $MONGO_INITDB_ROOT_PASSWORD --eval db.adminCommand(\"ping\")"]
}
initial_delay_seconds = 3
period_seconds = 1
}
}
}
}
}
}
// mongodb configmap
resource "kubernetes_config_map" "cm-mongodb" {
metadata {
name = "cm-mongodb"
namespace = kubernetes_namespace.demo-namespace.metadata.0.name
}
// improve creds with secret
data = {
MONGO_INITDB_DATABASE = "movies"
}
}
// monbodb secret
resource "kubernetes_secret" "scrt-mongodb" {
metadata {
name = "mongodb-creds"
}
data = {
MONGO_INITDB_ROOT_USERNAME = "root-user"
MONGO_INITDB_ROOT_PASSWORD = "secret"
}
type = "opaque"
}
This fails with:
kubernetes_config_map.cm-mongodb: Creation complete after 0s [id=my-demo-namespace/cm-mongodb]
kubernetes_deployment.mongodb: Creating...
Error: Failed to create deployment: Deployment.apps "mongodb" is invalid: spec.template.spec.containers[0].envFrom: Invalid value: "": may not have more than one field specified at a time
on template.tf line 12, in resource "kubernetes_deployment" "mongodb":
12: resource "kubernetes_deployment" "mongodb" {
What is wrong here?
You missed this line:
namespace = kubernetes_namespace.demo-namespace.metadata.0.name
You did not define the resource in the desired namespace so terraform failed to "find" the desired value.
// monbodb secret
resource "kubernetes_secret" "scrt-mongodb" {
metadata {
name = "mongodb-creds"
# -------------------------------------------------------------
# -------------------------------------------------------------
# Add the namespace here
namespace = kubernetes_namespace.demo-namespace.metadata.0.name
# -------------------------------------------------------------
# -------------------------------------------------------------
}
data = {
MONGO_INITDB_ROOT_USERNAME = "root-user"
MONGO_INITDB_ROOT_PASSWORD = "secret"
}
type = "opaque"
}