nginx ingress controller deployed through terraform fails - kubernetes

Question

I am using terraform to deploy a Kubernetes cluster with an nginx ingress controller which is hosted on EKS. The controller pod however fails and enters a CrashLoopBackOff status.
Based on the logs, it seems that the ingress controller is failing to watch and list *v1beta1.Ingress resources. The error message "failed to list *v1beta1.Ingress: the server could not find the requested resource" is repeated multiple times in the logs.
Is there a solution to this that I can apply in my terraform template?
main.tf
provider "kubernetes" {
host = data.aws_eks_cluster.default.endpoint
cluster_ca_certificate = base64decode(data.aws_eks_cluster.default.certificate_authority[0].data)
token = data.aws_eks_cluster_auth.default.token
config_path = "~/.kube/config"
}
provider "helm" {
kubernetes {
host = data.aws_eks_cluster.default.endpoint
cluster_ca_certificate = base64decode(data.aws_eks_cluster.default.certificate_authority[0].data)
token = data.aws_eks_cluster_auth.default.token
}
}
resource "local_sensitive_file" "kubeconfig" {
content = templatefile("${path.module}/kubeconfig.tpl", {
cluster_name = var.cluster_name,
clusterca = data.aws_eks_cluster.default.certificate_authority[0].data,
endpoint = data.aws_eks_cluster.default.endpoint,
})
filename = "./kubeconfig-${var.cluster_name}"
}
resource "kubernetes_namespace" "test" {
metadata {
name = "test"
}
}
resource "helm_release" "nginx_ingress" {
namespace = kubernetes_namespace.test.metadata.0.name
wait = true
timeout = 600
name = "ingress-nginx"
repository = "https://kubernetes.github.io/ingress-nginx"
chart = "ingress-nginx"
version = "3.30.0"
}
Error:
Warning: Helm release "ingress-nginx" was created but has a failed status. Use the `helm` command to investigate the error, correct it, then run Terraform again.
│
│ with helm_release.nginx_ingress,
│ on main.tf line 39, in resource "helm_release" "nginx_ingress":
│ 39: resource "helm_release" "nginx_ingress" {
│
╵
╷
│ Error: timed out waiting for the condition
│
│ with helm_release.nginx_ingress,
│ on main.tf line 39, in resource "helm_release" "nginx_ingress":
│ 39: resource "helm_release" "nginx_ingress" {
Logs:
➜ ~ kubectl get pods --namespace=test
NAME READY STATUS RESTARTS AGE
ingress-nginx-controller-554bcbbc94-d89bh 0/1 CrashLoopBackOff 13 (88s ago) 37m
➜ ~ kubectl logs -f ingress-nginx-controller-554bcbbc94-d89bh --namespace=test
-------------------------------------------------------------------------------
NGINX Ingress controller
Release: v0.46.0
Build: 6348dde672588d5495f70ec77257c230dc8da134
Repository: https://github.com/kubernetes/ingress-nginx
nginx version: nginx/1.19.6
-------------------------------------------------------------------------------
I0210 15:15:37.180002 7 flags.go:208] "Watching for Ingress" class="nginx"
W0210 15:15:37.180037 7 flags.go:213] Ingresses with an empty class will also be processed by this Ingress controller
W0210 15:15:37.180266 7 client_config.go:614] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I0210 15:15:37.180719 7 main.go:241] "Creating API client" host="https://XXX:443"
I0210 15:15:37.198460 7 main.go:285] "Running in Kubernetes cluster" major="1" minor="24+" git="v1.24.8-eks-ffeb93d" state="clean" commit="abb98ec0631dfe573ec5eae40dc48fd8f2017424" platform="linux/amd64"
I0210 15:15:37.203187 7 main.go:87] "Valid default backend" service="test/ingress-nginx-defaultbackend"
I0210 15:15:37.985389 7 main.go:105] "SSL fake certificate created" file="/etc/ingress-controller/ssl/default-fake-certificate.pem"
I0210 15:15:37.989186 7 main.go:115] "Enabling new Ingress features available since Kubernetes v1.18"
W0210 15:15:37.991201 7 main.go:127] No IngressClass resource with name nginx found. Only annotation will be used.
I0210 15:15:38.007946 7 ssl.go:532] "loading tls certificate" path="/usr/local/certificates/cert" key="/usr/local/certificates/key"
I0210 15:15:38.038537 7 nginx.go:254] "Starting NGINX Ingress controller"
I0210 15:15:38.048874 7 event.go:282] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"test", Name:"ingress-nginx-controller", UID:"b764504c-fcef-478e-aae3-684821ee8568", APIVersion:"v1", ResourceVersion:"54262", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap test/ingress-nginx-controller
E0210 15:15:39.148154 7 reflector.go:138] k8s.io/client-go#v0.20.2/tools/cache/reflector.go:167: Failed to watch *v1beta1.Ingress: failed to list *v1beta1.Ingress: the server could not find the requested resource
E0210 15:15:40.495320 7 reflector.go:138] k8s.io/client-go#v0.20.2/tools/cache/reflector.go:167: Failed to watch *v1beta1.Ingress: failed to list *v1beta1.Ingress: the server could not find the requested resource
E0210 15:15:42.544137 7 reflector.go:138] k8s.io/client-go#v0.20.2/tools/cache/reflector.go:167: Failed to watch *v1beta1.Ingress: failed to list *v1beta1.Ingress: the server could not find the requested resource
E0210 15:15:46.797783 7 reflector.go:138] k8s.io/client-go#v0.20.2/tools/cache/reflector.go:167: Failed to watch *v1beta1.Ingress: failed to list *v1beta1.Ingress: the server could not find the requested resource
E0210 15:15:54.929779 7 reflector.go:138] k8s.io/client-go#v0.20.2/tools/cache/reflector.go:167: Failed to watch *v1beta1.Ingress: failed to list *v1beta1.Ingress: the server could not find the requested resource
^C
➜ ~ kubectl describe pod ingress-nginx-controller-554bcbbc94-d89bh --namespace=test
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 42m default-scheduler Successfully assigned test/ingress-nginx-controller-554bcbbc94-d89bh to ip-XXX.eu-west-1.compute.internal
Normal Killing 41m kubelet Container controller failed liveness probe, will be restarted
Normal Pulled 41m (x2 over 42m) kubelet Container image "k8s.gcr.io/ingress-nginx/controller:v0.46.0#sha256:52f0058bed0a17ab0fb35628ba97e8d52b5d32299fbc03cc0f6c7b9ff036b61a" already present on machine
Normal Created 41m (x2 over 42m) kubelet Created container controller
Normal Started 41m (x2 over 42m) kubelet Started container controller
Warning Unhealthy 22m (x46 over 41m) kubelet Liveness probe failed: HTTP probe failed with statuscode: 500
Warning Unhealthy 7m4s (x94 over 41m) kubelet Readiness probe failed: HTTP probe failed with statuscode: 500
Warning BackOff 2m7s (x118 over 35m) kubelet Back-off restarting failed container

Answer 1

Your Ingress-nginx Helm chart is seems outdated as it is trying to find Ingress resources with API version v1beta1.Ingress which have been deprecated in newer k8s versions and moved to networking.k8s.io/v1.
So you need to update your helm-chart.