Multiple ingress controller is not working - kubernetes

Question

I'm creating multiple ingress controller in different namespaces. Initially, it's creating a load balancer in AWS and attached pod IP addresses to target groups. After some days it is not updating the new pod IP to the target group. I've attached the ingress controller logs here.
E0712 15:02:30.516295 1 leaderelection.go:270] error retrieving resource lock namespace1/ingress-controller-leader-alb: configmaps "ingress-controller-le │
│ ader-alb" is forbidden: User "system:serviceaccount:namespace1:fc-serviceaccount-icalb" cannot get resource "configmaps" in API group "" in the namespace "namespace1"
ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: "fc-ingress"
annotations:
kubernetes.io/ingress.class: alb-namespace1
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/subnets:
alb.ingress.kubernetes.io/certificate-arn:
alb.ingress.kubernetes.io/ssl-policy:
alb.ingress.kubernetes.io/security-groups:
alb.ingress.kubernetes.io/target-type: ip
alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS": 443}]'
alb.ingress.kubernetes.io/healthcheck-protocol: HTTP
alb.ingress.kubernetes.io/healthcheck-port: traffic-port
alb.ingress.kubernetes.io/healthcheck-path: '/'
alb.ingress.kubernetes.io/healthcheck-timeout-seconds: '2'
alb.ingress.kubernetes.io/healthcheck-interval-seconds: '5'
alb.ingress.kubernetes.io/success-codes: '200'
alb.ingress.kubernetes.io/healthy-threshold-count: '5'
alb.ingress.kubernetes.io/unhealthy-threshold-count: '2'
alb.ingress.kubernetes.io/load-balancer-attributes: access_logs.s3.enabled=false
alb.ingress.kubernetes.io/load-balancer-attributes: deletion_protection.enabled=false
alb.ingress.kubernetes.io/load-balancer-attributes: routing.http2.enabled=true
alb.ingress.kubernetes.io/target-group-attributes: slow_start.duration_seconds=0
alb.ingress.kubernetes.io/target-group-attributes: deregistration_delay.timeout_seconds=300
alb.ingress.kubernetes.io/target-group-attributes: stickiness.enabled=false
labels:
app: fc-label-app-ingress
spec:
rules:
- host: "hostname1.com"
http:
paths:
- backend:
serviceName: service1
servicePort: 80
- host: "hostname2.com"
http:
paths:
- backend:
serviceName: service2
servicePort: 80
- host: "hostname3.com"
http:
paths:
- backend:
serviceName: service3
servicePort: 80
ingress_controller.yaml
# Application Load Balancer (ALB) Ingress Controller Deployment Manifest.
# This manifest details sensible defaults for deploying an ALB Ingress Controller.
# GitHub: https://github.com/kubernetes-sigs/aws-alb-ingress-controller
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: fc-label-app-icalb
name: fc-ingress-controller-alb
namespace: namespace1
# Namespace the ALB Ingress Controller should run in. Does not impact which
# namespaces it's able to resolve ingress resource for. For limiting ingress
# namespace scope, see --watch-namespace.
spec:
replicas: 1
selector:
matchLabels:
app: fc-label-app-icalb
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 1
type: RollingUpdate
template:
metadata:
creationTimestamp: null
labels:
app: fc-label-app-icalb
spec:
containers:
- args:
# Limit the namespace where this ALB Ingress Controller deployment will
# resolve ingress resources. If left commented, all namespaces are used.
- --watch-namespace=namespace1
# Setting the ingress-class flag below ensures that only ingress resources with the
# annotation kubernetes.io/ingress.class: "alb" are respected by the controller. You may
# choose any class you'd like for this controller to respect.
- --ingress-class=alb-namespace1
# Name of your cluster. Used when naming resources created
# by the ALB Ingress Controller, providing distinction between
# clusters.
- --cluster-name=$EKS_CLUSTER_NAME
# AWS VPC ID this ingress controller will use to create AWS resources.
# If unspecified, it will be discovered from ec2metadata.
# - --aws-vpc-id=vpc-xxxxxx
# AWS region this ingress controller will operate in.
# If unspecified, it will be discovered from ec2metadata.
# List of regions: http://docs.aws.amazon.com/general/latest/gr/rande.html#vpc_region
# - --aws-region=us-west-1
# Enables logging on all outbound requests sent to the AWS API.
# If logging is desired, set to true.
# - ---aws-api-debug
# Maximum number of times to retry the aws calls.
# defaults to 10.
# - --aws-max-retries=10
env:
# AWS key id for authenticating with the AWS API.
# This is only here for examples. It's recommended you instead use
# a project like kube2iam for granting access.
#- name: AWS_ACCESS_KEY_ID
# value: KEYVALUE
# AWS key secret for authenticating with the AWS API.
# This is only here for examples. It's recommended you instead use
# a project like kube2iam for granting access.
#- name: AWS_SECRET_ACCESS_KEY
# value: SECRETVALUE
# Repository location of the ALB Ingress Controller.
image: docker.io/amazon/aws-alb-ingress-controller:v1.1.4
imagePullPolicy: Always
name: server
resources: {}
terminationMessagePath: /dev/termination-log
dnsPolicy: ClusterFirst
restartPolicy: Always
securityContext: {}
terminationGracePeriodSeconds: 30
serviceAccountName: fc-serviceaccount-icalb
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
app: fc-label-app-icalb
name: fc-clusterrole-icalb
rules:
- apiGroups:
- ""
- extensions
resources:
- configmaps
- endpoints
- events
- ingresses
- ingresses/status
- services
verbs:
- create
- get
- list
- update
- watch
- patch
- apiGroups:
- ""
- extensions
resources:
- nodes
- pods
- secrets
- services
- namespaces
verbs:
- get
- list
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
labels:
app: fc-label-app-icalb
name: fc-clusterrolebinding-icalb
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: fc-clusterrole-icalb
subjects:
- kind: ServiceAccount
name: fc-serviceaccount-icalb
namespace: namespace1
---
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
app: fc-label-app-icalb
name: fc-serviceaccount-icalb
namespace: namespace1

Answer 1

I have had an issue like that on AKS. I have two Nginx Ingress Controllers:
external-nginx-ingress
internal-nginx-ingress
Only one worked at a time, Internal or external.
After specifying a unique election-id for each one the problem was fixed.
I use the following HELM chart:
Repository = "https://kubernetes.github.io/ingress-nginx"
Chart = "ingress-nginx"
Chart_version = "4.1.3"
K8s Version = "1.22.4"
Deployment
kubectl get deploy -n ingress
NAME READY UP-TO-DATE AVAILABLE
external-nginx-ingress-controller 3/3 3 3
internal-nginx-ingress-controller 1/1 1 1
IngressClass
kubectl get ingressclass
NAME CONTROLLER PARAMETERS
external-nginx k8s.io/ingress-nginx <none>
internal-nginx k8s.io/internal-ingress-nginx <none>
Deployment for External
apiVersion: apps/v1
kind: Deployment
metadata:
name: external-nginx-ingress-controller
namespace: ingress
annotations:
meta.helm.sh/release-name: external-nginx-ingress
meta.helm.sh/release-namespace: ingress
spec:
replicas: 3
selector:
matchLabels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: external-nginx-ingress
app.kubernetes.io/name: ingress-nginx
template:
spec:
containers:
- name: ingress-nginx-external-controller
image: >-
k8s.gcr.io/ingress-nginx/controller:v1.2.1
args:
- /nginx-ingress-controller
- >-
--publish-service=$(POD_NAMESPACE)/external-nginx-ingress-controller
- '--election-id=external-ingress-controller-leader'
- '--controller-class=k8s.io/ingress-nginx'
- '--ingress-class=external-nginx'
- '--ingress-class-by-name=true'
Deployment for Internal
apiVersion: apps/v1
kind: Deployment
metadata:
name: internal-nginx-ingress-controller
namespace: ingress
annotations:
meta.helm.sh/release-name: internal-nginx-ingress
meta.helm.sh/release-namespace: ingress
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: internal-nginx-ingress
app.kubernetes.io/name: ingress-nginx
template:
spec:
containers:
- name: ingress-nginx-internal-controller
image: >-
k8s.gcr.io/ingress-nginx/controller:v1.2.1
args:
- /nginx-ingress-controller
- >-
--publish-service=$(POD_NAMESPACE)/internal-nginx-ingress-controller
- '--election-id=internal-ingress-controller-leader'
- '--controller-class=k8s.io/internal-ingress-nginx'
- '--ingress-class=internal-nginx'
- '--ingress-class-by-name=true'