I am a bit confused with the way how IngressClass works. I moved all annotations for ALB to IngressClass and made it the default one, however, I noticed that load balancer cannot be created as the certificate couldn't be found.
Default IngressClass:
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
annotations:
alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:ap-southeast-2:000045211111:certificate/ee65c0af-044b-4c48-abc6-b4b44d4a3c76
alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/target-type: ip
alb.ingress.kubernetes.io/wafv2-acl-arn: arn:aws:wafv2:ap-southeast-2:000045211111:regional/webacl/waf-api-regional-1e3042/d495cc4f-b24f-4465-afb4-ae5df32acb56
ingressclass.kubernetes.io/is-default-class: "true"
labels:
app.kubernetes.io/component: controller
name: alb-default
spec:
controller: ingress.k8s.aws/alb
When I move all these annotations to Ingress itself, the load balancer can be created successfully. I thought that annotations are taken from IngressClass and applied to Ingress itself when it is created.
I managed IngressClass from the terraform and populate these values during the infra provisioning so that I don't need to copy ARNs for the resources again and provide them when deploy service to k8s with Helm.
Am I missing anything? Is there any way to fix this?
Thank you.
https://github.com/kubernetes-sigs/aws-load-balancer-controller/pull/2963
solved this issue in this merge request
Related
We are deploying azure ingress.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: name1
annotations:
kubernetes.io/ingress.class: nginx
spec:
rules:
If we do not explicitly mention ingress class(kubernetes.io/ingress.class:) in the manifest, what would be the default ingress controller type?
Nginx documentation says default as N/A.
https://docs.nginx.com/nginx-ingress-controller/configuration/ingress-resources/advanced-configuration-with-annotations/
If you do not specify any ingress class explicitly, the value is omitted. You can specify a default ingress class for your cluster as you can read here.
You should also migrate to the IngressClassName field as the annotation is deprecated.
I deployed Istio using the operator and added a custom ingress gateway which is only accessible from a certain source range (our VPN).
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
namespace: istio-system
name: ground-zero-ingressgateway
spec:
profile: empty
components:
ingressGateways:
- name: istio-ingressgateway
enabled: true
- name: istio-vpn-ingressgateway
label:
app: istio-vpn-ingressgateway
istio: vpn-ingressgateway
enabled: true
k8s:
serviceAnnotations:
...
service:
loadBalancerSourceRanges:
- "x.x.x.x/x"
Now I want to configure Istio to expose a service outside of the service mesh cluster, using the Kubernetes Ingress resource. I use the kubernetes.io/ingress.class annotation to tell the Istio gateway controller that it should handle this Ingress.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: my-ingress
annotations:
kubernetes.io/ingress.class: istio
spec:
...
Kubernetes version (EKS): 1.19
Istio version: 1.10.3
Which ingress gateway controller is now used (istio-ingressgateway or istio-vpn-ingressgateway)? Is there a way to specify which one should be used?
P.S. I know that I could create a VirtualService and specify the correct gateway but we want to write a manifest that also works without Istio by specifying the correct ingress controller with an annotation.
You can create an ingress class that references the ingress controller that is deployed by default in the istio-system namespace. This configuration with ingress will work, however to my current knowledge, this is only used for backwards compatibility. If you want to use istio ingress controller functionality, you should use istio gateway and virtual service instead:
Using the Istio Gateway, rather than Ingress, is recommended to make use of the full feature set that Istio offers, such as rich traffic management and security features.
If this solution is not optimal for you, you should use e.g. nginx ingress controller and you can still bind it with annotations (deprecated) or using IngressClass. To my present knowledge it is not possible to bind this ingress class with an additional ingress controller. If you need an explanation, documentation, you should create an issue on github.
Summary: The recommended option is to use the gateway with virtual service. Another possibility is to use nginx alone ingress with different classes and an ingress resource for them.
I am setting up my ingress controller, ingress class and ingress to expose a service outside the cluster. This is fresh cluster setup.
I have setup the nginx-ingress controller using
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.41.0/deploy/static/provider/baremetal/deploy.yaml
The next step based on my understanding is to create the ingress class https://v1-18.docs.kubernetes.io/docs/concepts/services-networking/ingress/#ingress-class
apiVersion: networking.k8s.io/v1beta1
kind: IngressClass
metadata:
name: external-lb
spec:
controller: example.com/ingress-controller
parameters:
apiGroup: k8s.example.com/v1alpha
kind: IngressParameters
name: external-lb
How did they get the name of the controller example.com/ingress-controller?
I have run multiple scenarios with IngressClass, Ingress and Nginx Ingress Controller.
Scenario 1
IngressClass with custom name
Nginx Ingress Controller with default --ingress-class value which is nginx
Ingress using ingressClassName same as IngressClass name
Output: Response 404
Scenario 2
IngressClass with custom name
Nginx Ingress Controller with owningress-class ingress-test
Ingress using ingressClassName same as IngressClass name
Output: Response 404
Scenario 3
IngressClass with test name
Nginx Ingress Controller --ingress-class with value test
Ingress using test in ingressClassName
Output: Proper response
Senario 4
IngressClass with nginx name
Nginx Ingress Controller --ingress-class with value nginx
Ingress using nginx in ingressClassName
Output: Proper response
Conclusion
First of all, please keep in mind that there are 3 types of Nginx. Open Source Nginx Ingress Controller, you are probably using it. Nginx Incorporaton (nginx inc) and Nginx Incorporaton Plus.
In one of the scenarios, when I have used spec.controller: nginx.org/ingress-controller with Nginx Ingress Controller with argument --ingress-class=nginx, in Nginx Ingress Controller pod you will see entry which is pointing to k8s.io/ingress-nginx.
To reproduce this behavior, you will need to deploy IngressClass with specific controller and then deploy nginx.
apiVersion: networking.k8s.io/v1beta1
kind: IngressClass
metadata:
name: nginx
spec:
controller: nginx.org/ingress-controller
After deploying Nginx Ingress Controller, controller pod will be in CrashLoopBackOff state. In logs you will find entry:
E1118 15:42:19.008911 8 main.go:134] Invalid IngressClass (Spec.Controller) value "nginx.org/ingress-controller". Should be "k8s.io/ingress-nginx"
It works only when IngressClass name is set to nginx.
I would say that nginx.org/ingress-controller is for Nginx Incorporated and k8s.io/ingress-nginx for Open Source Nginx Ingress.
If custom value is used for --ingress-class argument in the controller Deployment manifest, presence or absence of IngressClass object with the same name doesn't made any difference in, how the cluster works, if only you keep Ingress spec.ingressClass value the same with controller argument. Moreover, if it's present, IngressClass spec.controller can have any value that match the required pattern "domain like" and that didn't affect Ingress workflow behavior on my cluster at all.
In addition, Ingress works fine if I put the correct value of the ingress-class either to spec.ingressClass property or to metadata.annotation.kubernetes.io/ingress.class accordingly. It gives an error like the following if you try to put both values to the same Ingres object:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: test-ingress
annotations:
kubernetes.io/ingress.class: nginx
spec:
ingressClassName: nginx
The Ingress "test-ingress" is invalid: annotations.kubernetes.io/ingress.class: Invalid value: "nginx": can not be set when the class field is also set
Please keep in mind it was tested only for Nginx Ingress Controlle. If you would like to use IngressClass with other Ingress Controllers like Traefik or Ambasador, you would check their release notes.
You will create the IngressClass as part of the Installation with Manifests steps (Step 3 here). That will create an IngressClass that looks like:
apiVersion: networking.k8s.io/v1beta1
kind: IngressClass
metadata:
name: nginx
# annotations:
# ingressclass.kubernetes.io/is-default-class: "true"
spec:
controller: nginx.org/ingress-controller
Then, to configure an Ingress resource to be consumed by nginx, just specify ingressClassName: nginx in the Ingress spec, as shown here, and pasted again below:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: cafe-ingress
spec:
ingressClassName: nginx
tls:
- hosts:
- cafe.example.com
secretName: cafe-secret
rules:
- host: cafe.example.com
. . .
I have gone through kong-ingress-controller deployment and getting started doc and done everything mentioned.
Update User Permissions
Deploy Kong Ingress Controller
Setup environment variables
Created Ingress with Routes
Everything works fine, I can access my applications based on the routes. But when I added rate-limit plugins or any other plugins it does not take any effect.
ingress.yaml :
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: my-ingress
namespace: default
annotations:
kubernetes.io/ingress.class: kong
plugins.konghq.com: http-ratelimit, http-auth
spec:
rules:
- host: foo.bar
http:
paths:
- path: /users
backend:
serviceName: my-service
servicePort: 80
rate-limit.yaml :
apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
name: http-ratelimit
labels:
global: 'true'
config:
minute: 5
plugin: rate-limiting
But the rate limit plugin has no effect on my ingress.
NB: The kong-ingress-controller is in kong namespace but the other resources are in default namespace. I tried to move everything to kong namespace then the plugins works but service does not work as it is in default namespace.
Thanks in advance.
Looking at the Kong docs, the rate-limit YAML looks correct. If the resource is configured correctly, Kong is not matching the request against the ingress resource because the user is not sending the correct request.
KongPlugin, KongIngress should be in same namespace as Service. YAML provides looks correct.
There must be something wrong in ingress yamls annotation and configuration.Is your service annotated with Ingress object?
I think you need to add this annotation to your KongPlugin:
annotations:
kubernetes.io/ingress.class: kong
So try with
apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
name: http-ratelimit
annotations:
kubernetes.io/ingress.class: kong
[...]
In my scenario, I wanted to apply the KongPlugin on a specific Ingress Resource/Route.
What worked for me was to create the KongPlugin object in the same namespace where the Ingress Resource (and therefore, the target service) lived.
I have a setup of kubernetes on a coreos baremetal.
For now I did the connection from outside world to service with a nginx reverse-proxy.
I'm trying the new Ingress resource.
for now I have added a simple ingress:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: kube-ui
spec:
backend:
serviceName: kube-ui
servicePort: 80
that starts like this:
INGRESS
NAME RULE BACKEND ADDRESS
kube-ui - kube-ui:80
My question is how to connect from the outside internet to that ingress point as this resource have no ADDRESS ... ?
POSTing this to the API server will have no effect if you have not configured an Ingress controller. You need to choose the ingress controller implementation that is the best fit for your cluster, or implement one. Examples and instructions can be found here.
check this gist
This is for the ingress-nginx, not kubernetes-ingress
Pre-requirement
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/mandatory.yaml
Exposing hostNetwork (hope you know what you are doing. As documented, other than this, you can use nodePort or loadbalancer.)
kubectl edit deployment.apps/nginx-ingress-controller -n ingress-nginx
add
template:
spec:
hostNetwork: true
port forwarding
apiVersion: v1
kind: ConfigMap
metadata:
name: tcp-services
namespace: ingress-nginx
data:
9000: "default/example-go:8080"
Also, you can use ingress object to expose the service