I want to change the IP address of my LoadBalancer ingress-nginx-controller in Google Cloud. I have now assigned the IP address via LoadBalancer. See the screenshot. Unfortunately it is not adopted in GKE. Why? Is that a bug?
GKE lb IP address change
I have verified this on my GKE test cluster.
When you Reserving a static external IP address it isn't assigned to any of your VMs. Depends on how you created cluster/reserved ip (standard or premium) you can get error like below:
Error syncing load balancer: failed to ensure load balancer: failed to create forwarding rule for load balancer (a574130f333b143a2a62281ef47c8dbb(default/nginx-ingress-controller)): googleapi: Error 400: PREMIUM network tier (the project's default network tier) is not supported: The network tier of specified IP address is STANDARD, that of Forwarding Rule must be the same., badRequest
In this scenario I've used cluster based in us-central-1c and reserved IP as Network Service Tier: Premium, Type: Regional and used region where my cluster is based - us-central-1. My ExternalIP: 34.66.79.1X8
NOTE Reserved IP must be in the same reagion as your cluster
Option 1: - Use Helm chart
Deploy Nginx
helm install nginx-ingress stable/nginx-ingress --set controller.service.loadBalancerIP=34.66.79.1X8,rbac.create=true
Service output:
$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.8.0.1 <none> 443/TCP 5h49m
nginx-ingress-controller LoadBalancer 10.8.5.158 <pending> 80:31898/TCP,443:30554/TCP 27s
nginx-ingress-default-backend ClusterIP 10.8.13.209 <none> 80/TCP 27s
Service describe output:
$ kubectl describe svc nginx-ingress-controller
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal EnsuringLoadBalancer 32s service-controller Ensuring load balancer
Normal EnsuredLoadBalancer 5s service-controller Ensured load balancer
Final output:
$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.8.0.1 <none> 443/TCP 5h49m
nginx-ingress-controller LoadBalancer 10.8.5.158 34.66.79.1X8 80:31898/TCP,443:30554/TCP 35s
nginx-ingress-default-backend ClusterIP 10.8.13.209 <none> 80/TCP 35s
Option 2 - Editing Nginx YAMLs before deploying Nginx
As per docs:
Initialize your user as a cluster-admin with the following command:
kubectl create clusterrolebinding cluster-admin-binding \
--clusterrole cluster-admin \
--user $(gcloud config get-value account)
Download YAML
$ wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.35.0/deploy/static/provider/cloud/deploy.yaml
Edit LoadBalancer service and add loadBalancerIP: <your-reserved-ip> like below:
# Source: ingress-nginx/templates/controller-service.yaml
apiVersion: v1
kind: Service
metadata:
labels:
helm.sh/chart: ingress-nginx-2.13.0
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 0.35.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: controller
name: ingress-nginx-controller
namespace: ingress-nginx
spec:
type: LoadBalancer
loadBalancerIP: 34.66.79.1x8 #This line
externalTrafficPolicy: Local
ports:
Deploy it kubectl apply -f deploy.yaml. Service output below:
$ kubectl get svc -A
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default kubernetes ClusterIP 10.8.0.1 <none> 443/TCP 6h6m
ingress-nginx ingress-nginx-controller LoadBalancer 10.8.5.165 <pending> 80:31226/TCP,443:31161/TCP 17s
ingress-nginx ingress-nginx-controller-admission ClusterIP 10.8.9.216 <none> 443/TCP 18s
6h6m
...
Describe output:
$ kubectl describe svc ingress-nginx-controller -n ingress-nginx
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal EnsuringLoadBalancer 40s service-controller Ensuring load balancer
Normal EnsuredLoadBalancer 2s service-controller Ensured load balancer
Service with reserved IP:
$ kubectl get svc -n ingress-nginx
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx-controller LoadBalancer 10.8.5.165 34.66.79.1X8 80:31226/TCP,443:31161/TCP 2m22s
ingress-nginx-controller-admission ClusterIP 10.8.9.216 <none> 443/TCP 2m23s
In Addition
Also please keep in mind that you should add annotations: kubernetes.io/ingress.class: nginx in your ingress resource when you want force GKE to use Nginx Ingress features, like rewrite.
Related
I deployed a (LoadBalancer) service for a pod on my minikube cluster, then exposed it via minikube service [my_service] command. Now I tried to "turn off the exposure" but couldn't find any way to do this, except deleting it what I would like to avoid. Is it possible to just turn off the exposure, not deleting (and redeploying) the existing already exposed service?
Background
In Kubernetes documentation regarding ServiceTypes, you can find information that if you want to expose your cluster outside you have to use NodePort or LoadBalancer.
If you want to keep your service/application in cluster, you should use ClusterIP:
Exposes the Service on a cluster-internal IP. Choosing this value makes the Service only reachable from within the cluster. This is the default ServiceType.
Depends on your version, you can edit it or use workaround. For example in K8s 1.16 you won't be able as some errors might occurs.
$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.8.0.1 <none> 443/TCP 3d20h
my-nginx LoadBalancer 10.8.14.224 34.121.77.108 80:32039/TCP 3m16s
$ kubectl patch service my-nginx -p '{"spec":{"type":"ClusterIP"}}'
The Service "my-nginx" is invalid: spec.ports[0].nodePort: Forbidden: may not be used when `type` is 'ClusterIP'
Solutions
However as you are using Minikube so I guess you are using newer version (1.20), so you can change it using:
kubectl edit
kubectl edit svc <yourSvcName> and change type to ClusterIP.
It will open Vi editor, where you can change spec.type to ClusterIP or just remove it as default type for Kubernetes service is ClusterIP so if it won't be specified, Kubernetes will automatically use ClusterIP.
$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 25s
my-nginx LoadBalancer 10.107.129.201 <pending> 80:30173/TCP 8s
minikube-new:~$ kubectl edit svc my-nginx
service/my-nginx edited
sekreta#minikube-new:~$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 46s
my-nginx ClusterIP 10.107.129.201 <none> 80/TCP 29s
kubectl patch
$ kubectl patch service <yourServiceName> -p '{"spec":{"type":"ClusterIP"}}'
$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 10m
my-nginx LoadBalancer 10.107.129.201 <pending> 80:30456/TCP 2m
minikube-new:~$ kubectl patch service my-nginx -p '{"spec":{"type":"ClusterIP"}}'
service/my-nginx patched
minikube-new:~$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 10m
my-nginx ClusterIP 10.107.129.201 <none> 80/TCP 3m
kubectl apply
You can edit your Yaml and remove spec.type or have 2 Yamls with ClusterIP and LoadBalancer and switch them depends on your needs.
$ kubectl apply -f svc.yaml
service/my-nginx configured
minikube-new:~$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 22m
my-nginx ClusterIP 10.111.237.133 <none> 80/TCP 16s
Use some 3rd party software to make changes in your cluster like Helm and using templates.
Using nodePort service with ingress, I success expose the service to the out world.
--- service
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)
default kubernetes ClusterIP 10.96.0.1 <none> 443/TCP
default postgres ClusterIP 10.106.182.170 <none> 5432/TCP
default user-api NodePort 10.99.12.136 <none> 3000:32099/TCP
ingress-nginx ingress-nginx NodePort 10.110.104.0 <none> 80:31691/TCP,443:30593/TCP
--- ingress
NAME HOSTS ADDRESS PORTS AGE
app-ingress example.com 10.110.104.0 80 3h27m
The rule of ingress show below.
Host Path Backends
---- ---- --------
example.com
/user-api user-api:3000 (172.16.117.201:3000)
If my user-api have a restful api /v1/health interface, how to access this api inside and outside server?
From the inside, http://user-api.default:3000/user-api. From the outside, use any node external IP (see kubectl get node -o wide for a list of them).
I've applied the yaml for the kubernetes dashboard.
Now I want to expose this service with the public IP of my server: https://kubernetes.io/docs/tutorials/stateless-application/expose-external-ip-address/#objectives
But there is no service/deployment on my cluster:
$ sudo kubectl get services kubernetes
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 63d
$ sudo kubectl get deployment
NAME READY UP-TO-DATE AVAILABLE AGE
What did I do wrong?
Thanks for the help
The command that you ran is fetching objects in default namespace.
However, Dashboard is deployed on kube-system namespace.
kubectl -n kube-system get services kubernetes
kubectl -n kube-system get deployment
I am giving you this info according to the link that you share kubernetes dashboard . And namely the YAML file
Oky thanks, now I get the right name:
sudo kubectl -n kube-system get deployment
NAME READY UP-TO-DATE AVAILABLE AGE
calico-kube-controllers 1/1 1 1 63d
coredns 2/2 2 2 63d
kubernetes-dashboard 1/1 1 1 103m
tiller-deploy 0/1 1 0 63d
But I still can't expose the service
sudo kubectl expose deployment kubernetes-dashboard
Error from server (NotFound): deployments.extensions "kubernetes-dashboard" not found
As mentionned here
SO, to reproduce and show how does it works - I spawned new fresh cluster on GKE.
Lets see what we have after applying dashboard yaml:
kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.1/src/deploy/recommended/kubernetes-dashboard.yaml
secret/kubernetes-dashboard-certs created
serviceaccount/kubernetes-dashboard created
role.rbac.authorization.k8s.io/kubernetes-dashboard-minimal created
rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard-minimal created
deployment.apps/kubernetes-dashboard created
service/kubernetes-dashboard created
kubectl get deployment kubernetes-dashboard -n kube-system
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
kubernetes-dashboard 1 1 1 1 3m22s
kubectl get services kubernetes-dashboard -n kube-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes-dashboard ClusterIP 10.0.6.26 <none> 443/TCP 5m1
kubectl describe service kubernetes-dashboard -n kube-system
Name: kubernetes-dashboard
Namespace: kube-system
Labels: k8s-app=kubernetes-dashboard
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"k8s-app":"kubernetes-dashboard"},"name":"kubernetes-dashboard"...
Selector: k8s-app=kubernetes-dashboard
Type: ClusterIP
IP: 10.0.6.26
Port: <unset> 443/TCP
TargetPort: 8443/TCP
Endpoints: 10.40.1.5:8443
Session Affinity: None
Events: <none>
During this deployment:
1) kubernetes-dashboard deployment has been created. Note that it was created with the k8s-app=kubernetes-dashboard label.
2) kubernetes-dashboard service was created and works using k8s-app=kubernetes-dashboard [selector](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/.
So basically when you receive such an error - this is expected. Because kubectl expose deployment kubernetes-dashboard -n kube-system is trying to create new service with the kubernetes-dashboard name.
Just to play with it - you can easily expose the same, but use another service names, for example:
kubectl expose deployment kubernetes-dashboard -n kube-system --name kube-dashboard-service2
service/kube-dashboard-service2 exposed
Note that default kubernetes-dashboard service is created using ClusterIP type - so you are able right now to access it
1) withing the cluster
2) using kubectl proxy from local machine
$ kubectl proxy
In browser: http://localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/
If you want to expose the same - you can use:
1) Ingress
2) Nodeport service type
In 2 words: edit clusterIP --> Nodeport type while kubectl -n kube-system edit service kubernetes-dashboard and access dashboard using https://[node_ip]:[port]
More detailed article is here: How To Access Kubernetes Dashboard Externally
3) Loadbalancer service type. This is Cloud specific feature, so it will work only with cloud providers
Traffic from the external load balancer is directed at the backend
Pods. The cloud provider decides how it is load balanced.
Some cloud providers allow you to specify the loadBalancerIP. In those
cases, the load-balancer is created with the user-specified
loadBalancerIP. If the loadBalancerIP field is not specified, the
loadBalancer is set up with an ephemeral IP address. If you specify a
loadBalancerIP but your cloud provider does not support the feature,
the loadbalancerIP field that you set is ignored.
When I run kubectl get svc -n kube-system it tells me:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kube-dns ClusterIP xx.xx.xx.xx <none> 53/UDP,53/TCP 13h
But when I try to kubectl edit svc/kube-dns -n kube-system:
error: services "kube-dns" is invalid
A copy of your changes has been stored to "/tmp/kubectl-edit-4p5gn.yaml"
error: Edit cancelled, no valid changes were saved.
I am unable to change it to a LoadBalancer...any ideas?
I also tried to create a new kube-dns also but I am unable to get an external-ip; it stays stuck in pending state.
kind: Service
metadata:
name: kubedns-bkp
namespace: kube-system
labels:
k8s-app: kube-dns
spec:
type: LoadBalancer
ports:
- port: 53
protocol: UDP
selector:
k8s-app: kube-dns
kubectl get svc -n kube-system reports:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubedns-bkp LoadBalancer xx.xx.xx.xx <pending> 53:32115/UDP 5h
Note: I have created k8s cluster with ELB integration, for other services I successfully get external IPs.
So, two things here:
As they advised you in the yaml validation errors that you chose not to share with us, one cannot change the type: of an existing Service; you have to create a new one, or delete the existing one and recreate it.
However, I would strongly, strongly, strongly advise against deleting the kube-dns Service -- you are more than welcome to create a new Service of type: LoadBalancer and point it at the same selector: as kube-dns is using. That way anyone who wishes to use the load balanced service can, but the things in the cluster who depend on kube-dns being a ClusterIP with (likely) that existing xx.xx.xx.xx value can continue as before.
I've created a test k8s cluster using kubespray (3 nodes, virtualbox
centos vm based) and have been trying to follow the guide for setting up nginx ingress, but i never seem to get an external address assigned to my service:
I can see that the ingress controller is apparently installed:
[root#k8s-01 ~]# kubectl get pods --all-namespaces -l app=ingress-nginx
NAMESPACE NAME READY STATUS RESTARTS AGE
ingress-nginx nginx-ingress-controller-58c9df5856-v6hml 1/1 Running 0 28m
And following the prerequisites docs, i have set up the http-svc sample service:
[root#k8s-01 ~]# kubectl get po
NAME READY STATUS RESTARTS AGE
http-svc-794dc89f5-f2vlx 1/1 Running 0 27m
[root#k8s-01 ~]# kubectl get svc http-svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
http-svc LoadBalancer 10.233.25.131 <pending> 80:30055/TCP 27m
[root#k8s-01 ~]# kubectl describe svc http-svc
Name: http-svc
Namespace: default
Labels: app=http-svc
Annotations: <none>
Selector: app=http-svc
Type: LoadBalancer
IP: 10.233.25.131
Port: http 80/TCP
TargetPort: 8080/TCP
NodePort: http 30055/TCP
Endpoints: 10.233.65.5:8080
Session Affinity: None
External Traffic Policy: Cluster
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Type 27m service-controller ClusterIP -> LoadBalancer
As far as i know, i should see a LoadBalancer Ingress entry, but the External IP for the service still appears to be pending, so something isn't working, but i'm at a loss where to diagnose what has gone wrong
Since you are creating your cluster locally, exposing your service as type LoadBalancer will not provision a loadbalancer for you. Use the type LoadBalancer if you are creating your cluster in a cloud environment such as AWS or GKE. In AWS it will auto-provision you an loadbalancer (ELB) and assign an external ip for the service.
To make your service work with current settings and environment change your service type from Loadbalancer to NodePort.