Set static external IP for my load balancer on GKE - kubernetes

I am trying to set up a static external IP for my load balancer on GKE but having no luck. Here is my Kubernetes service config file:
kind: Service
apiVersion: v1
metadata:
name: myAppService
spec:
selector:
app: myApp
ports:
- protocol: TCP
port: 3001
targetPort: 3001
type: LoadBalancer
loadBalancerIP: *********
This doesn't work. I expect to see my external IP as ********* but it just says pending:
➜ git:(master) kubectl get services
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ********* <none> 443/TCP 5m
myAppService ********* <pending> 3001:30126/TCP 5m
More details:
➜ git:(master) kubectl describe services
Name: kubernetes
Namespace: default
Labels: component=apiserver
provider=kubernetes
Annotations: <none>
Selector: <none>
Type: ClusterIP
IP: *********
Port: https 443/TCP
Endpoints: *********
Session Affinity: ClientIP
Events: <none>
Name: myAppService
Namespace: default
Labels: <none>
Annotations: <none>
Selector: app=myApp
Type: LoadBalancer
IP: *********
Port: <unset> 3001/TCP
NodePort: <unset> 30126/TCP
Endpoints:
Session Affinity: None
Events:
FirstSeen LastSeen Count From SubObjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
5m 20s 7 service-controller Normal CreatingLoadBalancer Creating load balancer
5m 19s 7 service-controller Warning CreatingLoadBalancerFailed Error creating load balancer (will retry): Failed to create load balancer for service default/myAppService: Cannot EnsureLoadBalancer() with no hosts
Any ideas?

I've encountered the same problem, but after reading the docs carefully, it turned out that I was just reserving the static IP incorrectly.
A service of type LoadBalancer creates a network load balancer, which is regional. Therefore, also the static IP address you reserve needs to be regional also (in the regoin of your cluster).
When I changed to this solution, everything worked fine for me...

This got me stuck as well, I hope someone finds this helpful.
In addition to what Dirk said, if you happen to reserve a global static IP address as oppose to a regional one; you need to use Ingres as describe here in documentation: Configuring Domain Names with Static IP Addresses specifically step 2b.
So basically you reserve the static ip gcloud compute addresses create helloweb-ip --global
and add an Ingres:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: helloweb
# this is where you you add your reserved ip
annotations:
kubernetes.io/ingress.global-static-ip-name: helloweb-ip
labels:
app: hello
spec:
backend:
serviceName: helloweb-backend
servicePort: 8080
---
apiVersion: v1
kind: Service
metadata:
name: helloweb-backend
labels:
app: hello
spec:
type: NodePort
selector:
app: hello
tier: web
ports:
- port: 8080
targetPort: 8080
The doc also describe how to assign a static ip if you choose type "LoadBalancer" under step 2a.

Related

Why can't load balancer connect to service in GKE?

I am deploying an application on GKE cluster and try to deploy a load balancer to make clients able to call this application.
My application spec is:
apiVersion: apps/v1
kind: Deployment
metadata:
name: api
namespace: default
spec:
replicas: 1
selector:
matchLabels:
name: api
template:
metadata:
labels:
name: api
spec:
serviceAccountName: docker-sa
containers:
- name: api
image: zhaoyi0113/rancher-go-api
ports:
- containerPort: 8080
apiVersion: v1
kind: Service
metadata:
name: api
annotations:
cloud.google.com/neg: '{"ingress": true}'
spec:
selector:
name: api
ports:
- port: 80
targetPort: 8080
protocol: TCP
type: NodePort
It listens on the port 8080 and a service open port 80 and use the targetPort 8080 to connect to the application.
And I have a ingress spec:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: sidecar
namespace: default
spec:
defaultBackend:
service:
name: api
port:
number: 80
After deploy, I am able to see the ip address from kubectl get ingress. But when I send a request to the ip, I got 502 error.
$ kubectl get ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
sidecar <none> * 107.178.245.193 80 28m
$ kubectl describe ingress sidecar
Name: sidecar
Labels: <none>
Namespace: default
Address: 107.178.245.193
Default backend: api:80 (10.0.1.14:8080)
Rules:
Host Path Backends
---- ---- --------
* * api:80 (10.0.1.14:8080)
Annotations: ingress.kubernetes.io/backends: {"k8s1-5ae02eec-default-api-80-28d7bbec":"Unknown"}
ingress.kubernetes.io/forwarding-rule: k8s2-fr-krllp0c9-default-sidecar-9a9n4r5m
ingress.kubernetes.io/target-proxy: k8s2-tp-krllp0c9-default-sidecar-9a9n4r5m
ingress.kubernetes.io/url-map: k8s2-um-krllp0c9-default-sidecar-9a9n4r5m
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Sync 29m loadbalancer-controller UrlMap "k8s2-um-krllp0c9-default-sidecar-9a9n4r5m" created
Normal Sync 28m loadbalancer-controller TargetProxy "k8s2-tp-krllp0c9-default-sidecar-9a9n4r5m" created
Normal Sync 28m loadbalancer-controller ForwardingRule "k8s2-fr-krllp0c9-default-sidecar-9a9n4r5m" created
Normal IPChanged 28m loadbalancer-controller IP is now 107.178.245.193
Normal Sync 3m51s (x7 over 29m) loadbalancer-controller Scheduled for sync
Below is the curl error response:
$ curl -i http://107.178.245.193/health
HTTP/1.1 502 Bad Gateway
Content-Type: text/html; charset=UTF-8
Referrer-Policy: no-referrer
Content-Length: 332
Date: Tue, 16 Aug 2022 10:40:31 GMT
<html><head>
<meta http-equiv="content-type" content="text/html;charset=utf-8">
<title>502 Server Error</title>
</head>
<body text=#000000 bgcolor=#ffffff>
<h1>Error: Server Error</h1>
<h2>The server encountered a temporary error and could not complete your request.<p>Please try again in 30 seconds.</h2>
<h2></h2>
</body></html>
When I describe the service api, I got below error:
$ kubectl describe service api
Name: api
Namespace: default
Labels: <none>
Annotations: cloud.google.com/neg: {"ingress": true}
cloud.google.com/neg-status: {"network_endpoint_groups":{"80":"k8s1-29362abf-default-api-80-f2f1248a"},"zones":["australia-southeast2-a"]}
field.cattle.io/publicEndpoints: [{"port":30084,"protocol":"TCP","serviceName":"default:api","allNodes":true}]
Selector: name=api
Type: NodePort
IP Family Policy: SingleStack
IP Families: IPv4
IP: 10.3.253.54
IPs: 10.3.253.54
Port: <unset> 80/TCP
TargetPort: 8080/TCP
NodePort: <unset> 30084/TCP
Endpoints: 10.0.1.17:8080
Session Affinity: None
External Traffic Policy: Cluster
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning AttachFailed 7s neg-controller Failed to Attach 2 network endpoint(s) (NEG "k8s1-29362abf-default-api-80-f2f1248a" in zone "australia-southeast2-a"): googleapi: Error 400: Invalid value for field 'resource.ipAddress': '10.0.1.18'. Specified IP address 10.0.1.18 doesn't belong to the (sub)network default or to the instance gke-gcp-cqrs-gcp-cqrs-node-pool-6b30ca5c-41q8., invalid
Warning RetryFailed 7s neg-controller Failed to retry NEG sync for "default/api-k8s1-29362abf-default-api-80-f2f1248a--/80-8080-GCE_VM_IP_PORT-L7": maximum retry exceeded
Does anyone know what could be the root course?
I created a new GKE cluster and tried setting up the same resources you are configuring. However, I used the following image for the container gcr.io/google-samples/hello-app:1.0. Everything else remains the same - leaving the gcp-setup.yaml file I used below for reference.
gcp-setup.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: api
namespace: default
spec:
replicas: 1
selector:
matchLabels:
name: api
template:
metadata:
labels:
name: api
spec:
containers:
- name: api
image: gcr.io/google-samples/hello-app:1.0
ports:
- containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
name: api
annotations:
cloud.google.com/neg: '{"ingress": true}'
spec:
selector:
name: api
ports:
- port: 80
targetPort: 8080
protocol: TCP
type: NodePort
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: sidecar
namespace: default
spec:
defaultBackend:
service:
name: api
port:
number: 80
There is also a small thing I had to change in your configuration, which is the annotation block - when I first tried to apply your configuration, I got the below error. Hence, I had to adjust the annotation entry to be annotations.
> kubectl apply -f gcp-setup.yaml
deployment.apps/api created
error: error validating "gcp-setup.yaml": error validating data: ValidationError(Service.metadata): unknown field "annotation" in io.k8s.apimachinery.pkg.apis.meta.v1.ObjectMeta; if you choose to ignore these errors, turn validation off with --validate=false
Afterwards, I was able to successfully provision all of the resources, and your configuration worked perfectly fine. It took around 3 minutes I believe for the Ingress resource to get an IP address assigned (masked as XX.XXX.XXX.XX below).
> kubectl get pods
NAME READY STATUS RESTARTS AGE
api-7d6fdd9845-8dwqc 1/1 Running 0 7m13s
> kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
api NodePort 10.36.4.150 <none> 80:30142/TCP 7m1s
kubernetes ClusterIP 10.36.0.1 <none> 443/TCP 12m
> kubectl get ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
sidecar <none> * XX.XXX.XXX.XX 80 7m18s
> kubectl describe ingress
Name: sidecar
Namespace: default
Address: XX.XXX.XXX.XX
Default backend: api:80 (10.32.0.10:8080)
Rules:
Host Path Backends
---- ---- --------
* * api:80 (10.32.0.10:8080)
Annotations: ingress.kubernetes.io/backends: {"k8s1-05f3ce8b-default-api-80-82dd4d72":"HEALTHY"}
ingress.kubernetes.io/forwarding-rule: k8s2-fr-9k4w4ytx-default-sidecar-9m5g4dex
ingress.kubernetes.io/target-proxy: k8s2-tp-9k4w4ytx-default-sidecar-9m5g4dex
ingress.kubernetes.io/url-map: k8s2-um-9k4w4ytx-default-sidecar-9m5g4dex
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Sync 7m13s loadbalancer-controller UrlMap "k8s2-um-9k4w4ytx-default-sidecar-9m5g4dex" created
Normal Sync 7m10s loadbalancer-controller TargetProxy "k8s2-tp-9k4w4ytx-default-sidecar-9m5g4dex" created
Normal Sync 6m59s loadbalancer-controller ForwardingRule "k8s2-fr-9k4w4ytx-default-sidecar-9m5g4dex" created
Normal IPChanged 6m59s loadbalancer-controller IP is now XX.XXX.XXX.XX
Normal Sync 28s (x6 over 8m3s) loadbalancer-controller Scheduled for sync
After the Ingress resource became healthy, I was able to navigate in my browser to the assigned IP address XX.XXX.XXX.XX and got a successful response back from the workload I deployed (gcr.io/google-samples/hello-app:1.0).
Browser Output
Hello, world!
Version: 1.0.0
Hostname: api-7d6fdd9845-8dwqc
As a conclusion, make sure to update your Service definition from metadata.annotation to metadata.annotations. It was the only change I had to do to make your configuration work. Furthermore, I recommend turning resource definition validation on to make sure that you catch such errors when defining new resources.
If the error still persists, I would recommend running kubectl describe ingress sidecar and analyze the output, assuming it is related to the Ingress resource definition.
EDIT1
To make sure that this is not a zone-related issue, I provisioned a VPC-native, Public cluster in the same zone that you are using (australia-southeast2-a). I then applied the same configuration, and it was successful, thus ruling out the zone-related topic.
Based on the additional information you included in the post, my best guess for some potential root causes for the Service error you're getting when running kubectl describe service would be:
Your GKE cluster is not VPC-native - I see this is a core requirement to be able to leverage NEG
Your GKE cluster has been provisioned as a Private cluster, and as a consequence, NEG tries to assign an IP address from the available Private subnet ranges. This would explain the 10.0.1.18 IP address that NEG tries to assign to the resource definition

Kubernetes service URL not responding to API call

I've been following multiple tutorials on how to deploy my (Spring Boot) api on Minikube. I already got it (user-service running on 8081) working in a docker container with an api gateway (port 8080) and eureka (port 8087), but for starters I just want it to run without those. Steps I took:
Push docker container or image (?) to docker hub, I don't know the proper term.
Create a deployment.yaml:
apiVersion: v1
kind: Service
metadata:
name: kwetter-service
spec:
type: LoadBalancer
selector:
app: kwetter
ports:
- protocol: TCP
port: 8080
targetPort: 8081
nodePort: 30070
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: kwetter-deployment
labels:
app: kwetter
spec:
replicas: 1
selector:
matchLabels:
app: kwetter
template:
metadata:
labels:
app: kwetter
spec:
containers:
- name: user-api
image: cazhero/s6-kwetter-backend_user:latest
ports:
- containerPort: 8081 #is the port it runs on when I manually start it up
kubectl apply -f deployment.yaml
minikube service kwetter-service
It takes me to an empty site with url: http://192.168.49.2:30070 which I thought I could use to make API calls to, but apparently not. How do I make api calls to my application running on minikube?
Get svc returns:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 4d4h
kwetter-service LoadBalancer 10.106.42.56 <pending> 8080:30070/TCP 4d
describe svc kwetter-service:
Name: kwetter-service
Namespace: default
Labels: <none>
Annotations: <none>
Selector: app=kwetter
Type: NodePort
IP Family Policy: SingleStack
IP Families: IPv4
IP: 10.106.42.56
IPs: 10.106.42.56
Port: <unset> 8080/TCP
TargetPort: 8081/TCP
NodePort: <unset> 30070/TCP
Endpoints: 172.17.0.4:8081
Session Affinity: None
External Traffic Policy: Cluster
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Type 6s service-controller LoadBalancer -> NodePort
Made an Ingress in the yaml, used kubectl get ing:
NAME CLASS HOSTS ADDRESS PORTS AGE
kwetter-ingress <none> * 80 49m
To make some things clear:
You need to have pushed your docker image cazhero/s6-kwetter-backend_user:latest to docker hub, check that at https://hub.docker.com/, in your personal repository.
What's the output of minikube service kwetter-service, does it print the URL http://192.168.49.2:30070?
Make sure your pod is running correctly by the following minikube command:
# check pod status
minikube kubectl -- get pods
# if the pod is running, check its container logs
minikube kubectl -- logs po kwetter-deployment-xxxx-xxxx
I see that you are using LoadBalancer service, where a LoadBalancer service is the standard way to expose a service to the internet. With this method, each service gets its own IP address.
Check external IP
kubectl get svc
Use the external IP and the port number in this format to access the
application.
http://REPLACE_WITH_EXTERNAL_IP:8080
If you want to access the application using Nodeport (30070), use the Nodeport service instead of LoadBalancer service.
Refer to this documentation for more information on accessing applications through Nodeport and LoadBalancer services.

Kubernetes service is configured but get connection refused

I have service configurd on my kuberntes cluster but when I try to curl ip:port I get connection refused
the following service configured :
apiVersion: v1
kind: Service
metadata:
name: frontend
namespace: production
spec:
type: NodePort
selector:
app: frontend
ports:
- name: control-center-web
port: 8008
protocol: TCP
targetPort: 8008
$ kubectl describe svc frontend
Name: frontend
Namespace: production
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"name":"frontend","namespace":"production"},"spec":{"ports":[{"name":...
Selector: app=frontend
Type: NodePort
IP: <ip>
Port: control-center-web 8008/TCP
TargetPort: 8008/TCP
NodePort: control-center-web 7851/TCP
Endpoints: <none>
Session Affinity: None
External Traffic Policy: Cluster
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Type 14m service-controller LoadBalancer -> NodePort
Why do I keep getting connection refuse for ip and port that I took from the svc?
The Endpoints in service has got None instead of IPs of the pods. This happens when the selector in service app: frontend does not match with the selector in pod spec,

Kubernetes services endpoints healtcheck

I've created a Kubernetes Service whose backend nodes aren't part of the Cluster but a fixed set of nodes (having fixed IPs), so I've also created an Endpoints resource with the same name:
apiVersion: v1
kind: Service
metadata:
name: elk-svc
spec:
ports:
- port: 9200
targetPort: 9200
protocol: TCP
---
kind: Endpoints
apiVersion: v1
metadata:
name: elk-svc
subsets:
-
addresses:
- { ip: 172.21.0.40 }
- { ip: 172.21.0.41 }
- { ip: 172.21.0.42 }
ports:
- port: 9200
Description of Service and Endpoints:
$ kubectl describe svc elk-svc
Name: elk-svc
Namespace: default
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"name":"elk-svc","namespace":"default"},"spec":{"ports":[{"port":9200,"protocol":"TCP"...
Selector: <none>
Type: ClusterIP
IP: 10.233.17.18
Port: <unset> 9200/TCP
Endpoints: 172.21.0.40:9200,172.21.0.41:9200,172.21.0.42:9200
Session Affinity: None
Events: <none>
$ kubectl describe ep elk-svc
Name: elk-svc
Namespace: default
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","kind":"Endpoints","metadata":{"annotations":{},"name":"elk-svc","namespace":"default"},"subsets":[{"addresses":[{"ip":"172.21.0.40"...
Subsets:
Addresses: 172.21.0.40,172.21.0.41,172.21.0.42
NotReadyAddresses: <none>
Ports:
Name Port Protocol
---- ---- --------
<unset> 9200 TCP
Events: <none>
My pods are able to communicate with ElasticSearch using the internal cluster IP 10.233.17.18. Everything goes fine !
My question is about if there's any way to have some kind of healthCheck mechanism for that Service I've created so if one of my ElasticSearch nodes goes down, ie: 172.21.0.40, then the Service is aware of that and and will no longer route traffic to that node, but to the others. Is that possible ?
Thanks.
This is not supported in k8s.
For more clarification refer this issue raised on your requirement ::
https://github.com/kubernetes/kubernetes/issues/77738#issuecomment-491560980
For this use-case best practice would be to use a loadbalancer like haproxy
My suggestion will be to have a reverse proxy such as nginx or haproxy infront of the elastic nodes which will do health check for those nodes.

Unable to connect to external load balancer even after exposing service in kubernetes

I have the following deployment file
apiVersion: apps/v1
kind: Deployment
metadata:
name: family-tree-deployment
labels:
app: familytree
spec:
replicas: 1
selector:
matchLabels:
app: familytree
template:
metadata:
labels:
app: familytree
spec:
containers:
- name: familytree
image: index.docker.io/koustubh/familytree:v1.0
ports:
- containerPort: 8080
I could successfully create the deployment using kubectl create -f deploy.yml
Now, I simply exposed this deployment with the following command
kubectl expose deployment family-tree-deployment --type=LoadBalancer --name=familytree-service
The service was successfully created.
The output is
$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
familytree-service LoadBalancer 10.51.244.161 35.221.113.235 8080:30505/TCP 1h
$ kubectl describe svc familytree-service
Name: familytree-service
Namespace: default
Labels: app=familytree
Annotations: <none>
Selector: app=familytree
Type: LoadBalancer
IP: 10.51.244.161
LoadBalancer Ingress: 35.221.113.235
Port: <unset> 8080/TCP
TargetPort: 8080/TCP
NodePort: <unset> 30505/TCP
Endpoints: 10.48.4.7:8080
Session Affinity: None
External Traffic Policy: Cluster
Events: <none>
I could login to the pod and I made sure the service is working.
However, when I use the external ip of the load balancer and query my api, the connection times out.
I have made sure firewall allows port 8080.
My application is running on port 8080
The generated Service object looks perfectly valid, so we can exclude a label issue or a missing public IP address. Besides you can access your Service internally, which means the firewall rule was applied incorrectly, most likely.
Please ensure you allow incoming traffic as follows
from the internet to the load balancer on TCP port 8080
from the load balancer to all Kubernetes nodes on TCP port 30505