I am new to Kubernetes and started reading through the documentation.
There often the term 'endpoint' is used but the documentation lacks an explicit definition.
What is an 'endpoint' in terms of Kubernetes? Where is it located?
I could image the 'endpoint' is some kind of access point for an individual 'node' but that's just a guess.
While you're correct that in the glossary there's indeed no entry for endpoint, it is a well defined Kubernetes network concept or abstraction. Since it's of secondary nature, you'd usually not directly manipulate it. There's a core resource Endpoint defined and it's also supported on the command line:
$ kubectl get endpoints
NAME ENDPOINTS AGE
kubernetes 192.168.64.13:8443 10d
And there you see what it effectively is: an IP address and a port. Usually, you'd let a service manage endpoints (one EP per pod the service routes traffic to) but you can also manually manage them if you have a use case that requires it.
Pods expose themselves through endpoints to a service.
It is if you will part of a pod.
Source: Services and Endpoints
An endpoint is a resource that gets the IP addresses of one or more pods dynamically assigned to it, along with a port. An endpoint can be viewed using kubectl get endpoints.
An endpoint resource is referenced by a kubernetes service, so that the service has a record of the internal IPs of pods in order to be able to communicate with them.
We need endpoints as an abstraction layer because the 'service' in kubernetes acts as part of the orchestration to ensure distribution of traffic to pods (including only sending traffic to healthy pods). For example if a pod dies, a replacement pod will be generated, with a new IP address. Conceptually, the dead pod IP will be removed from the endpoint object, and the IP of the newly created pod will be added, so that the service is updated and 'knows' which pods to connect to.
Read 'Exposing pods to the cluster', then 'Creating a Service' here - https://kubernetes.io/docs/concepts/services-networking/connect-applications-service/#exposing-pods-to-the-cluster
An easy way to investigate and see the relationship is:
kubectl describe pods - and observe the IP addresses of your pods
kubectl get ep - and observe the IP addresses assigned to your endpoint
kubectl describe service myServiceName - and observe the Endpoints associated with your service
So no, the endpoint isn't anything to do with the IP of an individual node. I find it useful to understand the overall structure of kubernetes and the relationship between the cluster, nodes, services, endpoints and pods. This diagram summarises it nicely, and shows an ingress flow that results in the OSI layer 4 (the TCP layer) reaching a back end Node 1, with the OSI layer 7 (http layer) ingress ultimately reaching 'Web Container 1' in Pod 1:
I'll answer your questions one by one:
What is an 'endpoint' in terms of Kubernetes?
(The resource name in K8S is Endpoints).
Endpoints is an object in kubernetes which represents a… List of Endpoints.
Those endpoints can be:
An internal pod running inside the cluster - this is the form that is more familiar.
It is created automatically behind the scenes for us when we create service and pods and match the service label selector to the pods labels.
An external IP which is not a pod - this is the least known option.
The external IP can reside outside the cluster - for example external web server or database.
It can also reside in a different namespace - if you want to point your Service to a Service in a different Namespace inside your cluster.
Regarding external Endpoints - If you do not specify a label selector in your service - Kubernetes can’t create the list of endpoints because he doesn’t know which pods should be included and proxied by the service.
Where is it located?
Like the great diagrams provided here are shown - it sits between the service and an internal (pods) or external (web server, databases etc’)
resources.
I could image the 'endpoint' is some kind of access point for an
individual 'node' Its an access point to a resource that sits inside
one of the nodes in your cluster.
An Endpoint can reside inside one of the nodes in your cluster, or outside your cluster / environment.
If its an internal endpoint (which means that the pod label matches a service label selector) - you can reach it with:
$kubectl describe svc/my-service
Name: my-service
Namespace: default
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"name":" my-service","namespace":"...
Selector: run=some-run
Type: NodePort
IP: 10.100.92.162
Port: <unset> 8080/TCP
TargetPort: 80/TCP
NodePort: <unset> 31300/TCP
Endpoints: 172.21.21.2:80,172.21.38.56:80,172.21.39.160:80
Session Affinity: None
External Traffic Policy: Cluster
Events: <none>
Or directly with:
$kubectl get endpoints my-service
NAME ENDPOINTS AGE
my-service 172.21.21.2:80,172.21.38.56:80,172.21.39.160:80 63d
Regarding external Enpoints:
You create a service without a label selector:
apiVersion: v1
kind: Service
metadata:
name: my-service #<------ Should match the name of Endpoints object
spec:
ports:
- protocol: TCP
port: 8080
targetPort: 9376
So the corresponding Endpoint object will not be created automatically and you manually add the Endpoints object and map the Service to the desired network address and port where the external resource is running:
apiVersion: v1
kind: Endpoints
metadata:
name: my-service #<------ Should match the name of Service
subsets:
- addresses:
- ip: 192.0.2.45
ports:
- port: 9376
(Notice:) I used the term internal for the auto-generated Endpoints for pods that has a match with the label selector and the term external for Endpoints that were created manually.
I could used the terms auto-generated and manual instead - that would have been more accurate but I think more confusing also.
In most cases, when the Endpoints are related to pods inside our cluster - we would want them to be also managed by K8S - in this case they will also need to be generated by K8S.
Think of Endpoints as 'final destination to reach an app' or 'smth at the very end'
As you can see in bellow example: pod-IP = 10.32.0.2, service-Port* = 3306, endpoint = [pod-IP]:[service-Port]
Therefore for User Bob to reach the MySql application it should address to 10.32.0.2:3306 that is the last node in the network where he can find his required information.
An simplistic example: I want to access Google Mail in this case for me/browser the endpoint will be gmail.com:443 similar with above example [pod-IP]:[service-Port]
Endpoints track the IP Addresses of the objects the service send traffic to.
When a service selector matches a pod label, that IP Address is added to your endpoints.
Source: https://theithollow.com/2019/02/04/kubernetes-endpoints/
In k8s, Endpoints is consisted of a distributed API like "[IP]:[Port]" and other things. However,in SOAP,Endpoint is a distributed API like URL.
from Google Cloud:
Endpoints is a distributed API management system. It provides an API console, hosting, logging, monitoring, and other features to help you create, share, maintain, and secure your APIs.
Related
I am new to kubernetes. I have created a cluster of db of kubernetes with 2 nodes. I can access those kubernetes pods from thin client like dbeaver to check the data. But I can not access those kubernetes nodes externally. I am currently trying to run a thick client which will load the data into cluster on kubernetes.
kubectl describe svc <svc>
I can see cluster-Ip assigned to the service. Type of my service is loadbalancer. I tried to use that but still not connecting. I read about using nodeport but without any IP address how to access that
So what is the best way to connect any node or cluster from outside.
Thank you in advance
Regards
#KrishnaChaurasia is right but I would like to explain it in more detail with the help of the official docs.
I strongly recommend going through the following sources:
NodePort Type Service: Exposes the Service on each Node's IP at a static port (the NodePort). A ClusterIP Service, to which the NodePort Service routes, is automatically created. You'll be able to contact the NodePort Service, from outside the cluster, by requesting <NodeIP>:<NodePort>. Here is an example of the NodePort Service:
apiVersion: v1
kind: Service
metadata:
name: my-service
spec:
type: NodePort
selector:
app: MyApp
ports:
# By default and for convenience, the `targetPort` is set to the same value as the `port` field.
- port: 80
targetPort: 80
# Optional field
# By default and for convenience, the Kubernetes control plane will allocate a port from a range (default: 30000-32767)
nodePort: 30007
Accessing services running on the cluster: You have several options for connecting to nodes, pods and services from outside the cluster:
Access services through public IPs.
Use a service with type NodePort or LoadBalancer to make the service reachable outside the cluster. See the services and kubectl expose documentation.
Depending on your cluster environment, this may just expose the service to your corporate network, or it may expose it to the internet. Think about whether the service being exposed is secure. Does it do its own authentication?
Place pods behind services. To access one specific pod from a set of replicas, such as for debugging, place a unique label on the pod and create a new service which selects this label.
In most cases, it should not be necessary for application developer to directly access nodes via their nodeIPs.
A supplement example: Use a Service to Access an Application in a Cluster: This page shows how to create a Kubernetes Service object that external clients can use to access an application running in a cluster.
These will help you to better understand the concepts of different Service Types, how to expose and access them from outside the cluster.
The example is described here - https://kubernetes.io/docs/tutorials/stateful-application/mysql-wordpress-persistent-volume/
The Service object for the wordpress-mysql is:
apiVersion: v1
kind: Service
metadata:
name: wordpress-mysql
labels:
app: wordpress
spec:
ports:
- port: 3306
selector:
app: wordpress
tier: mysql
clusterIP: None
The headless services are documented here - https://kubernetes.io/docs/concepts/services-networking/service/#headless-services The Service definition defines selectors, so I suppose the following passage applies:
For headless Services that define selectors, the endpoints controller
creates Endpoints records in the API, and modifies the DNS
configuration to return records (addresses) that point directly to the
Pods backing the Service
I have followed the example on a 3 node managed k8s cluster in Azure:
C:\work\k8s\mysql-wp-demo> kubectl.exe get ep
NAME ENDPOINTS AGE
kubernetes 52.186.94.71:443 47h
wordpress 10.244.0.10:80 5h33m
wordpress-mysql 10.244.3.28:3306 5h33m
C:\work\k8s\mysql-wp-demo> kubectl.exe get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
wordpress-584f8d8666-rlbf5 1/1 Running 0 5h33m 10.244.0.10 aks-nodepool1-30294001-vmss000001 <none> <none>
wordpress-mysql-55c74969cd-4l8d4 1/1 Running 0 5h33m 10.244.3.28 aks-nodepool1-30294001-vmss000003 <none> <none>
C:\work\k8s\mysql-wp-demo>
As far as I understand there is no difference from the endpoints perspective.
Can someone explain to me - what is the point of headless services in general and in this example in particular?
A regular service has a virtual Service IP that exists as iptables or ipvs rules on each node. A new connection to this service IP is then routed with DNAT to one of the Pod endpoints, to support a form of load balancing across multiple pods.
A headless service (that isn't an ExternalName) will create DNS A records for any endpoints with matching labels or name. Connections will go directly to a single pod/endpoint without traversing the service rules.
A service with a type of ExternalName is just a DNS CNAME record in kubernetes DNS. These are headless by definition as they are names for an IP external to the cluster.
The linked myql deployment/service example is leading into StatefulSet's. This Deployment is basically a single pod statefulset. When you do move to a StatefulSet with multiple pods, you will mostly want to address individual members of the StatefulSet with a specific name (see mdaniels comment).
Another reason to set clusterIP: None is to lessen the load on iptables processing which slows down as the number of services (i.e. iptables rules) increases. Applications that don't need multiple pods, don't need the Service IP. Setting up a cluster to use IPVS alleviates the slow down issue somewhat.
I want to be able to deploy several, single pod, apps and access them on a single IP address leaning on Kubernetes to assign the ports as they are when you use a NodePort service.
Is there a way to use NodePort with a load balancer?
Honestly, NodePort might work by itself, but GKE seems to block direct access to the nodes. There doesn't seem to be firewall controls like on their unmanaged VMs.
Here's a service if we need something to base an answer on. In this case, I want to deploy 10 these services which are different applications, on the same IP, each publicly accessible on a different port, each proxying port 80 of the nginx container.
---
apiVersion: v1
kind: Service
metadata:
name: foo-svc
spec:
selector:
app: nginx
ports:
- name: foo
protocol: TCP
port: 80
type: NodePort
GKE seems to block direct access to the nodes.
GCP allows creating the FW rules that allow incoming traffic either to 'All Instances in the Network' or 'Specified Target Tags/Service Account' in your VPC Network.
Rules are persistent unless the opposite is specified under the organization's policies.
Node's external IP address can be checked at Cloud Console --> Compute Engine --> VM Instances or with kubectl get nodes -o wide.
I run GKE (managed k8s) and can access all my assets externally.
I have opened all the needed ports in my setup. below is the quickest example.
Below you can find my setup:
$ kubectl get nodes -o wide
NAME AGE VERSION INTERNAL-IP EXTERNAL-IP
gke--mnnv 43d v1.14.10-gke.27 10.156.0.11 34.89.x.x
gke--nw9v 43d v1.14.10-gke.27 10.156.0.12 35.246.x.x
kubectl get svc -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) SELECTOR
knp-np NodePort 10.0.11.113 <none> 8180:30008/TCP 8180:30009/TCP app=server-go
$ curl 35.246.x.x:30008/test
Hello from ServerGo. You requested: /test
That is why it looks like a bunch of NodePort type Services would be sufficient (each one serves requests for particular selector)
If for some reason it's not possible to set up the FW rules to allow traffic directly to your Nodes it's possible to configure GCP TCP LoadBalancer.
Cloud Console --> Network Services --> Load Balancing --> Create LB --> TCP Load Balancing.
There you can select your GKE Nodes (or pool of nodes) as a 'Backend' and specify all the needed ports for the 'Frontend'. For the Frontend you can Reserve Static IP right during the configuration and specify 'Port' range as two port numbers separated by a dash (assuming you have multiple ports to be forwarded to your node pool). Additionally, you can create multiple 'Frontends' if needed.
I hope that helps.
Is there a way to use NodePort with a load balancer?
Kubernetes LoadBalancer type service builds on top of NodePort. So internally LoadBalancer uses NodePort meaning when a loadBalancer type service is created it automatically maps to the NodePort. Although it's tricky but possible to create NodePort type service and manually configure the Google provided loadbalancer to point to NodePorts.
I'm deploying a helm chart that consists of a service with three replica containers. I've been following these directions for exposing a service to an external IP address.
How do I expose a port per container or per pod? I explicitly do not want to expose a load balancer that maps that port onto some (but any) pod in the service. The service in question is part of a stateful set, and to clients on the outside it matters which of the three are being contacted, so I can't abstract that away behind a load balancer.
You need to create a new service for every pod in you stateful set. To distinguish pods you need to label them with their names like described here
When you have separate services you can use them individually in ingress.
Just adding the official Kubernetes documentation about creating a service:
https://kubernetes.io/docs/concepts/services-networking/service/
A Service in Kubernetes is a REST object, similar to a Pod. Like all of the REST objects, a Service definition can be POSTed to the apiserver to create a new instance. For example, suppose you have a set of Pods that each expose port 9376 and carry a label "app=MyApp".
kind: Service
apiVersion: v1
metadata:
name: my-service
spec:
selector:
app: MyApp
ports:
- protocol: TCP
port: 80
targetPort: 9376
This specification will create a new Service object named “my-service” which targets TCP port 9376 on any Pod with the "app=MyApp" label. This Service will also be assigned an IP address (sometimes called the “cluster IP”), which is used by the service proxies (see below). The Service’s selector will be evaluated continuously and the results will be POSTed to an Endpoints object also named “my-service”.
Note that a Service can map an incoming port to any targetPort. By default the targetPort will be set to the same value as the port field. Perhaps more interesting is that targetPort can be a string, referring to the name of a port in the backend Pods. The actual port number assigned to that name can be different in each backend Pod. This offers a lot of flexibility for deploying and evolving your Services. For example, you can change the port number that pods expose in the next version of your backend software, without breaking clients.
Kubernetes Services support TCP and UDP for protocols. The default is TCP.
I've setup a NodePort service using the following config:
wordpress-service.yaml
apiVersion: v1
kind: Service
metadata:
labels:
app: wordpress
name: wordpress
spec:
type: NodePort
ports:
- port: 80
targetPort: 80
protocol: TCP
selector:
app: wordpress
Is this sufficient to access the service externally, if so how can I now access the service? What details do I need - and how do I determine them - for example node IP.
For Kubernetes on GCE:
We had the same question regarding services of type NodePort: How do we access node port services from our own host?
#ivan.sim 's answer (nodeIp:nodePort) is on mark however, you still wouldn't be able to access your service unless you add a firewall ingress (inbound to google cloud) traffic rule on the VPC network console to allow your host to be able to access your compute node
the above rule is dangerous and should be used only during development
You can find the node port using either the Google Cloud console or by running subsequent kubectl commands to find out the node running your pod which has your container. i.e kubectl get pods , kubectl describe pod your-pod-name, kubectl describe node node-that-runs-you-pod .status.addresses has your ExternalIP
It would be great if we could extract the node ip running our container in the pod using only a label/selector and a few line of commands, so here is what we did, in this case our selector is app: your-label:
$ nodename=$(kubectl get pods -o jsonpath='{.items[?(#.metadata.labels.app=="your-label")].spec.nodeName}')
$ nodeIp=$(kubectl get nodes -o jsonpath='{.items[?(#.metadata.name=="'$(echo $nodename)'")].status.addresses[?(#.type=="ExternalIP")].address}')
$ echo nodeIp
notice: we used json path to extract the information we desired, for more on json path see: json path
You could certainly turn this into a script that takes a label/selector as input and outputs an external ip of the node running your container !!!
To get the nodeport just type:
$ kubectl get services
under the PORT(S) columns you will see something like tagetPort:nodePort. this nodeport is what you want .
nodeIp:nodePort
When you define a service as type NodePort, every node in your cluster will proxy that port to your service. If you nodes are reachable from outside the Kubernetes cluster, you should be able to access the service at nodeIP:nodePort.
To determine nodeIP of a particular node, you can use either kubectl get no <node> -o yaml or kubectl describe no <node>. The status.Addresses field will be of interest. Generally, you will see fields like HostName, ExternalIP and InternalIP there.
To determine nodePort of your service, you can use either kubectl get svc wordpress -o yaml or kubectl describe svc wordpress. The spec.ports.nodePort is the port you need.
Service defined like this got assgned a high port number and is exposed on all your cluster nodes on that port (probably something like 3xxxx). Hard to tell the rest without proper knowledge of how your cluster is provisioned. kubectl get nodes should give you some knowledge about your nodes.
Although I assume you want to expose the service to the outside world. In the long run I suggest getting familiar with LoadBalancer type services and Ingress / IngressController