How do I be able to go to a specific Pod in a DaemonSet without hostNetwork? The reason is my Pods in the DaemonSet are stateful, and I prefer to have at most one worker on each Node (that's why I used DaemonSet).
My original implementation was to use hostNetwork so the worker Pods can be found by Node IP by outside clients. But in many production environment hostNetwork is disabled, so we have to create one NodePort service for each Pod of the DaemonSet. This is not flexible and obviously cannot work in the long run.
Some more background on how my application is stateful
The application works in an HDFS-taste, where Workers(datanodes) register with Masters(namenodes) with their hostname. The masters and outside clients need to go to a specific worker for what it's hosting.
hostNetwork is an optional setting and is not necessary. You can connect to your pods without specifying it.
To communicate with pods in DaemonSet you can specify hostPort in the DaemonSet’s pod spec to expose it on the node. You can then communicate with it directly by using the IP of the node it is running on.
Another approach to connect to stateful application is StatefulSet. It allows you to specify network identifiers. However it requires headless service for network identity of the Pods and you are responsible for creating such services.
Related
As per the Headless service definition:
Kubernetes allows clients to discover pod IPs through DNS lookups. Usually, when you perform a DNS lookup for a service, the DNS server returns a single IP which is the service’s cluster IP. But if you don’t need the cluster IP for your service, you can set ClusterIP to None , then the DNS server will return the individual pod IPs instead of the service IP.Then client can connect to any of them
Looks like its similar to creating a clusterIP with one backend pod. If so why should we use clusterIP with one backend pod?
You don't usually control the Pod's name, and you can't guarantee that there will always be exactly one matching Pod.
There are a couple of standard reasons to not directly create Pods, but instead to rely on higher-level constructs. A Deployment can do zero-downtime upgrades; a StatefulSet manages associated PersistentVolumeClaims; a Job can retry on failure. A Pod also can't be edited once created, so you'd need to delete and recreate the Pod on any update. If you're using a Deployment you'll have Pod names like deployment-name-12345678-abcde, but you don't control the ending parts at all. A Service will give you a consistent name.
During an upgrade there won't necessarily be exactly the requested number of replicas:. A Deployment by default will create an additional Pod, wait for it to pass its readiness checks, then tear down the old Pod; an associated Service will route traffic correctly in this case. The alternative is to tear down the old Pod first. In both cases, you could have zero or two matching Pods and not just one.
Having the Service as the standard pattern also helps if you do ever decide to increase the replica count; for example you might choose to run multiple replicas just for additional resiliency in case ons of the Nodes fails. Communicating via a Service works exactly the same way whether you have just one Pod or several.
There is a difference. A DNS lookup for Headless service returns as many IP addresses as there are pods running, while a DNS lookup for ClusterIP service always returns exactly one IP address.
So yes, if you ever have only one Pod then Headless service may be sufficient.
But if you have more than one pod, for Headless service the client must implement some kind of load balancing while for ClusterIP kubernetes will do the load balancing.
One use case for Headless service with multiple pods is when you want to send requests to every pod, e.g. local cache draining.
I see in an article that I can access to pods from kubeproxy, so what is the role of kubernetes service here? and what is the difference between Kube Proxy and service? finally,
is kube proxy part of service?
As far as I understand:
Service is a Kubernetes object that has a stable name and stable IP and sits in front of a set of pods. All requests sent to the pods should go to the service.
Kube-proxy is a networking component running on every cluster node(basically its a Daemonset). It implements the low-level rules to allow communication to pods from inside as well as outside the Kubernetes Cluster. We can say that kube-proxy is a part of service.
So when a user tries to reach an application deployed on Kubernetes first it reaches the service and then forwards the request one of the underlying pods. This is done by using the rules that Kube proxy created.
For more understanding refer this video : Kube proxy & blog
Closer look at Kube proxy
From my understanding
If you are only accessing the pod ports inside the cluster, then there are no Service involved, as you need Service objects to expose your pods outside of your Cluster
Service exposes your pods outside of your Cluster. Service provides a stable virtual IP address. A controller keeps track of the pods that are associated with the Service. While kube-proxy is a daemon running on each node and watches the service resources defined in the cluster and manages the rules for the requests on a Service’s backend pods
kube-proxy interacts with the Service so kube-proxy can change the iptable rules when there are changes on Service objects. Hence they are separate entities.
We can discuss this for a while, but let's short a long story.
Requests come to Service
Then Service passes it to Kube-Proxy
Kube-Proxy decides to which Pod this request go
How requests are forwarded from Service to Pod
Kube Proxy forwards the request
Responsible for maintaining a list of Service IPs and corresponding Pod IPs
Check this section for more details...
I need for a service in a K8 pod to be able to make HTTP calls to downstream services, load balanced by a NodePort, within the same cluster and namespace.
My constraints are these:
I can do this only through manipulation of deployment and service
entities (no ingress. I don't have that level of access to the
cluster)
I cannot add any K8 plugins
The port that the NodePort exposes must be randomized, not hard coded
This whole thing must be automated. I can't set the deployment with the literal value of
the exposed port. It needs to be set by some sort of variable, or
similar process.
Is this possible, and, if so, how?
It probably can be done but it will not be straight forward and you might have to add some custom automation. A NodePort service is meant to be used by an entity outside your cluster.
For inter-cluster communication, a regular service (with a ClusterIP) will work as designed. Your service can reach another service using DNS service discovery. For example. svc-name.mynamespace.svc.cluster.local would be the DNS entry for a svc-name in the mynamespace namespace.
If you can only do a NodePort which essentially is a port on your K8s nodes, you could create another Deployment or Pod of something like nginx or haproxy. Then have this deployment being serviced by regular K8s service with a ClusterIP. Then have nginx or haproxy point to the NodePort on all your nodes in your Kubernetes cluster. Also, have it configured so that it only forwards to listening NodePorts with some kind of healthcheck.
The above seems like an extra necessary step, but if NodePort from within the cluster is what you need (for some reason), it should do the trick.
I was wondering how pods are accessed when no service is defined for that specific pod. If it's through the environment variables, how does the cluster retrieve these?
Also, when services are defined, where on the master node is it stored?
Kind regards,
Charles
If you define a service for your app , you can access it outside the cluster using that service
Services are of several types , including nodePort , where you can access that port on any cluster node and you will have access to the service regardless of the actual location of the pod
you can access the endpoints or actual pod ports inside the cluster as well , but not outside
all of the above uses the kubernetes service discovery
There are two type of service dicovery though
Internal Service discovery
External Service Discovery.
You cannot "access" a pods container port(s) without a service. Services are objects that define the desired state of an ultimate set of iptable rule(s).
Also, services, like all other objects, are stored in etcd and maintained through your master(s).
You could however manually create an iptable rule forwarding traffic to the local container port that docker has exposed.
Hope this helps! If you still have any questions drop them here.
Just for debugging purposes, you can forward a port from your machine to one in the pod:
kubectl port-forward POD_NAME HOST_PORT:POD_PORT
If you have to access it from anywhere, you should use services, but you got to have a deployment created
Create deployment
kubectl create -f https://raw.githubusercontent.com/kubernetes/website/master/content/en/examples/service/networking/run-my-nginx.yaml
Expose the deployment with a NodePort service
kubectl expose deployment deployment/my-nginx --type=NodePort --name=nginx-service
Then list the services and get the port of the service
kubectl get services | grep nginx-service
All cluster data is stored in etcd which is a distributed key-value store. If etcd goes down, cluster becomes unstable and no new pods can come up.
Kubernetes has a way to access any pod within the cluster. Service is a logical way to access a set of pods bound by a selector. An individual pod can still be accessed irrespective of the service. Further service can be created to access the pods from outside the cluster (NodePort service)
I have a Kubernetes deployment that has 3 replicas. It starts 3 pods which are distributed across a given cluster. I would like to know how to reliably get one pod to contact another pod within the same ReplicaSet.
The deployment above is already wrapped up in a Kubernetes Service. But Services do not cover my use case. I need each instance of my container (each Pod) to start-up a local in memory cache and have these cache communicate/sync with other cache instances running on other Pods. This is how I see a simple distributed cache working on for my service. Pod to pod communication within the same cluster is allowed as per the Kubernetes Network Model but I cannot see a reliable way to address each a pod from another pod.
I believe I can use a StatefulSet, however, I don't want to lose the ClusterIP assigned to the service which is required by Ingress for load balancing.
Ofcourse you can use statefulset, and ingress doesn't need ClusterIP that assigned to the service, since it uses the endpoints, so 'headless service' is ok.