With the support of Windows server 2019 in Kubernetes 1.14 it seems possible to have nodes of different OS. For example a Ubuntu 18.04 Node, RHEL 7 Node, Windows Server Node within one cluster.
In my use case I would like to have pre-configured queue system with a queue per OS type. The nodes would feed off their specific queues processing the job.
With the above in my mind is it possible to configure a Job to go to a specific queue and in turn a specific OS node?
Kubernetes nodes come populated with a standard set of labels, this includes kubernetes.io/os
Pods can then be assigned to certain places via a nodeSelector, podAffinity and podAntiAffinity.
apiVersion: extensions/v1beta1
kind: Pod
metadata:
name: anapp
spec:
containers:
- image: docker.io/me/anapp
name: anapp
ports:
- containerPort: 8080
nodeSelector:
kubernetes.io/os: linux
If you need finer grained control (for example choosing between Ubuntu/RHEL) you will need to add custom labels in your kubernetes node deployment to select from. This level of selection is rare as container runtimes try and hide most of the differences from you, but if you have a particular case then add extra label metadata to the nodes.
I would recommend using the ID and VERSION_ID fields from cat /etc/*release* as most Linux distros populate this information in some form.
kubectl label node thenode softey.com/release-id=debian
kubectl label node thenode softey.com/release-version-id=9
Related
i need to isolate dedicated node for monitoring on our Kubernetes cluster anybody know what is the best practise for that?so that it is not impacted by auto scaling
Yes this can be achieved, that's where Labels and Selectors come in handy.
You can label your nodes with key/value pairs, in-order to isolate your nodes based on behaviour of your application, Node isoloation/restriction.
There are several ways one can label their nodes. But this the common and simplest example.
kubectl label nodes <node-name> <label-key>=<label-value>
Eg: kubectl label nodes k8-node-101 instance-type=monitoring
And in your deployment make sure you add the nodeSelector, something like this.
apiVersion: v1
kind: Pod
metadata:
name: monitoring-pod
spec:
containers:
- name: mon-test
image: "cr.mycompany.io/monitoring:v0.1"
nodeSelector:
instance-type: monitoring
This is one example on how you can isolate your node for specific needs or specific project purposes.
Hope this helps.
I'm kind of new to the Kubernetes world. In my project we are planning to use windows containers(.net full framework) in short term and linux containers(.net core) for the long run.
We have a K8 cluster provided by infrastructure team and the cluster has mix of Linux and Windows nodes. I just wanted to know how my windows containers will only be deployed to windows nodes in the K8 cluster. Is it handled by K8 or Do I need anything else ?
Below are the details from the Kubernetes Windows Documentation.
Because your cluster has both Linux and Windows nodes, you must explicitly set the nodeSelector constraint to be able to schedule pods to Windows nodes. You must set nodeSelector with the label beta.kubernetes.io/os to the value windows; see the following example:
apiVersion: v1
kind: Pod
metadata:
name: iis
labels:
name: iis
spec:
containers:
- name: iis
image: microsoft/iis:windowsservercore-1709
ports:
- containerPort: 80
nodeSelector:
"kubernetes.io/os": windows
You would need to add following lines to your YAML file. Details are available here https://kubernetes.io/docs/getting-started-guides/windows/
nodeSelector:
"beta.kubernetes.io/os": windows
I m trying to have cloudera manager and cloudera agents on openshift, in order to run the installation I need to get all the pods communicating with each other.
Manually, I modified the /etc/hosts on the manager and add all the agents and on the agents I added the manager and all the other agents.
Now I wanted to automate this, let suppose I add a new agent, I want it to resolve the manager and the host (I can get a part of it done, by passing the manager name as an env variable and with a shell script add it to the /etc/hosts, not the ideal way but still solution). But the second part would be more difficult, to get the manager to resolve every new agent, and also to resolve every other agent on the same service.
I was wondering if there is a way so every pod on the cluster can resolve the others names ?
I have to services cloudera-manager with one pod, and an other service cloudera-agent with -let's say- 3 agents.
do you have any idea ?
thank you.
Not sure, but it looks like you could benefit from StatefulSets.
There are other ways to get the other pods ips (like using a headless service or requesting to the serverAPI directly ) but StatefulSets provide :
Stable, unique network identifiers
Stable, persistent storage.
Lots of other functionality that facilitates the deployment of a special kind of clusters like distributed databases. Not sure my term 'distributed' here is correct, but it helps me remind what they are for :).
If you want to get all Pods running under a certain Service, make sure to use a headless Service (i.e. set clusterIP: None). Then, you can query your local DNS-Server for the Service and will receive A-Records for all Pods assigned to it:
---
apiVersion: v1
kind: Service
metadata:
name: my-sv
namespace: my-ns
labels:
app: my-app
spec:
clusterIP: None
selector:
app: my-app
Then start your Pods (make sure to give app: labels for assignment) and query your DNS-Server from any of them:
kubectl exec -ti my-pod --namespace=my-ns -- /bin/bash
$ nslookup my-sv.my-ns.svc.cluster.local
Server: 10.255.3.10
Address: 10.255.3.10#53
Name: my-sv.my-ns.svc.cluster.local
Address: 10.254.24.11
Name: my-sv.my-ns.svc.cluster.local
Address: 10.254.5.73
Name: my-sv.my-ns.svc.cluster.local
Address: 10.254.87.6
I have a kubernetes cluster of 3 hosts where each Host has a unique id label.
On this cluster, there is a software that has 3 instances (replicas).
Each replica requires to talk to all other replicas. In addition, there is a service that contains all pods so that this application is permanently available.
So I have:
Instance1 (with labels run: theTool,instanceid: 1)
Instance2 (with labels run: theTool,instanceid: 2)
Instance3 (with labels run: theTool,instanceid: 3)
and
Service1 (selecting pods with label instanceid=1)
Service2 (selecting pods with label instanceid=2)
Service3 (selecting pods with label instanceid=3)
Service (selecting pods with label run=theTool)
This approach works but have I cannot scale or use the rolling-update feature.
I would like to define a deployment with 3 replicas, where each replicate gets a unique generic label (for instance the replica-id like 1/3, 2/3 and so on).
Within the services, I could use the selector to fetch this label which will exist even after an update.
Another solution might be to select the pod/deployment, depending on the host where it is running on. I could use a DaemonSet or just a pod/deployment with affinity to ensure that each host has an exact one replica of my deployment.
But I didn't know how to select a pod based on a host label where it runs on.
Using the hostname is not an option as hostnames will change in different environments.
I have searched the docs but didn't find anything matching this use case. Hopefully, someone here has an idea how to solve this.
The feature you're looking for is called StatefulSets, which just launched to beta with Kubernetes 1.5 (note that it was previously available in alpha under a different name, PetSets).
In a StatefulSet, each replica has a unique name that is persisted across restarts. In your example, these would be instance-1, instance-2, instance-3. Since the instance names are persisted (even if the pod is recreated on another node), you don't need a service-per-instance.
The documentation has more details:
Using StatefulSets
Scaling a StatefulSet
Deleting a StatefulSet
Debugging a StatefulSet
You can map NodeIP:NodePort with PodIP:PodPort. Your pod is running on some Node(Instance/VM).
Assign Label to your nodes ,
http://kubernetes.io/docs/user-guide/node-selection/
Write a service for your pod , for example
service.yaml:
apiVersion: v1
kind: Service
metadata:
name: mysql-service
labels:
label: mysql-service
spec:
type: NodePort
ports:
- port: 3306 #Port on which your service is running
nodePort: 32001 # Node port on which you can access it statically
targetPort: 3306
protocol: TCP
name: http
selector:
name: mysql-selector #bind pod here
Add node selector (in spec field) to your deployment.yaml
deployment.yaml:
spec:
nodeSelector:
nodename: mysqlnode #labelkey=labelname assigned in first step
With this you will be able to access your pod service with Nodeip:Nodeport. If I labeled node 10.11.20.177 with ,
nodename=mysqlnode
I will add in node selector ,
nodeSelector:
nodename : mysqlnode
I specified in service nodePort so now I can access pod service (Which is running in container)
10.11.20.177:32001
But this node should be in same network so it can access pod. For outside access make 32001 accessible publicaly with firewall configuration. It is static forever. Label will take care of your dynamic pod ips.
How does kubernetes choose the minion among many available for a given pod creation command? Is it something that can be controlled/tweaked ?
If replicated pods are submitted for deployment, is kubernetes intelligent enough to place them in different minions if they expose the same container/host port pair? Or does it always place different replicas in different minions ?
What about corner cases like what if two different pods (not necessarily replicas) that expose same host/container port pair are submitted? Will they carefully be placed on different minions ?
If a pod requires specific compute/memory requirements, can it be placed in a minion/host that has sufficient resources left to meet those requirement?
In summary, is there detailed documentation on kubernetes pod placement strategy?
Pods are scheduled to ports using the algorithm in generic_scheduler.go
There are rules that prevent host-port conflicts, and also to make sure that there are sufficient memory and cpu requirements. predicates.go
One way to choose minion for pod creation is using nodeSelector. Inside the yaml file of pod specify the label of minion for which you want to choose the minion.
apiVersion: v1
kind: Pod
metadata:
name: nginx1
labels:
env: test
spec:
containers:
- name: nginx
image: nginx
imagePullPolicy: IfNotPresent
nodeSelector:
key: value