i have a netcore webapi deployed on kubernetes. Every night at midnight i need to call an endpoint to do some operations on every pod, so i have deployed a cronjob that calls the api with curl and the method does the required operations.
apiVersion: batch/v1beta1
kind: CronJob
metadata:
name: test-cronjob
spec:
schedule: "0 0 * * *"
jobTemplate:
spec:
template:
spec:
containers:
- name: test-cronjob
image: curlimages/curl:7.74.0
imagePullPolicy: IfNotPresent
command:
- "/bin/sh"
- "-ec"
- |
date;
echo "doingOperation"
curl POST "serviceName/DailyTask"
restartPolicy: OnFailurey
But this only calls one pod, the one assigned by my ingress.
There is a way to call every pod contained in a service?
That is an expected behavior as when we do curl on a Kubernetes Service object, it is expected to pass the requests to only one of the endpoints (IP of the pods). To achieve, what you need, you need to write a custom script that first gets the endpoints associated with the services and then iteratively call curl over them one by one.
Note: The pods IP can be changed due to pod re-creation so you should fetch the endpoints associated with the service in each run of the cronjob.
You could run kubectl inside your job:
kubectl get pods -l mylabel=mylabelvalue \
-o go-template='{{range .items}}{{.status.podIP}}{{"\n"}}{{end}}'
This will return the internal IP of all the containers my the specific label.
You can then loop over the addresses and run your command.
Since enabling Pods with rights on the API-Server, e.g. to look at the actual service endpoints, is cumbersome and poses a security risk, I recommend a simple scripting solution here.
First up, install a headless service for the deployment in question (a service with clusterIP=None). This will make your internal DNS Server create several A records, each pointing at one of your Pod IPs.
Secondly, in order to ping each Pod in a round-robin fashion from your Cron-Job, employ a little shell script along the lines below (your will need to run this from a container with dig and curl installed on it):
dig +noall +answer <name-of-headless-service>.<namespace>.svc.cluster.local | awk -F$'\t' '{curl="curl <your-protocol>://"$2":<your-port>/<your-endpoint>"; print curl}' | source /dev/stdin
Related
In Google Cloud Platform, my goal is to have one cluster with a message queue and a pod to consume these in another cluster with MCS (Multi Cluster Service). When trying this out with only one cluster it went fairly smooth. I used the container name with the port number as the endpoint to connect to the redpanda message queue like this:
Now I want to do this between two clusters, but I'm having trouble configuring stuff right. This is my setup:
I followed this guide to set the clusters up which seemed to work (hard to tell, but no errors), and the redpanda application inside the pod is configured to be on localhost:9092. Unfortunately, I'm getting a Connection Error when running the consumer on my-service-export.my-ns.svc.clusterset.local:9092.
Is it correct to expose the pod with the message queue on its localhost?
Are there ways I can debug or test the connection between pods easier?
Ok, got it working. I obviously misread the setup at one point and had to re-do some stuff to get it working.
Also, the my-service-export should probably have the same name as the service you want to export, in my case redpanda.
A helpful tool to check the connection without running up a consumer is the simple dnsutils image. Use this deployment file and change the namespace to my-ns:
apiVersion: v1
kind: Pod
metadata:
name: dnsutils
namespace: my-ns # <----
spec:
containers:
- name: dnsutils
image: k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.3
command:
- sleep
- "3600"
imagePullPolicy: IfNotPresent
restartPolicy: Always
Then spin it up with apply, exec into it, then run host my-service-export.my-ns.svc.clusterset.local. If you get an IP back you are probably good.
I am developing an application that allows users to play around in their own sandboxes with a finite life time. Conceptually, it can be thought as if users were playing games of Pong. Users can interact with a web interface hosted at main/ to start a game of Pong. Each game of Pong will exist in its own pod. Since each game has a finite lifetime, the pods are dynamically created on-demand (through the Kubernetes API) as Kubernetes jobs with a single pod. There is therefore a one-to-one relationship between games of Pong and pods. Up to this point I have it all figured out.
My problem is, how do I set up an ingress to map dynamically created URLs, for example main/game1, to the corresponding pods? That is, if a user starts a game through the main interface, I would like him to be redirected to the URL of the corresponding pod where his game is hosted.
I could pre-allocate a set of urls, check if they have active jobs, and redirect if they do not, but the does not scale well. I am thinking dynamically assigning URLs is a common pattern in Kubernetes, so there must be a standard way to do this. I have looked at using nginx-ingress, but that is not a requirement.
Furthermore the comment, I created for you a little demo on minikube providing a working Ingress Class controller (enabled via minikube addons enable ingress).
Replicating the multiple Deployment that simulates the games.
kubectl create deployment deployment-1 --image=nginxdemos/hello
kubectl create deployment deployment-2 --image=nginxdemos/hello
kubectl create deployment deployment-3 --image=nginxdemos/hello
kubectl create deployment deployment-4 --image=nginxdemos/hello
kubectl create deployment deployment-5 --image=nginxdemos/hello
Same for Services resources:
kubectl create service clusterip deployment-1 --tcp=80:80
kubectl create service clusterip deployment-2 --tcp=80:80
kubectl create service clusterip deployment-3 --tcp=80:80
kubectl create service clusterip deployment-4 --tcp=80:80
kubectl create service clusterip deployment-5 --tcp=80:80
Finally, it's time for the Ingress one but we have to be quite hacky since we don't have the subcommand create available.
for number in `seq 5`; do echo "
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: deployment-$number
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /$1
spec:
rules:
- host: hello-world.info
http:
paths:
- path: /game$number
backend:
serviceName: deployment-$number
servicePort: 80
" | kubectl create -f -; done
Now you have Pod, Service and Ingress: obviously, you have to replicate the same result using Kubernetes API but, as I suggested in the comment, you should create a single Ingress resource and update accordingly Path subkey in a dynamic way.
However, if you try to simulate the cURL call faking the Host header, you can see the working result:
# curl `minikube ip`/game2 -sH 'Host: hello-world.info'|grep -i server
<p><span>Server address:</span> <span>172.17.0.5:80</span></p>
<p><span>Server name:</span> <span>deployment-2-5b98b954f6-8g5fl</span></p>
# curl `minikube ip`/game4 -sH 'Host: hello-world.info'|grep -i server
<p><span>Server address:</span> <span>172.17.0.7:80</span></p>
<p><span>Server name:</span> <span>deployment-4-767ff76774-d2fgj</span></p>
You can see the Pod IP and name as well.
I agree with Efrat Levitan. It's not the task for ingress/kubernetes itself.
You need another application (different layer of abstraction) to distinguish where the traffic should be routed for example istio and Routing rule for HTTP traffic based on cookies.
I wanted to know is it possible to have a job in Kubernetes that will run every hour, and will delete certain pods. I need this as a temporary
stop gap to fix an issue.
Yes, it's possible.
I think the easiest way is just to call the Kubernernes API directly from a job. Considering RBAC is configured, something like this:
apiVersion: batch/v1
kind: Job
metadata:
name: cleanup
spec:
serviceAccountName: service-account-that-has-access-to-api
template:
spec:
containers:
- name: cleanup
image: image-that-has-curl
command:
- curl
- -ik
- -X
- DELETE
- -H
- "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)"
- https://kubernetes.default.svc.cluster.local/api/v1/namespaces/{namespace}/pods/{name}
restartPolicy: Never
backoffLimit: 4
You can also run a kubectl proxy sidecar to connect to the cluster using localhost. More information here
Or even running plain kubectl in a pod is also an option: Kubernetes - How to run kubectl commands inside a container?
Use a CronJob (1, 2) to run the Job every hour.
K8S API can be accessed from Pod (3) with proper permissions. When a Pod is created a default ServiceAccount is assigned to it (4) by default. The default ServiceAccount has no RoleBinding and hence the default ServiceAccount and also the Pod has no permissions to invoke the API.
If a role (with permissions) is created and mapped to the default ServiceAccount, then all the Pods by default will get those permissions. So, it's better to create a new ServiceAccount instead of modifying the default ServiceAccount.
So, here are steps for RBAC (5)
Create a ServiceAccount
Create a Role with proper permissions (deleting pods)
Map the ServiceAccount with the Role using RoleBinding
Use the above ServiceAccount in the Pod definition
Create a pod/container with the code/commands to delete the pods
I know it's a bit confusing, but that's the way K8S works.
There is another workaround possibly.
You could create a liveness probe (super easy if you have none already) that doesn't run until after one hour and always fail.
livenessProbe:
tcpSocket:
port: 1234
initialDelaySeconds: 3600
This will wait 3600 seconds (1 hour) and then try to connect to port 1234 and if that fails it will kill the container (not the pod!).
I have created pods using below yaml.
apiVersion: v1
kind: Pod
metadata:
name: kubia-liveness
spec:
containers:
- image: luksa/kubia-unhealthy
name: kubia
livenessProbe:
httpGet:
path: /
port: 8080
Then I created pods using the below command.
$ kubectl create -f kubia-liveness-probe.yaml
It created a pod successfully.
Then I'm trying to create load balancer service to access from the external world.
For that I'm using the below command.
$ kubectl expose rc kubia-liveness --type=LoadBalancer --name kubia-liveness-http
For this, I'm getting below error.
Error from server (NotFound): replicationcontrollers "kubia-liveness" not found
I'm not sure how to create replicationControllers. Could anybody please give me the command to do the same.
You are mixing two approaches here, one is creating stuff from yaml definition, which is good by it self (but bare in mind that it is really rare to create a POD rather then Deployment or ReplicationController) with exposing via CLI, which has some assumptions made (ie. it expects replication controller) and with these assumptions it creates appropriate service. My suggestion would be to go for creating Service from yaml manifest as well, so you can tailor it to fit your case.
I m trying to have cloudera manager and cloudera agents on openshift, in order to run the installation I need to get all the pods communicating with each other.
Manually, I modified the /etc/hosts on the manager and add all the agents and on the agents I added the manager and all the other agents.
Now I wanted to automate this, let suppose I add a new agent, I want it to resolve the manager and the host (I can get a part of it done, by passing the manager name as an env variable and with a shell script add it to the /etc/hosts, not the ideal way but still solution). But the second part would be more difficult, to get the manager to resolve every new agent, and also to resolve every other agent on the same service.
I was wondering if there is a way so every pod on the cluster can resolve the others names ?
I have to services cloudera-manager with one pod, and an other service cloudera-agent with -let's say- 3 agents.
do you have any idea ?
thank you.
Not sure, but it looks like you could benefit from StatefulSets.
There are other ways to get the other pods ips (like using a headless service or requesting to the serverAPI directly ) but StatefulSets provide :
Stable, unique network identifiers
Stable, persistent storage.
Lots of other functionality that facilitates the deployment of a special kind of clusters like distributed databases. Not sure my term 'distributed' here is correct, but it helps me remind what they are for :).
If you want to get all Pods running under a certain Service, make sure to use a headless Service (i.e. set clusterIP: None). Then, you can query your local DNS-Server for the Service and will receive A-Records for all Pods assigned to it:
---
apiVersion: v1
kind: Service
metadata:
name: my-sv
namespace: my-ns
labels:
app: my-app
spec:
clusterIP: None
selector:
app: my-app
Then start your Pods (make sure to give app: labels for assignment) and query your DNS-Server from any of them:
kubectl exec -ti my-pod --namespace=my-ns -- /bin/bash
$ nslookup my-sv.my-ns.svc.cluster.local
Server: 10.255.3.10
Address: 10.255.3.10#53
Name: my-sv.my-ns.svc.cluster.local
Address: 10.254.24.11
Name: my-sv.my-ns.svc.cluster.local
Address: 10.254.5.73
Name: my-sv.my-ns.svc.cluster.local
Address: 10.254.87.6