Kubernetes delete pod job - kubernetes

I wanted to know is it possible to have a job in Kubernetes that will run every hour, and will delete certain pods. I need this as a temporary
stop gap to fix an issue.

Yes, it's possible.
I think the easiest way is just to call the Kubernernes API directly from a job. Considering RBAC is configured, something like this:
apiVersion: batch/v1
kind: Job
metadata:
name: cleanup
spec:
serviceAccountName: service-account-that-has-access-to-api
template:
spec:
containers:
- name: cleanup
image: image-that-has-curl
command:
- curl
- -ik
- -X
- DELETE
- -H
- "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)"
- https://kubernetes.default.svc.cluster.local/api/v1/namespaces/{namespace}/pods/{name}
restartPolicy: Never
backoffLimit: 4
You can also run a kubectl proxy sidecar to connect to the cluster using localhost. More information here
Or even running plain kubectl in a pod is also an option: Kubernetes - How to run kubectl commands inside a container?

Use a CronJob (1, 2) to run the Job every hour.
K8S API can be accessed from Pod (3) with proper permissions. When a Pod is created a default ServiceAccount is assigned to it (4) by default. The default ServiceAccount has no RoleBinding and hence the default ServiceAccount and also the Pod has no permissions to invoke the API.
If a role (with permissions) is created and mapped to the default ServiceAccount, then all the Pods by default will get those permissions. So, it's better to create a new ServiceAccount instead of modifying the default ServiceAccount.
So, here are steps for RBAC (5)
Create a ServiceAccount
Create a Role with proper permissions (deleting pods)
Map the ServiceAccount with the Role using RoleBinding
Use the above ServiceAccount in the Pod definition
Create a pod/container with the code/commands to delete the pods
I know it's a bit confusing, but that's the way K8S works.

There is another workaround possibly.
You could create a liveness probe (super easy if you have none already) that doesn't run until after one hour and always fail.
livenessProbe:
tcpSocket:
port: 1234
initialDelaySeconds: 3600
This will wait 3600 seconds (1 hour) and then try to connect to port 1234 and if that fails it will kill the container (not the pod!).

Related

Openshift - Run pod only for specific time period

I'm new to Openshfit. We are using openshift deployments to deploy our multiple microservice (SpringBoot application). The deployment is done from docker image.
We have a situation that we need to stop one micro service alone from Midnight till morning 5 AM ( due to an external dependency ).
Could someone suggest a way to do this automatically?
I was able to run
oc scale deployment/sampleservice--replicas=0 manually to make number of pods as zero and scale up to 1 manually later.
I'm not sure how to run this command on specific time automatically. The CronJob in Openshift should be able to do this. But not sure how to configure cronjob to execute an OC command.
Any guidance will be of great help
Using a cronjob is a good option.
First, you'll need an image that has the oc command line client available. I'm sure there's a prebuilt one out there somewhere, but since this will be running with privileges in your OpenShift cluster you want something you trust, which probably means building it yourself. I used:
FROM quay.io/centos/centos:8
RUN curl -o /tmp/openshift-client.tar.gz \
https://mirror.openshift.com/pub/openshift-v4/clients/ocp/latest/openshift-client-linux.tar.gz; \
tar -C /bin -xf /tmp/openshift-client.tar.gz oc kubectl; \
rm -f /tmp/openshift-client.tar.gz
ENTRYPOINT ["/bin/oc"]
In order to handle authentication correctly, you'll need to create a ServiceAccount and then assign it appropriate privileges through a Role and a RoleBinding. I created a ServiceAccount named oc-client-sa:
apiVersion: v1
kind: ServiceAccount
metadata:
name: oc-client-sa
namespace: oc-client-example
A Role named oc-client-role that grants privileges to Pod and Deployment objects:
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: oc-client-role
namespace: oc-client-example
rules:
- verbs:
- get
- list
- create
- watch
- patch
apiGroups:
- ''
resources:
- pods
- verbs:
- get
- list
- create
- watch
- patch
apiGroups:
- 'apps'
resources:
- deployments
- deployments/scale
And a RoleBinding that connects the oc-client-sa ServiceAccount
to the oc-client-role Role:
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: oc-client-rolebinding
namespace: oc-client-example
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: oc-client-role
subjects:
- kind: ServiceAccount
name: oc-client-sa
With all this in place, we can write a CronJob like this that will
scale down a deployment at a specific time. Note that we're running
the jobs using the oc-client-sa ServiceAccount we created earlier:
apiVersion: batch/v1
kind: CronJob
metadata:
name: scale-web-down
namespace: oc-client-example
spec:
schedule: "00 00 * * *"
concurrencyPolicy: Forbid
jobTemplate:
spec:
template:
spec:
serviceAccountName: oc-client-sa
restartPolicy: Never
containers:
- image: docker.io/larsks/openshift-client
args:
- scale
- deployment/sampleservice
- --replicas=0
name: oc-scale-down
You would write a similar one to scale things back up at 5AM.
The oc client will automatically use the credentials provided to your pod by Kubernetes because of the serviceAccountName setting.
API
You can use the OC rest api client and write the simple python code which will scale down replicas. Pack this python into a docker image and run it as a cronjob inside the OC cluster.
Simple Curl
Run a simple curl inside the cronjob to scale up & down deployment at a certain time.
Here is a simple Curl to scale the deployment: https://docs.openshift.com/container-platform/3.7/rest_api/apis-apps/v1beta1.Deployment.html#Get-apis-apps-v1beta1-namespaces-namespace-deployments-name-scale
API documentation : https://docs.openshift.com/container-platform/3.7/rest_api/apis-apps/v1beta1.Deployment.html
CLI
If you don't want to run code as docker image in cronjob of K8s, you can also run the command, in that case, use the docker image inside cronjob, and fire the command
OC-cli : https://hub.docker.com/r/widerin/openshift-cli
Dont forget authentication is required in both cases either API or running a command inside the cronjob.

Kubernetes cronjob that calls all pods in a service

i have a netcore webapi deployed on kubernetes. Every night at midnight i need to call an endpoint to do some operations on every pod, so i have deployed a cronjob that calls the api with curl and the method does the required operations.
apiVersion: batch/v1beta1
kind: CronJob
metadata:
name: test-cronjob
spec:
schedule: "0 0 * * *"
jobTemplate:
spec:
template:
spec:
containers:
- name: test-cronjob
image: curlimages/curl:7.74.0
imagePullPolicy: IfNotPresent
command:
- "/bin/sh"
- "-ec"
- |
date;
echo "doingOperation"
curl POST "serviceName/DailyTask"
restartPolicy: OnFailurey
But this only calls one pod, the one assigned by my ingress.
There is a way to call every pod contained in a service?
That is an expected behavior as when we do curl on a Kubernetes Service object, it is expected to pass the requests to only one of the endpoints (IP of the pods). To achieve, what you need, you need to write a custom script that first gets the endpoints associated with the services and then iteratively call curl over them one by one.
Note: The pods IP can be changed due to pod re-creation so you should fetch the endpoints associated with the service in each run of the cronjob.
You could run kubectl inside your job:
kubectl get pods -l mylabel=mylabelvalue \
-o go-template='{{range .items}}{{.status.podIP}}{{"\n"}}{{end}}'
This will return the internal IP of all the containers my the specific label.
You can then loop over the addresses and run your command.
Since enabling Pods with rights on the API-Server, e.g. to look at the actual service endpoints, is cumbersome and poses a security risk, I recommend a simple scripting solution here.
First up, install a headless service for the deployment in question (a service with clusterIP=None). This will make your internal DNS Server create several A records, each pointing at one of your Pod IPs.
Secondly, in order to ping each Pod in a round-robin fashion from your Cron-Job, employ a little shell script along the lines below (your will need to run this from a container with dig and curl installed on it):
dig +noall +answer <name-of-headless-service>.<namespace>.svc.cluster.local | awk -F$'\t' '{curl="curl <your-protocol>://"$2":<your-port>/<your-endpoint>"; print curl}' | source /dev/stdin

Pod cannot pull image from private docker registry

I am having some real trouble getting my pods to pull images from a private docker registry that I have setup and am able to authenticate to (I can do docker login https://my.website.com/ and I get Login Succeeded without having to put in my username:password) (I am able to run docker pull my.website.com:5000/human/forum and see all the layers being downloaded.) .
I use https://github.com/bazelbuild/rules_k8s#aliasing-eg-k8s_deploy where I specify the namespace to be "default".
I made sure to put "HTTPS://my.website.com:5000/V2/" (in lowercase) in the auth section in the docker config file before I generated the regcred secret.
Notice that I specify the imagePullSecrets below:
# deployment.yaml
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: angular-bazel-example-prod
spec:
replicas: 1
template:
metadata:
labels:
app: angular-bazel-example-prod
spec:
containers:
- name: angular-bazel-example
image: human/forum:dev
imagePullPolicy: Always
ports:
- containerPort: 8080
imagePullSecrets:
- name: regcred # Notice
I made sure to update my certificate authority certificates:
cp /etc/docker/certs.d/my.website.com\:5000/ca.crt /usr/local/share/ca-certificates/my.website.registry.com/
sudo update-ca-certificates
I see sudo curl --user testuser:testpassword --cacert /usr/local/share/ca-certificates/my.website.registry.com/ca.crt -X GET https://mywebsite.com:5000/v2/_catalog
> {"repositories":["human/forum"]}
I see sudo curl --user testuser:testpassword --cacert /usr/local/share/ca-certificates/mywebsite.registry.com/ca.crt -X GET https://mywebsite.com:5000/v2/human/forum/tags/list
> {"name":"a/repository","tags":["dev"]}
There must be a way to troubleshoot this but I don't know how.
One thing I am curious about is
kubectl describe pod my-first-pod...
...
Volumes:
default-token-mtz9g:
Type: Secret
Where can I find this volume? I can't kubectl exec into a container because none is running.. because the pod can't pull the image.
Do you have any ideas on how I could troubleshoot this?
Thank you!
Slackware
Create a kubernetes secret to access the custom repository. One way is to provide the server, user and password manually. Documentation link.
kubectl create secret docker-registry $YOUR_REGISTRY_NAME --docker-server=https://$YOUR_SERVER_DNS/v2/ --docker-username=$YOUR_USER --docker-password=$YOUR_PASSWORD --docker-email=whatever#gmail.com --namespace=default
Then use it in your yaml
...
imagePullSecrets:
- name: $YOUR_REGISTRY_NAME
Regarding the default-token that you see mounted, it belongs to the service account that your pod uses to talk to the kubernetes api. You can find the service account by running kubectl get sa and kubectl describe sa $DEFAULT_SERVICE_ACCOUNT_ID. Find the token by running kubectl get secrets and kubectl describe secret $SECRET_ID. To clarify this service account and token have nothing to do with the docker registry unless you specify it. To include the registry in the service account follow this guide link

How Kubernetes knows resource requests and limits?

Here is a yaml file that has been created to be deployed in kubernetes. I would like to know since there is no resource request and limits in the file, how kubernetes knows the resource requests and limits to run it? How can I fetch that information?
apiVersion: v1
kind: Pod
metadata:
name: rss-site
labels:
app: web
spec:
containers:
- name: front-end
image: nginx
ports:
- containerPort: 80
- name: rss-reader
image: nickchase/rss-php-nginx:v1
ports:
- containerPort: 88
You can "kubectl describe" your pod and see what actual resources got assigned. With LimitRange Kubernetes can assign default requests and limits to pod if not part of its spec.
If there are no requests/limits assigned - your pod will become of Best Effort quality of service and can be Evicted in case of resource pressure on node.
you can use the below steps to fetch the resource limits assigned to the pod.
Create the pod
-------------------
kubectl run test-resource-limits --image=busybox --limits "memory=100Mi" \
--command -- /bin/sh -c "while true; do sleep 2; done"
Test the resource limits that are specified
-------------------------------------------
kubectl get pods test-resource-limits-7b8b46c8c7-jdjgs \
-o=jsonpath='{.spec.containers[0].resources}'
If you don't specify resource requests and limits. Kubernetes will run your workload without them. meaning your pod could potentially use all the CPU and RAM on the node.
Caveat to that; if your namespace has defaults set with a limitRange the defaults will be applied to workloads that don't specify resource spec.

Error from server (NotFound): replicationcontrollers "kubia-liveness" not found

I have created pods using below yaml.
apiVersion: v1
kind: Pod
metadata:
name: kubia-liveness
spec:
containers:
- image: luksa/kubia-unhealthy
name: kubia
livenessProbe:
httpGet:
path: /
port: 8080
Then I created pods using the below command.
$ kubectl create -f kubia-liveness-probe.yaml
It created a pod successfully.
Then I'm trying to create load balancer service to access from the external world.
For that I'm using the below command.
$ kubectl expose rc kubia-liveness --type=LoadBalancer --name kubia-liveness-http
For this, I'm getting below error.
Error from server (NotFound): replicationcontrollers "kubia-liveness" not found
I'm not sure how to create replicationControllers. Could anybody please give me the command to do the same.
You are mixing two approaches here, one is creating stuff from yaml definition, which is good by it self (but bare in mind that it is really rare to create a POD rather then Deployment or ReplicationController) with exposing via CLI, which has some assumptions made (ie. it expects replication controller) and with these assumptions it creates appropriate service. My suggestion would be to go for creating Service from yaml manifest as well, so you can tailor it to fit your case.