I'm trying to execute a few simple commands on a kubernetes pod in Azure. I've successfully done so with the localhost + pod-as-module-parameter syntax:
---
- hosts: localhost
connection: kubectl
collections:
- kubernetes.core
gather_facts: False
tasks:
- name: Get pod
k8s_info:
kind: Pod
namespace: my-namespace
register: pod_list
- name: Run command
k8s_exec:
pod: "{{pod_list.resources[0].metadata.name}}"
namespace: my_namespace
command: "/bin/bash -c 'echo Hello world'"
However, I want to avoid the repetition of specifying pod and namespace for every kubernetes.core module call, as well as parsing the namespace explicitly in every playbook.
So I got the kubernetes dynamic inventory plugin to work, and can see the desired pod in a group label_app_some-predictable-name, as confirmed by output of ansible-inventory.
What I don't get is if at this point I should be able to run regular command module (I couldn't get that to work at all), or if I need to keep using k8s_exec, which still requires pod and namespace to be specified explicitly (albeit now I can refer to the guest facts populated by the inventory plugin), on top of now requiring delegate_to: localhost:
---
- name: Execute command
hosts: label_app_some-predicatable-name
connection: kubectl
gather_facts: false
collections:
- kubernetes.core
tasks:
- name: Execute command via kubectl
delegate_to: localhost
k8s_exec:
command: "/bin/sh -c 'echo Hello world'"
pod: "{{ansible_kubectl_pod}}"
namespace: "{{ansible_kubectl_namespace}}"
What am I missing? Is there a playbook example that makes use of the kubernetes dynamic inventory?
Related
Scenario is like this:
I have a pod running in a node in K8s cluster in GCP. cluster is created using kops and pod is created using kne_cli.
I know only the name of the pod e.g. "test-pod".
My requirement is to configure something in the node where this pod is running. e.g. I want to update "iptables -t nat" table in node.
how to access the node and configure it from within a pod?
any suggestion will be helpful.
You the Job or deployment or POD, not sure how POD is getting managed. If you just want to run that task Job is good fir for you.
One option is to use SSH way :
You can run one POD inside that you get a list of Nodes or specific node as per need and run SSH command to connect with that node.
That way you will be able to access Node from POD and run commands top of Node.
You can check this document for ref : https://alexei-led.github.io/post/k8s_node_shell/
Option two :
You can mount sh file on Node with IP table command and invoke that shell script from POD to execute which will run the command whenever you want.
Example :
apiVersion: v1
kind: ConfigMap
metadata:
name: command
data:
command.sh: |
#!/bin/bash
echo "running sh script on node..!"
---
apiVersion: batch/v1beta1
kind: CronJob
metadata:
name: command
spec:
schedule: "*/1 * * * *"
jobTemplate:
spec:
template:
spec:
serviceAccountName: cron-namespace-admin
containers:
- name: command
image: IMAGE:v1
imagePullPolicy: IfNotPresent
volumeMounts:
- name: commandfile
mountPath: /test/command.sh
subPath: command.sh
- name: script-dir
mountPath: /test
restartPolicy: OnFailure
volumes:
- name: commandfile
configMap:
name: command
defaultMode: 0777
- name: script-dir
hostPath:
path: /var/log/data
type: DirectoryOrCreate
Use privileged mode
securityContext:
privileged: true
Privileged - determines if any container in a pod can enable
privileged mode. By default a container is not allowed to access any
devices on the host, but a "privileged" container is given access to
all devices on the host. This allows the container nearly all the same
access as processes running on the host. This is useful for containers
that want to use linux capabilities like manipulating the network
stack and accessing devices.
Read more : https://kubernetes.io/docs/concepts/security/pod-security-policy/#privileged
You might be better off using GKE and configuring the ip-masq-agent as described here: https://cloud.google.com/kubernetes-engine/docs/how-to/ip-masquerade-agent
In case you stick with kops on GCE, I would suggest following the guide for ip-masq-agent here instead of the GKE docs: https://kubernetes.io/docs/tasks/administer-cluster/ip-masq-agent/
In case you really need to run custom iptables rules on the host then your best option is to create a DaemonSet with pods that are privileged and have hostNetwork: true. That should allow you to modify iptable rules directly on the host from the pod.
I'm running a pod with 3 containers (telegraf, fluentd and an in-house agent) that makes use of shareProcessNamespace: true.
I've written a python script to fetch the initial config for telegraf and fluentd from a central controller API endpoint. Since this is a one time operation, I plan to use helm post-install hook.
apiVersion: batch/v1
kind: Job
metadata:
name: agent-postinstall
annotations:
"helm.sh/hook-weight": "3"
"helm.sh/hook": "post-install"
spec:
template:
spec:
containers:
- name: agent-postinstall
image: "{{ .Values.image.agent.repository }}:{{ .Values.image.agent.tag | default .Chart.AppVersion }}"
imagePullPolicy: IfNotPresent
command: ['python3', 'getBaseCfg.py']
volumeMounts:
- name: config-agent-volume
mountPath: /etc/config
volumes:
- name: config-agent-volume
configMap:
name: agent-cm
restartPolicy: Never
backoffLimit: 1
It is required for the python script to check if telegraf/fluentd/agent processes are up, before getting the config. I intend to wait (with a timeout) until pgrep <telegraf/fluentd/agent> returns true and then fire APIs. Is there a way to enable shareProcessNamespace for the post-install hook as well? Thanks.
PS: Currently, the agent calls the python script along with its own startup script. It works, but it is kludgy. I'd like to move it out of agent container.
shareProcessNamespace
Most important part of this flag is it works only within one pod, all containers within one pod will share processes between each other.
In described approach job is supposed to be used. Job creates a separate pod so it won't work this way. Container should be a part of the "main" pod with all other containers to have access to running processes of that pod.
More details about process sharing.
Possible way to solution it
It's possible to get processes from the containers directly using kubectl command.
Below is an example how to check state of the processes using pgrep command. The pgrepContainer container needs to have the pgrep command already installed.
job.yaml:
apiVersion: batch/v1
kind: Job
metadata:
name: "{{ .Release.Name }}-postinstall-hook"
annotations: "helm.sh/hook": post-install
spec:
template:
spec:
serviceAccountName: config-user # service account with appropriate permissions is required using this approach
volumes:
- name: check-script
configMap:
name: check-script
restartPolicy: Never
containers:
- name: post-install-job
image: "bitnami/kubectl" # using this image with kubectl so we can connect to the cluster
command: ["bash", "/mnt/script/checkScript.sh"]
volumeMounts:
- name: check-script
mountPath: /mnt/script
And configmap.yaml which contains script and logic which check three processes in loop for 60 iterations per 10 seconds each:
apiVersion: v1
kind: ConfigMap
metadata:
name: check-script
data:
checkScript.sh: |
#!/bin/bash
podName=test
pgrepContainer=app-1
process1=sleep
process2=pause
process3=postgres
attempts=0
until [ $attempts -eq 60 ]; do
kubectl exec ${podName} -c ${pgrepContainer} -- pgrep ${process1} 1>/dev/null 2>&1 \
&& kubectl exec ${podName} -c ${pgrepContainer} -- pgrep ${process2} 1>/dev/null 2>&1 \
&& kubectl exec ${podName} -c ${pgrepContainer} -- pgrep ${process3} 1>/dev/null 2>&1
if [ $? -eq 0 ]; then
break
fi
attempts=$((attempts + 1))
sleep 10
echo "Waiting for all containers to be ready...$[ ${attempts}*10 ] s"
done
if [ $attempts -eq 60 ]; then
echo "ERROR: Timeout"
exit 1
fi
echo "All containers are ready !"
echo "Configuring telegraf and fluentd services"
Final result will look like:
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
test 2/2 Running 0 20m
test-postinstall-hook-dgrc9 0/1 Completed 0 20m
$ kubectl logs test-postinstall-hook-dgrc9
Waiting for all containers to be ready...10 s
All containers are ready !
Configuring telegraf and fluentd services
Above is an another approach, you can use its logic as base to achieve your end goal.
postStart
Also postStart hook can be considered to be used where some logic will be located. It will run after container is created. Since main application takes time to start and there's already logic which waits for it, it's not an issue that:
there is no guarantee that the hook will execute before the container ENTRYPOINT
I am writing ansible scripts for deploying services using Kubernetes, I am stuck with a step that is for the post-deployment process:
I have deployed a service having "replicas: 3", and all the replicas are up and running now my problem is to I have to do a migration for which I have to get into the container and run a script already present there.
I can do it manually by getting into the container individually and then run the script but this will again require manual intervention.
What I want to achieve is once the deployment is done and all the replicas are up and running I want to run the scripts by getting into the containers and all these steps should be performed by ansible script and no manual effort required.
Is there a way to do this?
Take a look at k8s_exec module.
- name: Check RC status of command executed
community.kubernetes.k8s_exec:
namespace: myproject
pod: busybox-test
command: cmd_with_non_zero_exit_code
register: command_status
ignore_errors: True
- name: Check last command status
debug:
msg: "cmd failed"
when: command_status.return_code != 0
#Vasili Angapov is right - k8s_exec module is probably the best solution in this case but I would like to add some useful notes.
To use k8s_exec we need to know the exact Pod name (we need to pass it as pod parameter in ansible task). As you wrote, I assume that your Pods are managed by Deployment, so every Pod has random string in its name added by ReplicaSet. Therefore, you have to find the full names of the Pods somehow.
I've created simple playbook to illustrate how we can find Pod names for all Pods with label: app=web and then run sample touch file123456789 command on these Pods.
---
- hosts: localhost
collections:
- community.kubernetes
tasks:
- name: "Search for all Pods labelled app=web"
k8s_info:
kind: Pod
label_selectors:
- app = web
register: pod_names
- name: "Get Pod names"
set_fact:
pod_names: "{{ pod_names | json_query('resources[*].metadata.name') }}"
- name: "Run command on every Pod labelled app=web"
k8s_exec:
namespace: default
pod: "{{ item }}"
command: touch file123456789
with_items: "{{ pod_names }}"
NOTE: Instead of k8s_exec module you can use command module as well.
In our example instead of k8s_exec task we can have:
- name: "Run command on every Pod labelled app=web"
command: >
kubectl exec "{{ item }}" -n default -- touch file123456789
with_items: "{{ pod_names }}"
Can anyone point out how to connect to the mongo db instance using mongo client using either command line client or from .net core programs with connection strings?
We have created a sample cluster in digitalocean with a namespace, let's say mongodatabase.
We installed the mongo statefulset with 3 replicas. We are able to successfully connect with the below command
kubectl --kubeconfig=configfile.yaml -n mongodatabase exec -ti mongo-0 mongo
But when we connect from a different namespace or from default namespace with the pod names in the below format, it doesn't work.
kubectl --kubeconfig=configfile.yaml exec -ti mongo-0.mongo.mongodatabase.cluster.svc.local mongo
where mongo-0.mongo.mongodatabase.cluster.svc.local is in pod-0.service_name.namespace.cluster.svc.local (also tried pod-0.statfulset_name.namespace.cluster.svc.local and pod-0.service_name.statefulsetname.namespace.cluster.svc.local) etc.,
Can any one help with the correct dns name/connection string to be used while connecting with mongo client in command line and also from the programs like java/.net core etc.,?
Also should we use kubernetes deployment instead of statefulsets here?
You need to reference the mongo service by namespaced dns. So if your mongo service is mymongoapp and it is deployed in mymongonamespace, you should be able to access it as mymongoapp.mymongonamespace.
To test, I used the bitnami/mongodb docker client. As follows:
From within mymongonamespace, this command works
$ kubectl config set-context --current --namespace=mymongonamespace
$ kubectl run mongodbclient --rm --tty -i --image bitnami/mongodb --command -- mongo --host mymongoapp
But when I switched to namespace default it didn't work
$ kubectl config set-context --current --namespace=default
$ kubectl run mongodbclient --rm --tty -i --image bitnami/mongodb --command -- mongo --host mymongoapp
Qualifying the host with the namespace then works
$ kubectl run mongodbclient --rm --tty -i --image bitnami/mongodb --command -- mongo --host mymongoapp.mymongonamespace
This is how you can get inside mongo-0 pod
kubectl --kubeconfig=configfile.yaml exec -ti mongo-0 sh
I think you are looking for this DNS for Services and Pods.
You can have a fully qualified domain name (FQDN) for a Services or for a Pod.
Also please have a look at this kubernetes: Service located in another namespace, as I think it will provide you with answer on how to access it from different namespace.
An example would look like this:
apiVersion: v1
kind: Service
metadata:
name: default-subdomain
spec:
selector:
name: busybox
clusterIP: None
ports:
- name: foo # Actually, no port is needed.
port: 1234
targetPort: 1234
---
apiVersion: v1
kind: Pod
metadata:
name: busybox1
labels:
name: busybox
spec:
hostname: busybox-1
subdomain: default-subdomain
containers:
- image: busybox:1.28
command:
- sleep
- "3600"
name: busybox
---
apiVersion: v1
kind: Pod
metadata:
name: busybox2
labels:
name: busybox
spec:
hostname: busybox-2
subdomain: default-subdomain
containers:
- image: busybox:1.28
command:
- sleep
- "3600"
name: busybox
If there exists a headless service in the same namespace as the pod and with the same name as the subdomain, the cluster’s KubeDNS Server also returns an A record for the Pod’s fully qualified hostname. For example, given a Pod with the hostname set to “busybox-1” and the subdomain set to “default-subdomain”, and a headless Service named “default-subdomain” in the same namespace, the pod will see its own FQDN as “busybox-1.default-subdomain.my-namespace.svc.cluster.local”. DNS serves an A record at that name, pointing to the Pod’s IP. Both pods “busybox1” and “busybox2” can have their distinct A records.
The Endpoints object can specify the hostname for any endpoint addresses, along with its IP.
Note: Because A records are not created for Pod names, hostname is required for the Pod’s A record to be created. A Pod with no hostname but with subdomain will only create the A record for the headless service (default-subdomain.my-namespace.svc.cluster.local), pointing to the Pod’s IP address. Also, Pod needs to become ready in order to have a record unless publishNotReadyAddresses=True is set on the Service.
Your question about Deployments vs StatefulSets should be a different question. But the answer is that the StatefulSet is used when you want "Stable Persistent Storage" kubernetes.io.
Also from the same page "stable is synonymous with persistence across Pod (re)scheduling". So basically your mongo instance is backed by a PeristentVolume and you want the volume reattached after the pod is rescheduled.
I'm doing a prototype where one service depends on an availability of other. Scenario:
Service A is assumed to be already available in a local network. It was either deployed by K8S or manually (or even a managed one provided by AWS etc.).
Service B depends on environment variable SERVICE_A_IP and won't start without it. It's treated as a black box and can't be modified.
I want to pass Service A IP to Service B through K8S YAML configuration file. Perfect syntax for this occasion:
...
env:
- name: SERVICE_A_IP
valueFrom:
k8sDeployment:
name: service_a
key: deploymentIP
...
During the prototyping stage Service A is an another K8S deployment but it might not be so in a production environment. Thus I need to decouple from SERVICE_A_SERVICE_IP that will be available to Service B (given it's deployed after Service A). I'm not into DNS discovery as well as it would require container modification which is far from a perfect solution.
If I would do it manually with kubectl (or with a shell script) it would be like the following:
$ kubectl run service_a --image=service_a:latest --port=8080
$ kubectl expose deployment service_a
$ SERVICE_A_IP="$(kubectl describe service service_a | \
grep IP: | \
cut -f2 -d ':' | \
xargs)"
$ kubectl run service_b --image=service_b:latest --port=8080 \
--env="SERVICE_A_IP=${SERVICE_A_IP}"
It works. Though I want to do the same using YAML configuration without injecting SERVICE_A_IP into configuration file with shell (basically modifying the file).
Is there any way to do so? Please take the above setting as set in stone.
UPDATE
Not the best way though still:
$ kubectl create -f service_a.yml
deployment "service_a" created
service "service_a" created
$ SERVICE_A_IP="$(kubectl describe service service_a | \
grep IP: | \
cut -f2 -d ':' | \
xargs)"
$ kubectl create configmap service_a_meta \
--from-literal="SERVICE_A_IP=${SERVICE_A_IP}"
And then in service_b.yml:
...
env:
- name: SERVICE_A_IP
valueFrom:
configMapKeyRef:
name: service_a_meta
key: SERVICE_A_IP
...
That will work but still involves some shell and generally feels way too hax.
You can use attach handlers to lifecycle events for update your environment variables on start.
Here is an example:
apiVersion: v1
kind: Pod
metadata:
name: appB
spec:
containers:
- name: appB
image: nginx
lifecycle:
postStart:
exec:
command: ["/bin/sh", "-c", "export SERVICE_B_IP=$(host <SERVICE_B>.<SERVICE_B_NAMESPACE>.svc.cluster.local)"]
Kubernetes will run preStart script each time when pod with your appB container is starting right in appB container before execution of the main application.
But, because of that description:
PostStart
This hook executes immediately after a container is created. However, there is no guarantee that the hook will execute before the container ENTRYPOINT. No parameters are passed to the handler.
You need to add some sleep for your main app before the real start just to be sure that hook will be finished before application will be started.