Running a command on all kubernetes pods of a service - kubernetes

Hey I'm running a kubernetes cluster and I want to run a command on all pods that belong to a specific service.
As far as I know kubectl exec can only run on a pod and tracking all my pods is a ridiculous amount of work (which is one of the benefits of services).
Is there any way or tool that gives you the ability to "broadcast" to all pods in a service?
Thanks in advance for your help!

Here's a simple example with kubectl pipe to xargs, printing env of each pod:
k get pod \
-l {your label selectors} \
--field-selector=status.phase=Running \
-o custom-columns=name:metadata.name --no-headers \
| xargs -I{} kubectl exec {} env

As Bal Chua wrote, kubectl has no way to do this, but you can use bash script to do this:
#!/usr/bin/env bash
PROGNAME=$(basename $0)
function usage {
echo "usage: $PROGNAME [-n NAMESPACE] [-m MAX-PODS] -s SERVICE -- COMMAND"
echo " -s SERVICE K8s service, i.e. a pod selector (required)"
echo " COMMAND Command to execute on the pods"
echo " -n NAMESPACE K8s namespace (optional)"
echo " -m MAX-PODS Max number of pods to run on (optional; default=all)"
echo " -q Quiet mode"
echo " -d Dry run (don't actually exec)"
}
function header {
if [ -z $QUIET ]; then
>&2 echo "###"
>&2 echo "### $PROGNAME $*"
>&2 echo "###"
fi
}
while getopts :n:s:m:qd opt; do
case $opt in
d)
DRYRUN=true
;;
q)
QUIET=true
;;
m)
MAX_PODS=$OPTARG
;;
n)
NAMESPACE="-n $OPTARG"
;;
s)
SERVICE=$OPTARG
;;
\?)
usage
exit 0
;;
esac
done
if [ -z $SERVICE ]; then
usage
exit 1
fi
shift $(expr $OPTIND - 1)
while test "$#" -gt 0; do
if [ "$REST" == "" ]; then
REST="$1"
else
REST="$REST $1"
fi
shift
done
if [ "$REST" == "" ]; then
usage
exit 1
fi
PODS=()
for pod in $(kubectl $NAMESPACE get pods --output=jsonpath={.items..metadata.name}); do
echo $pod | grep -qe "^$SERVICE" >/dev/null 2>&1
if [ $? -eq 0 ]; then
PODS+=($pod)
fi
done
if [ ${#PODS[#]} -eq 0 ]; then
echo "service not found in ${NAMESPACE:-default}: $SERVICE"
exit 1
fi
if [ ! -z $MAX_PODS ]; then
PODS=("${PODS[#]:0:$MAX_PODS}")
fi
header "{pods: ${#PODS[#]}, command: \"$REST\"}"
for i in "${!PODS[#]}"; do
pod=${PODS[$i]}
header "{pod: \"$(($i + 1))/${#PODS[#]}\", name: \"$pod\"}"
if [ "$DRYRUN" != "true" ]; then
kubectl $NAMESPACE exec $pod -- $REST
fi
done

Here:
kubectl -n alex get pods -l app=alex-admin-api -o name | xargs -I{} kubectl -n alex exec {} -- cat alexAdminApi.log >> alex-admin-api_pods.logs

I have written a simple kubectl plugin that "boardcast"s commands to all pods, using Tmux. Assuming that all your pods in the service should share the same labels in their spec, app=foobar for instance, you can use the command below,
kubectl tmux-exec -l app=foobar bash
The plugin is available on Github: predatorray/kubectl-tmux-exec. Hope it will help you!

there is also a kubectl plugin called kubectl-exec-all https://github.com/jpdasma/kubectl-exec-all that is doing a good job.
Can't work with krew index but you can save the content of https://raw.githubusercontent.com/jpdasma/kubectl-exec-all/master/unix/exec-all.sh to /usr/local/bin/kubectl-exec_all and don't forget to make it executable by running
chmod +x /usr/local/bin/kubectl-exec_all
then it work like this:
kubectl plugin exec-all daemonset docker-daemon -- docker system prune -a -f

Related

How to retrieve the pod/container in which run a given process

Using crictl an containerd, is there an easy way to find to which pod/container belongs a given process, using it's PID` on the host machine?
For example, how can I retrieve the name of the pod which runs the process below (1747):
root#k8s-worker-node:/# ps -ef | grep mysql
1000 1747 1723 0 08:58 ? 00:00:01 mysqld
Assuming that you're looking at the primary process in a pod, you could do something like this:
crictl ps -q | while read cid; do
if crictl inspect -o go-template --template '{{ .info.pid }}' $cid | grep -q $target_pid; then
echo $cid
fi
done
This walks through all the crictl managed pods and checks the pod pid against the value of the $target_pid value (which you have set beforehand to the host pid in which you are interested).
1. Using pid2pod
pid2pod is a dedicated tool: https://github.com/k8s-school/pid2pod
Example:
# Install
$ curl -Lo ./pid2pod https://github.com/k8s-school/pid2pod/releases/download/v0.0.1/pid2pod-linux-amd64
$ chmod +x ./pid2pod
$ mv ./pid2pod /some-dir-in-your-PATH/pid2pod
# Run
$ ./pid2pod 1525
NAMESPACE POD CONTAINER PRIMARY PID
kube-system calico-node-6kt29 calico-node 1284
2. Using sysdig OSS
Install sysdig and run:
sudo csysdig -pc
You'll get something in the htop's style:
3. Custom script
Using #Iarsks answer, I propose a solution based on command belown which provides the pod name for a given PID:
$ pid=1254
$ nsenter -t $pid -u hostname
coredns-558bd4d5db-rxz9r
This solution display namespace, pod, container and container's primary PID. It is possible to copy paste the script below in a file named get_pid.sh and then run ./get_pid.sh 2345 for example.
#!/bin/bash
# Display pod information about a process, using its host PID as input
set -euo pipefail
usage() {
cat << EOD
Usage: `basename $0` PID
Available options:
-h this message
Display pod information about a process, using its host PID as input:
- display namespace, pod, container, and primary process pid for this container if the process is running in a pod
- else exit with code 1
EOD
}
if [ $# -ne 1 ] ; then
usage
exit 2
fi
pid=$1
is_running_in_pod=false
pod=$(nsenter -t $pid -u hostname 2>&1)
if [ $? -ne 0 ]
then
printf "%s %s:\n %s" "nsenter command failed for pid" "$pid" "$pod"
fi
cids=$(crictl ps -q)
for cid in $cids
do
current_pod=$(crictl inspect -o go-template --template '{{ index .info.config.labels "io.kubernetes.pod.name"}}' "$cid")
if [ "$pod" == "$current_pod" ]
then
tmpl='NS:{{ index .info.config.labels "io.kubernetes.pod.namespace"}} POD:{{ index .info.config.labels "io.kubernetes.pod.name"}} CONTAINER:{{ index .info.config.labels "io.kubernetes.container.name"}} PRIMARY PID:{{.info.pid}}'
crictl inspect --output go-template --template "$tmpl" "$cid"
is_running_in_pod=true
break
fi
done
if [ "$is_running_in_pod" = false ]
then
echo "Process $pid is not running in a pod."
exit 1
fi
WARNING: this solution does not work if two pods have the same name (even in different namespaces)

what is the root password of telepresence in kubernetes remote debugging

I am using telepresence to remote debugging the kubernetes cluster, and I am log in cluster using command:
telepresence
but when I want to install some software in the telepresence pod:
sudo apt-get install wget
and I did not know the password of telepresence pod, so what should I do to install software?
you could using this script to login pod as root:
#!/usr/bin/env bash
set -xe
POD=$(kubectl describe pod "$1")
NODE=$(echo "$POD" | grep -m1 Node | awk -F'/' '{print $2}')
CONTAINER=$(echo "$POD" | grep -m1 'Container ID' | awk -F 'docker://' '{print $2}')
CONTAINER_SHELL=${2:-bash}
set +e
ssh -t "$NODE" sudo docker exec --user 0 -it "$CONTAINER" "$CONTAINER_SHELL"
if [ "$?" -gt 0 ]; then
set +x
echo 'SSH into pod failed. If you see an error message similar to "executable file not found in $PATH", please try:'
echo "$0 $1 sh"
fi
login like this:
./login-k8s-pod.sh flink-taskmanager-54d85f57c7-wd2nb

running kubectl create command from inside a shell script

I am trying to call kubectl create -n $test_namespace -f <absolute-path-to-tea-rc-file>.yaml from a shell script.
#!/bin/bash
current_dir=$(pwd)
echo "current_dir is $current_dir"
parentdir="$(dirname "$current_dir")"
echo $parentdir
kubectl create -n $test_namespace -f $parentdir/deployment/common/tea-rc.yaml
It gives error as below:
Bhagyashrees-MacBook-Pro:execution_controller bhagyashree$ sh test.sh
current_dir is /Users/bhagyashree/Documents/FDrepo/armada-ingress-ctl-test/execution_controller
/Users/bhagyashree/Documents/FDrepo/armada-ingress-ctl-test
error: unknown command "/Users/bhagyashree/Documents/FDrepo/armada-ingress-ctl-test/deployment/common/tea-rc.yaml"
See 'kubectl create -h' for help and examples.
the same command works when it is executed from a terminal.
kubectl create -n testnamespace -f /Users/bhagyashree/Documents/FDrepo/armada-ingress-ctl-test/deployment/common/tea-rc.yaml
What I am i missing here?
I think it's because the variable $test_namespace has not been set.
If you change the last line to echo "kubectl create -n $test_namespace -f $parentdir/deployment/common/tea-rc.yaml" you'll see what it's trying to run, it will look like kubctl create -n -f /path/to/dir/deployment/common/tea-rc.yaml. The Namespace can not be left blank.
You could also try adding a line like echo $test_namespace to check.

Executing multiple commands( or from a shell script) in a kubernetes pod

I'm writing a shell script which needs to login into the pod and execute a series of commands in a kubernetes pod.
Below is my sample_script.sh:
kubectl exec octavia-api-worker-pod-test -c octavia-api bash
unset http_proxy https_proxy
mv /usr/local/etc/octavia/octavia.conf /usr/local/etc/octavia/octavia.conf-orig
/usr/local/bin/octavia-db-manage --config-file /usr/local/etc/octavia/octavia.conf upgrade head
After running this script, I'm not getting any output.
Any help will be greatly appreciated
Are you running all these commands as a single line command? First of all, there's no ; or && between those commands. So if you paste it as a multi-line script to your terminal, likely it will get executed locally.
Second, to tell bash to execute something, you need: bash -c "command".
Try running this:
$ kubectl exec POD_NAME -- bash -c "date && echo 1"
Wed Apr 19 19:29:25 UTC 2017
1
You can make it multiline like this:
$ kubectl exec POD_NAME -- bash -c "date && \
echo 1 && \
echo 2"
The following should work
kubectl -it exec podname -- bash -c "ls && ls"
bin dev etc home proc root run sys tmp usr var bin
dev etc home proc root run sys tmp usr var
If above command doesn't work then try too replace bash with one of the following /bin/bash, sh or /bin/sh
-t
can solve your task
For example, I run here few cmd:
kubectl get pods |grep nginx|cut -f1 -d\ |\
while read pod; \
do echo "$pod writing:";\
kubectl exec -t $pod -- bash -c \
"dd if=/dev/zero of=/feeds/test.bin bs=260K count=4 2>&1|\
grep copi |cut -d, -f4; \
a=$SECONDS; echo -ne 'reading:'; cat /feeds/test.bin >/dev/null ; \
let a=SECONDS-a ; \
echo $a sec"
done
p.s. your example will be:
kubectl exec -t octavia-api-worker-pod-test -c octavia-api -- bash -c "unset http_proxy https_proxy ; mv /usr/local/etc/octavia/octavia.conf /usr/local/etc/octavia/octavia.conf-orig ; /usr/local/bin/octavia-db-manage --config-file /usr/local/etc/octavia/octavia.conf ; upgrade ; head"
Posting here because google search still brings you to this post...
I'd like to throw out using a HEREDOC as an additional possibility.
kubectl exec -i --tty-false PODNAME -- bash << EOF
echo "insert all your commands here."
echo "this subprocess will even pickup any variables you have in"
echo "the shell script that is calling this"
EOF

init.d celery script for CentOS?

I'm writing a Django app that uses celery. So far I've been running on Ubuntu, but I'm trying to deploy to CentOS.
Celery comes with a nice init.d script for Debian-based distributions, but it doesn't work on RedHat-based distributions like CentOS because it uses start-stop-daemon. Does anybody have an equivalent one for RedHat that uses the same variable conventions so I can reuse my /etc/default/celeryd file?
Is better solved here:
Celery CentOS init script
You should be good using that one
Since I didn't get an answer, I tried to roll my own:
#!/bin/sh
#
# chkconfig: 345 99 15
# description: celery init.d script
# Defines the following variables
# CELERYD_CHDIR
# DJANGO_SETTINGS_MODULE
# CELERYD
# CELERYD_USER
# CELERYD_GROUP
# CELERYD_LOG_FILE
CELERYD_PIDFILE=/var/run/celery.pid
if test -f /etc/default/celeryd; then
. /etc/default/celeryd
fi
# Source function library.
. /etc/init.d/functions
# Celery options
CELERYD_OPTS="$CELERYD_OPTS -f $CELERYD_LOG_FILE -l $CELERYD_LOG_LEVEL"
if [ -n "$2" ]; then
CELERYD_OPTS="$CELERYD_OPTS $2"
fi
start () {
cd $CELERYD_CHDIR
daemon --user $CELERYD_USER --pidfile $CELERYD_PIDFILE $CELERYD $CELERYD_OPTS &
}
stop () {
if [[ -s $CELERYD_PIDFILE ]] ; then
echo "Stopping Celery"
killproc -p $CELERYD_PIDFILE python
echo "done!"
rm -f $CELERYD_PIDFILE
else
echo "Celery not running."
fi
}
check_status() {
status -p $CELERYD_PIDFILE python
}
case "$1" in
start)
start
;;
stop)
stop
;;
restart)
stop
start
;;
status)
check_status
;;
*)
echo "Usage: $0 {start|stop|restart|status}"
exit 1
;;
esac