How to retrieve the pod/container in which run a given process - kubernetes

Using crictl an containerd, is there an easy way to find to which pod/container belongs a given process, using it's PID` on the host machine?
For example, how can I retrieve the name of the pod which runs the process below (1747):
root#k8s-worker-node:/# ps -ef | grep mysql
1000 1747 1723 0 08:58 ? 00:00:01 mysqld

Assuming that you're looking at the primary process in a pod, you could do something like this:
crictl ps -q | while read cid; do
if crictl inspect -o go-template --template '{{ .info.pid }}' $cid | grep -q $target_pid; then
echo $cid
fi
done
This walks through all the crictl managed pods and checks the pod pid against the value of the $target_pid value (which you have set beforehand to the host pid in which you are interested).

1. Using pid2pod
pid2pod is a dedicated tool: https://github.com/k8s-school/pid2pod
Example:
# Install
$ curl -Lo ./pid2pod https://github.com/k8s-school/pid2pod/releases/download/v0.0.1/pid2pod-linux-amd64
$ chmod +x ./pid2pod
$ mv ./pid2pod /some-dir-in-your-PATH/pid2pod
# Run
$ ./pid2pod 1525
NAMESPACE POD CONTAINER PRIMARY PID
kube-system calico-node-6kt29 calico-node 1284
2. Using sysdig OSS
Install sysdig and run:
sudo csysdig -pc
You'll get something in the htop's style:
3. Custom script
Using #Iarsks answer, I propose a solution based on command belown which provides the pod name for a given PID:
$ pid=1254
$ nsenter -t $pid -u hostname
coredns-558bd4d5db-rxz9r
This solution display namespace, pod, container and container's primary PID. It is possible to copy paste the script below in a file named get_pid.sh and then run ./get_pid.sh 2345 for example.
#!/bin/bash
# Display pod information about a process, using its host PID as input
set -euo pipefail
usage() {
cat << EOD
Usage: `basename $0` PID
Available options:
-h this message
Display pod information about a process, using its host PID as input:
- display namespace, pod, container, and primary process pid for this container if the process is running in a pod
- else exit with code 1
EOD
}
if [ $# -ne 1 ] ; then
usage
exit 2
fi
pid=$1
is_running_in_pod=false
pod=$(nsenter -t $pid -u hostname 2>&1)
if [ $? -ne 0 ]
then
printf "%s %s:\n %s" "nsenter command failed for pid" "$pid" "$pod"
fi
cids=$(crictl ps -q)
for cid in $cids
do
current_pod=$(crictl inspect -o go-template --template '{{ index .info.config.labels "io.kubernetes.pod.name"}}' "$cid")
if [ "$pod" == "$current_pod" ]
then
tmpl='NS:{{ index .info.config.labels "io.kubernetes.pod.namespace"}} POD:{{ index .info.config.labels "io.kubernetes.pod.name"}} CONTAINER:{{ index .info.config.labels "io.kubernetes.container.name"}} PRIMARY PID:{{.info.pid}}'
crictl inspect --output go-template --template "$tmpl" "$cid"
is_running_in_pod=true
break
fi
done
if [ "$is_running_in_pod" = false ]
then
echo "Process $pid is not running in a pod."
exit 1
fi
WARNING: this solution does not work if two pods have the same name (even in different namespaces)

Related

Kubectl port forward reliably in a shell script

I am using kubectl port-forward in a shell script but I find it is not reliable, or doesn't come up in time:
kubectl port-forward ${VOLT_NODE} ${VOLT_CLUSTER_ADMIN_PORT}:${VOLT_CLUSTER_ADMIN_PORT} -n ${NAMESPACE} &
if [ $? -ne 0 ]; then
echo "Unable to start port forwarding to node ${VOLT_NODE} on port ${VOLT_CLUSTER_ADMIN_PORT}"
exit 1
fi
PORT_FORWARD_PID=$!
sleep 10
Often after I sleep for 10 seconds, the port isn't open or forwarding hasn't happened. Is there any way to wait for this to be ready. Something like kubectl wait would be ideal, but open to shell options also.
I took #AkinOzer's comment and turned it into this example where I port-forward a postgresql database's port so I can make a pg_dump of the database:
#!/bin/bash
set -e
localport=54320
typename=service/pvm-devel-kcpostgresql
remoteport=5432
# This would show that the port is closed
# nmap -sT -p $localport localhost || true
kubectl port-forward $typename $localport:$remoteport > /dev/null 2>&1 &
pid=$!
# echo pid: $pid
# kill the port-forward regardless of how this script exits
trap '{
# echo killing $pid
kill $pid
}' EXIT
# wait for $localport to become available
while ! nc -vz localhost $localport > /dev/null 2>&1 ; do
# echo sleeping
sleep 0.1
done
# This would show that the port is open
# nmap -sT -p $localport localhost
# Actually use that port for something useful - here making a backup of the
# keycloak database
PGPASSWORD=keycloak pg_dump --host=localhost --port=54320 --username=keycloak -Fc --file keycloak.dump keycloak
# the 'trap ... EXIT' above will take care of kill $pid

Why does kubectl cp command terminates with exit code 126?

I am trying to copy files from the pod to local using following command:
kubectl cp /namespace/pod_name:/path/in/pod /path/in/local
But the command terminates with exit code 126 and copy doesn't take place.
Similarly while trying from local to pod using following command:
kubectl cp /path/in/local /namespace/pod_name:/path/in/pod
It throws the following error:
OCI runtime exec failed: exec failed: container_linux.go:367: starting container process caused: exec: "tar": executable file not found in $PATH: unknown
Please help through this.
kubectl cp is actually a very small wrapper around kubectl exec whatever tar c | tar x. A side effect of this is that you need a working tar executable in the target container, which you do not appear to have.
In general kubectl cp is best avoided, it's usually only good for weird debugging stuff.
kubectl cp requires the tar to be present in your container, as the help says:
!!!Important Note!!!
Requires that the 'tar' binary is present in your container
image. If 'tar' is not present, 'kubectl cp' will fail.
Make sure your container contains the tar binary in its $PATH
An alternative way to copy a file from local filesystem into a container:
cat [local file path] | kubectl exec -i -n [namespace] [pod] -c [container] "--" sh -c "cat > [remote file path]"
Useful command to copy the file from pod to local
kubectl exec -n <namespace> <pod> -- cat <filename with path> > <filename>
For me the cat worked like this:
cat <file name> | kubectl exec -i <pod-id> -- sh -c "cat > <filename>"
Example:
cat file.json | kubectl exec -i server-77b7976cc7-x25s8 -- sh -c "cat > /tmp/file.json"
Didn't need to specify namespace since I run the command from a specific project, and since we have one container, didn't need to specify it

what is the root password of telepresence in kubernetes remote debugging

I am using telepresence to remote debugging the kubernetes cluster, and I am log in cluster using command:
telepresence
but when I want to install some software in the telepresence pod:
sudo apt-get install wget
and I did not know the password of telepresence pod, so what should I do to install software?
you could using this script to login pod as root:
#!/usr/bin/env bash
set -xe
POD=$(kubectl describe pod "$1")
NODE=$(echo "$POD" | grep -m1 Node | awk -F'/' '{print $2}')
CONTAINER=$(echo "$POD" | grep -m1 'Container ID' | awk -F 'docker://' '{print $2}')
CONTAINER_SHELL=${2:-bash}
set +e
ssh -t "$NODE" sudo docker exec --user 0 -it "$CONTAINER" "$CONTAINER_SHELL"
if [ "$?" -gt 0 ]; then
set +x
echo 'SSH into pod failed. If you see an error message similar to "executable file not found in $PATH", please try:'
echo "$0 $1 sh"
fi
login like this:
./login-k8s-pod.sh flink-taskmanager-54d85f57c7-wd2nb

Running a command on all kubernetes pods of a service

Hey I'm running a kubernetes cluster and I want to run a command on all pods that belong to a specific service.
As far as I know kubectl exec can only run on a pod and tracking all my pods is a ridiculous amount of work (which is one of the benefits of services).
Is there any way or tool that gives you the ability to "broadcast" to all pods in a service?
Thanks in advance for your help!
Here's a simple example with kubectl pipe to xargs, printing env of each pod:
k get pod \
-l {your label selectors} \
--field-selector=status.phase=Running \
-o custom-columns=name:metadata.name --no-headers \
| xargs -I{} kubectl exec {} env
As Bal Chua wrote, kubectl has no way to do this, but you can use bash script to do this:
#!/usr/bin/env bash
PROGNAME=$(basename $0)
function usage {
echo "usage: $PROGNAME [-n NAMESPACE] [-m MAX-PODS] -s SERVICE -- COMMAND"
echo " -s SERVICE K8s service, i.e. a pod selector (required)"
echo " COMMAND Command to execute on the pods"
echo " -n NAMESPACE K8s namespace (optional)"
echo " -m MAX-PODS Max number of pods to run on (optional; default=all)"
echo " -q Quiet mode"
echo " -d Dry run (don't actually exec)"
}
function header {
if [ -z $QUIET ]; then
>&2 echo "###"
>&2 echo "### $PROGNAME $*"
>&2 echo "###"
fi
}
while getopts :n:s:m:qd opt; do
case $opt in
d)
DRYRUN=true
;;
q)
QUIET=true
;;
m)
MAX_PODS=$OPTARG
;;
n)
NAMESPACE="-n $OPTARG"
;;
s)
SERVICE=$OPTARG
;;
\?)
usage
exit 0
;;
esac
done
if [ -z $SERVICE ]; then
usage
exit 1
fi
shift $(expr $OPTIND - 1)
while test "$#" -gt 0; do
if [ "$REST" == "" ]; then
REST="$1"
else
REST="$REST $1"
fi
shift
done
if [ "$REST" == "" ]; then
usage
exit 1
fi
PODS=()
for pod in $(kubectl $NAMESPACE get pods --output=jsonpath={.items..metadata.name}); do
echo $pod | grep -qe "^$SERVICE" >/dev/null 2>&1
if [ $? -eq 0 ]; then
PODS+=($pod)
fi
done
if [ ${#PODS[#]} -eq 0 ]; then
echo "service not found in ${NAMESPACE:-default}: $SERVICE"
exit 1
fi
if [ ! -z $MAX_PODS ]; then
PODS=("${PODS[#]:0:$MAX_PODS}")
fi
header "{pods: ${#PODS[#]}, command: \"$REST\"}"
for i in "${!PODS[#]}"; do
pod=${PODS[$i]}
header "{pod: \"$(($i + 1))/${#PODS[#]}\", name: \"$pod\"}"
if [ "$DRYRUN" != "true" ]; then
kubectl $NAMESPACE exec $pod -- $REST
fi
done
Here:
kubectl -n alex get pods -l app=alex-admin-api -o name | xargs -I{} kubectl -n alex exec {} -- cat alexAdminApi.log >> alex-admin-api_pods.logs
I have written a simple kubectl plugin that "boardcast"s commands to all pods, using Tmux. Assuming that all your pods in the service should share the same labels in their spec, app=foobar for instance, you can use the command below,
kubectl tmux-exec -l app=foobar bash
The plugin is available on Github: predatorray/kubectl-tmux-exec. Hope it will help you!
there is also a kubectl plugin called kubectl-exec-all https://github.com/jpdasma/kubectl-exec-all that is doing a good job.
Can't work with krew index but you can save the content of https://raw.githubusercontent.com/jpdasma/kubectl-exec-all/master/unix/exec-all.sh to /usr/local/bin/kubectl-exec_all and don't forget to make it executable by running
chmod +x /usr/local/bin/kubectl-exec_all
then it work like this:
kubectl plugin exec-all daemonset docker-daemon -- docker system prune -a -f

Executing multiple commands( or from a shell script) in a kubernetes pod

I'm writing a shell script which needs to login into the pod and execute a series of commands in a kubernetes pod.
Below is my sample_script.sh:
kubectl exec octavia-api-worker-pod-test -c octavia-api bash
unset http_proxy https_proxy
mv /usr/local/etc/octavia/octavia.conf /usr/local/etc/octavia/octavia.conf-orig
/usr/local/bin/octavia-db-manage --config-file /usr/local/etc/octavia/octavia.conf upgrade head
After running this script, I'm not getting any output.
Any help will be greatly appreciated
Are you running all these commands as a single line command? First of all, there's no ; or && between those commands. So if you paste it as a multi-line script to your terminal, likely it will get executed locally.
Second, to tell bash to execute something, you need: bash -c "command".
Try running this:
$ kubectl exec POD_NAME -- bash -c "date && echo 1"
Wed Apr 19 19:29:25 UTC 2017
1
You can make it multiline like this:
$ kubectl exec POD_NAME -- bash -c "date && \
echo 1 && \
echo 2"
The following should work
kubectl -it exec podname -- bash -c "ls && ls"
bin dev etc home proc root run sys tmp usr var bin
dev etc home proc root run sys tmp usr var
If above command doesn't work then try too replace bash with one of the following /bin/bash, sh or /bin/sh
-t
can solve your task
For example, I run here few cmd:
kubectl get pods |grep nginx|cut -f1 -d\ |\
while read pod; \
do echo "$pod writing:";\
kubectl exec -t $pod -- bash -c \
"dd if=/dev/zero of=/feeds/test.bin bs=260K count=4 2>&1|\
grep copi |cut -d, -f4; \
a=$SECONDS; echo -ne 'reading:'; cat /feeds/test.bin >/dev/null ; \
let a=SECONDS-a ; \
echo $a sec"
done
p.s. your example will be:
kubectl exec -t octavia-api-worker-pod-test -c octavia-api -- bash -c "unset http_proxy https_proxy ; mv /usr/local/etc/octavia/octavia.conf /usr/local/etc/octavia/octavia.conf-orig ; /usr/local/bin/octavia-db-manage --config-file /usr/local/etc/octavia/octavia.conf ; upgrade ; head"
Posting here because google search still brings you to this post...
I'd like to throw out using a HEREDOC as an additional possibility.
kubectl exec -i --tty-false PODNAME -- bash << EOF
echo "insert all your commands here."
echo "this subprocess will even pickup any variables you have in"
echo "the shell script that is calling this"
EOF