After make docker image of cassandra, cassandra-setup, application and zookeeper from my custom thingsboard.
I tried to deploy that to Kubernetes Engine, there's no error, but not running well.
Here is my command for yaml from my github:
curl -L https://raw.githubusercontent.com/Firdauzfan/ThingsboardGSPE/master/docker/k8s/common.yaml > common.yaml
curl -L https://raw.githubusercontent.com/Firdauzfan/ThingsboardGSPE/master/docker/k8s/cassandra.yaml > cassandra.yaml
curl -L https://raw.githubusercontent.com/Firdauzfan/ThingsboardGSPE/master/docker/k8s/zookeeper.yaml > zookeeper.yaml
curl -L https://raw.githubusercontent.com/Firdauzfan/ThingsboardGSPE/master/docker/k8s/tb.yaml > tb.yaml
curl -L https://raw.githubusercontent.com/Firdauzfan/ThingsboardGSPE/master/docker/k8s/cassandra-setup.yaml > cassandra-setup.yaml
and here is my docker image:
https://hub.docker.com/u/firdauzfanani/
Example: when i run command kubectl create -f cassandra.yaml, cassandra engine just show running but not ready.
Status screenshot here
If it is shown as not ready even if it is running with no issue (es: you can ssh into it and all the services are running), could be an misconfiguration of your redinessprobe that I see defined in the YAML file as follow, but I have no clue regarding its behaviour. Consider that accordingly to documentation it should return 0.
readinessProbe:
exec:
command:
- /bin/bash
- -c
- /ready-probe.sh
On the other hand, if when you try to access the pod you face some kind of errors, I would suggest you if you didn't do it already to retrieve further information to carry on the troubleshooting running the following commands:
$ kubectl describe deployments
$ kubectl describe pods
$ kubectl describe services
This series of commands could help you in order to understand better what is going on.
Please run them and edit your initial post with the output and I can take a look to them.
To ssh into the pod run:
$ kubectl get pods (to retrieve pod name)
$ kubectl exec -ti PODNAME /bn/bash
UPDATE
I deployed your YAML files, the pods is running correctly (I believe) what is failing is the probe whose content is the following:
cat /ready-probe.sh
if [[ $(nodetool status | grep $POD_IP) == *"UN"* ]]; then
if [[ $DEBUG ]]; then
echo "UN";
fi
exit 0;
else
if [[ $DEBUG ]]; then
echo "Not Up";
fi
exit 1;
Related
I was able to view the app container logs using kubectl -f logs and was able to login to the container using "k exec --stdin --tty -- /bin/bash".
After injecting linkerd, I could not login to the container. However my goal is to check the app logs.
When I use this "k logs -f linkerd-proxy" I could not see the app-related logs.
I tried injecting debug-sidecar as well.
Tried this - "k logs deploy/ linkerd-debug - " and as well as this "k exec -it -c linkerd-debug -- tshark -i any -f "tcp" -V -Y "http.request"
still I couldn't see the exact logs for my app in the pod. Please suggest.
Linkerd works by injecting an additional container into your pods; this is known as the "sidecar" pattern. Your application (or better said container) logs are still accessible, however, as a result of having more than one container in the pod, kubectl requires you to explicitly specify the container name.
For example, assuming you have a pod with two containers (linkerd-proxy and app), you'd have to specify app as the name of the container:
$ kubectl logs -f <pod-name> -c app
# You can specify the container name without the -c flag
$ kubectl logs -f <pod-name> app
# This will work for 'exec' too
$ kubectl exec <pod-name> -c app -it -- sh
I am trying to run this command in my Argo workflow
kubectl cp /tmp/appendonly.aof redis-node-0:/data/appendonly.aof -c redis -n redis
but I get this error
Error from server (InternalError): an error on the server ("invalid upgrade response: status code 200") has prevented the request from succeeding (get pods redis-node-0)
surprisingly when I am copying the file from a pod to local system then it is working, like this command kubectl cp redis-node-0:/data/appendonly.aof tmp/appendonly.aof -c redis -n redis
Any idea what might be causing it?
Solution -
Not sure what was causing this issue but found this command in the docs that worked fine
tar cf - appendonly.aof | kubectl exec -i -n redis redis-node-0 -- tar xf - -C /data
I often run tasks like:
Read the log of the service X
or
Attach a shell inside the service Y
I always use something in my history like:
kubectl logs `kubectl get pods --no-headers -o custom-columns=":metadata.name" | grep <partial_name>`
or
kubectl exec -it `kubectl get pods --no-headers -o custom-columns=":metadata.name" | grep <partial_name>` bash
Do you know if kubectl has already something in place for this? Or should I create my own set of aliases?
Kubernetes instances are loosely coupled by the means of labels (key-value pairs). Because of that Kubernetes provides various functionalities that can help you to operate on sets of objects based on labels.
In case you have several pods of the same service good chances that they are managed by some ReplicaSet with the use of some specific label. You should see it if you run:
kubectl get pods --show-labels
Now for aggregating logs for instance you could use label selector like:
kubectl logs -l key=value
For more info please see: https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/ .
added to my .zshconfig
sshpod () {
kubectl exec --stdin --tty `kubectl get pods --no-headers -o custom-columns=":metadata.name" | grep ${1} | head -n 1` -- /bin/bash
}
usage
sshpod podname
this
finds all pods
greps needed name
picks the first
sshs into the pod
You can go access a pod by its deployment/service/etc:
kubectl exec -it svc/foo -- bash
kubectl exec -it deployment/bar -- bash
Kubernetes will pick a pod that matches the criteria and send you to it.
You can enable shell autocompletion. Kubectl provides this support for Bash and Zsh which will save you a lot of typing (you will use TAB to get the suggestion/completion).
Kuberentes documentations has a great set of information about how to enable autocompletion under Optional kubectl configurations. It covers Bash on Linux, Bash on MacOS and Zsh.
Looking at documentation for installing Knative requires a Kubernetes cluster v1.11 or newer with the MutatingAdmissionWebhook admission controller enabled. So checking the documentation for this I see the following command:
kube-apiserver -h | grep enable-admission-plugins
However, kube-apiserver is running inside a docker container on master. Logging in as admin to master, I am not seeing this on the command line after install. What steps do I need to take to to run this command? Its probably a basic docker question but I dont see this documented anywhere in Kubernetes documentation.
So what I really need to know is if this command line is the best way to set these plugins and also how exactly to enter the container to execute the command line.
Where is kube-apiserver located
Should I enter the container? What is name of container and how do I enter it to execute the command?
I think that answer from #embik that you've pointed out in the initial question is quite decent, but I'll try to shed light on some aspects that can be useful for you.
As #embik mentioned in his answer, kube-apiserver binary actually resides on particular container within K8s api-server Pod, therefore you can free to check it, just execute /bin/sh on that Pod:
kubectl exec -it $(kubectl get pods -n kube-system| grep kube-apiserver|awk '{print $1}') -n kube-system -- /bin/sh
You might be able to propagate the desired enable-admission-plugins through kube-apiserver command inside this Pod, however any modification will disappear once api-server Pod re-spawns, i.e. master node reboot, etc.
The essential api-server config located in /etc/kubernetes/manifests/kube-apiserver.yaml. Node agent kubelet controls kube-apiserver runtime Pod, and each time when health checks are not successful kubelet sents a request to K8s Scheduler in order to re-create this affected Pod from primary kube-apiserver.yaml file.
This is old, still if its in the benefit of a needy. The a #Nick_Kh's answer is good enough, just want to extend it.
In case the api-server pod fails to give you the shell access, you may directly execute the command using kubectl exec like this:
kubectl exec -it kube-apiserver-rhino -n kube-system -- kube-apiserver -h | grep enable-admission-plugins
In this case, I wanted to know what are the default admission plugins enabled and every time I tried accessing pod's shell (bash, sh, etc.), ended up with error like this:
[root#rhino]# kubectl exec -it kube-apiserver-rhino -n kube-system -- /bin/sh
OCI runtime exec failed: exec failed: container_linux.go:367: starting container process caused: exec: "/bin/sh": stat /bin/sh: no such file or directory: unknown
command terminated with exit code 126
In IBM Cloud Private EE, I need to go to the Web UI User > Configure client, copy the kubectl config commands and then run these 5 commands on my client machine.
I deployed the IBM Cloud private EE on 5 VMs and have access to the master node. I am wondering if there is a way to capture these kubectl config commands directly from the docker containers without having a need to go to the Web UI.
For example: I did not want to download the kubectl client from google (as I just want to use same kubectl version which is in the ICP containers) and I used the following command to get it from the container itself.
docker run --rm -v $(pwd):/data -e LICENSE=accept \
ibmcom/icp-inception:2.1.0.1-ee \
cp -r /usr/local/bin/kubectl /data
Then, I copied this to all VM guests so that I could access kubectl from any guest.
chmod +x kubectl
for host in $(awk '/192.168.142/ {print $3}' /etc/hosts)
do
scp kubectl $host:/bin
done
Where - 192.168.142 is the subnet of my VM guests.
But, I could not figure out how to get Configure Client commands without having to go to the Web UI. I need this to automate client kubectl command so that my environment is ready for kubectl commands through simple scripts.
You should use Vagrant to automate those steps.
For instance, IBM/deploy-ibm-cloud-private/Vagrantfile has this section:
install_kubectl = <<SCRIPT
echo "Pulling #{image_repo}/kubernetes:v#{k8s_version}..."
sudo docker run -e LICENSE=#{license} --net=host -v /usr/local/bin:/data #{image_repo}/kubernetes:v#{k8s_version} cp /kubectl /data &> /dev/null
kubectl config set-credentials icpadmin --username=admin --password=admin &> /dev/null
kubectl config set-cluster icp --server=http://127.0.0.1:8888 --insecure-skip-tls-verify=true &> /dev/null
kubectl config set-context icp --cluster=icp --user=admin --namespace=default &> /dev/null
kubectl config use-context icp &> /dev/null
SCRIPT
See more at "Kubernetes, IBM Cloud Private, and Vagrant, oh my!", from Tim Pouyer.
#VonC provided useful tips. This is how the service account token can be obtained.
Get the token from a running container - Tip from this link.
RUNNIGCONTAINER=$(docker ps | grep k8s_cloudiam-apikeys_auth | awk '{print $1}')
TOKEN=$(docker exec -t $RUNNIGCONTAINER cat /var/run/secrets/kubernetes.io/serviceaccount/token)
I already know the name of the IBM Cloud Private cluster name, master node and the default user name. The only missing link was the token. Please note that the script used by Tim is using password and the only difference was - I wanted to use token instead of the password.
So use the scripts.
kubectl config set-cluster ${CLUSTERNAME}.icp --server=https://$MASTERNODE:8001 --insecure-skip-tls-verify=true
kubectl config set-context ${CLUSTERNAME}.icp-context --cluster=${CLUSTERNAME}.icp
kubectl config set-credentials admin --token=$TOKEN
kubectl config set-context ${CLUSTERNAME}.icp-context --user=$DEFAULTUSERNAME --namespace=default
kubectl config use-context ${CLUSTERNAME}.icp-context
# get token
icp_auth_token=`curl -s -k -H "Content-Type: application/x-www-form-urlencoded;charset=UTF-8" \
-d "grant_type=password&username=${myuser}&password=${mypass}&scope=openid" \
https://${icp_server}:8443/idprovider/v1/auth/identitytoken --insecure | \
sed 's/{//g;s/}//g;s/\"//g' | \
awk -F ':' '{print $7}'`
# setup context
kubectl config set-cluster ${icp_server} --server=https://${icp_server}:8001 --insecure-skip-tls-verify=true
kubectl config set-credentials ${icp_server}-user --token=${icp_auth_token}
kubectl config set-context ${icp_server}-context --cluster=${icp_server} --user=${icp_server}-user
kubectl config use-context ${icp_server}-context