I am bit stuck with a problem I have.
I am struggling to set up the cni-plugin for k8. I was trying to install different cni-plugins that now I think many things are messed up.
Is there a way to neatly delete everything connected with a cni-plugin so that I can have a clean starting point? The goal is to avoid formatting my whole machine.
From this Stack Question.
steps to remove old calico configs from kubernetes without kubeadm reset:
clear ip route: ip route flush proto bird
remove all calico links in all nodes ip link list | grep cali | awk '{print $2}' | cut -c 1-15 | xargs -I {} ip link delete {}
remove ipip module modprobe -r ipip
remove calico configs rm /etc/cni/net.d/10-calico.conflist && rm /etc/cni/net.d/calico-kubeconfig
restart kubelet service kubelet restart
After those steps all the running pods won't be connect, then I have to delete all the pods, then all the pods works. This has litter influence if you are using replicaset
Or else you can use:
kubeadm reset command. this will un-configure the kubernetes cluster.
How can I distribute the kube config file on worker nodes?
Only my master node has it (file under ~/.kube/config), and I'm not sure what's the proper way to programmatically copy the file over to the worker nodes, so that I can use kubectl on all nodes.
You can use the scp command in order to copy a file from one machine to another one.
run the following command from your master node for each worker node :
[user#k8s-master]$ scp ~/.kube/config username#k8s-worker1:~/.kube/config
It is not recommended that you have ~/.kube/config on the worker nodes. If a worker node is compromised due to a vulnerable pod, it could compromise the cluster using this config.
Thats why it is recommended to use a bastion host and use kube context.
However, for non-prod environments,
you can do something like this
kubectl get no --no-headers | egrep -v "master|controlplane" | awk '{print $1}' | while read line; do
scp -pr ~/.kube/ ${line}:~/.kube;
done
scp -pr will create the .kube directory if it doesn't exist on the worker nodes
I tried this command:
kubectl logs --tail
I got this error/help output:
Error: flag needs an argument: --tail
Aliases:
logs, log
Examples:
# Return snapshot logs from pod nginx with only one container
kubectl logs nginx
# Return snapshot logs for the pods defined by label app=nginx
kubectl logs -lapp=nginx
# Return snapshot of previous terminated ruby container logs from pod web-1
kubectl logs -p -c ruby web-1
# Begin streaming the logs of the ruby container in pod web-1
kubectl logs -f -c ruby web-1
# Display only the most recent 20 lines of output in pod nginx
kubectl logs --tail=20 nginx
# Show all logs from pod nginx written in the last hour
kubectl logs --since=1h nginx
# Return snapshot logs from first container of a job named hello
kubectl logs job/hello
# Return snapshot logs from container nginx-1 of a deployment named nginx
kubectl logs deployment/nginx -c nginx-1
ummm I just want to see all the logs, isn't this a common thing to want to do? How can I tail all the logs for a cluster?
kail from the top answer is Linux and macOS only, but Stern also works on Windows.
It can do pod matching based on e.g. a regex match for the name, and then can follow the logs.
To follow ALL pods without printing any prior logs from the default namespace you would run e.g.:
stern ".*" --tail 0
For absolutely everything, incl. internal stuff happening in kube-system namespace:
stern ".*" --all-namespaces --tail 0
Alternatively you could e.g. follow all login-.* containers and get some context with
stern "login-.*" --tail 25
If you don't mind using a third party tool, kail does exactly what you're describing.
Streams logs from all containers of all matched pods. [...] With no arguments, kail matches all pods in the cluster.
The only thing you can do is to get logs of multiple pods using label selectors like this:
kubectl logs -f -l app=nginx -l app=php
For getting all logs of the entire cluster you have to setup centralized log collection like Elasticsearch, Fluentd and Kibana. Simplest way to do it is installation using Helm charts like described here: https://linux-admin.tech/kubernetes/logging/2018/10/24/elk-stack-installation.html
I would recommend using a nice bash script named kubetail.
You can just download the bash script and add it to in your project and run for example:
$ ./some-tools-directory/kubetail.sh --selector app=user --since 10m
To see all pods with the label app=user.
Notice the nice display of colors per pod:
(*) Run ./tools/kubetail.sh -h to see some nice execution options.
kubetail.sh <search term> [-h] [-c] [-n] [-t] [-l] [-d] [-p] [-s] [-b] [-k] [-v] [-r] [-i] -- tail multiple Kubernetes pod logs at the same time
where:
-h, --help Show this help text
-c, --container The name of the container to tail in the pod (if multiple containers are defined in the pod).
Defaults to all containers in the pod. Can be used multiple times.
-t, --context The k8s context. ex. int1-context. Relies on ~/.kube/config for the contexts.
-l, --selector Label selector. If used the pod name is ignored.
-n, --namespace The Kubernetes namespace where the pods are located (defaults to "default")
-f, --follow Specify if the logs should be streamed. (true|false) Defaults to true.
-d, --dry-run Print the names of the matched pods and containers, then exit.
-p, --previous Return logs for the previous instances of the pods, if available. (true|false) Defaults to false.
-s, --since Only return logs newer than a relative duration like 5s, 2m, or 3h. Defaults to 10s.
-b, --line-buffered This flags indicates to use line-buffered. Defaults to false.
-e, --regex The type of name matching to use (regex|substring)
-j, --jq If your output is json - use this jq-selector to parse it.
example: --jq ".logger + \" \" + .message"
-k, --colored-output Use colored output (pod|line|false).
pod = only color pod name, line = color entire line, false = don't use any colors.
Defaults to line.
-z, --skip-colors Comma-separated list of colors to not use in output
If you have green foreground on black, this will skip dark grey and some greens -z 2,8,10
Defaults to: 7,8
--timestamps Show timestamps for each log line
--tail Lines of recent log file to display. Defaults to -1, showing all log lines.
-v, --version Prints the kubetail version
-r, --cluster The name of the kubeconfig cluster to use.
-i, --show-color-index Show the color index before the pod name prefix that is shown before each log line.
Normally only the pod name is added as a prefix before each line, for example "[app-5b7ff6cbcd-bjv8n]",
but if "show-color-index" is true then color index is added as well: "[1:app-5b7ff6cbcd-bjv8n]".
This is useful if you have color blindness or if you want to know which colors to exclude (see "--skip-colors").
Defaults to false.
examples:
kubetail.sh my-pod-v1
kubetail.sh my-pod-v1 -c my-container
kubetail.sh my-pod-v1 -t int1-context -c my-container
kubetail.sh '(service|consumer|thing)' -e regex
kubetail.sh -l service=my-service
kubetail.sh --selector service=my-service --since 10m
kubetail.sh --tail 1
I have hardly ever seen anyone pulling all logs from entire clusters, because you usually either need logs to manually search for certain issues or follow (-f) a routine, or collect audit information, or stream all logs to a log sink to have them prepared for monitoring (e.g. prometheus).
However, if there's a need to fetch all logs, using the --tail option is not what you're looking for (tail only shows the last number of lines of a certain log source and avoids spilling the entire log history of a single log source to your terminal).
For kubernetes, you can write a simple script in a language of your choice (bash, Python, whatever) to kubectl get all --show-all --all-namespaces and iterate over the pods to run kubectl -n <namespace> logs <pod>; but be aware that there might be multiple containers in a pod with individual logs each, and also logs on the cluster nodes themselves, state changes in the deployments, extra meta information that changes, volume provisioning, and heaps more.
That's probably the reason why it's quite uncommon to pull all logs from an entire cluster and thus there's no easy (shortcut) way to do so.
# assumes you have pre-set the KUBECONFIG or using the default one ...
do_check_k8s_logs(){
# set the desired namespaces here vvvv
for namespace in `echo apiv2 default kube-system`; do
while read -r pod ; do
while read -r container ; do
kubectl -n $namespace logs $pod $container | tail -n 2000
done < <(kubectl -n $namespace get pods -o json | jq -r ".items[]|select(.metadata.name | contains ( \"$pod\"))| .status.containerStatuses[].name") ;
done < <(kubectl -n $namespace get pods -o json | jq -r '.items[].metadata.name') \
| tee -a ~/Desktop/k8s-$namespace-logs.`date "+%Y%m%d_%H%M%S"`.log
done
}
do_check_k8s_logs
For your applications data, you probably just want to tail all the pods in the cluster.
But if you want logs for the control-plane of a cluster - you can use:
https://aws.amazon.com/about-aws/whats-new/2019/04/amazon-eks-now-delivers-kubernetes-control-plane-logs-to-amazon-/
I'm trying to get access to mongodb replicaset via kubectl, so I won't expose it to internet, I can't use OpenVPN since Calico blocks it.
So I'm using this script:
export MONGO_POD_NAME1=$(kubectl get pods --namespace develop -l "app=mongodb-replicaset" -o jsonpath="{.items[0].metadata.name}")
export MONGO_POD_NAME2=$(kubectl get pods --namespace develop -l "app=mongodb-replicaset" -o jsonpath="{.items[1].metadata.name}")
export MONGO_POD_NAME3=$(kubectl get pods --namespace develop -l "app=mongodb-replicaset" -o jsonpath="{.items[2].metadata.name}")
echo $MONGO_POD_NAME1, $MONGO_POD_NAME2, $MONGO_POD_NAME3
kubectl port-forward --namespace develop $MONGO_POD_NAME1 27020:27017 & p3=$!
kubectl port-forward --namespace develop $MONGO_POD_NAME2 27021:27017 & p4=$!
kubectl port-forward --namespace develop $MONGO_POD_NAME3 27022:27017 & p5=$!
wait -n
[ "$?" -gt 1 ] || kill "$p3" "$p4" "$p5"
wait
And my connection string looks like this:
mongodb://LOGIN:PW#localhost:27020,localhost:27021,localhost:27022/animedb?replicaSet=rs0
However, I still can't connect to my mongodb replicaset, it says:
connection error: { MongoNetworkError: failed to connect to server
[anime-data-develop-mongodb-replicaset-0.anime-data-develop-mongodb-replicaset.develop.svc.cluster.local:27017]
on first connect [MongoNetworkError: getaddrinfo ENOTFOUND
anime-data-develop-mongodb-replicaset-0.anime-data-develop-mongodb-replicaset.develop.svc.cluster.local
anime-data-develop-mongodb-replicaset-0.anime-data-develop-mongodb-replicaset.develop.svc.cluster.local:27017]
But if I use direct connection, I still can connect to each node!
What might be a problem here? How can I connect to mongodb for development?
Port Forwarding will make a local port on your machine redirect (forward) traffic to some pod. In your case, you've asked Kubernetes to forward traffic on 127.0.0.1:27020 to your pod's 27017 port.
The issue happen because the Replica Set configuration points to the other nodes using your internal cluster IPs, so you will see something like [ReplicaSetMonitor-TaskExecutor] changing hosts to rs0/<ClusterIP-1>:27017,<ClusterIP-2>:27017,<ClusterIP-3>:27017 from rs/localhost:27020,localhost:27021,localhost:27022 on your mongo client session, and your machine can't reach your Cluster's IPs, of course.
For development purposes, you'd have to connect to your primary Mongo node only (as in mongodb://localhost:27020/animedb), which will replicate your data into your secondaries. That's safe enough for development/debugging, but not suitable for production!
If you need to set it up for permanent/production access, you should update your replicaSet settings so they find each other using public IPs or hostnames, see https://docs.mongodb.com/manual/tutorial/change-hostnames-in-a-replica-set/.
So I have a Kubernetes cluster, and I am using Flannel for an overlay network. It has been working fine (for almost a year actually) then I modified a service to have 2 ports and all of a sudden I get this about a completely different service, one that was working previously and I did not edit:
<Timestamp> <host> flanneld[873]: I0407 18:36:51.705743 00873 vxlan.go:345] L3 miss: <Service's IP>
<Timestamp> <host> flanneld[873]: I0407 18:36:51.705865 00873 vxlan.go:349] Route for <Service's IP> not found
Is there a common cause to this? I am using Kubernetes 1.0.X and Flannel 0.5.5 and I should mention only one node is having this issue, the rest of the nodes are fine. The bad node's kube-proxy is also saying it can't find the service's endpoint.
Sometime flannel will change it's subnet configuration... you can tell this if the IP and MTU from cat /run/flannel/subnet.env doesn't match ps aux | grep docker (or cat /etc/default/docker)... in which case you will need to reconfigure docker to use the new flannel config.
First you have to delete the docker network interface
sudo ip link set dev docker0 down
sudo brctl delbr docker0
Next you have to reconfigure docker to use the new flannel config.
Note: sometimes this step has to be done manually (i.e. read the contents of /run/flannel/subnet.env and then alter /etc/default/docker)
source /run/flannel/subnet.env
echo DOCKER_OPTS=\"-H tcp://127.0.0.1:4243 -H unix:///var/run/docker.sock --bip=${FLANNEL_SUBNET} --mtu=${FLANNEL_MTU}\" > /etc/default/docker
Finally, restart docker
sudo service docker restart