Where to execute kube-proxy command? - kubernetes

From this article, I can specify 'userspace' as my proxy-mode, but I am unable to understand what command I need to use for it and at what stage? Like after creating deployment or service?
I am running a minikube cluster currently.

kube-proxy is a process that runs on each kubernetes node to manage network connections coming into and out of kubernetes.
You don't run the command as such, but your deployment method (usually kubeadm) configures the options for it to run.
As #Hang Du mentioned, in minikube you can modify it's options by editing the kube-proxy configmap and changing mode to userspace
kubectl -n kube-system edit configmap kube-proxy
Then delete the Pod.
kubectl -n kube-system get pod
kubectl -n kube-system delete pod kube-proxy-XXXXX

If you are using minikube, you can find a DaemonSet named kube-proxy like followings:
$ kubectl get ds -n kube-system kube-proxy -o yaml
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
...
labels:
k8s-app: kube-proxy
name: kube-proxy
namespace: kube-system
...
spec:
...
spec:
containers:
- command:
- /usr/local/bin/kube-proxy
- --config=/var/lib/kube-proxy/config.conf
- --hostname-override=$(NODE_NAME)
env:
- name: NODE_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: spec.nodeName
image: k8s.gcr.io/kube-proxy:v1.15.0
imagePullPolicy: IfNotPresent
name: kube-proxy
...
volumeMounts:
- mountPath: /var/lib/kube-proxy
name: kube-proxy
- mountPath: /run/xtables.lock
name: xtables-lock
- mountPath: /lib/modules
name: lib-modules
readOnly: true
dnsPolicy: ClusterFirst
...
volumes:
- configMap:
defaultMode: 420
name: kube-proxy
name: kube-proxy
- hostPath:
path: /run/xtables.lock
type: FileOrCreate
name: xtables-lock
- hostPath:
path: /lib/modules
type: ""
name: lib-modules
...
Look at the .spec.template.spec.containers[].command, the container runs the kube-proxy command. You can provide the flag --proxy-mode=userspace in the command array.
- command:
- /usr/local/bin/kube-proxy
- --config=/var/lib/kube-proxy/config.conf
- --hostname-override=$(NODE_NAME)
- --proxy-mode=userspace

Related

How to deploy logstash with persistent volume on kubernetes?

Using GKE to deploy logstash by statefulset kind with pvc. Also need to install an output plugin.
When don't use while true; do sleep 1000; done; in container's command args, it can't deploy with pvc successfully.
The pod will cause CrashLoopBackOff error.
Normal Created 13s (x2 over 14s) kubelet Created container logstash
Normal Started 13s (x2 over 13s) kubelet Started container logstash
Warning BackOff 11s (x2 over 12s) kubelet Back-off restarting failed container
From here I found it can try to add sleep. So the statefulset with pvc can deploy successfully.
But when check its logs will find:
/bin/sh: bin/logstash-plugin: No such file or directory
/bin/sh: bin/logstash: No such file or directory
How to do it in a good way to start container to install an logstash output plugin?
The whole manifest file:
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: logstash-to-gcs
namespace: logging
spec:
serviceName: "logstash"
selector:
matchLabels:
app: logstash
updateStrategy:
type: RollingUpdate
replicas: 3
template:
metadata:
labels:
app: logstash
spec:
containers:
- name: logstash
image: docker.elastic.co/logstash/logstash:7.10.0
resources:
limits:
memory: 2Gi
ports:
- containerPort: 5044
volumeMounts:
- name: config-volume
mountPath: /usr/share/logstash/config
- name: logstash-pipeline-volume
mountPath: /usr/share/logstash/pipeline
- name: logstash-data
mountPath: /usr/share/logstash
command: ["/bin/sh","-c"]
args:
- bin/logstash-plugin install logstash-output-google_cloud_storage;
bin/logstash -f /usr/share/logstash/pipeline/logstash.conf;
while true; do sleep 1000; done;
volumes:
- name: config-volume
configMap:
name: logstash-configmap
items:
- key: logstash.yml
path: logstash.yml
- name: logstash-pipeline-volume
configMap:
name: logstash-configmap
items:
- key: logstash.conf
path: logstash.conf
volumeClaimTemplates:
- metadata:
name: logstash-data
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 1Gi
For my containers I have a script I have ran on Container Deployments and it works to install the plugins and then run logstash.
Post Arg> sh /usr/share/logstash/config/plugins.sh
#!binbash
echo running post install scripts for plugins..;
logstash-plugin install logstash-filter-sentimentalizer
logstash-plugin install logstash-input-mysql
echo finished post install scripts for plugins..;
sleep 1
exec logstash

Get value of configMap from mountPath

I created configmap this way.
kubectl create configmap some-config --from-literal=key4=value1
After that i created pod which looks like this
.
I connect to this pod this way
k exec -it nginx-configmap -- /bin/sh
I found the folder /some/path but i could get value from key4.
If you refer to your ConfigMap in your Pod this way:
apiVersion: v1
kind: Pod
metadata:
name: mypod
spec:
containers:
- name: myfrontend
image: nginx
volumeMounts:
- mountPath: "/var/www/html"
name: config-volume
volumes:
- name: config-volume
configMap:
name: some-config
it will be available in your Pod as a file /var/www/html/key4 with the content of value1.
If you rather want it to be available as an environment variable you need to refer to it this way:
apiVersion: v1
kind: Pod
metadata:
name: mypod
spec:
containers:
- name: myfrontend
image: nginx
envFrom:
- configMapRef:
name: some-config
As you can see you don't need for it any volumes and volume mounts.
Once you connect to such Pod by running:
kubectl exec -ti mypod -- /bin/bash
You will see that your environment variable is defined:
root#mypod:/# echo $key4
value1

Can't modify ETCD manifest for Kubernetes static pod

I'd like to modify etcd pod to listening 0.0.0.0(or host machine IP) instead of 127.0.0.1.
I'm working on a migration from a single master to multi-master kubernetes cluster, but I faced with an issue that after I modified /etc/kubernetes/manifests/etcd.yaml with correct settings and restart kubelet and even docker daemons, etcd still working on 127.0.0.1.
Inside docker container I'm steel seeing that etcd started with --listen-client-urls=https://127.0.0.1:2379 instead of host IP
cat /etc/kubernetes/manifests/etcd.yaml
apiVersion: v1
kind: Pod
metadata:
annotations:
scheduler.alpha.kubernetes.io/critical-pod: ""
creationTimestamp: null
labels:
component: etcd
tier: control-plane
name: etcd
namespace: kube-system
spec:
containers:
- command:
- etcd
- --advertise-client-urls=https://192.168.22.9:2379
- --cert-file=/etc/kubernetes/pki/etcd/server.crt
- --client-cert-auth=true
- --data-dir=/var/lib/etcd
- --initial-advertise-peer-urls=https://192.168.22.9:2380
- --initial-cluster=test-master-01=https://192.168.22.9:2380
- --key-file=/etc/kubernetes/pki/etcd/server.key
- --listen-client-urls=https://192.168.22.9:2379
- --listen-peer-urls=https://192.168.22.9:2380
- --name=test-master-01
- --peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt
- --peer-client-cert-auth=true
- --peer-key-file=/etc/kubernetes/pki/etcd/peer.key
- --peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
- --snapshot-count=10000
- --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
image: k8s.gcr.io/etcd-amd64:3.2.18
imagePullPolicy: IfNotPresent
livenessProbe:
exec:
command:
- /bin/sh
- -ec
- ETCDCTL_API=3 etcdctl --endpoints=https://[192.168.22.9]:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt
--cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt --key=/etc/kubernetes/pki/etcd/healthcheck-client.key
get foo
failureThreshold: 8
initialDelaySeconds: 15
timeoutSeconds: 15
name: etcd
resources: {}
volumeMounts:
- mountPath: /var/lib/etcd
name: etcd-data
- mountPath: /etc/kubernetes/pki/etcd
name: etcd-certs
hostNetwork: true
priorityClassName: system-cluster-critical
volumes:
- hostPath:
path: /var/lib/etcd
type: DirectoryOrCreate
name: etcd-data
- hostPath:
path: /etc/kubernetes/pki/etcd
type: DirectoryOrCreate
name: etcd-certs
status: {}
[root#test-master-01 centos]# kubectl -n kube-system get po etcd-test-master-01 -o yaml
apiVersion: v1
kind: Pod
metadata:
annotations:
kubernetes.io/config.hash: c3eef2d48a776483adc00311df8cb940
kubernetes.io/config.mirror: c3eef2d48a776483adc00311df8cb940
kubernetes.io/config.seen: 2019-05-24T13:50:06.335448715Z
kubernetes.io/config.source: file
scheduler.alpha.kubernetes.io/critical-pod: ""
creationTimestamp: 2019-05-24T14:08:14Z
labels:
component: etcd
tier: control-plane
name: etcd-test-master-01
namespace: kube-system
resourceVersion: "6288"
selfLink: /api/v1/namespaces/kube-system/pods/etcd-test-master-01
uid: 5efadb1c-7e2d-11e9-adb7-fa163e267af4
spec:
containers:
- command:
- etcd
- --advertise-client-urls=https://127.0.0.1:2379
- --cert-file=/etc/kubernetes/pki/etcd/server.crt
- --client-cert-auth=true
- --data-dir=/var/lib/etcd
- --initial-advertise-peer-urls=https://127.0.0.1:2380
- --initial-cluster=test-master-01=https://127.0.0.1:2380
- --key-file=/etc/kubernetes/pki/etcd/server.key
- --listen-client-urls=https://127.0.0.1:2379
- --listen-peer-urls=https://127.0.0.1:2380
- --name=test-master-01
- --peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt
- --peer-client-cert-auth=true
- --peer-key-file=/etc/kubernetes/pki/etcd/peer.key
- --peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
- --snapshot-count=10000
- --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
image: k8s.gcr.io/etcd-amd64:3.2.18
imagePullPolicy: IfNotPresent
livenessProbe:
exec:
command:
- /bin/sh
- -ec
- ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt
--cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt --key=/etc/kubernetes/pki/etcd/healthcheck-client.key
get foo
First check your kubelet option --pod-manifest-path, put your correct yaml in this path.
To make sure etcd pod has been deleted, move yaml file out of pod-manifest-path, wait this pod has been deleted by docker ps -a. Then put your correct yaml file into pod-manifest-path.
Reviewed my automation scripts step by step and found that I've performed a backup of etcd yaml in the same folder with .bak extension. Looks like kubelet daemon uploads all the files inside the manifests folder and despite the file extension.

Hosting local directory to Kubernetes Pod

I have a single node Kubernetes cluster. I want the pod I make to have access to /mnt/galahad on my local computer (which is the host for the cluster).
Here is my Kubernetes config yaml:
apiVersion: v1
kind: Pod
metadata:
name: galahad-test-distributor
namespace: galahad-test
spec:
volumes:
- name: place-for-stuff
hostPath:
path: /mnt/galahad
containers:
- name: galahad-test-distributor
image: vergilkilla/distributor:v9
volumeMounts:
- name: place-for-stuff
mountPath: /mnt
resources:
limits:
memory: "200Mi"
requests:
memory: "100Mi"
I start my pod like such:
kubectl apply -f ./create-distributor.yaml -n galahad-test
I get a terminal into my newly-made pod:
kubectl exec -it galahad-test-distributor -n galahad-test -- /bin/bash
I go to /mnt in my pod and it doesn't have anything from /mnt/galahad. I make a new file in the host /mnt/galahad folder - doesn't reflect in the pod. How do I achieve this functionality to have the host path files/etc. reflect in the pod? Is this possible in the somewhat straightforward way I am trying here (defining it per-pod definition without creating separate PersistentVolumes and PersistentVolumeRequests)?
Your yaml file looks good.
Using this configuration:
apiVersion: v1
kind: Pod
metadata:
name: galahad-test-distributor
namespace: galahad-test
spec:
volumes:
- name: place-for-stuff
hostPath:
path: /mnt/galahad
containers:
- name: galahad-test-distributor
image: busybox
args: [/bin/sh, -c,
'i=0; while true; do echo "$i: $(date)"; i=$((i+1)); sleep 1; done']
volumeMounts:
- name: place-for-stuff
mountPath: /mnt
resources:
limits:
memory: "200Mi"
requests:
memory: "100Mi"
I ran this and everything worked as expected:
>>> kubectl apply -f create-distributor.yaml # side node: you don't need
# to specify the namespace here
# since it's inside the yaml file
pod/galahad-test-distributor created
>>> touch /mnt/galahad/file
>>> kubectl -n galahad-test exec galahad-test-distributor ls /mnt
file
Are you sure you are adding your files in the right place? For instance, if you are running your cluster inside a VM (e.g. minikube), make sure you are adding the files inside the VM, not on the machine hosting the VM.

kuberentes communication is not working cross nodes

I have a procedure of installing kubernetes cluster via kubeadm and it worked multiple times.
for some reason now I have a cluster which I installed and for some reason the nodes are having trouble communicating.
the problem reflect in couple of ways :
sometimes the cluster is unable to resolve global dns records such as mirrorlist.centos.org
sometimes one pod from a specific node has no connectivity to another pod in different node
my kubernetes version is 1.9.2
my hosts are centOS 7.4
I use flannel as cni plugin in version 0.9.1
my cluster is built on AWS
mt debugging so far was :
kubectl get nodes -o jsonpath='{.items[*].spec.podCIDR}' - to see subnets
10.244.0.0/24 10.244.1.0/24
I tried adding configurations to kubedns ( even though it is needed in all my other clusters ) like https://kubernetes.io/docs/tasks/administer-cluster/dns-custom-nameservers/#configure-stub-domain-and-upstream-dns-servers
I tried installing busybox and ding nslookup to cluster kubernetes.default and it only works of busybox is on the same node as the dns ( tried this link https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/
I even tried creating an AMI from other running environments and deploying it as a node to this cluster and it still fails.
I tried checking if some port is missing so I even opened all ports between nodes
I also disabled iptables and firewall and all nodes just to make sure it is not the reason
nothing helps.
please any tip would help
edit :
I added my flannel configuration:
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: flannel
rules:
- apiGroups:
- ""
resources:
- pods
verbs:
- get
- apiGroups:
- ""
resources:
- nodes
verbs:
- list
- watch
- apiGroups:
- ""
resources:
- nodes/status
verbs:
- patch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: flannel
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: flannel
subjects:
- kind: ServiceAccount
name: flannel
namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: flannel
namespace: kube-system
---
kind: ConfigMap
apiVersion: v1
metadata:
name: kube-flannel-cfg
namespace: kube-system
labels:
tier: node
app: flannel
data:
cni-conf.json: |
{
"name": "cbr0",
"type": "flannel",
"delegate": {
"isDefaultGateway": true
}
}
net-conf.json: |
{
"Network": "10.244.0.0/16",
"Backend": {
"Type": "vxlan"
}
}
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
name: kube-flannel-ds
namespace: kube-system
labels:
tier: node
app: flannel
spec:
template:
metadata:
labels:
tier: node
app: flannel
spec:
hostNetwork: true
nodeSelector:
beta.kubernetes.io/arch: amd64
tolerations:
- key: node-role.kubernetes.io/master
operator: Exists
effect: NoSchedule
serviceAccountName: flannel
initContainers:
- name: install-cni
image: quay.io/coreos/flannel:v0.9.1-amd64
command:
- cp
args:
- -f
- /etc/kube-flannel/cni-conf.json
- /etc/cni/net.d/10-flannel.conf
volumeMounts:
- name: cni
mountPath: /etc/cni/net.d
- name: flannel-cfg
mountPath: /etc/kube-flannel/
containers:
- name: kube-flannel
image: quay.io/coreos/flannel:v0.9.1-amd64
command: [ "/opt/bin/flanneld", "--ip-masq", "--kube-subnet-mgr" ]
securityContext:
privileged: true
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
volumeMounts:
- name: run
mountPath: /run
- name: flannel-cfg
mountPath: /etc/kube-flannel/
volumes:
- name: run
hostPath:
path: /run
- name: cni
hostPath:
path: /etc/cni/net.d
- name: flannel-cfg
configMap:
name: kube-flannel-cfg
the issues was that the AWS machines were provisioned not by me and the team that provisioned the machines assured that all internal traffic is opened.
after a lot of debugging with nmap I found out that UDP ports are not opened and since flannel requires UDP traffic the communication was not working properly.
once UDP was opened issues got solved.