I've tried to change kube-proxy configMap and kube-proxy command to set metricsBindAddress but kubernetes resets these changes(without any warnings) after couple seconds.
kubectl edit cm kube-proxy-config -n kube-system => add metricsBindAddress => wait couple seconds and open the config - there is empty metricsBindAddress
kubectl edit ds kube-proxy -n kube-system => add --metrics-bind-address to command => wait couple seconds => the command was reset to default
How to change kube-proxy config and keep these changes ?
Kubernetes version 1.17
UPDATE(as you can, after several seconds metricsBindAddress was changed to empty string):
UPDATE 2(pay attention on metricsBinAddress, it's changed after ~40-50 seconds):
FINAL UPDATE:
Answer from cloud provider(Yandex) - kube-proxy pod it is on the host's network, so to prevent security problems, it listens exclusively on the loopback address and therefore the parameter will be reset
p.s. https://github.com/helm/charts/tree/master/stable/prometheus-operator#kubeproxy - I want to make kube-proxy accessible by prometheus
First edit:
kubectl edit cm/kube-proxy -n kube-system
.....
metricsBindAddress: 0.0.0.0:10249
.....
Then,
kubectl rollout restart ds kube-proxy -n kube-system
You have to restart the pods otherwise they do not get the configuration.
You can check the status by:
kubectl rollout status ds kube-proxy -n kube-system
I am posting this Community Wiki because root cause of the issue has been determined.
Usually to change of metricsBindAddress: can be achieved by editing ConfigMap and delete kube-proxy pod or use rollout restart on DaemonSet.
Root cause of this issue was that this change was blocked by OP's environment - Yandex Cloud.
OP received feedback from Yandex Support
kube-proxy pod it is on the host's network, so to prevent security problems, it listens exclusively on the loopback address and therefore the parameter will be reset
Related
I have a coredns pod running in kube-system namespace. I need to restart the coredns pod without downtime. I am aware that we can do delete coredns pod using below command and new coredns pod will spin up automatically.
kubectl delete pods -n kube-system -l k8s-app=kube-dns
But it is creating downtime. So I am expecting to do a restart of this coredns pod without downtime.
Usecase: I have set TTL as 1hr i.e, cache 3600. I have an external dns server where I will forward any requests to that external dns server using forward plugin. And whenever I need to get recent changes in external dns entries before TTL gets expired, I think we need to restart the coredns pod. Is there any other way to achieve this? If restart is the only way, how can I do it without downtime? It would be really helpful when someone helps me on this. Thanks in advance!
Normally, the result of this command kubectl get deployment coredns --namespace kube-system --output jsonpath='{.spec.strategy.rollingUpdate.maxUnavailable}' will return 1; means for deployment of 2 pods (typical coredns setup), pod will be replace 1 at a time, leaving the other one serving request. In this case, you can run kubectl rollout restart deployment coredns --namespace kube-system to restart without downtime, no explicitly delete or hike coredns pod count required.
So this has been working forever. I have a few simple services running in GKE and they refer to each other via the standard service.namespace DNS names.
Today all DNS name resolution stopped working. I haven't changed anything, although this may have been triggered by a master upgrade.
/ambassador # nslookup ambassador-monitor.default
nslookup: can't resolve '(null)': Name does not resolve
nslookup: can't resolve 'ambassador-monitor.default': Try again
/ambassador # cat /etc/resolv.conf
search default.svc.cluster.local svc.cluster.local cluster.local c.snowcloud-01.internal google.internal
nameserver 10.207.0.10
options ndots:5
Master version 1.14.7-gke.14
I can talk cross-service using their IP addresses, it's just DNS that's not working.
Not really sure what to do about this...
The easiest way to verify if there is a problem with your Kube DNS is to look at the logs StackDriver [https://cloud.google.com/logging/docs/view/overview].
You should be able to find DNS resolution failures in the logs for the pods, with a filter such as the following:
resource.type="container"
("UnknownHost" OR "lookup fail" OR "gaierror")
Be sure to check logs for each container. Because the exact names and numbers of containers can change with the GKE version, you can find them like so:
kubectl get pod -n kube-system -l k8s-app=kube-dns -o \
jsonpath='{range .items[*].spec.containers[*]}{.name}{"\n"}{end}' | sort -u kubectl get pods -n kube-system -l k8s-app=kube-dns
Has the pod been restarted frequently? Look for OOMs in the node console. The nodes for each pod can be found like so:
kubectl get pod -n kube-system -l k8s-app=kube-dns -o \
jsonpath='{range .items[*]}{.spec.nodeName} pod={.metadata.name}{"\n"}{end}'
The kube-dns pod contains four containers:
kube-dns process watches the Kubernetes master for changes in Services and Endpoints, and maintains in-memory lookup structures to serve DNS requests,
dnsmasq adds DNS caching to improve performance,
sidecar provides a single health check endpoint while performing dual health checks (for dnsmasq and kubedns). It also collects dnsmasq metrics and exposes them in the Prometheus format,
prometheus-to-sd scraping the metrics exposed by sidecar and sending them to Stackdriver.
By default, the dnsmasq container accepts 150 concurrent requests. Requests beyond this are simply dropped and result in failed DNS resolution, including resolution for metadata. To check for this, view the logs with the following filter:
resource.type="container"resource.labels.cluster_name="<cluster-name>"resource.labels.namespace_id="kube-system"logName="projects/<project-id>/logs/dnsmasq""Maximum number of concurrent DNS queries reached"
If legacy stackdriver logging of cluster is disabled, use the following filter:
resource.type="k8s_container"resource.labels.cluster_name="<cluster-name>"resource.labels.namespace_name="kube-system"resource.labels.container_name="dnsmasq""Maximum number of concurrent DNS queries reached"
If Stackdriver logging is disabled, execute the following:
kubectl logs --tail=1000 --namespace=kube-system -l k8s-app=kube-dns -c dnsmasq | grep 'Maximum number of concurrent DNS queries reached'
Additionally, you can try to use the command [dig ambassador-monitor.default #10.207.0.10] from each nodes to verify if this is only impacting one node. If it is, you can simple re-create the impacted node.
It appears that I hit a bug that caused the gke-metadata server to start crash pooling (which in turn prevented kube-dns from working).
Creating a new pool with a previous version (1.14.7-gke.10) and migrating to it fixed everything.
I am told a fix has already been submitted.
Thank you for your suggestions.
Start by debugging your kubernetes services [1]. This will tell you whether is a k8s resource issue or kubernetes itself is failing. Once you understand that, you can proceed to fix it. You can post results here if you want to follow up.
[1] https://kubernetes.io/docs/tasks/debug-application-cluster/debug-service/
I am running kubernetes inside 'Docker Desktop' on Mac OS High Sierra.
Is it possible to change the flags given to the kubernetes api-server with this setup?
I can see that the api-server is running.
I am able to exec into the api-server container. When I kill the api-server so I could run it with my desired flags, the container is immediately killed.
Try this to find the name of apiserver deployment:
kubectl -n kube-system get deploy | grep apiserver
Grab the name of deployment and edit its configuration:
kubectl -n kube-system edit deploy APISERVER_DEPLOY_NAME
When you do that the editor will open and from there you can change apiserver command line flags. After editing you should save and close editor, then your changes will be applied.
I there is no a deployment for kube-apiserver since those pods are static so they are created and managed by kubelet.
The way to change kube-api's parameters is like #hanx mentioned:
ssh into the master node (not a container);
update the file under - /etc/kubernetes/manifests/;
restart kubelet - systemctl restart kubelet;
I keep reading documentation that gives parameters for kube-proxy, but does not explain how where these parameters are supposed to be used. I create my cluster using az aks create with the azure-cli program, then I get credentials and use kubectl. So far everything I've done has involved yaml for services and deployments and such, but I can't figure out where all this kube-proxy stuff fits into all of this.
I've googled for days. I've opened question issues on github with AKS. I've asked on the kubernetes slack channel, but nobody has responded.
The kube-proxy on all your Kubernetes nodes runs as a Kubernetes DaemonSet and its configuration is stored on a Kubernetes ConfigMap. To make any changes or add/remove options you will have to edit the kube-proxy DaemonSet or ConfigMap on the kube-system namespace.
$ kubectl -n kube-system edit daemonset kube-proxy
or
$ kubectl -n kube-system edit configmap kube-proxy
For a reference on the kube-proxy command line options you can refer to here.
As of Kubernetes 1.2, kube-proxy is now a pod running in the kube-system namespace.
The old init script /etc/init.d/kube-proxy has been removed.
Aside from simply resetting the GCE instance, is there a good way to restart kube-proxy?
I just added an annotation to change the proxy mode, and I need to restart kube-proxy for my change to take effect.
The kube-proxy is run as an addon pod, meaning the Kubelet will automatically restart it if it goes away. This means you can restart the kube-proxy pod by simply deleting it:
$ kubectl delete pod --namespace=kube-system kube-proxy-${NODE_NAME}
Where $NODE_NAME is the node you want to restart the proxy on (this is assuming a default configuration, otherwise kubectl get pods --kube-system should include the list of kube-proxy pods).
If the restarted kube-proxy is missing your annotation change, you may need to update the manifest file, usually found in /etc/kubernetes/manifests on the node.