No response when "externalTrafficPolicy" is set to "Local" - kubernetes

I cannot reach the following Kubernetes service when externalTrafficPolicy: Local is set. I access it directly through the NodePort but always get a timeout.
{
"kind": "Service",
"apiVersion": "v1",
"metadata": {
"name": "echo",
"namespace": "default",
"selfLink": "/api/v1/namespaces/default/services/echo",
"uid": "c1b66aca-cc53-11e8-9062-d43d7ee2fdff",
"resourceVersion": "5190074",
"creationTimestamp": "2018-10-10T06:14:33Z",
"labels": {
"k8s-app": "echo"
}
},
"spec": {
"ports": [
{
"name": "tcp-8080-8080-74xhz",
"protocol": "TCP",
"port": 8080,
"targetPort": 3333,
"nodePort": 30275
}
],
"selector": {
"k8s-app": "echo"
},
"clusterIP": "10.101.223.0",
"type": "NodePort",
"sessionAffinity": "None",
"externalTrafficPolicy": "Local"
},
"status": {
"loadBalancer": {}
}
}
I know that for this pods of the service need to be available on a node because traffic is not routed to other nodes. I checked this.

Not sure where you are connecting from and what command you are typing to test connectivity or what's your environment like. But this is most likely due to this known issue where the node ports are not reachable with externalTrafficPolicy set to Local if the kube-proxy cannot find the IP address for the node where it's running on.
This link sheds more light into the problem. Apparently --hostname-override on the kube-proxy is not working as of K8s 1.10. You have to specify the HostnameOverride option in the kube-proxy ConfigMap. There's also a fix described here that will make it upstream at some point in the future from this writing.

As Jagrut said, the link shared in Rico's answer does not contain the desired section with the patch anymore, so I'll share a different thread where stacksonstacks' answer worked for me: here . This solution consists in editing kube-proxy.yaml to include the HOST_IP argument.

In my case, request to node cluster/public ip which own deployment/pod.
spec:
clusterIP: 10.9x.x.x <-- request this ip
clusterIPs:
- 10.9x.x.x
externalTrafficPolicy: Local
or
NAME STATUS ... EXTERNAL-IP
master-node Ready 3.2x.x.x
worker-node Ready 13.1x.x.x <-- request this ip
additionally, alway to request same node's ip, use nodeSelector in Deployment.

Related

kubectl proxy is unable to reach pod services: steps I should attempt to investigate/diagnose?

I have a baremetal k8s cluster (1.23.5, calico as CNI), and after some uptime I encountered that kubectl proxy is unable to reach pod services:
proxying:
kubectl proxy --port=8008
Starting to serve on 127.0.0.1:8008
requesting:
curl http://localhost:8008/api/v1/namespaces/emz/pods/nginx-6b78d5b64c-km5lr:80/proxy/
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {},
"status": "Failure",
"message": "error trying to reach service: dial tcp 10.200.3.21:80: i/o timeout",
"reason": "ServiceUnavailable",
"code": 503
}
No matter what pod, service or namespace - the result is the same. Pods are running without errors, pods can ping each other among different namespaces, DNS resolve is working inside pods, and api objects are browseable inside k8s api (I omit most of the sample output because it's fully normal and cumbersome):
> curl http://localhost:8008/api/v1/namespaces/emz/pods/nginx-6b78d5b64c-km5lr
{
"kind": "Pod",
"apiVersion": "v1",
"metadata": {
"name": "nginx-6b78d5b64c-km5lr",
"generateName": "nginx-6b78d5b64c-",
"namespace": "emz",
"uid": "819b94de-7aa6-441a-a40e-805a0f79e9a8",
"resourceVersion": "8070751",
[...]
"containers": [
{
"name": "nginx",
"image": "nginx",
"ports": [
{
"name": "nginx",
"containerPort": 80,
"protocol": "TCP"
}
],
[...]
"status": {
"phase": "Running",
[...]
"podIP": "10.200.3.21",
"podIPs": [
{
"ip": "10.200.3.21"
}
],
What is even more weird - is that kubectl port-forward is able to reach said services:
port-forward:
> kubectl port-forward -n emz nginx-6b78d5b64c-km5lr 8008:80
Forwarding from 127.0.0.1:8008 -> 80
Forwarding from [::1]:8008 -> 80
request:
> curl 127.0.0.1:8008
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
nginx.org.<br/>
Commercial support is available at
nginx.com.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
Can anyone please point me to the direction on how to diagnose/investigate the reason for the kubectl proxy to stuck in these timeouts ? Cluster seems to be fully operating besides that.
Thanks.
Update: Seems like the reason of this, direct or relatedб may be in the fact that all of the three control nodes lost connectivity via the internal pod network to other pods. I have drained and rebooted one node, but this didn't help at all.
Update 2: In fact it did help to restore the connectivity from this node to the indernal network. Weird, but calico pods/deployments were not complaining about loss of connectivity.

Second NIC with Macvlan on GKE

I need to add a second interface to some of the specific K8s pods on GKE that need to be accessible directly from public users on the Internet. So I used Multus and defined a Macvlan cni like this:
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
name: macvlan-conf
spec:
config: '{
"cniVersion": "0.3.1",
"type": "macvlan",
"master": "eth0",
"mode": "bridge",
"ipam": {
"type": "host-local",
"subnet": "10.162.0.0/20",
"rangeStart": "10.162.0.100",
"rangeEnd": "10.162.0.150",
"routes": [
{ "dst": "0.0.0.0/0" }
],
"gateway": "10.162.0.1"
}
}'
10.162.0.1 is the default gateway of my K8s nodes in GCP. So, I imagine that in this case, pods should have the access to the outside. But in pods, just there is one default gateway that routes the internal pods traffic. Also, I can't add any route because of the privileges issues.
Question:
My expectation is wrong? How I should use Macvlan to create a public interface for those pods?

How to access the application deployed in minikube k8s cluster

I have installed the minikube, deployed the hello-minikube application and opened the port. Basically I have followed the getting started tutorial at https://kubernetes.io/docs/setup/learning-environment/minikube/#quickstart.
The problem starts when I want to open the URL where the deployed application is running obtained by running minikube service hello-minikube --url.
I get http://172.17.0.7:31198 and that URI cannot be opened, since that IP does not exist locally. Changing it to http://localhost:31198 does not work either (so adding an entry to hosts file won't work I guess).
The application is running, I can query the cluster and obtain the info through http://127.0.0.1:50501/api/v1/namespaces/default/services/hello-minikube:
{
"kind": "Service",
"apiVersion": "v1",
"metadata": {
"name": "hello-minikube",
"namespace": "default",
"selfLink": "/api/v1/namespaces/default/services/hello-minikube",
"uid": "56845ce6-bbba-45e5-a1b6-d094949438cf",
"resourceVersion": "1578",
"creationTimestamp": "2020-03-10T10:33:41Z",
"labels": {
"app": "hello-minikube"
}
},
"spec": {
"ports": [
{
"protocol": "TCP",
"port": 8080,
"targetPort": 8080,
"nodePort": 31198
}
],
"selector": {
"app": "hello-minikube"
},
"clusterIP": "10.108.152.177",
"type": "NodePort",
"sessionAffinity": "None",
"externalTrafficPolicy": "Cluster"
},
"status": {
"loadBalancer": {
}
}
}
λ kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
hello-minikube NodePort 10.108.152.177 <none> 8080:31198/TCP 4h34m
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 4h42m
How to access the application deployed in minikube k8s cluster on localhost? Also minikube is running as a docker container on the machine with following ports 32770:2376 32769:8443 32771:22 exposed.
Found the solution in another thread - port forwarding
kubectl port-forward svc/hello-minikube 31191:8080
The first port is port that you will use on your machine (in the browser) and 8080 is the port defined when running the service.

How to get list of metrics available for HPA?

I have GCP cluster which contains GKE application:
I want to scale application using HPA
Based on supporting for metrics
HPA able to read metrics from
metrics.k8s.io (resource metrics)
custom.metrics.k8s.io(custom metrics)
external.metrics.k8s.io(external metrics)
How could I check what metrics available? How could try this API on my own ? Is it possible at all?
P.S.
Based on suggested answer I executed command:
kubectl get --raw https://MY-KUBE-APISERVER-IP:6443/apis/metrics.k8s.io/v1beta1/namespaces/default/pods
Response is:
{
"items": [
{
"metadata": {
"name": "prometheus-adapter-69fcdd56bc-2plh7",
"namespace": "default",
"selfLink": "/\r\napis/metrics.k8s.io/v1beta1/namespaces/default/pods/prometheus-adapter-69fcdd56bc-2plh7",
"creationTimestamp": "2020-02-05T10:56:02Z"
},
"timestamp": "2020-02-05T10:55:22Z",
"window": "30s",
"containers": [
{
"name": "prometheus-adapter",
"usage": {
"cpu": "15\r\n31939n",
"memory": "10408Ki"
}
}
]
},
{
"metadata": {
"name": "stackdriver-exporter-76fdbc9d8f-c285l",
"namespace": "default",
"selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/default/pods/stackdriver-exporter-76fdbc9d8f-c285l",
"creationTimestamp": "2020-0\r\n2-05T10:56:02Z"
},
"timestamp": "2020-02-05T10:55:22Z",
"window": "30s",
"containers": [
{
"name": "stackdriver-exporter",
"usage": {
"cpu": "79340n",
"memory": "2000Ki"
}
}
]
}
],
"kind": "PodMetricsList",
"apiVersion": "metrics.k8s.io/v1beta1",
"metadata": {
"selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/default/pods"
}
}
$ kubectl top pods
NAME CPU(cores) MEMORY(bytes)
prometheus-adapter-69fcdd56bc-2plh7 2m 10Mi
stackdriver-exporter-76fdbc9d8f-c285l 1m 1Mi
But I still don't see all metrics available for HPA
Metrics server exposes metrics via below APIs.
/nodes - all node metrics; type []NodeMetrics
/nodes/{node} - metrics for a specified node; type NodeMetrics
/namespaces/{namespace}/pods - all pod metrics within namespace with
support for all-namespaces; type []PodMetrics
/namespaces/{namespace}/pods/{pod} - metrics for a specified pod;
type PodMetrics
You can view available metrics as below for example
$ kubectl get --raw https://KUBE-APISERVER-IP:6443/apis/metrics.k8s.io/v1beta1
{
"kind": "APIResourceList",
"apiVersion": "v1",
"groupVersion": "metrics.k8s.io/v1beta1",
"resources": [
{
"name": "nodes",
"singularName": "",
"namespaced": false,
"kind": "NodeMetrics",
"verbs": [
"get",
"list"
]
},
{
"name": "pods",
"singularName": "",
"namespaced": true,
"kind": "PodMetrics",
"verbs": [
"get",
"list"
]
}
]
}
$ kubectl get --raw https://KUBE-APISERVER-IP:6443/apis/metrics.k8s.io/v1beta1/namespaces/default/pods
{
"kind": "PodMetricsList",
"apiVersion": "metrics.k8s.io/v1beta1",
"metadata": {
"selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/default/pods"
},
"items": []
}
$ kubectl get --raw https://KUBE-APISERVER-IP:6443/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods
{
"kind": "PodMetricsList",
"apiVersion": "metrics.k8s.io/v1beta1",
"metadata": {
"selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods"
},
"items": [
{
"metadata": {
"name": "coredns-bcccf59f-jfl6x",
"namespace": "kube-system",
"selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/coredns-bcccf59f-jfl6x",
"creationTimestamp": "2021-02-17T20:31:29Z"
},
"timestamp": "2021-02-17T20:30:27Z",
"window": "30s",
"containers": [
{
"name": "coredns",
"usage": {
"cpu": "1891053n",
"memory": "8036Ki"
}
}
]
},
{
"metadata": {
"name": "coredns-bcccf59f-vmfvv",
"namespace": "kube-system",
"selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/coredns-bcccf59f-vmfvv",
"creationTimestamp": "2021-02-17T20:31:29Z"
},
"timestamp": "2021-02-17T20:30:25Z",
"window": "30s",
"containers": [
{
"name": "coredns",
"usage": {
"cpu": "1869226n",
"memory": "8096Ki"
}
}
]
}
]
}
You can also use command kubectl top pods which internally calls the above API.
Custom Metrics
These are provided by adapters developed by vendors and what metrics are available will depend on the adapter. Once you know the metrics name You can use API to access it.
You can view available metrics as below and get the metrics name.
kubectl get --raw https://KUBE-APISERVER-IP:6443/apis/custom.metrics.k8s.io/v1beta1
External Metrics
These are provided by adapters developed by vendors and what metrics are available will depend on the adapter. Once you know the metrics name You can use API to access it.
You can view available metrics as below and get the metrics name.
kubectl get --raw https://KUBE-APISERVER-IP:6443/apis/external.metrics.k8s.io/v1beta1
Edit:
You already have Prometheus adapter but if the metric is not exposed as custom metrics to be consumable by HPA then you need to expose the required metrics. Refer to this guide for this.
On GKE case is bit different.
As default Kubernetes have some built-in metrics (CPU and Memory). If you want to use HPA based on this metric you will not have any issues.
In GCP concept:
Custom Metrics are used when you want to use metrics exported by Kubernetes workload or metric attached to Kubernetes object such as Pod or Node.
External Metrics- Metrics sent to Workspaces with a metric type beginning external.googleapis.com are known as external metrics. The metrics are typically exported by open-source projects and third-party providers. More details can be found here.
Stackdriver Monitoring treats external metrics the same as custom metrics, with one exception. For external metrics, a resource_type of global is invalid and results in the metric data being discarded.
As GKE is integrated with Stackdriver
Google Kubernetes Engine (GKE) includes native integration with Stackdriver Monitoring and Stackdriver Logging. When you create a GKE cluster, Stackdriver Kubernetes Engine Monitoring is enabled by default and provides a monitoring dashboard specifically tailored for Kubernetes.
With Stackdriver Kubernetes Engine Monitoring, you can control whether or not Stackdriver Logging collects application logs. You also have the option to disable the Stackdriver Monitoring and Stackdriver Logging integration altogether.
Check Available Metrics
As you are using cloud environment - GKE, you can find all default available metrics by curiling localhost on proper port. You have to SSH to one of Nodes and then curl metric-server $ curl localhost:10255/metrics.
Second way is to check available metrics documentation.
IMPORTANT
You can see available metrics, however to use them in HPA you need deploy Adapters like Stackdriver adapter or Prometheus adapter. In default (version 1.13.11) GKE cluster you have already deployed metrics-server, heapster(deprecated in newer versions) and prometheus-to-sd-XXX. If you would like to use Stackdriver, you would have many config already applied, but if you want to use Prometheus you will need adjust Prometheus operators, adapters, deployments. Details can be found here.
On GKE docs, you can find some tutorials to use HPA with Custom Metrics or with HPA with External Metrics. You can also read about GKE monitoring with Prometheus and Stackdriver, depends on your needs.
As GKE is integrated with Stackdriver you can read article about Enabling Monitoring.

Should I add an iptables rule on the random kubernetes node port if I want to access from outside world

I have an app deployed on kubernetes cluster.
The kubernetes node port is 30010 which will be redirect to 41018.
[root#kubernetes-slave ~]# iptables -L -n -t nat
Chain KUBE-NODEPORT-HOST (1 references)
target prot opt source destination
DNAT tcp -- 0.0.0.0/0 0.0.0.0/0 /* default/myservice:http */ tcp dpt:30010 to:<server_ip>:41018
Every thing works fine if I disable the firewall. But if I enable it with service firewalld start. I can not access http://<server_ip>:30010.
Even if I execute firewall-cmd --zone=public --add-port=30010/tcp, It is still "Bad GateWay"
After two days of debug, I finally figured out that I should open 41018 port instead of 30010, then every thing works.
But the question here is that the 30010 port can be configured by user, but the port 41018 is randomly chosen by kubernetes on each deployment.
I think it is not a good choice to open 41018 port. Is there any suggestion? Or Is there some problem in my use of kubernetes?
You can set a static nodePort in your Service definition that will not change on each deploy.
For example:
{
"kind": "Service",
"apiVersion": "v1",
"metadata": {
"name": "my-service"
},
"spec": {
"selector": {
"app": "MyApp"
},
"ports": [
{
"protocol": "TCP",
"port": 80,
"targetPort": 9376,
"nodePort": 30061
}
],
"type": "nodePort"
}
}
Should always create a nodePort of 30061 for your service.