k8s dashboard metric does not work (Metric client health check failed) - kubernetes-dashboard

In my case, dashboard-metrics-scraper does not work. Is there any suggestion?
My test environment is kubernetes 1.20.4.
kubectl top nodes/pods are works fine
Following commands do NOT work
on kubernetes-dashboard log
2021/03/22 06:58:35 Metric client health check failed: the server is currently unable to handle the request (get services dashboard-metrics-scraper). Retrying in 30 seconds.
2021/03/22 06:59:05 Metric client health check failed: the server is currently unable to handle the request (get services dashboard-metrics-scraper). Retrying in 30 seconds.
on console
with adding /proxy/healthz goes bad request. If it is removing this (/proxy/healthz), it works fine.
$ kubectl get --raw "/api/v1/namespaces/kubernetes-dashboard/services/dashboard-metrics-scraper/proxy/healthz"
Error from server (BadRequest): the server rejected our request for an unknown reason
Following commands WORK FINE
wget from container
kubectl exec -it busybox -- wget --spider http://dashboard-metrics-scraper.kubernetes-dashboard.svc.cluster.local:8000
Connecting to dashboard-metrics-scraper.kubernetes-dashboard.svc.cluster.local:8000 (10.110.73.42:8000)
kubectl proxy seems work, but it is redirected.
$ curl http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/http:dashboard-metrics-scraper:/proxy/healthz
<HTML><HEAD>
<TITLE>Redirect</TITLE>
</HEAD>
<BODY>
<FONT face="Helvetica">
<big><strong></strong></big><BR>
</FONT>
<blockquote>
<TABLE border=0 cellPadding=1 width="80%">
<TR><TD>
<FONT face="Helvetica">
<big>Redirect (authentication_redirect_to_virtual_host)</big>
<BR>
<BR>
</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica">
You are being redirected to the authentication virtual host.
</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica">
</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica" SIZE=2>
<BR>
</FONT>
</TD></TR>
</TABLE>
</blockquote>
</FONT>
</BODY></HTML>
kubectl get --raw to dashboard-metrics-scraper works fine (but if it is adding /proxy/healthz, it does not work.
$ kubectl get --raw "/api/v1/namespaces/kubernetes-dashboard/services/dashboard-metrics-scraper"
{"kind":"Service","apiVersion":"v1","metadata":{"name":"dashboard-metrics-scraper","namespace":"kubernetes-dashboard","uid":"43f83d61-81b2-4d57-8321-516158444cd1","resourceVersion":"6033988","creationTimestamp":"2021-03-22T02:53:15Z","labels":{"k8s-app":"dashboard-metrics-scraper"},"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"v1\",\"kind\":\"Service\",\"metadata\":{\"annotations\":{},\"labels\":{\"k8s-app\":\"dashboard-metrics-scraper\"},\"name\":\"dashboard-metrics-scraper\",\"namespace\":\"kubernetes-dashboard\"},\"spec\":{\"ports\":[{\"port\":8000,\"targetPort\":8000}],\"selector\":{\"k8s-app\":\"dashboard-metrics-scraper\"}}}\n"},"managedFields":[{"manager":"kubectl-client-side-apply","operation":"Update","apiVersion":"v1","time":"2021-03-22T02:53:15Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:kubectl.kubernetes.io/last-applied-configuration":{}},"f:labels":{".":{},"f:k8s-app":{}}},"f:spec":{"f:ports":{".":{},"k:{\"port\":8000,\"protocol\":\"TCP\"}":{".":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{".":{},"f:k8s-app":{}},"f:sessionAffinity":{},"f:type":{}}}}]},"spec":{"ports":[{"protocol":"TCP","port":8000,"targetPort":8000}],"selector":{"k8s-app":"dashboard-metrics-scraper"},"clusterIP":"10.110.73.42","clusterIPs":["10.110.73.42"],"type":"ClusterIP","sessionAffinity":"None"},"status":{"loadBalancer":{}}}
dashboard-metrics-scraper service seems work
$ kubectl describe svc -n kubernete
s-dashboard dashboard-metrics-scraper
Name: dashboard-metrics-scraper
Namespace: kubernetes-dashboard
Labels: k8s-app=dashboard-metrics-scraper
Annotations: <none>
Selector: k8s-app=dashboard-metrics-scraper
Type: ClusterIP
IP Families: <none>
IP: 10.110.73.42
IPs: 10.110.73.42
Port: <unset> 8000/TCP
TargetPort: 8000/TCP
Endpoints: 172.16.8.142:8000
Session Affinity: None
Events: <none>

I solved it:
Add hostNetwork: true in deploy dashboard-metrics-scraper
k8s: v1.25.4
kind: Deployment
apiVersion: apps/v1
metadata:
labels:
k8s-app: dashboard-metrics-scraper
name: dashboard-metrics-scraper
namespace: kubernetes-dashboard
spec:
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
k8s-app: dashboard-metrics-scraper
template:
metadata:
labels:
k8s-app: dashboard-metrics-scraper
spec:
securityContext:
seccompProfile:
type: RuntimeDefault
containers:
- name: dashboard-metrics-scraper
image: kubernetesui/metrics-scraper:v1.0.8
ports:
- containerPort: 8000
protocol: TCP
livenessProbe:
httpGet:
scheme: HTTP
path: /
port: 8000
initialDelaySeconds: 30
timeoutSeconds: 30
volumeMounts:
- mountPath: /tmp
name: tmp-volume
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
runAsUser: 1001
runAsGroup: 2001
serviceAccountName: kubernetes-dashboard
hostNetwork: true
nodeSelector:
"kubernetes.io/os": linux
# Comment the following tolerations if Dashboard must not be deployed on master
tolerations:
- key: node-role.kubernetes.io/master
effect: NoSchedule
volumes:
- name: tmp-volume
emptyDir: {}

If you are testing it on your own kube setup, the metric server tries to connect on dns and as your setup does not have dns for nodes so you need to modify the configuration to allow the metric server to communicate on internal IP
add following flags to deployment
- --kubelet-insecure-tls
- --kubelet-preferred-address-types=InternalIP
so overall deployment will look like this
apiVersion: apps/v1
kind: Deployment
metadata:
name: metrics-server
namespace: kube-system
labels:
k8s-app: metrics-server
spec:
selector:
matchLabels:
**k8s-app: metrics-server**
template:
metadata:
name: metrics-server
labels:
k8s-app: metrics-server
spec:
serviceAccountName: metrics-server
volumes:
# mount in tmp so we can safely use from-scratch images and/or read-only containers
- name: tmp-dir
emptyDir: {}
containers:
- name: metrics-server
image: k8s.gcr.io/metrics-server-amd64:v0.3.6
imagePullPolicy: IfNotPresent
args:
- --cert-dir=/tmp
- --secure-port=4443
- --kubelet-insecure-tls
- --kubelet-preferred-address-types=InternalIP
ports:
- name: main-port
containerPort: 4443
protocol: TCP
securityContext:
readOnlyRootFilesystem: true
runAsNonRoot: true
runAsUser: 1000
volumeMounts:
- name: tmp-dir
mountPath: /tmp
nodeSelector:
kubernetes.io/os: linux
kubernetes.io/arch: "amd64"

Related

Kubernetes LoadBalancer

I setup my raspberry PI cluster and have installed metallb. I have the following Wordpress services running. I am confused why I cannot get this working via browser or wget process
pi#master:~ $ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 6d3h
mysql ClusterIP None <none> 3306/TCP 41m
wordpress LoadBalancer 10.101.63.209 192.168.1.50 80:32499/TCP 4m35s
When I try to do a wget to my website it keeps on trying to go out thru port 30820
What am I doing wrong here?
pi#master:~ $ wget 192.168.1.50
--2020-04-05 03:29:56-- http://192.168.1.50/
Connecting to 192.168.1.50:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: http://192.168.1.50:30820/ [following]
--2020-04-05 03:29:57-- http://192.168.1.50:30820/
Connecting to 192.168.1.50:30820... failed: No route to host.
Here is my deployment. Does this look OK?
apiVersion: apps/v1
kind: Deployment
metadata:
name: wordpress
#namespace: wordpress
labels:
app: wordpress
spec:
selector:
matchLabels:
app: wordpress
tier: frontend
strategy:
type: Recreate
replicas: 3
template:
metadata:
labels:
app: wordpress
tier: frontend
spec:
containers:
- image: wordpress:4.8-apache
imagePullPolicy: IfNotPresent
name: wordpress
env:
- name: WORDPRESS_DB_HOST
value: mysql
- name: WORDPRESS_DB_PASSWORD
valueFrom:
secretKeyRef:
name: mysql-pass # generated before in secret.yml
key: password
ports:
- containerPort: 80
name: wordpress
volumeMounts:
- name: wordpress-persistent-storage
mountPath: "/var/www/html" # which data will be stored
resources:
limits:
cpu: '1'
memory: '512Mi'
requests:
cpu: '500m'
memory: '256Mi'
volumes:
- name: wordpress-persistent-storage
persistentVolumeClaim:
claimName: wordpress-persistent-storage
tolerations:
- effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists
tolerationSeconds: 300
- effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
tolerationSeconds: 300
---
apiVersion: v1
kind: Service
metadata:
name: wordpress
#namespace: wordpress
labels:
app: wordpress
tier: frontend
spec:
selector:
app: wordpress
ports:
- protocol: 'TCP'
port: 80
targetPort: 80
#externalTrafficPolicy: Local
type: LoadBalancer
It may be your Wordpress settings which implies redirect. Neither MetalLB nor k8s Service don't have redirect functionality, as both work on network level.

Two kubernetes deployments in the same namespace are not able to communicate

I'm deploying ELK stack (oss) to kubernetes cluster. Elasticsearch deployment and service starts correctly and API is reacheble. Kibana deployment starts but can't access elasticsearch:
From Kibana container logs:
{"type":"log","#timestamp":"2019-05-08T22:49:26Z","tags":["error","elasticsearch","admin"],"pid":1,"message":"Request error, retrying\nHEAD http://elasticsearch:9200/ => getaddrinfo ENOTFOUND elasticsearch elasticsearch:9200"}
{"type":"log","#timestamp":"2019-05-08T22:50:44Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","#timestamp":"2019-05-08T22:50:44Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"}
Both deployments are in the same namespace "observability". I also tried to reference elasticsearch container as elasticsearch.observability.svc.cluster.local but it's not working too.
What I'am doing wrong? How to reference elasticsearch container from kibana container?
More info:
kubectl --context=19team-observability-admin-context -n observability get pods
NAME READY STATUS RESTARTS AGE
elasticsearch-9d495b84f-j2297 1/1 Running 0 15s
kibana-65bc7f9c4-s9cv4 1/1 Running 0 15s
kubectl --context=19team-observability-admin-context -n observability get service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
elasticsearch NodePort 10.104.250.175 <none> 9200:30083/TCP,9300:30059/TCP 1m
kibana NodePort 10.102.124.171 <none> 5601:30124/TCP 1m
I start my containers with command
kubectl --context=19team-observability-admin-context -n observability apply -f .\elasticsearch.yaml -f .\kibana.yaml
elasticsearch.yaml
apiVersion: v1
kind: Service
metadata:
name: elasticsearch
namespace: observability
spec:
type: NodePort
ports:
- name: "9200"
port: 9200
targetPort: 9200
- name: "9300"
port: 9300
targetPort: 9300
selector:
app: elasticsearch
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: elasticsearch
namespace: observability
spec:
replicas: 1
selector:
matchLabels:
app: elasticsearch
template:
metadata:
labels:
app: elasticsearch
spec:
initContainers:
- name: set-vm-max-map-count
image: busybox
imagePullPolicy: IfNotPresent
command: ['sysctl', '-w', 'vm.max_map_count=262144']
securityContext:
privileged: true
resources:
requests:
memory: "512Mi"
cpu: "1"
limits:
memory: "724Mi"
cpu: "1"
containers:
- name: elasticsearch
image: docker.elastic.co/elasticsearch/elasticsearch-oss:6.7.1
ports:
- containerPort: 9200
- containerPort: 9300
resources:
requests:
memory: "3Gi"
cpu: "1"
limits:
memory: "3Gi"
cpu: "1"
kibana.yaml
apiVersion: v1
kind: Service
metadata:
name: kibana
namespace: observability
spec:
type: NodePort
ports:
- name: "5601"
port: 5601
targetPort: 5601
selector:
app: observability_platform_kibana
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
labels:
app: observability_platform_kibana
name: kibana
namespace: observability
spec:
replicas: 1
template:
metadata:
labels:
app: observability_platform_kibana
spec:
containers:
- env:
# THIS IS WHERE WE SET CONNECTION BETWEEN KIBANA AND ELASTIC
- name: ELASTICSEARCH_HOSTS
value: http://elasticsearch:9200
- name: SERVER_NAME
value: kibana
image: docker.elastic.co/kibana/kibana-oss:6.7.1
name: kibana
ports:
- containerPort: 5601
resources:
requests:
memory: "512Mi"
cpu: "1"
limits:
memory: "724Mi"
cpu: "1"
restartPolicy: Always
UPDATE 1
As gonzalesraul proposed I've created second service for elastic with ClusterIP type:
apiVersion: v1
kind: Service
metadata:
labels:
app: elasticsearch
name: elasticsearch-local
namespace: observability
spec:
type: ClusterIP
ports:
- port: 9200
protocol: TCP
targetPort: 9200
selector:
app: elasticsearch
Service is created:
kubectl --context=19team-observability-admin-context -n observability get service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
elasticsearch NodePort 10.106.5.94 <none> 9200:31598/TCP,9300:32018/TCP 26s
elasticsearch-local ClusterIP 10.101.178.13 <none> 9200/TCP 26s
kibana NodePort 10.99.73.118 <none> 5601:30004/TCP 26s
And reference elastic as "http://elasticsearch-local:9200"
Unfortunately it does not work, in kibana container:
{"type":"log","#timestamp":"2019-05-09T10:13:54Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch-local:9200/"}
Do not use a NodePort service, instead use a ClusterIP. If you need to expose as a Nodeport your service, create a second service besides, for instance:
---
apiVersion: v1
kind: Service
metadata:
labels:
app: elasticsearch
name: elasticsearch-local
namespace: observability
spec:
type: ClusterIP
ports:
- port: 9200
protocol: TCP
targetPort: 9200
selector:
app: elasticsearch
Then update the kibana manifest to point to the ClusterIP service:
# ...
# THIS IS WHERE WE SET CONNECTION BETWEEN KIBANA AND ELASTIC
- name: ELASTICSEARCH_HOSTS
value: http://elasticsearch-local:9200
# ...
The nodePort services do not create a 'dns entry' (ex. elasticsearch.observability.svc.cluster.local) on kubernetes
Edit the server name value in kibana.yaml and set it to kibana:5601.
I think if you don't do this, by default it is trying to go to port 80.
This is what looks like now kibana.yaml:
...
spec:
containers:
- env:
- name: ELASTICSEARCH_HOSTS
value: http://elasticsearch:9200
- name: SERVER_NAME
value: kibana:5601
image: docker.elastic.co/kibana/kibana-oss:6.7.1
imagePullPolicy: IfNotPresent
name: kibana
...
And this is the output now:
{"type":"log","#timestamp":"2019-05-09T10:37:16Z","tags":["status","plugin:console#6.7.1","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","#timestamp":"2019-05-09T10:37:16Z","tags":["status","plugin:interpreter#6.7.1","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","#timestamp":"2019-05-09T10:37:16Z","tags":["status","plugin:metrics#6.7.1","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","#timestamp":"2019-05-09T10:37:16Z","tags":["status","plugin:tile_map#6.7.1","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","#timestamp":"2019-05-09T10:37:16Z","tags":["status","plugin:timelion#6.7.1","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","#timestamp":"2019-05-09T10:37:16Z","tags":["status","plugin:elasticsearch#6.7.1","info"],"pid":1,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","#timestamp":"2019-05-09T10:37:17Z","tags":["listening","info"],"pid":1,"message":"Server running at http://0:5601"}
UPDATE
I just tested it on a bare metal cluster (bootstraped through kubeadm), and worked again.
This is the output:
{"type":"log","#timestamp":"2019-05-09T11:09:59Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"}
{"type":"log","#timestamp":"2019-05-09T11:10:01Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","#timestamp":"2019-05-09T11:10:01Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"}
{"type":"log","#timestamp":"2019-05-09T11:10:04Z","tags":["status","plugin:elasticsearch#6.7.1","info"],"pid":1,"state":"green","message":"Status changed from red to green - Ready","prevState":"red","prevMsg":"Unable to connect to Elasticsearch."}
{"type":"log","#timestamp":"2019-05-09T11:10:04Z","tags":["info","migrations"],"pid":1,"message":"Creating index .kibana_1."}
{"type":"log","#timestamp":"2019-05-09T11:10:06Z","tags":["info","migrations"],"pid":1,"message":"Pointing alias .kibana to .kibana_1."}
{"type":"log","#timestamp":"2019-05-09T11:10:06Z","tags":["info","migrations"],"pid":1,"message":"Finished in 2417ms."}
{"type":"log","#timestamp":"2019-05-09T11:10:06Z","tags":["listening","info"],"pid":1,"message":"Server running at http://0:5601"}
Note that it passed from "No Living Connections" to "Running". I am running the nodes on GCP. I had to open the firewalls for it to work. What's your environment?

Service not exposing in kubernetes

I have a deployment and a service in GKE. I exposed the deployment as a Load Balancer but I cannot access it through the service (curl or browser). I get an:
curl: (7) Failed to connect to <my-Ip-Address> port 443: Connection refused
I can port forward directly to the pod and it works fine:
kubectl --namespace=redfalcon port-forward web-service-rf-76967f9c68-2zbhm 9999:443 >> /dev/null
curl -k -v --request POST --url https://localhost:9999/auth/login/ --header 'content-type: application/json' --header 'x-profile-key: ' --data '{"email":"<testusername>","password":"<testpassword>"}'
I have most likely misconfigured my service but cannot see how. Any help on what I did would be very much appreciated.
Service Yaml:
---
apiVersion: v1
kind: Service
metadata:
name: red-falcon-lb
namespace: redfalcon
spec:
type: LoadBalancer
ports:
- name: https
port: 443
protocol: TCP
selector:
app: web-service-rf
Deployment YAML
apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
name: web-service-rf
spec:
selector:
matchLabels:
app: web-service-rf
replicas: 2 # tells deployment to run 2 pods matching the template
template:
metadata:
labels:
app: web-service-rf
spec:
initContainers:
- name: certificate-init-container
image: proofpoint/certificate-init-container:0.2.0
imagePullPolicy: Always
env:
- name: NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
args:
- "-namespace=$(NAMESPACE)"
- "-pod-name=$(POD_NAME)"
- "-query-k8s"
volumeMounts:
- name: tls
mountPath: /etc/tls
containers:
- name: web-service-rf
image: gcr.io/redfalcon-186521/redfalcon-webserver-minimal:latest
# image: gcr.io/redfalcon-186521/redfalcon-webserver-full:latest
command:
- "./server"
- "--port=443"
imagePullPolicy: Always
env:
- name: GOOGLE_APPLICATION_CREDENTIALS
value: /var/secrets/google/key.json
ports:
- containerPort: 443
resources:
limits:
memory: "500Mi"
cpu: "100m"
volumeMounts:
- mountPath: /etc/tls
name: tls
- mountPath: /var/secrets/google
name: google-cloud-key
volumes:
- name: tls
emptyDir: {}
- name: google-cloud-key
secret:
secretName: pubsub-key
output: kubectl describe svc red-falcon-lb
Name: red-falcon-lb
Namespace: redfalcon
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"name":"red-falcon-lb","namespace":"redfalcon"},"spec":{"ports":[{"name":"https","port...
Selector: app=web-service-rf
Type: LoadBalancer
IP: 10.43.245.9
LoadBalancer Ingress: <EXTERNAL IP REDACTED>
Port: https 443/TCP
TargetPort: 443/TCP
NodePort: https 31524/TCP
Endpoints: 10.40.0.201:443,10.40.0.202:443
Session Affinity: None
External Traffic Policy: Cluster
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal EnsuringLoadBalancer 39m service-controller Ensuring load balancer
Normal EnsuredLoadBalancer 38m service-controller Ensured load balancer
I figured out what it was...
My golang app was listening on localhost instead of 0.0.0.0. This meant that port forwarding on kubectl worked but any service exposure didn't work.
I had to add "--host 0.0.0.0" to my k8s command and it then listened to requests from outside localhost.
My command ended up being...
"./server --port 8080 --host 0.0.0.0"

kube-registry-proxy doesn't expose port 5000 on any nodes

I'm using private Docker registry addon in my kubernetes cluster, and I would like to expose port 5000 on each node to pull image from localhost:5000 easily. So I placed a pod manifest file /etc/kubernetes/manifests/kube-registry-proxy.manifest on every node to start a local proxy for port 5000. It works when I manually deployed kubernetes on bare metal ubuntu few months ago, but failed when I try kargo, the port 5000 not listening.
I'm using kargo with calico network plugin, the docker registry's configurations are:
kind: PersistentVolume
apiVersion: v1
metadata:
name: kube-system-kube-registry-pv
labels:
kubernetes.io/cluster-service: "true"
spec:
capacity:
storage: 500Gi
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Retain
hostPath:
path: /registry
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: kube-registry-pvc
namespace: kube-system
labels:
kubernetes.io/cluster-service: "true"
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 500Gi
apiVersion: v1
kind: ReplicationController
metadata:
name: kube-registry-v0
namespace: kube-system
labels:
k8s-app: kube-registry
version: v0
kubernetes.io/cluster-service: "true"
spec:
replicas: 1
selector:
k8s-app: kube-registry
version: v0
template:
metadata:
labels:
k8s-app: kube-registry
version: v0
kubernetes.io/cluster-service: "true"
spec:
containers:
- name: registry
image: registry:2.5.1
resources:
# keep request = limit to keep this container in guaranteed class
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 100m
memory: 100Mi
env:
- name: REGISTRY_HTTP_ADDR
value: :5000
- name: REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY
value: /var/lib/registry
volumeMounts:
- name: image-store
mountPath: /var/lib/registry
ports:
- containerPort: 5000
name: registry
protocol: TCP
volumes:
- name: image-store
persistentVolumeClaim:
claimName: kube-registry-pvc
apiVersion: v1
kind: Service
metadata:
name: kube-registry
namespace: kube-system
labels:
k8s-app: kube-registry
kubernetes.io/cluster-service: "true"
kubernetes.io/name: "KubeRegistry"
spec:
selector:
k8s-app: kube-registry
ports:
- name: registry
port: 5000
protocol: TCP
I have created a pod manifest file /etc/kubernetes/manifests/kube-registry-proxy.manifest before run kargo:
apiVersion: v1
kind: Pod
metadata:
name: kube-registry-proxy
namespace: kube-system
spec:
containers:
- name: kube-registry-proxy
image: gcr.io/google_containers/kube-registry-proxy:0.3
resources:
limits:
cpu: 100m
memory: 50Mi
env:
- name: REGISTRY_HOST
value: kube-registry.kube-system.svc.cluster.local
- name: REGISTRY_PORT
value: "5000"
- name: FORWARD_PORT
value: "5000"
ports:
- name: registry
containerPort: 5000
hostPort: 5000
kube-registry-proxy is running on all nodes, but nothing listen on port 5000. Some output:
ubuntu#k8s15m1:~$ kubectl get all --all-namespaces | grep registry-proxy
kube-system po/kube-registry-proxy-k8s15m1 1/1 Running 1 1h
kube-system po/kube-registry-proxy-k8s15m2 1/1 Running 0 1h
kube-system po/kube-registry-proxy-k8s15s1 1/1 Running 0 1h
ubuntu#k8s15m1:~$ docker ps | grep registry
756fcf674288 gcr.io/google_containers/kube-registry-proxy:0.3 "/usr/bin/run_proxy" 19 minutes ago Up 19 minutes k8s_kube-registry-proxy.bebf6da1_kube-registry-proxy-k8s15m1_kube-system_a818b22dc7210ecd31414e328ae28e43_7221833c
ubuntu#k8s15m1:~$ docker logs 756fcf674288 | tail
waiting for kube-registry.kube-system.svc.cluster.local to come online
starting proxy
ubuntu#k8s15m1:~$ netstat -ltnp | grep 5000
ubuntu#k8s15m1:~$ curl -v localhost:5000/v1/
* Trying 127.0.0.1...
* connect to 127.0.0.1 port 5000 failed: Connection refused
* Failed to connect to localhost port 5000: Connection refused
* Closing connection 0
curl: (7) Failed to connect to localhost port 5000: Connection refused
ubuntu#k8s15m1:~$ kubectl get po kube-registry-proxy-k8s15m1 --namespace=kube-system -o wide
NAME READY STATUS RESTARTS AGE IP NODE
kube-registry-proxy-k8s15m1 1/1 Running 3 1h 10.233.69.64 k8s15m1
ubuntu#k8s15m1:~$ curl -v 10.233.69.64:5000/v1/
* Trying 10.233.69.64...
* Connected to 10.233.69.64 (10.233.69.64) port 5000 (#0)
> GET /v1/ HTTP/1.1
> Host: 10.233.69.64:5000
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 404 Not Found
< Content-Type: text/plain; charset=utf-8
< Docker-Distribution-Api-Version: registry/2.0
< X-Content-Type-Options: nosniff
< Date: Tue, 14 Mar 2017 16:41:56 GMT
< Content-Length: 19
<
404 page not found
* Connection #0 to host 10.233.69.64 left intact
I think there are a couple of things going on here.
Foremost, be aware that Kubernetes Services come in 3 flavors: ClusterIP (which is the default), NodePort (which sounds very much like what you were expecting to happen), and LoadBalancer (which I won't mention further, but the docs do).
I would expect that if you updated your Service to explicitly request type: NodePort, you'll get closer to what you had in mind (but be aware that unless you changed it, NodePort ports are limited to 30000-32767.
Thus:
apiVersion: v1
kind: Service
metadata:
name: kube-registry
namespace: kube-system
labels:
k8s-app: kube-registry
kubernetes.io/cluster-service: "true"
kubernetes.io/name: "KubeRegistry"
spec:
type: NodePort # <--- updated line
selector:
k8s-app: kube-registry
ports:
- name: registry
port: 5000
nodePort: 30500 # <-- you can specify, or omit this
protocol: TCP
If you have an opinion about the port you want the Service to listen on, feel free to specify it, or just leave it off and Kubernetes will pick one from the available space.
I'm going to mention this next thing for completeness, but what I'm about to say is a bad practice, so please don't do it. You can also have the Pods listen directly on the actual TCP/IP stack of the Node, by specifying hostPort; so in your case, it would be hostPort: 5000 right below containerPort: 5000, causing the Pod to behave like a normal docker -p 5000:5000 command would. But doing that makes scheduling Pods a nightmare, so please don't.
Secondly, about your 404 from curl:
I'm going to assume, based on the output of your curl command that 10.233.69.x is your Service CIDR, which explains why port 5000 responded with anything. The request was in the right spirit, but /v1/ was an incorrect URI to attempt. The Docker Registry API docs contains a section about checking it is a V2 API instance. My favorite curl of a registry is https://registry.example.com/v2/_catalog because it will return the name of every image therein, ensuring that my credentials are correct, that the registry server is operating correctly, and so on.
I know it's a lot to take in, so if you feel I glossed over something, let me know and I'll try to address it. Good luck!

How do I run traefik behind Kubernetes on Google Container Engine?

So I have Traefik "running" on Kubernetes:
apiVersion: v1
kind: Service
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: '{"kind":"Service","apiVersion":"v1","metadata":{"name":"traefik","namespace":"kube-system","creationTimestamp":null,"labels":{"k8s-app":"traefik-ingress-lb"}},"spec":{"ports":[{"name":"http","port":80,"targetPort":80},{"name":"https","port":443,"targetPort":443}],"selector":{"k8s-app":"traefik-ingress-lb"},"type":"LoadBalancer"},"status":{"loadBalancer":{}}}'
creationTimestamp: 2016-11-30T23:15:49Z
labels:
k8s-app: traefik-ingress-lb
name: traefik
namespace: kube-system
resourceVersion: "9672"
selfLink: /api/v1/namespaces/kube-system/services/traefik
uid: ee07b957-b752-11e6-88fa-42010af00083
spec:
clusterIP: 10.11.251.200
ports:
- name: http
nodePort: 30297
port: 80
protocol: TCP
targetPort: 80
- name: https
nodePort: 30247
port: 443
protocol: TCP
targetPort: 443
selector:
k8s-app: traefik-ingress-lb
sessionAffinity: None
type: LoadBalancer
status:
loadBalancer:
ingress:
- ip: # IP THAT IS ALLOCATED BY k8s BUT NOT ASSIGNED
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: "1"
kubectl.kubernetes.io/last-applied-configuration: '###'
creationTimestamp: 2016-11-30T22:59:07Z
generation: 2
labels:
k8s-app: traefik-ingress-lb
name: traefik-ingress-controller
namespace: kube-system
resourceVersion: "23438"
selfLink: /apis/extensions/v1beta1/namespaces/kube-system/deployments/traefik-ingress-controller
uid: 9919ff46-b750-11e6-88fa-42010af00083
spec:
replicas: 1
selector:
matchLabels:
k8s-app: traefik-ingress-lb
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 1
type: RollingUpdate
template:
metadata:
creationTimestamp: null
labels:
k8s-app: traefik-ingress-lb
name: traefik-ingress-lb
version: v1.1
spec:
containers:
- args:
- --web
- --kubernetes
- --configFile=/etc/config/traefik.toml
- --logLevel=DEBUG
image: gcr.io/myproject/traefik
imagePullPolicy: Always
name: traefik-ingress-lb
ports:
- containerPort: 80
hostPort: 80
protocol: TCP
- containerPort: 8080
protocol: TCP
resources:
limits:
cpu: 200m
memory: 30Mi
requests:
cpu: 100m
memory: 20Mi
terminationMessagePath: /dev/termination-log
volumeMounts:
- mountPath: /etc/config
name: config-volume
- mountPath: /etc/traefik
name: traefik-volume
dnsPolicy: ClusterFirst
restartPolicy: Always
securityContext: {}
terminationGracePeriodSeconds: 60
volumes:
- configMap:
defaultMode: 420
name: traefik-config
name: config-volume
- emptyDir: {}
name: traefik-volume
status:
observedGeneration: 2
replicas: 1
unavailableReplicas: 1
updatedReplicas: 1
My problem is that the external IP assigned by Kubernetes does not actually forward to Traefik; in fact, the IP assigned does not even show up in my Google Cloud Platform console. How do I get Traefik working with a load-balancer on Google Container Engine?
If you do
kubectl describe svc/traefik
Do the endpoints match the IPs from:
kubectl get po -lk8s-app=traefik-ingress-lb -o wide
What happens when you do hit the LB IP? Does it load indefinitely, or do you get a different service?
There was a new fork of Traefik that supposedly fixes some kubernetes issues. However, I did further reading and talking about Traefik and uncovered some (recent) advisories of potential instability. While this is to be expected in new software, I decided to switch to NGINX to handle my reverse proxy. It's been working wonderfully, so I'm going to go ahead and close this question.