kube-registry-proxy doesn't expose port 5000 on any nodes - kubernetes

Question

I'm using private Docker registry addon in my kubernetes cluster, and I would like to expose port 5000 on each node to pull image from localhost:5000 easily. So I placed a pod manifest file /etc/kubernetes/manifests/kube-registry-proxy.manifest on every node to start a local proxy for port 5000. It works when I manually deployed kubernetes on bare metal ubuntu few months ago, but failed when I try kargo, the port 5000 not listening.
I'm using kargo with calico network plugin, the docker registry's configurations are:
kind: PersistentVolume
apiVersion: v1
metadata:
name: kube-system-kube-registry-pv
labels:
kubernetes.io/cluster-service: "true"
spec:
capacity:
storage: 500Gi
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Retain
hostPath:
path: /registry
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: kube-registry-pvc
namespace: kube-system
labels:
kubernetes.io/cluster-service: "true"
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 500Gi
apiVersion: v1
kind: ReplicationController
metadata:
name: kube-registry-v0
namespace: kube-system
labels:
k8s-app: kube-registry
version: v0
kubernetes.io/cluster-service: "true"
spec:
replicas: 1
selector:
k8s-app: kube-registry
version: v0
template:
metadata:
labels:
k8s-app: kube-registry
version: v0
kubernetes.io/cluster-service: "true"
spec:
containers:
- name: registry
image: registry:2.5.1
resources:
# keep request = limit to keep this container in guaranteed class
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 100m
memory: 100Mi
env:
- name: REGISTRY_HTTP_ADDR
value: :5000
- name: REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY
value: /var/lib/registry
volumeMounts:
- name: image-store
mountPath: /var/lib/registry
ports:
- containerPort: 5000
name: registry
protocol: TCP
volumes:
- name: image-store
persistentVolumeClaim:
claimName: kube-registry-pvc
apiVersion: v1
kind: Service
metadata:
name: kube-registry
namespace: kube-system
labels:
k8s-app: kube-registry
kubernetes.io/cluster-service: "true"
kubernetes.io/name: "KubeRegistry"
spec:
selector:
k8s-app: kube-registry
ports:
- name: registry
port: 5000
protocol: TCP
I have created a pod manifest file /etc/kubernetes/manifests/kube-registry-proxy.manifest before run kargo:
apiVersion: v1
kind: Pod
metadata:
name: kube-registry-proxy
namespace: kube-system
spec:
containers:
- name: kube-registry-proxy
image: gcr.io/google_containers/kube-registry-proxy:0.3
resources:
limits:
cpu: 100m
memory: 50Mi
env:
- name: REGISTRY_HOST
value: kube-registry.kube-system.svc.cluster.local
- name: REGISTRY_PORT
value: "5000"
- name: FORWARD_PORT
value: "5000"
ports:
- name: registry
containerPort: 5000
hostPort: 5000
kube-registry-proxy is running on all nodes, but nothing listen on port 5000. Some output:
ubuntu#k8s15m1:~$ kubectl get all --all-namespaces | grep registry-proxy
kube-system po/kube-registry-proxy-k8s15m1 1/1 Running 1 1h
kube-system po/kube-registry-proxy-k8s15m2 1/1 Running 0 1h
kube-system po/kube-registry-proxy-k8s15s1 1/1 Running 0 1h
ubuntu#k8s15m1:~$ docker ps | grep registry
756fcf674288 gcr.io/google_containers/kube-registry-proxy:0.3 "/usr/bin/run_proxy" 19 minutes ago Up 19 minutes k8s_kube-registry-proxy.bebf6da1_kube-registry-proxy-k8s15m1_kube-system_a818b22dc7210ecd31414e328ae28e43_7221833c
ubuntu#k8s15m1:~$ docker logs 756fcf674288 | tail
waiting for kube-registry.kube-system.svc.cluster.local to come online
starting proxy
ubuntu#k8s15m1:~$ netstat -ltnp | grep 5000
ubuntu#k8s15m1:~$ curl -v localhost:5000/v1/
* Trying 127.0.0.1...
* connect to 127.0.0.1 port 5000 failed: Connection refused
* Failed to connect to localhost port 5000: Connection refused
* Closing connection 0
curl: (7) Failed to connect to localhost port 5000: Connection refused
ubuntu#k8s15m1:~$ kubectl get po kube-registry-proxy-k8s15m1 --namespace=kube-system -o wide
NAME READY STATUS RESTARTS AGE IP NODE
kube-registry-proxy-k8s15m1 1/1 Running 3 1h 10.233.69.64 k8s15m1
ubuntu#k8s15m1:~$ curl -v 10.233.69.64:5000/v1/
* Trying 10.233.69.64...
* Connected to 10.233.69.64 (10.233.69.64) port 5000 (#0)
> GET /v1/ HTTP/1.1
> Host: 10.233.69.64:5000
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 404 Not Found
< Content-Type: text/plain; charset=utf-8
< Docker-Distribution-Api-Version: registry/2.0
< X-Content-Type-Options: nosniff
< Date: Tue, 14 Mar 2017 16:41:56 GMT
< Content-Length: 19
<
404 page not found
* Connection #0 to host 10.233.69.64 left intact

Answer 1

I think there are a couple of things going on here.
Foremost, be aware that Kubernetes Services come in 3 flavors: ClusterIP (which is the default), NodePort (which sounds very much like what you were expecting to happen), and LoadBalancer (which I won't mention further, but the docs do).
I would expect that if you updated your Service to explicitly request type: NodePort, you'll get closer to what you had in mind (but be aware that unless you changed it, NodePort ports are limited to 30000-32767.
Thus:
apiVersion: v1
kind: Service
metadata:
name: kube-registry
namespace: kube-system
labels:
k8s-app: kube-registry
kubernetes.io/cluster-service: "true"
kubernetes.io/name: "KubeRegistry"
spec:
type: NodePort # <--- updated line
selector:
k8s-app: kube-registry
ports:
- name: registry
port: 5000
nodePort: 30500 # <-- you can specify, or omit this
protocol: TCP
If you have an opinion about the port you want the Service to listen on, feel free to specify it, or just leave it off and Kubernetes will pick one from the available space.
I'm going to mention this next thing for completeness, but what I'm about to say is a bad practice, so please don't do it. You can also have the Pods listen directly on the actual TCP/IP stack of the Node, by specifying hostPort; so in your case, it would be hostPort: 5000 right below containerPort: 5000, causing the Pod to behave like a normal docker -p 5000:5000 command would. But doing that makes scheduling Pods a nightmare, so please don't.
Secondly, about your 404 from curl:
I'm going to assume, based on the output of your curl command that 10.233.69.x is your Service CIDR, which explains why port 5000 responded with anything. The request was in the right spirit, but /v1/ was an incorrect URI to attempt. The Docker Registry API docs contains a section about checking it is a V2 API instance. My favorite curl of a registry is https://registry.example.com/v2/_catalog because it will return the name of every image therein, ensuring that my credentials are correct, that the registry server is operating correctly, and so on.
I know it's a lot to take in, so if you feel I glossed over something, let me know and I'll try to address it. Good luck!