I have installed docker-registry on Kubernetes via helm.
I am able to docker push to docker push 0.0.0.0:5000/<my-container>:v1 using port-forward.
Now how do I reference the images in the registry from a deployment.yaml?
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: <my-container>-deployment-v1
spec:
replicas: 1
template:
metadata:
labels:
app: <my-container>-deployment
version: v1
spec:
containers:
- name: <my-container>
image: 0.0.0.0:5000/<my-container>:v1 # <<< ????
imagePullPolicy: Always
ports:
- containerPort: 80
imagePullSecrets:
- name: private-docker-registry-secret
This do list my containers:
curl -X GET http://0.0.0.0:5000/v2/_catalog
I keep getting ImagePullBackOff when deploying.
I tyied using internal service name and cluster ip address, still not working.
Then tried using secrets:
{
"kind": "Secret",
"apiVersion": "v1",
"metadata": {
"name": "running-buffoon-docker-registry-secret",
"namespace": "default",
"selfLink": "/api/v1/namespaces/default/secrets/running-buffoon-docker-registry-secret",
"uid": "127c93c1-53df-11e9-8ede-a63ad724d5b9",
"resourceVersion": "216488",
"creationTimestamp": "2019-03-31T18:01:56Z",
"labels": {
"app": "docker-registry",
"chart": "docker-registry-1.7.0",
"heritage": "Tiller",
"release": "running-buffoon"
}
},
"data": {
"haSharedSecret": "xxx"
},
"type": "Opaque"
}
And added the secret to to deployment.yaml:
imagePullSecrets:
- name: running-buffoon-docker-registry-secret
Then I get:
image "x.x.x.x/:<my-container>v1": rpc error: code = Unknown desc = Error response from daemon: Get https://x.x.x.x/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
You need to get the cluster-ip of your local docker registry.
You will find this in the dashboard - just visit the registry pod page and then to the associated service. Replace your image spec's 0.0.0.0 with the cluster ip. Also make sure the port matches - generally the port exposed by registry service is different from the actual port exposed inside the cluster. If you have authentication set up in your registry, you will need imagepullsecret as well.
I have blogged about minikube setup with a local registry - might be helpful. https://amritbera.com/journal/minikube-insecure-registry.html
Related
I am currently trying to move my calico based clusters to the new Dataplane V2, which is basically a managed Cilium offering.
For local testing, I am running k3d with open source cilium installed, and created a set of NetworkPolicies (k8s native ones, not CiliumPolicies), which lock down the desired namespaces.
My current issue is, that when porting the same Policies on a GKE cluster (with DataPlane enabled), those same policies don't work.
As an example let's take a look into the connection between some app and a database:
---
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
name: db-server.db-client
namespace: BAR
spec:
podSelector:
matchLabels:
policy.ory.sh/db: server
policyTypes:
- Ingress
ingress:
- ports: []
from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: FOO
podSelector:
matchLabels:
policy.ory.sh/db: client
---
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
name: db-client.db-server
namespace: FOO
spec:
podSelector:
matchLabels:
policy.ory.sh/db: client
policyTypes:
- Egress
egress:
- ports:
- port: 26257
protocol: TCP
to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: BAR
podSelector:
matchLabels:
policy.ory.sh/db: server
Moreover, using GCP monitoring tools we can see the expected and actual effect the policies have on connectivity:
Expected:
Actual:
And logs from the application trying to connect to the DB, and getting denied:
{
"insertId": "FOO",
"jsonPayload": {
"count": 3,
"connection": {
"dest_port": 26257,
"src_port": 44506,
"dest_ip": "172.19.0.19",
"src_ip": "172.19.1.85",
"protocol": "tcp",
"direction": "egress"
},
"disposition": "deny",
"node_name": "FOO",
"src": {
"pod_name": "backoffice-automigrate-hwmhv",
"workload_kind": "Job",
"pod_namespace": "FOO",
"namespace": "FOO",
"workload_name": "backoffice-automigrate"
},
"dest": {
"namespace": "FOO",
"pod_namespace": "FOO",
"pod_name": "cockroachdb-0"
}
},
"resource": {
"type": "k8s_node",
"labels": {
"project_id": "FOO",
"node_name": "FOO",
"location": "FOO",
"cluster_name": "FOO"
}
},
"timestamp": "FOO",
"logName": "projects/FOO/logs/policy-action",
"receiveTimestamp": "FOO"
}
EDIT:
My local env is a k3d cluster created via:
k3d cluster create --image ${K3SIMAGE} --registry-use k3d-localhost -p "9090:30080#server:0" \
-p "9091:30443#server:0" foobar \
--k3s-arg=--kube-apiserver-arg="enable-admission-plugins=PodSecurityPolicy,NodeRestriction,ServiceAccount#server:0" \
--k3s-arg="--disable=traefik#server:0" \
--k3s-arg="--disable-network-policy#server:0" \
--k3s-arg="--flannel-backend=none#server:0" \
--k3s-arg=feature-gates="NamespaceDefaultLabelName=true#server:0"
docker exec k3d-server-0 sh -c "mount bpffs /sys/fs/bpf -t bpf && mount --make-shared /sys/fs/bpf"
kubectl taint nodes k3d-ory-cloud-server-0 node.cilium.io/agent-not-ready=true:NoSchedule --overwrite=true
skaffold run --cache-artifacts=true -p cilium --skip-tests=true --status-check=false
docker exec k3d-server-0 sh -c "mount --make-shared /run/cilium/cgroupv2"
Where cilium itself is being installed by skaffold, via helm with the following parameters:
name: cilium
remoteChart: cilium/cilium
namespace: kube-system
version: 1.11.0
upgradeOnChange: true
wait: false
setValues:
externalIPs.enabled: true
nodePort.enabled: true
hostPort.enabled: true
hubble.relay.enabled: true
hubble.ui.enabled: true
UPDATE:
I have setup a third environment: a GKE cluster using the old calico CNI (Legacy dataplane) and installed cilium manually as shown here. Cilium is working fine, even hubble is working out of the box (unlike with the dataplane v2...) and I found something interesting. The rules behave the same as with the GKE managed cilium, but with hubble working I was able to see this:
For some reason cilium/hubble cannot identify the db pod and decipher its labels. And since the labels don't work, the policies that rely on those labels, also don't work.
Another proof of this would be the trace log from hubble:
Here the destination app is only identified via an IP, and not labels.
The question now is why is this happening?
Any idea how to debug this problem? What could be difference coming from? Do the policies need some tuning for the managed Cilium, or is a bug in GKE?
Any help/feedback/suggestion appreciated!
Update: I was able to solve the mystery and it was ArgoCD all along. Cilium is creating an Endpoint and Identity for each object in the namespace, and Argo was deleting them after deploying the applications.
For anyone who stumbles on this, the solution is to add this exclusion to ArgoCD:
resource.exclusions: |
- apiGroups:
- cilium.io
kinds:
- CiliumIdentity
- CiliumEndpoint
clusters:
- "*"
First all of, for some reasons, I'm using an unsupported and obsolete version of Kubernetes (1.12), and I can't upgrade.
I'm trying to configure the scheduler to avoid running pods on some nodes by changing the node score when the scheduler try to find the best available node, and I would like to do that on scheduler level and not by using nodeAffinity at deployment, replicaset, pod, etc level (therefore all pods will be affected by this change).
After reading the k8s docs here: https://kubernetes.io/docs/reference/scheduling/config/#scheduling-plugins and checking that some options were already present in 1.12, I'm trying to use the NodePreferAvoidPods plugins.
In the documentation the plugin specifies:
Scores nodes according to the node annotation scheduler.alpha.kubernetes.io/preferAvoidPods
Which if understand correctly should do the work.
So, i've updated the static manifest for kube-scheduler.yaml to use the following config:
apiVersion: kubescheduler.config.k8s.io/v1alpha1
kind: KubeSchedulerConfiguration
profiles:
- plugins:
score:
enabled:
- name: NodePreferAvoidPods
weight: 100
clientConnection:
kubeconfig: /etc/kubernetes/scheduler.conf
But adding the following annotation
scheduler.alpha.kubernetes.io/preferAvoidPods: to the node doesn't seem to work.
For testing I'm made a basic nginx deployment with a replica equal to the number of worker nodes (4).
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
labels:
app: nginx
spec:
replicas: 4
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.21
ports:
- containerPort: 80
Then I check where the pods where created with kubectl get pods -owide
So, I believe some options are required for this annotation to works.
I've tried to set the annotation to "true", "1" but k8s refuse my change and I can't figure what are the valid options for this annotation and I can't find any documentation about that.
I've checked within git release for 1.12, this plugin was already present (at least there are some lines of codes), I don't think the behavior or settings changed much since.
Thanks.
So from source Kubernetes codes here a valid value for this annoation:
{
"preferAvoidPods": [
{
"podSignature": {
"podController": {
"apiVersion": "v1",
"kind": "ReplicationController",
"name": "foo",
"uid": "abcdef123456",
"controller": true
}
},
"reason": "some reason",
"message": "some message"
}
]
}`
But there is no details on how to predict the uid and no answer where given when asked by another one on github years ago: https://github.com/kubernetes/kubernetes/issues/41630
For my initial question which was to avoid scheduling pods on node, I found an other method by using the well-known taint node.kubernetes.io/unschedulable and the value PreferNoSchedule
Tainting a node with this command do the job and this taint seem persistent across cordon/uncordon (a cordon will set to NoSchedule and uncordon will set it back to PreferNoSchedule).
kubectl taint node NODE_NAME node.kubernetes.io/unschedulable=:PreferNoSchedule
I learned recently that Kubernetes has a feature called Init Containers. Awesome, because I can use this feature to wait for my postgres service and create/migrate the database before my web application service runs.
However, it appears that Init Containers can only be configured in a Pod yaml file. Is there a way I can do this via a Deployment yaml file? Or do I have to choose?
To avoid confusion, ill answer your specific question. i agree with oswin that you may want to consider another method.
Yes, you can use init containers with a deployment. this is an example using the old style (pre 1.6) but it should work
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: 'nginx'
spec:
replicas: 1
selector:
matchLabels:
app: 'nginx'
template:
metadata:
labels:
app: 'nginx'
annotations:
pod.beta.kubernetes.io/init-containers: '[
{
"name": "install",
"image": "busybox",
"imagePullPolicy": "IfNotPresent",
"command": ["wget", "-O", "/application/index.html", "http://kubernetes.io/index.html"],
"volumeMounts": [
{
"name": "application",
"mountPath": "/application"
}
]
}
]'
spec:
volumes:
- name: 'application'
emptyDir: {}
containers:
- name: webserver
image: 'nginx'
ports:
- name: http
containerPort: 80
volumeMounts:
- name: 'application'
mountPath: '/application'
You probably want to use readiness probes instead of init containers for this use case. Check out this link and a blog. Also note that a deployment will not send traffic to a pod that is not reported ready - If that was your worry.
This is a well known pattern and a readiness probe in the web server would simply check the DB endpoint / data availability before reporting ready. This is a simple solution as opposed to the complexity of an extra init container and has the advantage of detecting DB outages correctly as well.
My Kubernetes version is :
# kubectl --version
Kubernetes v1.4.0
I am planning to use Prometheus to monitor my Kube cluster. For this, I need to annotate the metrics URL.
My current metrics URL is like :
http://172.16.33.7:8080/metrics
But I want it to be like :
http://172.16.33.7:8080/websocket/metrics
First I tried to do this manually ::
kubectl annotate pods websocket-backend-controller-db83999c5b534b277b82badf6c152cb9m1 prometheus.io/path=/websocket/metrics
kubectl annotate pods websocket-backend-controller-db83999c5b534b277b82badf6c152cb9m1 prometheus.io/scrape='true'
kubectl annotate pods websocket-backend-controller-db83999c5b534b277b82badf6c152cb9m1 prometheus.io/port='8080'
All these commands work perfectly fine and I am able to see the annotations.
{
"metadata": {
"name": "websocket-backend-controller-v1krf",
"generateName": "websocket-backend-controller-",
"namespace": "default",
"selfLink": "/api/v1/namespaces/default/pods/websocket-backend-controller-v1krf",
"uid": "e323994b-4081-11e7-8bd0-0050569b6f44",
"resourceVersion": "27534379",
"creationTimestamp": "2017-05-24T13:07:06Z",
"labels": {
"name": "websocket-backend"
},
"annotations": {
"kubernetes.io/created-by": "{\"kind\":\"SerializedReference\",\"apiVersion\":\"v1\",\"reference\":{\"kind\":\"ReplicationController\",\"namespace\":\"default\",\"name\":\"websocket-backend-controller\",\"uid\":\"e321f1a8-4081-11e7-8bd0-0050569b6f44\",\"apiVersion\":\"v1\",\"resourceVersion\":\"27531840\"}}\n",
"prometheus.io/path": "/websocket/metrics",
"prometheus.io/port": "8080",
"prometheus.io/scrape": "true"
}
But since I want this configuration to remain permanent, I am setting the following annotations in my services files.
# cat websocket-service.yaml
apiVersion: v1
kind: Service
metadata:
name: websocket-service
labels:
baseApi: websocket
annotations:
prometheus.io/scrape: 'true'
prometheus.io/path: /websocket/metrics
prometheus.io/port: '8080'
spec:
selector:
name: websocket-backend
ports:
- port: 8080
targetPort: 8080
nodePort: 30800
protocol: TCP
type: NodePort
clusterIP: 10.100.10.45
I restarted my websocket service and the corresponding pods but these configs do not seem to be taking effect.
kubectl create -f websocket-service.yaml
kubectl create -f ../controllers/websocket-replication-controller.yaml
The result does not show the annotations configured.
{
"metadata": {
"name": "websocket-backend-controller-v1krf",
"generateName": "websocket-backend-controller-",
"namespace": "default",
"selfLink": "/api/v1/namespaces/default/pods/websocket-backend-controller-v1krf",
"uid": "e323994b-4081-11e7-8bd0-0050569b6f44",
"resourceVersion": "27531879",
"creationTimestamp": "2017-05-24T13:07:06Z",
"labels": {
"name": "websocket-backend"
},
"annotations": {
"kubernetes.io/created-by": "{\"kind\":\"SerializedReference\",\"apiVersion\":\"v1\",\"reference\":{\"kind\":\"ReplicationController\",\"namespace\":\"default\",\"name\":\"websocket-backend-controller\",\"uid\":\"e321f1a8-4081-11e7-8bd0-0050569b6f44\",\"apiVersion\":\"v1\",\"resourceVersion\":\"27531840\"}}\n"
}
All I am doing is rather than using a command line, I am setting the configs using services config but it does not seem to be working.
If you annotate the service, it doesn't take any effect on the possibly matched pods. Your pods are managed either by a ReplicationController, or over a ReplicaSet / Deployment. In that case, annotate these resources to make the annotations reach the pods. In example of deployments, you must use the template section, like:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
# Unique key of the Deployment instance
name: deployment-example
spec:
# 3 Pods should exist at all times.
replicas: 3
# Keep record of 2 revisions for rollback
revisionHistoryLimit: 2
template:
metadata:
annotations:
prometheus.io/scrape: 'true'
prometheus.io/path: /websocket/metrics
prometheus.io/port: '8080'
I am trying to prepare a dev environment for my team, so we can develop, stage and deploy with the same (or near same) environment.
Getting a Kubernetes Cluster running locally via http://kubernetes.io/v1.0/docs/getting-started-guides/docker.html was nice and simple. I could then use kubectl to start the pods and services for my application.
However, the services IP addresses are going to be different each time you start up. Which is a problem, if your code needs to use them. In Google Container Engine kube DNS means you can access a service by name. Which means the code that uses the service can remain constant between deployments.
Now, I know we could piece together the IP and PORT via environment variables, but I wanted to have an identical set up as possible.
So I followed some instructions found in various places, both here and in the Kubernetes repo like this.
Sure enough with a little editing of the yml files KubeDNS starts up.
But an nslookup on kubernetes.default fails. The health check on the DNS also fails (because it can't resolve the test look up) and the instance is shut down and restarted.
Running kubectl cluster-info results in:
Kubernetes master is running at http://localhost:8080
KubeDNS is running at http://localhost:8080/api/v1/proxy/namespaces/kube-system/services/kube-dns
So all good. However, hitting that endpoint results in:
{
kind: "Status",
apiVersion: "v1",
metadata: { },
status: "Failure",
message: "no endpoints available for "kube-dns"",
code: 500
}
I am now at a loss, and know it is something obvious or easy to fix as it seems to all be working. Here is how I start up the cluster and DNS.
# Run etcd
docker run --net=host \
-d gcr.io/google_containers/etcd:2.0.12 /usr/local/bin/etcd \
--addr=127.0.0.1:4001 --bind-addr=0.0.0.0:4001 --data-dir=/var/etcd/data
# Run the master
docker run \
--volume=/:/rootfs:ro \
--volume=/sys:/sys:ro \
--volume=/dev:/dev \
--volume=/var/lib/docker/:/var/lib/docker:ro \
--volume=/var/lib/kubelet/:/var/lib/kubelet:rw \
--volume=/var/run:/var/run:rw \
--net=host \
--privileged=true \
-d \
gcr.io/google_containers/hyperkube:v1.0.6 \
/hyperkube kubelet --containerized --hostname-override="127.0.0.1" \
--address="0.0.0.0" --api-servers=http://localhost:8080 \
--config=/etc/kubernetes/manifests \
--cluster_dns=10.0.0.10 --cluster_domain=cluster.local
# Run the service proxy
docker run -d --net=host --privileged gcr.io/google_containers/hyperkube:v1.0.6 \
/hyperkube proxy --master=http://127.0.0.1:8080 --v=2
# forward local port - after this you should be able to user kubectl locally
machine=default; ssh -i ~/.docker/machine/machines/$machine/id_rsa docker#$(docker-machine ip $machine) -L 8080:localhost:8080
All the containers spin up ok, kubectl get nodes reports ok. Note I pass in the dns flags.
I then start the DNS rc with this file, which is the edited version from here
apiVersion: v1
kind: ReplicationController
metadata:
name: kube-dns-v9
namespace: kube-system
labels:
k8s-app: kube-dns
version: v9
kubernetes.io/cluster-service: "true"
spec:
replicas: 1
selector:
k8s-app: kube-dns
version: v9
template:
metadata:
labels:
k8s-app: kube-dns
version: v9
kubernetes.io/cluster-service: "true"
spec:
containers:
- name: etcd
image: gcr.io/google_containers/etcd:2.0.9
resources:
limits:
cpu: 100m
memory: 50Mi
command:
- /usr/local/bin/etcd
- -data-dir
- /var/etcd/data
- -listen-client-urls
- http://127.0.0.1:2379,http://127.0.0.1:4001
- -advertise-client-urls
- http://127.0.0.1:2379,http://127.0.0.1:4001
- -initial-cluster-token
- skydns-etcd
volumeMounts:
- name: etcd-storage
mountPath: /var/etcd/data
- name: kube2sky
image: gcr.io/google_containers/kube2sky:1.11
resources:
limits:
cpu: 100m
memory: 50Mi
args:
# command = "/kube2sky"
- -domain=cluster.local
- name: skydns
image: gcr.io/google_containers/skydns:2015-10-13-8c72f8c
resources:
limits:
cpu: 100m
memory: 50Mi
args:
# command = "/skydns"
- -machines=http://localhost:4001
- -addr=0.0.0.0:53
- -ns-rotate=false
- -domain=cluster.local
ports:
- containerPort: 53
name: dns
protocol: UDP
- containerPort: 53
name: dns-tcp
protocol: TCP
livenessProbe:
httpGet:
path: /healthz
port: 8080
scheme: HTTP
initialDelaySeconds: 30
timeoutSeconds: 5
readinessProbe:
httpGet:
path: /healthz
port: 8080
scheme: HTTP
initialDelaySeconds: 1
timeoutSeconds: 5
- name: healthz
image: gcr.io/google_containers/exechealthz:1.0
resources:
limits:
cpu: 10m
memory: 20Mi
args:
- -cmd=nslookup kubernetes.default.svc.cluster.local 127.0.0.1 >/dev/null
- -port=8080
ports:
- containerPort: 8080
protocol: TCP
volumes:
- name: etcd-storage
emptyDir: {}
dnsPolicy: Default # Don't use cluster DNS.
Then start the service (again based on the file in the repo)
apiVersion: v1
kind: Service
metadata:
name: kube-dns
namespace: kube-system
labels:
k8s-app: kube-dns
kubernetes.io/cluster-service: "true"
kubernetes.io/name: "KubeDNS"
spec:
selector:
k8s-app: kube-dns
clusterIP: 10.0.0.10
ports:
- name: dns
port: 53
protocol: UDP
- name: dns-tcp
port: 53
protocol: TCP
I made the assumption based on another SO question that clusterIP is the value I passed into the master, and not the ip of the host machine. I am sure it has to be something obvious or simple that I have missed. Anyone out there who can help?
Thanks!
UPDATE
I found this closed issue over in the GitHub repo. Seems I have an identical problem.
I have added to the thread on GitHub, and tried lots of things but still no progress. I tried using different images, but they had different errors (or the same error representing itself differently, I couldn't tell).
Everything relating to this that I have found suggests IP restrictions, or firewall/security settings. So I decided to curl the api from the container itself.
docker exec 49705c38846a echo $(curl http://0.0.0.0:8080/api/v1/services?labels=)
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 908 100 908 0 0 314k 0 --:--:-- --:--:-- --:--:-- 443k
{ "kind": "ServiceList", "apiVersion": "v1", "metadata": { "selfLink": "/api/v1/services", "resourceVersion": "948" }, "items": [ { "metadata": { "name": "kubernetes", "namespace": "default", "selfLink": "/api/v1/namespaces/default/services/kubernetes", "uid": "369a9307-796e-11e5-87de-7a0704d1fdad", "resourceVersion": "6", "creationTimestamp": "2015-10-23T10:09:57Z", "labels": { "component": "apiserver", "provider": "kubernetes" } }, "spec": { "ports": [ { "protocol": "TCP", "port": 443, "targetPort": 443, "nodePort": 0 } ], "clusterIP": "10.0.0.1", "type": "ClusterIP", "sessionAffinity": "None" }, "status": { "loadBalancer": {} } } ] }
Seems like a valid response to me, so why the JSON parse error coming from kube2Sky!?
Failed to list *api.Service: couldn't get version/kind; json parse error: invalid character '<' looking for beginning of value
Failed to list *api.Endpoints: couldn't get version/kind; json parse error: invalid character '<' looking for beginning of value
The problem was with the networking and kube2sky not accessing the API, so couldn't get the services.
Changing the docker run for the master from,
--config=/etc/kubernetes/manifests
to
--config=/etc/kubernetes/manifests-multi
Then in the skydns-rc.yaml the for kube2sky as well as setting the domain, set the host IP address.
- -kube_master_url=http://192.168.99.100:8080 #<- your docker machine IP
Without the manifests-multi, the host IP is not accessible.
This was a simple change but took a bit to track down.
I have created a simple set up on GitHub and will maintain this so people don't have to go through this pain just to get a local dev environment up and running.
https://github.com/justingrayston/kubernetes-docker-dns
If you don't see any endpoints then most likely your skydns pod is not working. Try kubectl get pods --all-namespaces to see what is the status.
clusterIP can be any IP address which is not used yet. It will be used to connect to DNS service. It should not be the host IP.