I'm trying to get going with Kubernetes DaemonSets and not having any luck at all. I've searched for a solution to no avail. I'm hoping someone here can help out.
First, I've seen this ticket. Restarting the controller manager doesn't appear to help. As you can see here, the other kube processes have all been started after the apiserver and the api server has '--runtime-config=extensions/v1beta1=true' set.
kube 31398 1 0 08:54 ? 00:00:37 /usr/bin/kube-apiserver --logtostderr=true --v=0 --etcd_servers=http://dock-admin:2379 --address=0.0.0.0 --allow-privileged=false --portal_net=10.254.0.0/16 --admission_control=NamespaceAutoProvision,LimitRanger,ResourceQuota --runtime-config=extensions/v1beta1=true
kube 12976 1 0 09:49 ? 00:00:28 /usr/bin/kube-controller-manager --logtostderr=true --v=0 --master=http://127.0.0.1:8080 --cloud-provider=
kube 29489 1 0 11:34 ? 00:00:00 /usr/bin/kube-scheduler --logtostderr=true --v=0 --master=http://127.0.0.1:8080
However api-versions only shows version 1:
$ kubectl api-versions
Available Server Api Versions: v1
Kubernetes version is 1.2:
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"2", GitVersion:"v1.2.0", GitCommit:"86327329213fed4af2661c5ae1e92f9956b24f55", GitTreeState:"clean"}
Server Version: version.Info{Major:"1", Minor:"2", GitVersion:"v1.2.0", GitCommit:"86327329213fed4af2661c5ae1e92f9956b24f55", GitTreeState:"clean"}
The DaemonSet has been created, but appears to have no pods scheduled (status.desiredNumberScheduled).
$ kubectl get ds -o json
{
"kind": "List",
"apiVersion": "v1",
"metadata": {},
"items": [
{
"kind": "DaemonSet",
"apiVersion": "extensions/v1beta1",
"metadata": {
"name": "ds-test",
"namespace": "dvlp",
"selfLink": "/apis/extensions/v1beta1/namespaces/dvlp/daemonsets/ds-test",
"uid": "2d948b18-fa7b-11e5-8a55-00163e245587",
"resourceVersion": "2657499",
"generation": 1,
"creationTimestamp": "2016-04-04T15:37:45Z",
"labels": {
"app": "ds-test"
}
},
"spec": {
"selector": {
"app": "ds-test"
},
"template": {
"metadata": {
"creationTimestamp": null,
"labels": {
"app": "ds-test"
}
},
"spec": {
"containers": [
{
"name": "ds-test",
"image": "foo.vt.edu:1102/dbaa-app:v0.10-dvlp",
"ports": [
{
"containerPort": 8080,
"protocol": "TCP"
}
],
"resources": {},
"terminationMessagePath": "/dev/termination-log",
"imagePullPolicy": "IfNotPresent"
}
],
"restartPolicy": "Always",
"terminationGracePeriodSeconds": 30,
"dnsPolicy": "ClusterFirst",
"securityContext": {}
}
}
},
"status": {
"currentNumberScheduled": 0,
"numberMisscheduled": 0,
"desiredNumberScheduled": 0
}
}
]
}
Here is my yaml file to create the DaemonSet
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
name: ds-test
spec:
selector:
app: ds-test
template:
metadata:
labels:
app: ds-test
spec:
containers:
- name: ds-test
image: foo.vt.edu:1102/dbaa-app:v0.10-dvlp
ports:
- containerPort: 8080
Using that file to create the DaemonSet appears to work (I get 'daemonset "ds-test" created'), but no pods are created:
$ kubectl get pods -o json
{
"kind": "List",
"apiVersion": "v1",
"metadata": {},
"items": []
}
(I would have posted this as a comment, if I had enough reputation)
I am confused by your output.
kubectl api-versions should print out extensions/v1beta1 if it is enabled on the server. Since it does not, it looks like extensions/v1beta1 is not enabled.
But kubectl get ds should fail if extensions/v1beta1 is not enabled. So I can not figure out if extensions/v1beta1 is enabled on your server or not.
Can you try GET masterIP/apis and see if extensions is listed there?
You can also go to masterIP/apis/extensions/v1beta1 and see if daemonsets is listed there.
Also, I see kubectl version says 1.2, but then kubectl api-versions should not print out the string Available Server Api Versions (that string was removed in 1.1: https://github.com/kubernetes/kubernetes/pull/15796).
I have this issue in my cluster (k8s version: 1.9.7):
Daemonset controlled by "Daemonset controller" not "Scheduler", So I restart the controller manager, the problem sloved:
But I think this is a issue of kubernetes, some relation info:
Bug 1469037 - Sometime daemonset DESIRED=0 even this matched node
v1.7.4 - Daemonset DESIRED 0 (for node-exporter) #51785
I was facing a similar issue, then tried searching for the daemonset in the kube-system namespace, as mentioned here, https://github.com/kubernetes/kubernetes/issues/61342
I actually did get an output properly as well
For any case that the current state of pods is not equal to desired state (whether it was created by a DaemonSet, ReplicaSet, Deployment etc') I would first check the Kubelet on the current node:
$ sudo systemctl status kubelet
Or:
$ sudo journalctl -u kubelet
In many cases pods weren't created in my cluster because of errors like:
Couldn't parse as pod (Object 'Kind' is missing in 'null')
Which might occur after editing a resource's yaml in editor like vim.
Try:
$kubectl taint nodes --all node-role.kubernetes.io/master:NoSchedule-
master node cannot accept pods.
Related
I am currently trying to move my calico based clusters to the new Dataplane V2, which is basically a managed Cilium offering.
For local testing, I am running k3d with open source cilium installed, and created a set of NetworkPolicies (k8s native ones, not CiliumPolicies), which lock down the desired namespaces.
My current issue is, that when porting the same Policies on a GKE cluster (with DataPlane enabled), those same policies don't work.
As an example let's take a look into the connection between some app and a database:
---
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
name: db-server.db-client
namespace: BAR
spec:
podSelector:
matchLabels:
policy.ory.sh/db: server
policyTypes:
- Ingress
ingress:
- ports: []
from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: FOO
podSelector:
matchLabels:
policy.ory.sh/db: client
---
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
name: db-client.db-server
namespace: FOO
spec:
podSelector:
matchLabels:
policy.ory.sh/db: client
policyTypes:
- Egress
egress:
- ports:
- port: 26257
protocol: TCP
to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: BAR
podSelector:
matchLabels:
policy.ory.sh/db: server
Moreover, using GCP monitoring tools we can see the expected and actual effect the policies have on connectivity:
Expected:
Actual:
And logs from the application trying to connect to the DB, and getting denied:
{
"insertId": "FOO",
"jsonPayload": {
"count": 3,
"connection": {
"dest_port": 26257,
"src_port": 44506,
"dest_ip": "172.19.0.19",
"src_ip": "172.19.1.85",
"protocol": "tcp",
"direction": "egress"
},
"disposition": "deny",
"node_name": "FOO",
"src": {
"pod_name": "backoffice-automigrate-hwmhv",
"workload_kind": "Job",
"pod_namespace": "FOO",
"namespace": "FOO",
"workload_name": "backoffice-automigrate"
},
"dest": {
"namespace": "FOO",
"pod_namespace": "FOO",
"pod_name": "cockroachdb-0"
}
},
"resource": {
"type": "k8s_node",
"labels": {
"project_id": "FOO",
"node_name": "FOO",
"location": "FOO",
"cluster_name": "FOO"
}
},
"timestamp": "FOO",
"logName": "projects/FOO/logs/policy-action",
"receiveTimestamp": "FOO"
}
EDIT:
My local env is a k3d cluster created via:
k3d cluster create --image ${K3SIMAGE} --registry-use k3d-localhost -p "9090:30080#server:0" \
-p "9091:30443#server:0" foobar \
--k3s-arg=--kube-apiserver-arg="enable-admission-plugins=PodSecurityPolicy,NodeRestriction,ServiceAccount#server:0" \
--k3s-arg="--disable=traefik#server:0" \
--k3s-arg="--disable-network-policy#server:0" \
--k3s-arg="--flannel-backend=none#server:0" \
--k3s-arg=feature-gates="NamespaceDefaultLabelName=true#server:0"
docker exec k3d-server-0 sh -c "mount bpffs /sys/fs/bpf -t bpf && mount --make-shared /sys/fs/bpf"
kubectl taint nodes k3d-ory-cloud-server-0 node.cilium.io/agent-not-ready=true:NoSchedule --overwrite=true
skaffold run --cache-artifacts=true -p cilium --skip-tests=true --status-check=false
docker exec k3d-server-0 sh -c "mount --make-shared /run/cilium/cgroupv2"
Where cilium itself is being installed by skaffold, via helm with the following parameters:
name: cilium
remoteChart: cilium/cilium
namespace: kube-system
version: 1.11.0
upgradeOnChange: true
wait: false
setValues:
externalIPs.enabled: true
nodePort.enabled: true
hostPort.enabled: true
hubble.relay.enabled: true
hubble.ui.enabled: true
UPDATE:
I have setup a third environment: a GKE cluster using the old calico CNI (Legacy dataplane) and installed cilium manually as shown here. Cilium is working fine, even hubble is working out of the box (unlike with the dataplane v2...) and I found something interesting. The rules behave the same as with the GKE managed cilium, but with hubble working I was able to see this:
For some reason cilium/hubble cannot identify the db pod and decipher its labels. And since the labels don't work, the policies that rely on those labels, also don't work.
Another proof of this would be the trace log from hubble:
Here the destination app is only identified via an IP, and not labels.
The question now is why is this happening?
Any idea how to debug this problem? What could be difference coming from? Do the policies need some tuning for the managed Cilium, or is a bug in GKE?
Any help/feedback/suggestion appreciated!
Update: I was able to solve the mystery and it was ArgoCD all along. Cilium is creating an Endpoint and Identity for each object in the namespace, and Argo was deleting them after deploying the applications.
For anyone who stumbles on this, the solution is to add this exclusion to ArgoCD:
resource.exclusions: |
- apiGroups:
- cilium.io
kinds:
- CiliumIdentity
- CiliumEndpoint
clusters:
- "*"
According to this post:
https://netapp.io/2018/06/15/highly-secure-kubernetes-persistent-volumes/
You can't use/mount an NFS share in a pod if the pod is not having security context as privileged.
I am running a pod , with external NFS mounted but I have not specified any security context other than uid/gid. Working RW fine.
How can I check if my pod is a normal one or is privileged.
You can check this using kubectl get pods yourpod -o json under .spec.containers.securityContext or in metadata
As an example I created 2 nginx pods:
nginx(with privileged: true)
"metadata": {
"annotations": {
"cni.projectcalico.org/podIP": "10.48.2.3/32",
"kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"v1\",\"kind\":\"Pod\",\"metadata\":{\"annotations\":{},\"labels\":{\"app\":\"nginx\"},\"name\":\"nginx\",\"namespace\":\"default\"},\"spec\":{\"containers\":[{\"image\":\"nginx\",\"name\":\"nginx\",\"ports\":[{\"containerPort\":80}],\"securityContext\":{\"privileged\":true}}]}}\n",
"securityContext": {
"privileged": true
and
nginx-nonprivileged
"metadata": {
"annotations": {
"cni.projectcalico.org/podIP": "10.48.2.4/32",
"kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"v1\",\"kind\":\"Pod\",\"metadata\":{\"annotations\":{},\"labels\":{\"app\":\"nginx\"},\"name\":\"nginx-nonprivileged\",\"namespace\":\"default\"},\"spec\":{\"containers\":[{\"image\":\"nginx\",\"name\":\"nginx\",\"ports\":[{\"containerPort\":80}]}]}}\n",
I have installed docker-registry on Kubernetes via helm.
I am able to docker push to docker push 0.0.0.0:5000/<my-container>:v1 using port-forward.
Now how do I reference the images in the registry from a deployment.yaml?
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: <my-container>-deployment-v1
spec:
replicas: 1
template:
metadata:
labels:
app: <my-container>-deployment
version: v1
spec:
containers:
- name: <my-container>
image: 0.0.0.0:5000/<my-container>:v1 # <<< ????
imagePullPolicy: Always
ports:
- containerPort: 80
imagePullSecrets:
- name: private-docker-registry-secret
This do list my containers:
curl -X GET http://0.0.0.0:5000/v2/_catalog
I keep getting ImagePullBackOff when deploying.
I tyied using internal service name and cluster ip address, still not working.
Then tried using secrets:
{
"kind": "Secret",
"apiVersion": "v1",
"metadata": {
"name": "running-buffoon-docker-registry-secret",
"namespace": "default",
"selfLink": "/api/v1/namespaces/default/secrets/running-buffoon-docker-registry-secret",
"uid": "127c93c1-53df-11e9-8ede-a63ad724d5b9",
"resourceVersion": "216488",
"creationTimestamp": "2019-03-31T18:01:56Z",
"labels": {
"app": "docker-registry",
"chart": "docker-registry-1.7.0",
"heritage": "Tiller",
"release": "running-buffoon"
}
},
"data": {
"haSharedSecret": "xxx"
},
"type": "Opaque"
}
And added the secret to to deployment.yaml:
imagePullSecrets:
- name: running-buffoon-docker-registry-secret
Then I get:
image "x.x.x.x/:<my-container>v1": rpc error: code = Unknown desc = Error response from daemon: Get https://x.x.x.x/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
You need to get the cluster-ip of your local docker registry.
You will find this in the dashboard - just visit the registry pod page and then to the associated service. Replace your image spec's 0.0.0.0 with the cluster ip. Also make sure the port matches - generally the port exposed by registry service is different from the actual port exposed inside the cluster. If you have authentication set up in your registry, you will need imagepullsecret as well.
I have blogged about minikube setup with a local registry - might be helpful. https://amritbera.com/journal/minikube-insecure-registry.html
I cannot reach the following Kubernetes service when externalTrafficPolicy: Local is set. I access it directly through the NodePort but always get a timeout.
{
"kind": "Service",
"apiVersion": "v1",
"metadata": {
"name": "echo",
"namespace": "default",
"selfLink": "/api/v1/namespaces/default/services/echo",
"uid": "c1b66aca-cc53-11e8-9062-d43d7ee2fdff",
"resourceVersion": "5190074",
"creationTimestamp": "2018-10-10T06:14:33Z",
"labels": {
"k8s-app": "echo"
}
},
"spec": {
"ports": [
{
"name": "tcp-8080-8080-74xhz",
"protocol": "TCP",
"port": 8080,
"targetPort": 3333,
"nodePort": 30275
}
],
"selector": {
"k8s-app": "echo"
},
"clusterIP": "10.101.223.0",
"type": "NodePort",
"sessionAffinity": "None",
"externalTrafficPolicy": "Local"
},
"status": {
"loadBalancer": {}
}
}
I know that for this pods of the service need to be available on a node because traffic is not routed to other nodes. I checked this.
Not sure where you are connecting from and what command you are typing to test connectivity or what's your environment like. But this is most likely due to this known issue where the node ports are not reachable with externalTrafficPolicy set to Local if the kube-proxy cannot find the IP address for the node where it's running on.
This link sheds more light into the problem. Apparently --hostname-override on the kube-proxy is not working as of K8s 1.10. You have to specify the HostnameOverride option in the kube-proxy ConfigMap. There's also a fix described here that will make it upstream at some point in the future from this writing.
As Jagrut said, the link shared in Rico's answer does not contain the desired section with the patch anymore, so I'll share a different thread where stacksonstacks' answer worked for me: here . This solution consists in editing kube-proxy.yaml to include the HOST_IP argument.
In my case, request to node cluster/public ip which own deployment/pod.
spec:
clusterIP: 10.9x.x.x <-- request this ip
clusterIPs:
- 10.9x.x.x
externalTrafficPolicy: Local
or
NAME STATUS ... EXTERNAL-IP
master-node Ready 3.2x.x.x
worker-node Ready 13.1x.x.x <-- request this ip
additionally, alway to request same node's ip, use nodeSelector in Deployment.
My Kubernetes version is :
# kubectl --version
Kubernetes v1.4.0
I am planning to use Prometheus to monitor my Kube cluster. For this, I need to annotate the metrics URL.
My current metrics URL is like :
http://172.16.33.7:8080/metrics
But I want it to be like :
http://172.16.33.7:8080/websocket/metrics
First I tried to do this manually ::
kubectl annotate pods websocket-backend-controller-db83999c5b534b277b82badf6c152cb9m1 prometheus.io/path=/websocket/metrics
kubectl annotate pods websocket-backend-controller-db83999c5b534b277b82badf6c152cb9m1 prometheus.io/scrape='true'
kubectl annotate pods websocket-backend-controller-db83999c5b534b277b82badf6c152cb9m1 prometheus.io/port='8080'
All these commands work perfectly fine and I am able to see the annotations.
{
"metadata": {
"name": "websocket-backend-controller-v1krf",
"generateName": "websocket-backend-controller-",
"namespace": "default",
"selfLink": "/api/v1/namespaces/default/pods/websocket-backend-controller-v1krf",
"uid": "e323994b-4081-11e7-8bd0-0050569b6f44",
"resourceVersion": "27534379",
"creationTimestamp": "2017-05-24T13:07:06Z",
"labels": {
"name": "websocket-backend"
},
"annotations": {
"kubernetes.io/created-by": "{\"kind\":\"SerializedReference\",\"apiVersion\":\"v1\",\"reference\":{\"kind\":\"ReplicationController\",\"namespace\":\"default\",\"name\":\"websocket-backend-controller\",\"uid\":\"e321f1a8-4081-11e7-8bd0-0050569b6f44\",\"apiVersion\":\"v1\",\"resourceVersion\":\"27531840\"}}\n",
"prometheus.io/path": "/websocket/metrics",
"prometheus.io/port": "8080",
"prometheus.io/scrape": "true"
}
But since I want this configuration to remain permanent, I am setting the following annotations in my services files.
# cat websocket-service.yaml
apiVersion: v1
kind: Service
metadata:
name: websocket-service
labels:
baseApi: websocket
annotations:
prometheus.io/scrape: 'true'
prometheus.io/path: /websocket/metrics
prometheus.io/port: '8080'
spec:
selector:
name: websocket-backend
ports:
- port: 8080
targetPort: 8080
nodePort: 30800
protocol: TCP
type: NodePort
clusterIP: 10.100.10.45
I restarted my websocket service and the corresponding pods but these configs do not seem to be taking effect.
kubectl create -f websocket-service.yaml
kubectl create -f ../controllers/websocket-replication-controller.yaml
The result does not show the annotations configured.
{
"metadata": {
"name": "websocket-backend-controller-v1krf",
"generateName": "websocket-backend-controller-",
"namespace": "default",
"selfLink": "/api/v1/namespaces/default/pods/websocket-backend-controller-v1krf",
"uid": "e323994b-4081-11e7-8bd0-0050569b6f44",
"resourceVersion": "27531879",
"creationTimestamp": "2017-05-24T13:07:06Z",
"labels": {
"name": "websocket-backend"
},
"annotations": {
"kubernetes.io/created-by": "{\"kind\":\"SerializedReference\",\"apiVersion\":\"v1\",\"reference\":{\"kind\":\"ReplicationController\",\"namespace\":\"default\",\"name\":\"websocket-backend-controller\",\"uid\":\"e321f1a8-4081-11e7-8bd0-0050569b6f44\",\"apiVersion\":\"v1\",\"resourceVersion\":\"27531840\"}}\n"
}
All I am doing is rather than using a command line, I am setting the configs using services config but it does not seem to be working.
If you annotate the service, it doesn't take any effect on the possibly matched pods. Your pods are managed either by a ReplicationController, or over a ReplicaSet / Deployment. In that case, annotate these resources to make the annotations reach the pods. In example of deployments, you must use the template section, like:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
# Unique key of the Deployment instance
name: deployment-example
spec:
# 3 Pods should exist at all times.
replicas: 3
# Keep record of 2 revisions for rollback
revisionHistoryLimit: 2
template:
metadata:
annotations:
prometheus.io/scrape: 'true'
prometheus.io/path: /websocket/metrics
prometheus.io/port: '8080'