Kubernetes API server , serving pod logs - kubernetes

Question

The REST API requests , GET , POST , PUT etc to Kubernetes API server are request , responses and simple to understand , such as kubectl create <something>. I wonder how the API server serves the pod logs when I do kubectl logs -f <pod-name> ( and similar operations like kubectl attach <pod> ), Is it just an http response to GET in a loop?

Answer 1

My advice is to always check what kubectl does under the cover, and for that use -v=9 with your command. It will provide you with full request and responses that are going between the client and the server.

Answer 2

Yep, it looks like it's currently just a HTTP GET that kubectl is using, when looking at the source of logs.go although there seems to be a desire to unify and upgrade a couple of commands (exec, port-forward, logs, etc.) to WebSockets.
Showing Maciej's excellent suggestion in action:
$ kubectl run test --image centos:7 \
-- sh -c "while true ; do echo Work ; sleep 2 ; done"
$ kubectl get po
NAME READY STATUS RESTARTS AGE
test-769f6f8c9f-2nx7m 1/1 Running 0 2m
$ kubectl logs -v9 -f test-769f6f8c9f-2nx7m
I1019 13:49:34.282007 71247 loader.go:359] Config loaded from file /Users/mhausenblas/.kube/config
I1019 13:49:34.284698 71247 loader.go:359] Config loaded from file /Users/mhausenblas/.kube/config
I1019 13:49:34.292620 71247 loader.go:359] Config loaded from file /Users/mhausenblas/.kube/config
I1019 13:49:34.293136 71247 round_trippers.go:386] curl -k -v -XGET -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.12.0 (darwin/amd64) kubernetes/0ed3388" 'https://192.168.64.13:8443/api/v1/namespaces/default/pods/test-769f6f8c9f-2nx7m'
I1019 13:49:34.305016 71247 round_trippers.go:405] GET https://192.168.64.13:8443/api/v1/namespaces/default/pods/test-769f6f8c9f-2nx7m 200 OK in 11 milliseconds
I1019 13:49:34.305039 71247 round_trippers.go:411] Response Headers:
I1019 13:49:34.305047 71247 round_trippers.go:414] Date: Fri, 19 Oct 2018 12:49:34 GMT
I1019 13:49:34.305054 71247 round_trippers.go:414] Content-Type: application/json
I1019 13:49:34.305062 71247 round_trippers.go:414] Content-Length: 2390
I1019 13:49:34.305125 71247 request.go:942] Response Body: {"kind":"Pod","apiVersion":"v1","metadata":{"name":"test-769f6f8c9f-2nx7m","generateName":"test-769f6f8c9f-","namespace":"default","selfLink":"/api/v1/namespaces/default/pods/test-769f6f8c9f-2nx7m","uid":"0581b0fa-d39d-11e8-9827-42a64713caf8","resourceVersion":"892912","creationTimestamp":"2018-10-19T12:46:39Z","labels":{"pod-template-hash":"3259294759","run":"test"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"test-769f6f8c9f","uid":"057f3ad4-d39d-11e8-9827-42a64713caf8","controller":true,"blockOwnerDeletion":true}]},"spec":{"volumes":[{"name":"default-token-fbx4m","secret":{"secretName":"default-token-fbx4m","defaultMode":420}}],"containers":[{"name":"test","image":"centos:7","args":["sh","-c","while true ; do echo Work ; sleep 2 ; done"],"resources":{},"volumeMounts":[{"name":"default-token-fbx4m","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"minikube","securityContext":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}]},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2018-10-19T12:46:39Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2018-10-19T12:46:40Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":null},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2018-10-19T12:46:39Z"}],"hostIP":"192.168.64.13","podIP":"172.17.0.11","startTime":"2018-10-19T12:46:39Z","containerStatuses":[{"name":"test","state":{"running":{"startedAt":"2018-10-19T12:46:39Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"centos:7","imageID":"docker-pullable://centos#sha256:67dad89757a55bfdfabec8abd0e22f8c7c12a1856514726470228063ed86593b","containerID":"docker://5c25f5fce576d68d743afc9b46a9ea66f3cd245f5075aa95def623b6c2d93256"}],"qosClass":"BestEffort"}}
I1019 13:49:34.316531 71247 loader.go:359] Config loaded from file /Users/mhausenblas/.kube/config
I1019 13:49:34.317000 71247 round_trippers.go:386] curl -k -v -XGET -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.12.0 (darwin/amd64) kubernetes/0ed3388" 'https://192.168.64.13:8443/api/v1/namespaces/default/pods/test-769f6f8c9f-2nx7m/log?follow=true'
I1019 13:49:34.339341 71247 round_trippers.go:405] GET https://192.168.64.13:8443/api/v1/namespaces/default/pods/test-769f6f8c9f-2nx7m/log?follow=true 200 OK in 22 milliseconds
I1019 13:49:34.339380 71247 round_trippers.go:411] Response Headers:
I1019 13:49:34.339390 71247 round_trippers.go:414] Content-Type: text/plain
I1019 13:49:34.339407 71247 round_trippers.go:414] Date: Fri, 19 Oct 2018 12:49:34 GMT
Work
Work
Work
^C

Answer 3

If you extract any Kubernetes object using kubectl on the highest debugging level -v 9 with a streaming option -f, as for example kubectl logs -f <pod-name> -v 9, you can realize that kubectl passing follow=true flag to API request by acquiring logs from target Pod accordingly, and stream to the output as well:
curl -k -v -XGET -H "Accept: application/json, /" -H "User-Agent:
kubectl/v1.12.1 (linux/amd64) kubernetes/4ed3216"
'https://API_server_IP/api/v1/namespaces/default/pods/Pod-name/log?follow=true'
You can consider launching own API requests by following the next steps:
Obtain token for authorization purpose:
MY_TOKEN="$(kubectl get secret <default-secret> -o jsonpath='{$.data.token}' | base64 -d)"
Then you can retrieve manually the required data from API server directly:
curl -k -v -H "Authorization : Bearer $MY_TOKEN" https://API_server_IP/api/v1/namespaces/default/pods