Pod labeling not possible from within pod using Kubernetes on Docker-EE - kubernetes

Question

We are using a Apache-Kafka deployment on Kubernetes which is based on the ability to label pods after they have been created (see https://github.com/Yolean/kubernetes-kafka). The init container of the broker pods takes advantage of this feature to set a label on itself with its own numeric index (e.g. "0", "1", etc) as value. The label is used in the service descriptors to select exactly one pod.
This approach works fine on our DIND-Kubernetes environment. However, when tried to port the deployment onto a Docker-EE Kubernetes environment we ran into trouble because the command kubectl label pod generates a run time error which is completely misleading (also see https://github.com/fabric8io/kubernetes-client/issues/853).
In order to verify the run time error in a minimal setup we created the following deployment scripts.
First step: Successfully label pod using the Docker-EE-Host
# create a simple pod as a test target for labeling
> kubectl run -ti -n default --image alpine sh
# get the pod name for all further steps
> kubectl -n default get pods
NAME READY STATUS RESTARTS AGE
nfs-provisioner-7d49cdcb4f-8qx95 1/1 Running 1 7d
nginx-deployment-76dcc8c697-ng4kb 1/1 Running 1 7d
nginx-deployment-76dcc8c697-vs24j 1/1 Running 0 20d
sh-777f6db646-hrm65 1/1 Running 0 3m <--- This is the test pod
test-76bbdb4654-9wd9t 1/1 Running 2 6d
test4-76dbf847d5-9qck2 1/1 Running 0 5d
# get client and server versions
> kubectl version
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.5",
GitCommit:"32ac1c9073b132b8ba18aa830f46b77dcceb0723", GitTreeState:"clean",
BuildDate:"2018-06-21T11:46:00Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"8+", GitVersion:"v1.8.11- docker-8d637ae", GitCommit:"8d637aedf46b9c21dde723e29c645b9f27106fa5",
GitTreeState:"clean", BuildDate:"2018-04-26T16:51:21Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
# set label
kubectl -n default label pod sh-777f6db646-hrm65 "mylabel=hallo"
pod "sh-777f6db646-hrm65" labeled <---- successful execution
Everything works fine as expected.
Second step: Reproduce run-time error from within pod
Create Docker image containing kubectl 1.10.5
FROM debian:stretch-
slim#sha256:ea42520331a55094b90f6f6663211d4f5a62c5781673935fe17a4dfced777029
ENV KUBERNETES_VERSION=1.10.5
RUN set -ex; \
export DEBIAN_FRONTEND=noninteractive; \
runDeps='curl ca-certificates procps netcat'; \
buildDeps=''; \
apt-get update && apt-get install -y $runDeps $buildDeps --no-install- recommends; \
rm -rf /var/lib/apt/lists/*; \
\
curl -sLS -o k.tar.gz -k https://dl.k8s.io/v${KUBERNETES_VERSION}/kubernetes-client-linux-amd64.tar.gz; \
tar -xvzf k.tar.gz -C /usr/local/bin/ --strip-components=3 kubernetes/client/bin/kubectl; \
rm k.tar.gz; \
\
apt-get purge -y --auto-remove $buildDeps; \
rm /var/log/dpkg.log /var/log/apt/*.log
This image is deployed as 10.100.180.74:5000/test/kubectl-client-1.10.5 in a site local registry and will be referred to below.
Create a pod using the container above
apiVersion: apps/v1beta2
kind: StatefulSet
metadata:
name: pod-labeler
namespace: default
spec:
selector:
matchLabels:
app: pod-labeler
replicas: 1
serviceName: pod-labeler
updateStrategy:
type: OnDelete
template:
metadata:
labels:
app: pod-labeler
annotations:
spec:
terminationGracePeriodSeconds: 10
containers:
- name: check-version
image: 10.100.180.74:5000/test/kubectl-client-1.10.5
env:
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: POD_NAME
value: sh-777f6db646-hrm65
command: ["/usr/local/bin/kubectl", "version" ]
- name: label-pod
image: 10.100.180.74:5000/test/kubectl-client-1.10.5
env:
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: POD_NAME
value: sh-777f6db646-hrm65
command: ["/bin/bash", "-c", "/usr/local/bin/kubectl -n default label pod $POD_NAME 'mylabel2=hallo'" ]
Logging output
We get the following logging output
# Log of the container "check-version"
2018-07-18T11:11:10.791011157Z Client Version: version.Info{Major:"1",
Minor:"10", GitVersion:"v1.10.5",
GitCommit:"32ac1c9073b132b8ba18aa830f46b77dcceb0723", GitTreeState:"clean",
BuildDate:"2018-\
06-21T11:46:00Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
2018-07-18T11:11:10.791058997Z Server Version: version.Info{Major:"1",
Minor:"8+", GitVersion:"v1.8.11-docker-8d637ae",
GitCommit:"8d637aedf46b9c21dde723e29c645b9f27106fa5", GitTreeState:"clean",
BuildDate:"2018-04-26T16:51:21Z", GoVersion:"go1.8.3", Compiler:"gc",
Platform:"linux/amd64"}
and the run time error
2018-07-18T11:24:15.448695813Z The Pod "sh-777f6db646-hrm65" is invalid:
spec.tolerations: Forbidden: existing toleration can not be modified except its tolerationSeconds
Notes
This is not an authorization problem since we've given the default user of the default namespace full administrative rights. In case we don't, we get an error message referring to missing permissions.
Both client and servers versions "outside" (e.g on the docker host) and "inside" (e.g. the pod) are identical down to the GIT commit tag
We are using version 3.0.2 of the Universal Control Plane
Any ideas?

Answer 1

It was pointed out in one of the comments that the issue may be caused by a missing permission even though the error message does not insinuate so. We officially filed a ticket with Docker and actually got exactly this result: In order to be able to set/modify a label from within a pod the default user of the namespace must be given the "Scheduler" role on the swarm resource (which later shows up as \ in the GUI). Granting this permission fixes the problem. See added grant in Docker-EE-GUI below.
From my point of view, this is far from obvious. The Docker support representative offered to investigate if this is actually expected behavior or results from a bug. As soon as we learn more on this question I will include it into our answer.
As for using more debugging output: Unfortunately, adding --v=9 to the calls of kubectl does not return any useful information. There's too much output to be displayed here but the overall logging is very similar in both cases: It consists of a lot GET API requests which are all successful followed by a final PATCH API request which succeeds in the one case but fails in the other as described above.