Openshift - Applying RunOnlyAs SCC to pods within a deployment - deployment

I have an SCC defined that lets the developer user run containers as UID 1015:
kind: SecurityContextConstraints
apiVersion: v1
metadata:
name: developer
allowPrivilegedContainer: false
runAsUser:
type: MustRunAs
uid: 1015
seLinuxContext:
type: MustRunAs
uid: 1015
users:
- developer
This SCC works fine when I create a pod directly as the developer user:
apiVersion: v1
kind: Pod
metadata:
name: test-1-0
spec:
securityContext:
runAsUser: 1015
containers:
- name: test-1-0
image: test-image:1.0
imagePullPolicy: "Always"
volumeMounts:
- name: secret-dir
mountPath: "/secrets"
readOnly: true
volumes:
- name: secret-dir
secret:
secretName: test-1.0-configs
However, when I transition this to a deployment, I get an error. Here's my YAML:
kind: "DeploymentConfig"
apiVersion: "v1"
metadata:
name: "test-1-0"
spec:
template:
metadata:
labels:
name: "test-1-0"
spec:
securityContext:
runAsUser: 1015
containers:
- name: test-1-0
image: test-image:1.0
imagePullPolicy: "Always"
volumeMounts:
- name: secret-dir
mountPath: "/secrets"
readOnly: true
volumes:
- name: secret-dir
secret:
secretName: test-1.0-configs
replicas: 2
selector:
name: "test-1-0"
And the error I receive:
Error creating: pods "test-1-0-1-" is forbidden: unable to validate against any security context constraint: [securityContext.runAsUser: Invalid value: 1015: UID on container test-1-0 does not match required range. Found 1015, required min: 1000050000 max: 1000059999]
It's as if the deployment is deploying pods NOT as the user I created the deployment with. Is there any way to resolve that?

It is probably better to dictate what SCC is to be used for the deployment via a service account.
First off create the SCC. Using what I have tested as an example, I created uid1000.json containing.
{
"apiVersion": "v1",
"kind": "SecurityContextConstraints",
"metadata": {
"name": "uid1000"
},
"requiredDropCapabilities": [
"KILL",
"MKNOD",
"SYS_CHROOT",
"SETUID",
"SETGID"
],
"runAsUser": {
"type": "MustRunAs",
"uid": "1000"
},
"seLinuxContext": {
"type": "MustRunAs"
},
"fsGroup": {
"type": "MustRunAs"
},
"supplementalGroups": {
"type": "RunAsAny"
},
"volumes": [
"configMap",
"downwardAPI",
"emptyDir",
"persistentVolumeClaim",
"projected",
"secret"
]
}
Then ran:
oc create -f uid1000.json --as system:admin
Need to be admin to do that.
Next I created a service account in the target project just for running the application which needs this SCC.
oc create serviceaccount runasuid1000
I now say that anything run as this service account should use the new SCC.
oc adm policy add-scc-to-user uid1000 -z runasuid1000 --as system:admin
Again need to be admin to do that. The -z option means use current project, so make sure you are in the right one.
Finally I patch the existing deployment config.
oc patch dc/minimal-notebook --patch '{"spec":{"template":{"spec":{"serviceAccountName": "runasuid1000"}}}}'
If necessary, due to config change trigger being disabled, trigger a new deployment:
oc rollout latest minimal-notebook
This will force the container to run as uid 1000, overriding what the image even says it should run as according to USER definition.

Related

Populating a Containers environment values with mounted configMap in Kubernetes

I'm currently learning Kubernetes and recently learnt about using ConfigMaps for a Containers environment variables.
Let's say I have the following simple ConfigMap:
apiVersion: v1
data:
MYSQL_ROOT_PASSWORD: password
kind: ConfigMap
metadata:
creationTimestamp: null
name: mycm
I know that a container of some deployment can consume this environment variable via:
kubectl set env deployment mydb --from=configmap/mycm
or by specifying it manually in the manifest like so:
containers:
- env:
- name: MYSQL_ROOT_PASSWORD
valueFrom:
configMapKeyRef:
key: MYSQL_ROOT_PASSWORD
name: mycm
However, this isn't what I am after, since I'd to manually change the environment variables each time the ConfigMap changes.
I am aware that mounting a ConfigMap to the Pod's volume allows for the auto-updating of ConfigMap values. I'm currently trying to find a way to set a Container's environment variables to those stored in the mounted config map.
So far I have the following YAML manifest:
apiVersion: apps/v1
kind: Deployment
metadata:
creationTimestamp: null
labels:
app: mydb
name: mydb
spec:
replicas: 1
selector:
matchLabels:
app: mydb
strategy: {}
template:
metadata:
creationTimestamp: null
labels:
app: mydb
spec:
containers:
- image: mariadb
name: mariadb
resources: {}
args: ["export MYSQL_ROOT_PASSWORD=$(cat /etc/config/MYSQL_ROOT_PASSWORD)"]
volumeMounts:
- name: config-volume
mountPath: /etc/config
env:
- name: MYSQL_ROOT_PASSWORD
value: temp
volumes:
- name: config-volume
configMap:
name: mycm
status: {}
I'm attempting to set the MYSQL_ROOT_PASSWORD to some temporary value, and then update it to mounted value as soon as the container starts via args: ["export MYSQL_ROOT_PASSWORD=$(cat /etc/config/MYSQL_ROOT_PASSWORD)"]
As I somewhat expected, this didn't work, resulting in the following error:
/usr/local/bin/docker-entrypoint.sh: line 539: /export MYSQL_ROOT_PASSWORD=$(cat /etc/config/MYSQL_ROOT_PASSWORD): No such file or directory
I assume this is because the volume is mounted after the entrypoint. I tried adding a readiness probe to wait for the mount but this didn't work either:
readinessProbe:
exec:
command: ["sh", "-c", "test -f /etc/config/MYSQL_ROOT_PASSWORD"]
initialDelaySeconds: 5
periodSeconds: 5
Is there any easy way to achieve what I'm trying to do, or is it impossible?
So I managed to find a solution, with a lot of inspiration from this answer.
Essentially, what I did was create a sidecar container based on the alpine K8s image that mounts the configmap and constantly watches for any changes, since the K8s API automatically updates the mounted configmap when the configmap is changed. This required the following script, watch_passwd.sh, which makes use of inotifywait to watch for changes and then uses the K8s API to rollout the changes accordingly:
update_passwd() {
kubectl delete secret mysql-root-passwd > /dev/null 2>&1
kubectl create secret generic mysql-root-passwd --from-file=/etc/config/MYSQL_ROOT_PASSWORD
}
update_passwd
while true
do
inotifywait -e modify "/etc/config/MYSQL_ROOT_PASSWORD"
update_passwd
kubectl rollout restart deployment $1
done
The Dockerfile is then:
FROM docker.io/alpine/k8s:1.25.6
RUN apk update && apk add inotify-tools
COPY watch_passwd.sh .
After building the image (locally in this case) as mysidecar, I create the ServiceAccount, Role, and RoleBinding outlined here, adding rules for deployments so that they can be restarted by the sidecar.
After this, I piece it all together to create the following YAML Manifest (note that imagePullPolicy is set to Never, since I created the image locally):
apiVersion: apps/v1
kind: Deployment
metadata:
creationTimestamp: null
labels:
app: mydb
name: mydb
spec:
replicas: 3
selector:
matchLabels:
app: mydb
strategy: {}
template:
metadata:
creationTimestamp: null
labels:
app: mydb
spec:
serviceAccountName: secretmaker
containers:
- image: mysidecar
name: mysidecar
imagePullPolicy: Never
command:
- /bin/sh
- -c
- |
./watch_passwd.sh $(DEPLOYMENT_NAME)
env:
- name: DEPLOYMENT_NAME
valueFrom:
fieldRef:
fieldPath: metadata.labels['app']
volumeMounts:
- name: config-volume
mountPath: /etc/config
- image: mariadb
name: mariadb
resources: {}
envFrom:
- secretRef:
name: mysql-root-passwd
volumes:
- name: config-volume
configMap:
name: mycm
status: {}
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: secretmaker
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
labels:
app: mydb
name: secretmaker
rules:
- apiGroups: [""]
resources: ["secrets"]
verbs: ["create", "get", "delete", "list"]
- apiGroups: ["apps"]
resources: ["deployments"]
verbs: ["get", "list", "watch", "update", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
labels:
app: mydb
name: secretmaker
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: secretmaker
subjects:
- kind: ServiceAccount
name: secretmaker
namespace: default
---
It all works as expected! Hopefully this is able to help someone out in the future. Also, if anybody comes across this and has a better solution please feel free to let me know :)

The connection to the server x.x.x.x:6443 was refused - did you specify the right host or port? Kubernetes

Here is the scenario -
I have deployed kubernetes cluster and inside that I have created an application( say test) with "kubectl:1.20" image. I have created required clusterrole, rolebindings and sa for this test application to have the authorization for managing deployments.
Now my requirement is that i should be able to fetch and patch the deployments which are running in the kubernetes cluster from inside the test pod as part of a cronjob.
When i am running the cronjob, i am getting the below mentioned error in the test pod
"The connection to the server x.x.x.x:6443 was refused - did you specify the right host or port?"
I have passed the kubeconfig file of the cluster as configmap to the test pod.
below is the cronjob yml file -
kind: CronJob
apiVersion: batch/v1beta1
metadata:
name: test
spec:
schedule: "*/2 * * * *"
jobTemplate:
spec:
template:
spec:
serviceAccountName: local-kubectl-scale-deploy
containers:
- name: mycron-container
image: bitnami/kubectl:1.20
imagePullPolicy: IfNotPresent
env:
- name: KUBECONFIG
value: "/tmp/kubeconfig"
command: [ "/bin/sh" ]
args: [ "/var/httpd-init/script", "namespace" ]
tty: true
volumeMounts:
- name: script
mountPath: "/var/httpd-init/"
- name: kubeconfig
mountPath: "/tmp/"
volumes:
- name: script
configMap:
name: cronscript
defaultMode: 0777
- name: kubeconfig
configMap:
name: kubeconfig
restartPolicy: OnFailure
terminationGracePeriodSeconds: 0
concurrencyPolicy: Replace
below is the kubeconfig file which i am passing as configmap inside the pod.
kind: ConfigMap
apiVersion: v1
metadata:
name: kubeconfig
labels:
app: kubeconfig
data:
kubeconfig: |
apiVersion: v1
kind: Config
clusters:
- name: default-cluster
cluster:
certificate-authority-data: xxxxxxxxxxx
server: https://x.x.x.x:6443
contexts:
- name: default-context
context:
cluster: default-cluster
namespace: default
user: default-user
current-context: default-context
users:
- name: default-user
user:
token: xxxxxx
Am I missing some configuration from the pod to communicate to this x.x.x.x:6443 IP?
To diagnose the communication error to the address we would need a lot more detail from your network configuration, but instead we can fix the cronjob.
The proper way of achieving that is to set a service account with prober RBAC similar to https://stackoverflow.com/a/58378834/3930971

Get the Kubernetes uid of the Deployment that created the pod, from within the pod

I want to be able to know the Kubernetes uid of the Deployment that created the pod, from within the pod.
The reason for this is so that the Pod can spawn another Deployment and set the OwnerReference of that Deployment to the original Deployment (so it gets Garbage Collected when the original Deployment is deleted).
Taking inspiration from here, I've tried*:
Using field refs as env vars:
containers:
- name: test-operator
env:
- name: DEPLOYMENT_UID
valueFrom:
fieldRef: {fieldPath: metadata.uid}
Using downwardAPI and exposing through files on a volume:
containers:
volumeMounts:
- mountPath: /etc/deployment-info
name: deployment-info
volumes:
- name: deployment-info
downwardAPI:
items:
- path: "uid"
fieldRef: {fieldPath: metadata.uid}
*Both of these are under spec.template.spec of a resource of kind: Deployment.
However for both of these the uid is that of the Pod, not the Deployment. Is what I'm trying to do possible?
The behavior is correct, the Downward API is for pod rather than deployment/replicaset.
So I guess the solution is set the name of deployment manually in spec.template.metadata.labels, then adopt Downward API to inject the labels as env variables.
I think it's impossible to get the UID of Deployment itself, you can set any range of runAsUser while creating the deployment.
Try this command to get the UIDs of the existing pods:
kubectl get pod -o jsonpath='{range .items[*]}{#.metadata.name}{" runAsUser: "}{#.spec.containers[*].securityContext.runAsUser}{" fsGroup: "}{#.spec.securityContext.fsGroup}{" seLinuxOptions: "}{#.spec.securityContext.seLinuxOptions.level}{"\n"}{end}'
It's not the exact what you wanted to get, but it can be a hint for you.
To set the UID while creating the Deployment, see the example below:
apiVersion: apps/v1
kind: Deployment
metadata:
name: toolbox2
labels:
app: toolbox2
spec:
replicas: 3
selector:
matchLabels:
app: toolbox2
template:
metadata:
labels:
app: toolbox2
spec:
securityContext:
supplementalGroups: [1000620001]
seLinuxOptions:
level: s0:c25,c10
containers:
- name: net-toolbox
image: quay.io/wcaban/net-toolbox
ports:
- containerPort: 2000
securityContext:
runAsUser: 1000620001

Restart kubernetes deployment after changing configMap

I have a deployment which includes a configMap, persistentVolumeClaim, and a service. I have changed the configMap and re-applied the deployment to my cluster. I understand that this change does not automatically restart the pod in the deployment:
configmap change doesn't reflect automatically on respective pods
Updated configMap.yaml but it's not being applied to Kubernetes pods
I know that I can kubectl delete -f wiki.yaml && kubectl apply -f wiki.yaml. But that destroys the persistent volume which has data I want to survive the restart. How can I restart the pod in a way that keeps the existing volume?
Here's what wiki.yaml looks like:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: dot-wiki
spec:
accessModes:
- ReadWriteOnce
volumeMode: Filesystem
resources:
requests:
storage: 4Gi
---
apiVersion: v1
kind: ConfigMap
metadata:
name: wiki-config
data:
config.json: |
{
"farm": true,
"security_type": "friends",
"secure_cookie": false,
"allowed": "*"
}
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: wiki-deployment
spec:
replicas: 1
selector:
matchLabels:
app: wiki
template:
metadata:
labels:
app: wiki
spec:
securityContext:
runAsUser: 1000
runAsGroup: 1000
fsGroup: 1000
initContainers:
- name: wiki-config
image: dobbs/farm:restrict-new-wiki
securityContext:
runAsUser: 0
runAsGroup: 0
allowPrivilegeEscalation: false
volumeMounts:
- name: dot-wiki
mountPath: /home/node/.wiki
command: ["chown", "-R", "1000:1000", "/home/node/.wiki"]
containers:
- name: farm
image: dobbs/farm:restrict-new-wiki
command: [
"wiki", "--config", "/etc/config/config.json",
"--admin", "bad password but memorable",
"--cookieSecret", "any-random-string-will-do-the-trick"]
ports:
- containerPort: 3000
volumeMounts:
- name: dot-wiki
mountPath: /home/node/.wiki
- name: config-templates
mountPath: /etc/config
volumes:
- name: dot-wiki
persistentVolumeClaim:
claimName: dot-wiki
- name: config-templates
configMap:
name: wiki-config
---
apiVersion: v1
kind: Service
metadata:
name: wiki-service
spec:
ports:
- name: http
targetPort: 3000
port: 80
selector:
app: wiki
In addition to kubectl rollout restart deployment, there are some alternative approaches to do this:
1. Restart Pods
kubectl delete pods -l app=wiki
This causes the Pods of your Deployment to be restarted, in which case they read the updated ConfigMap.
2. Version the ConfigMap
Instead of naming your ConfigMap just wiki-config, name it wiki-config-v1. Then when you update your configuration, just create a new ConfigMap named wiki-config-v2.
Now, edit your Deployment specification to reference the wiki-config-v2 ConfigMap instead of wiki-config-v1:
apiVersion: apps/v1
kind: Deployment
# ...
volumes:
- name: config-templates
configMap:
name: wiki-config-v2
Then, reapply the Deployment:
kubectl apply -f wiki.yaml
Since the Pod template in the Deployment manifest has changed, the reapplication of the Deployment will recreate all the Pods. And the new Pods will use the new version of the ConfigMap.
As an additional advantage of this approach, if you keep the old ConfigMap (wiki-config-v1) around rather than deleting it, you can revert to a previous configuration at any time by just editing the Deployment manifest again.
This approach is described in Chapter 1 of Kubernetes Best Practices (O'Reilly, 2019).
For the specific question about restarting containers after the configuration is changed, as of kubectl v1.15 you can do this:
# apply the config changes
kubectl apply -f wiki.yaml
# restart the containers in the deployment
kubectl rollout restart deployment wiki-deployment
You should do nothing but change your ConfigMap, and wait for the changes to be applies. The answer you have posted the link is wrong. After a ConfigMap change, it doesn't apply the changes right away, but can take time. Like 5 minutes, or something like that.
If that doesn't happen, you can report a bug about that specific version of k8s.

Deployment not detecting change of container image tag in init-container

I have been using init-containers since they became available and find them super useful. My core image (below as web-dev) does not change much, but my init-container image (below as web-data-dev) does change often.
The init-container uses a container image with a version number. I change this version number to the latest value, and then do kubectl apply -f deployment.yaml
For instance, i change eu.gcr.io/project/web-data-dev:187 to eu.gcr.io/project/web-data-dev:188 before running kubectl apply.
When I do this however, no deployment happens, if i make any changes to the image the init-container uses, the deployment will still not happen. I assume this is because the init-container changes are not being detected.
I then tried to just put some garbage in the image field, like this: "image": "thisIsNotAnImage" and run kubectl apply -f again, but the update is still not applied.
My question is - How do i make kubectl apply -f detect an image tag change in an init-container? am i doing something wrong, is this a bug, or is this simply not implemented yet because init-containers are Alpha?
The full deployment YAML is below.
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: web-deployment
spec:
replicas: 1
strategy:
rollingUpdate:
maxUnavailable: 0
template:
metadata:
labels:
app: web
tier: frontend
annotations:
pod.alpha.kubernetes.io/init-containers: '[
{
"name": "initialiser1",
"image": "eu.gcr.io/project/web-data-dev:187",
"command": ["cp", "-r", "/data-in/", "/opt/"],
"volumeMounts": [
{
"name": "file-share",
"mountPath": "/opt/"
}
]
}
]'
spec:
containers:
- image: eu.gcr.io/project/web-dev:20
name: web
resources:
requests:
cpu: 10m
memory: 40Mi
ports:
- containerPort: 80
name: http
- containerPort: 443
name: https
volumeMounts:
- name: file-share
mountPath: /opt/
volumes:
- name: file-share
emptyDir: {}
If you are using Kubernetes 1.4, try to change pod.alpha.kubernetes.io/init-containers to pod.beta.kubernetes.io/init-containers.
I can't find a proper issue on GitHub, but behaviour of these two annotations is different. I can do kubectl apply -f with the second one and the deployment will be updated.
You can test it using the example below:
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: nginx
spec:
template:
metadata:
labels:
app: nginx
annotations:
pod.beta.kubernetes.io/init-containers: '[
{
"name": "install",
"image": "busybox",
"command": ["/bin/sh", "-c", "echo foo > /work-dir/index.html"],
"volumeMounts": [
{
"name": "workdir",
"mountPath": "/work-dir"
}
]
}
]'
spec:
volumes:
- name: workdir
emptyDir: {}
containers:
- name: nginx
image: nginx
ports:
- containerPort: 80
volumeMounts:
- name: workdir
mountPath: /usr/share/nginx/html
Try to change foo to bar and see the result:
$ cat nginx.yaml | kubectl apply -f -
deployment "nginx" created
$ curl $(minikube service nginx --url)
Waiting, endpoint for service is not ready yet...
foo
$ cat nginx.yaml | sed -e 's/foo/bar/g' | kubectl apply -f -
deployment "nginx" configured
$ curl $(minikube service nginx --url)
Waiting, endpoint for service is not ready yet...
bar
The same thing using pod.alpha.kubernetes.io/init-containers:
$ curl $(minikube service nginx --url)
Waiting, endpoint for service is not ready yet...
foo
$ cat nginx.yaml | sed -e 's/foo/bar/g' | kubectl apply -f -
deployment "nginx" configured
$ curl $(minikube service nginx --url)
foo