Whitelisting sysctls for containers in Kubernetes Kind - kubernetes

I'm trying to deploy a container in a Kubernetes Kind cluster. The container I'm trying to deploy needs a couple of sysctls flags to be set.
The deployment fails with
forbidden sysctl: "kernel.msgmnb" not whitelisted
UPDATE
I have since added a cluster policy as suggested, created a role that grants usage to it and assigned the Cluster Role to the default service account:
---
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
name: sysctl-psp
spec:
privileged: false # Don't allow privileged pods!
# The rest fills in some required fields.
seLinux:
rule: RunAsAny
supplementalGroups:
rule: RunAsAny
runAsUser:
rule: RunAsAny
fsGroup:
rule: RunAsAny
volumes:
- '*'
allowedUnsafeSysctls:
- kernel.msg*
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: role_allow_sysctl
rules:
- apiGroups: ['policy']
resources: ['podsecuritypolicies']
verbs: ['*']
resourceNames:
- sysctl-psp
- apiGroups: ['']
resources:
- replicasets
- services
- pods
verbs: ['*']
- apiGroups: ['apps']
resources:
- deployments
verbs: ['*']
The cluster role binding is like this:
kubectl -n <namespace> create rolebinding default:role_allow_sysctl --clusterrole=role_allow_sysctl --serviceaccount=<namespace>:default
I am then trying to create a deployment and a service in the same namespace:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: test-app
labels:
app: test-app
spec:
selector:
matchLabels:
app: test-app
tier: dev
strategy:
type: Recreate
template:
metadata:
labels:
app: test-app
tier: dev
spec:
securityContext:
sysctls:
- name: kernel.msgmnb
value: "6553600"
- name: kernel.msgmax
value: "1048800"
- name: kernel.msgmni
value: "32768"
- name: kernel.sem
value: "128 32768 128 4096"
containers:
- image: registry:5000/<container>:1.0.0
name: test-app
imagePullPolicy: IfNotPresent
ports:
- containerPort: 10666
name:port-1
---
The problem remains the same however, I'm getting multiple pods spawned, all failing with the same message forbidden sysctl: "kernel.msgmnb" not whitelisted

I don't think that --alowed-unsafe-sysctls flag could work with Kind nodes, because Kind nodes themselves are containers, whose sysctl FS is read-only.
My workaround is to change the needed sysctl values on my host machine. Kind nodes (and in turn their containers) will reuse these values.

Related

Pod assigned node role instead of service account role on AWS EKS

Before I get started I have seen questions this and this, and they did not help.
I have a k8s cluster on AWS EKS on which I am deploying a custom k8s controller for my application. Using instructions from eksworkshop.com, I created my service account with the appropriate IAM role using eksctl. I assign the role in my deployment.yaml as seen below. I also set the securityContext as that seemed to solve problem in another case as described here.
apiVersion: apps/v1
kind: Deployment
metadata:
name: tel-controller
namespace: tel
spec:
replicas: 2
selector:
matchLabels:
app: tel-controller
strategy:
rollingUpdate:
maxSurge: 50%
maxUnavailable: 50%
type: RollingUpdate
template:
metadata:
labels:
app: tel-controller
spec:
serviceAccountName: tel-controller-serviceaccount
securityContext:
fsGroup: 65534
containers:
- image: <image name>
imagePullPolicy: Always
name: tel-controller
args:
- --metrics-bind-address=:8080
- --health-probe-bind-address=:8081
- --leader-elect=true
ports:
- name: webhook-server
containerPort: 9443
protocol: TCP
- name: metrics-port
containerPort: 8080
protocol: TCP
- name: health-port
containerPort: 8081
protocol: TCP
securityContext:
readOnlyRootFilesystem: true
runAsNonRoot: true
allowPrivilegeEscalation: false
But this does not seem to be working. If I describe the pod, I see the correct role.
AWS_DEFAULT_REGION: us-east-1
AWS_REGION: us-east-1
AWS_ROLE_ARN: arn:aws:iam::xxxxxxxxx:role/eksctl-eks-tel-addon-iamserviceaccount-tel-t-Role1-3APV5KCV33U8
AWS_WEB_IDENTITY_TOKEN_FILE: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
Mounts:
/var/run/secrets/eks.amazonaws.com/serviceaccount from aws-iam-token (ro)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-6ngsr (ro)
But if I do a sts.GetCallerIdentityInput() from inside the controller application, I see the node role. And obviously i get an access denied error.
caller identity: (go string) {
Account: "xxxxxxxxxxxx",
Arn: "arn:aws:sts::xxxxxxxxxxx:assumed-role/eksctl-eks-tel-nodegroup-voice-NodeInstanceRole-BJNYF5YC2CE3/i-0694a2766c5d70901",
UserId: "AROAZUYK7F2GRLKRGGNXZ:i-0694a2766c5d70901"
}
This is how I created by service account
eksctl create iamserviceaccount --cluster ${EKS_CLUSTER_NAME} \
--namespace tel \
--name tel-controller-serviceaccount \
--attach-policy-arn arn:aws:iam::xxxxxxxxxx:policy/telcontrollerRoute53Policy \
--override-existing-serviceaccounts --approve
I have done this successfully in the past. The difference this time is that I also have role & role bindings attached to this service account. My rbac.yaml for this SA.
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: tel-controller-role
labels:
app: tel-controller
rules:
- apiGroups: [""]
resources: [events]
verbs: [create, delete, get, list, update, watch]
- apiGroups: ["networking.k8s.io"]
resources: [ingressclasses]
verbs: [get, list]
- apiGroups: ["", "networking.k8s.io"]
resources: [services, ingresses]
verbs: [create, get, list, patch, update, delete, watch]
- apiGroups: [""]
resources: [configmaps]
verbs: [create, delete, get, update]
- apiGroups: ["coordination.k8s.io"]
resources: ["leases"]
verbs: [get, create, update]
- apiGroups: [""]
resources: [pods]
verbs: [get, list, watch, update]
- apiGroups: ["", "networking.k8s.io"]
resources: [services/status, ingresses/status]
verbs: [update, patch]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: tel-controller-rolebinding
labels:
app: tel-controller
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: tel-controller-role
subjects:
- kind: ServiceAccount
name: tel-controller-serviceaccount
namespace: tel
What am I doing wrong here? Thanks.
PS: I am deploying using kubectl
PPS: from go.mod I am using github.com/aws/aws-sdk-go v1.44.28
So the Go SDK has 2 methods to create a session. Apparently I was using the deprecated method to create my session (session.New). Using the recommended method to create the new session, session.NewSession solved my problem.

Unable to create ALBIngressController in Kubernetes environment

I’m creating an Application Load Balancer (AWSALBIngressController-v1.1.6) in the Kubernetes environment. While creating that, for some reason I’m getting the following error -
E1111 06:02:13.117566 1 controller.go:217] kubebuilder/controller "msg"="Reconciler error" "error"="failed to reconcile LB managed SecurityGroup: failed to reconcile managed LoadBalancer securityGroup: NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors" "controller"="alb-ingress-controller" "request"={"Namespace”:”sampleNamespace”,”Name":"alb-ingress"}
Following are the ALB config files for reference-
ALB Controller Deployment file-
---
# Application Load Balancer (ALB) Ingress Controller Deployment Manifest.
# This manifest details sensible defaults for deploying an ALB Ingress Controller.
# GitHub: https://github.com/kubernetes-sigs/aws-alb-ingress-controller
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app.kubernetes.io/name: alb-ingress-controller
name: alb-ingress-controller
# Namespace the ALB Ingress Controller should run in. Does not impact which
# namespaces it's able to resolve ingress resource for. For limiting ingress
# namespace scope, see --watch-namespace.
namespace: kube-system
spec:
selector:
matchLabels:
app.kubernetes.io/name: alb-ingress-controller
template:
metadata:
labels:
app.kubernetes.io/name: alb-ingress-controller
spec:
containers:
- name: alb-ingress-controller
args:
- --ingress-class=alb
- --watch-namespace=sampleNamespace
- --cluster-name=ckuster-xl
- --aws-vpc-id=vpc-3d53e783
- --aws-region=us-east-1
- --default-tags=Name=tag1-xl-ALB,mgr=mgrname
# newer version (v1.1.7) of the alb-ingress-controller image requires iam permission to wafv2
# even when no wafv2 annotation is used
image: docker.io/amazon/aws-alb-ingress-controller:v1.1.6
resources:
requests:
cpu: 100m
memory: 90Mi
limits:
cpu: 200m
memory: 200Mi
serviceAccountName: alb-ingress-controller
RBAC yaml file-
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
app.kubernetes.io/name: alb-ingress-controller
name: alb-ingress-controller
rules:
- apiGroups:
- ""
- extensions
resources:
- configmaps
- endpoints
- events
- ingresses
- ingresses/status
- services
verbs:
- create
- get
- list
- update
- watch
- patch
- apiGroups:
- ""
- extensions
resources:
- nodes
- pods
- secrets
- services
- namespaces
verbs:
- get
- list
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
labels:
app.kubernetes.io/name: alb-ingress-controller
name: alb-ingress-controller
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: alb-ingress-controller
subjects:
- kind: ServiceAccount
name: alb-ingress-controller
namespace: kube-system
I tried few solutions like adding args - --aws-api-debug, - --aws-region args in ALB deployment file and --auto-discover-base-arn , --auto-discover-default-role in kiam server yaml file but it didn't work.

how to deploy a statefulset when a pod security policy is in place in kubernetes

I am trying to play around with PodSecurityPolicies in kubernetes so pods can't be created if they are using the root user.
This is my psp definition:
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
name: eks.restrictive
spec:
hostNetwork: false
seLinux:
rule: RunAsAny
supplementalGroups:
rule: RunAsAny
runAsUser:
rule: MustRunAsNonRoot
fsGroup:
rule: RunAsAny
volumes:
- '*'
and this is my statefulset definition
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: web
spec:
selector:
matchLabels:
app: nginx # has to match .spec.template.metadata.labels
serviceName: "nginx"
replicas: 3 # by default is 1
template:
metadata:
labels:
app: nginx # has to match .spec.selector.matchLabels
spec:
securityContext:
#only takes integers.
runAsUser: 1000
terminationGracePeriodSeconds: 10
containers:
- name: nginx
image: k8s.gcr.io/nginx-slim:0.8
ports:
- containerPort: 80
name: web
volumeMounts:
- name: www
mountPath: /usr/share/nginx/html
volumeClaimTemplates:
- metadata:
name: www
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: "my-storage-class"
resources:
requests:
storage: 1Gi
When trying to create this statefulset I get
create Pod web-0 in StatefulSet web failed error: pods "web-0" is forbidden: unable to validate against any pod security policy:
It doesn't specify what policy am I violating, and since I am specifying I want to run this on user 1000, I am not running this as root (Hence my understanding is that this statefulset pod definition is not violating any rules defined in the PSP). There is no USER specified in the Dockerfile used for this image.
The other weird part, is that this works fine for standard pods (kind: Pod, instead of kind:Statefulset), for example, this works just fine, when the same PSP exists:
apiVersion: v1
kind: Pod
metadata:
name: my-nodejs
spec:
securityContext:
runAsUser: 1000
containers:
- name: my-node
image: node
ports:
- name: web
containerPort: 80
protocol: TCP
command:
- /bin/sh
- -c
- |
npm install http-server-g
npx http-server
What am I missing / doing wrong?
You seems to have forgotten about binding this psp to a service account.
You need to apply the following:
cat << EOF | kubectl apply -f-
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: psp-role
rules:
- apiGroups: ['policy']
resources: ['podsecuritypolicies']
verbs: ['use']
resourceNames:
- eks.restrictive
EOF
cat << EOF | kubectl apply -f-
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: psp-role-binding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: psp-role
subjects:
- kind: ServiceAccount
name: default
namespace: default
EOF
If you dont want to use the default account you can create a separate service account and bind the role to it.
Read more about it k8s documentation - pod-security-policy.

Kubernetes Dashboard not connecting in the network

I've tried many options to access kubernetes dashboard from a different machine, which is within the network of my master server.
From https://kubernetes.io/docs/tasks/access-application-cluster/web-ui-dashboard/
I installed k8s dashboard using:
kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-beta8/aio/deploy/recommended.yaml
And opened dashboard by running kube proxy then changed the service yaml file from ClusterIP to NodePort and added a port: 32414. But, it is not working both inside and outside master.
So, deleted
kubectl delete -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-beta8/aio/deploy/recommended.yaml
Then, created dashboard using below yaml file.
# Copyright 2017 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# ------------------- Dashboard Secret ------------------- #
apiVersion: v1
kind: Secret
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard-certs
namespace: kube-system
type: Opaque
---
# ------------------- Dashboard Service Account ------------------- #
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kube-system
---
# ------------------- Dashboard Role & Role Binding ------------------- #
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: kubernetes-dashboard-minimal
namespace: kube-system
rules:
# Allow Dashboard to create 'kubernetes-dashboard-key-holder' secret.
- apiGroups: [""]
resources: ["secrets"]
verbs: ["create"]
# Allow Dashboard to create 'kubernetes-dashboard-settings' config map.
- apiGroups: [""]
resources: ["configmaps"]
verbs: ["create"]
# Allow Dashboard to get, update and delete Dashboard exclusive secrets.
- apiGroups: [""]
resources: ["secrets"]
resourceNames: ["kubernetes-dashboard-key-holder", "kubernetes-dashboard-certs"]
verbs: ["get", "update", "delete"]
# Allow Dashboard to get and update 'kubernetes-dashboard-settings' config map.
- apiGroups: [""]
resources: ["configmaps"]
resourceNames: ["kubernetes-dashboard-settings"]
verbs: ["get", "update"]
# Allow Dashboard to get metrics from heapster.
- apiGroups: [""]
resources: ["services"]
resourceNames: ["heapster"]
verbs: ["proxy"]
- apiGroups: [""]
resources: ["services/proxy"]
resourceNames: ["heapster", "http:heapster:", "https:heapster:"]
verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: kubernetes-dashboard-minimal
namespace: kube-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: kubernetes-dashboard-minimal
subjects:
- kind: ServiceAccount
name: kubernetes-dashboard
namespace: kube-system
---
# ------------------- Dashboard Deployment ------------------- #
kind: Deployment
apiVersion: apps/v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kube-system
spec:
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
k8s-app: kubernetes-dashboard
template:
metadata:
labels:
k8s-app: kubernetes-dashboard
spec:
containers:
- name: kubernetes-dashboard
image: k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.0
ports:
- containerPort: 8443
protocol: TCP
args:
- --auto-generate-certificates
# Uncomment the following line to manually specify Kubernetes API server Host
# If not specified, Dashboard will attempt to auto discover the API server and connect
# to it. Uncomment only if the default does not work.
# - --apiserver-host=http://my-address:port
volumeMounts:
- name: kubernetes-dashboard-certs
mountPath: /certs
# Create on-disk volume to store exec logs
- mountPath: /tmp
name: tmp-volume
livenessProbe:
httpGet:
scheme: HTTPS
path: /
port: 8443
initialDelaySeconds: 30
timeoutSeconds: 30
volumes:
- name: kubernetes-dashboard-certs
secret:
secretName: kubernetes-dashboard-certs
- name: tmp-volume
emptyDir: {}
serviceAccountName: kubernetes-dashboard
# Comment the following tolerations if Dashboard must not be deployed on master
tolerations:
- key: node-role.kubernetes.io/master
effect: NoSchedule
---
# ------------------- Dashboard Service ------------------- #
kind: Service
apiVersion: v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kube-system
spec:
type: NodePort
ports:
- port: 443
targetPort: 8443
nodePort: 32323
selector:
k8s-app: kubernetes-dashboard
Then , added sa_cluster_admin.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: dashboard-admin
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: cluster-admin-rolebinding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: dashboard-admin
namespace: kube-system
Applied both using kubectl apply -f <yaml_files>
Then, I can access the dashboard with the IP address and without running kubectl proxy.
But it is working only on the master.
When I tried to connect from another machine, it is not connecting.
https://masternode_ip:NodePort_added

How to create an additional NGNIX ingress controller if there is an existing controller

I have an existing Nginx controller in my EKS cluster. This is a cluster-wide ingress controller. I want to create another Nginx controller to do some testing. This is also going to be a public-facing ingress controller. Is it possible to do that? I tried creating it by creating a new namespace and then creating new resources under that namespace but it started logging the logs for all the ingresses that were already present. Any idea on how to do that?
you need to specify the annotationkubernetes.io/ingress.class: "$INGRESS_CONTROLLER"
for example here you are saying nginx will be responsible for this ingress
kind: Ingress
metadata:
name: foo
annotations:
kubernetes.io/ingress.class: "nginx"
if you do not define a class, your cloud provider may use a default ingress controller.
using-multiple-ingress-controllers
alb.ingress.kubernetes.io/scheme annotation is used for deciding internal or public.
List of ingress Annotation
You can create additional ingress by deploying nginx ingress image as deployment or daemonset. Below are the manifests for this example. Once you done with this, then you should be able to access the this ingress using nodeIP and nodeport. Then onwards having cloud Loadbalancer that is reachable internal and resolving into node IP's should get you working end to end.
Or you may be better off creating service of type loadbalancer in below example
==> config-map.yml <==
apiVersion: v1
kind: ConfigMap
metadata:
name: nginx-configuration
==> deployment.yml <==
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: nginx-ingress-controller
spec:
replicas: 1
selector:
matchLabels:
app: nginx-ingress
template:
metadata:
labels:
app: nginx-ingress
spec:
containers:
- name: nginx-ingress-controller
image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.21.0
args:
- /nginx-ingress-controller
- --configmap=${POD_NAMESPACE}/nginx-configuration
env:
- name: POD_NAM
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
ports:
- name: http
containerPort: 80
- name: https
containerPort: 443
==> service-account.yml <==
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: nginx-ingress
rules:
- apiGroups:
- ""
resources:
- services
- endpoints
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- secrets
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- configmaps
verbs:
- get
- list
- watch
- update
- create
- apiGroups:
- ""
resources:
- pods
verbs:
- list
- watch
- apiGroups:
- ""
resources:
- events
verbs:
- create
- patch
- apiGroups:
- extensions
resources:
- ingresses
verbs:
- list
- watch
- get
- apiGroups:
- "extensions"
resources:
- ingresses/status
verbs:
- update
- apiGroups:
- k8s.nginx.org
resources:
- virtualservers
- virtualserverroutes
verbs:
- list
- watch
- get
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: nginx-ingress
subjects:
- kind: ServiceAccount
name: nginx-ingress
namespace: nginx-ingress
roleRef:
kind: ClusterRole
name: nginx-ingress
apiGroup: rbac.authorization.k8s.io
==> service.yml <==
apiVersion: v1
kind: Service
metadata:
name: nginx-ingress
spec:
selector:
app: nginx-ingress
type: NodePort
ports:
- port: 80
targetPort: 80
protocol: TCP
- port: 443
targetPort: 443
protocol: TCP
name: https