setting up build pod: Timed out while waiting for ServiceAccount/<service_account_name> to be present in the cluster - kubernetes

I am using helm charts to deploy Gitlab Runner into Kubernetes cluster. I want that the created pods when runner is triggered to have a costume services account instead of the default one. I did create role and cluster role and did the role bindings.
However, I am getting the following error when running a CI job
From Gitlab CI
Running with gitlab-runner 15.0.0 (cetx4b)
on initial-runner -P-d1RhT
Preparing the "kubernetes" executor
00:00
Using Kubernetes namespace: namespace_test
Using Kubernetes executor with image registry.gitlab.com/docker-images/ubuntu-base:latest ...
Using attach strategy to execute scripts...
Preparing environment
00:05
ERROR: Job failed (system failure): prepare environment: setting up build pod: Timed out while waiting for ServiceAccount/gitlab-runner to be present in the cluster. Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information
list roles and services accounts
# get rolebindings & clusterrolebindings
kubectl get rolebindings,clusterrolebindings -n namespace_test | grep gitlab-runner
# output
# rolebinding.rbac.authorization.k8s.io/gitlab-runner Role/gitlab-runner
# clusterrolebinding.rbac.authorization.k8s.io/gitlab-runner ClusterRole/gitlab-runner
---
# get serviceaccounts
kubectl get serviceaccounts -n namespace_test
# output
# NAME SECRETS AGE
# default 1 6h50m
# gitlab-runner 1 24m
# kubernetes-dashboard 1 6h50m
# mysql 2 6h49m
helm values
runners:
concurrent: 8
name: initial-runner
config: |
[[runners]]
[runners.kubernetes]
namespace = "namespace_test"
image = "registry.gitlab.com/docker-images/ubuntu-base:latest"
service_account = "gitlab-runner"
tags: base
rbac:
create: false
serviceAccountName: gitlab-runner
any ideas on how to solve this issue?

In my case, I forgot to give the "gitlab-runner" cluster role the right permissions on "serviceaccounts" resource.

Ensure the role that is attached to your Gitlab runner has the following specification:
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: gitlab-runner
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["list", "get", "watch", "create", "delete"]
- apiGroups: [""]
resources: ["pods/exec"]
verbs: ["create"]
- apiGroups: [""]
resources: ["pods/log"]
verbs: ["get"]
- apiGroups: [""]
resources: ["secrets"]
verbs: ["list", "get", "create", "delete", "update"]
- apiGroups: [""]
resources: ["configmaps"]
verbs: ["list", "get", "create", "delete", "update"]
- apiGroups: [""]
resources: ["pods/attach"]
verbs: ["list", "get", "create", "delete", "update"]
- apiGroups: [""]
resources: ["serviceaccounts"]
verbs: ["list", "get", "create", "delete", "update"]

Related

User "system:serviceaccount:gitlab:default" cannot get resource "deployments" in API group "apps" in the namespace "gitlab"

I am trying to run a k8s deployment job from gitlab kubernetes executor.
I deployed the kubernetes runner using helm as following.
my values.yaml includes the following rbac rules:
rbac:
create: true
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["list", "get", "watch", "create", "delete"]
- apiGroups: [""]
resources: ["pods/exec"]
verbs: ["create"]
- apiGroups: [""]
resources: ["pods/log"]
verbs: ["get"]
- apiGroups: [""]
resources: ["pods/attach"]
verbs: ["list", "get", "create", "delete", "update"]
- apiGroups: [""]
resources: ["secrets"]
verbs: ["list", "get", "create", "delete", "update"]
- apiGroups: [""]
resources: ["configmaps"]
verbs: ["list", "get", "create", "delete", "update"]
- apiGroups: [""]
resources: ["services"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: ["apps"]
resources: ["deployments"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
clusterWideAccess: true
podSecurityPolicy:
enabled: false
resourceNames:
- gitlab-runner
then
helm install --namespace gitlab gitlab-runner -f values.yaml gitlab/gitlab-runner
and, my .gitlab-ci.yml has the following stage:
script:
- mkdir -p /etc/deploy
- echo $kube_config |base64 -d > $KUBECONFIG
- sed -i "s/IMAGE_TAG/$CI_PIPELINE_ID/g" deployment.yaml
- cat deployment.yaml
- kubectl apply -f deployment.yaml
and, I got the following error in pipeline logs:
$ kubectl apply -f deployment.yaml
Error from server (Forbidden): error when retrieving current configuration of:
Resource: "apps/v1, Resource=deployments", GroupVersionKind: "apps/v1, Kind=Deployment"
Name: "java-demo", Namespace: "gitlab"
Object: &{map["apiVersion":"apps/v1" "kind":"Deployment" "metadata":map["annotations":map["kubectl.kubernetes.io/last-applied-configuration":""] "name":"java-demo" "namespace":"gitlab"] "spec":map["replicas":'\x01' "selector":map["matchLabels":map["app":"java-demo"]] "template":map["metadata":map["labels":map["app":"java-demo"]] "spec":map["containers":[map["image":"square2019/dummy-repo:555060965" "imagePullPolicy":"Always" "name":"java-demo" "ports":[map["containerPort":'\u1f90']]]]]]]]}
from server for: "deployment.yaml": deployments.apps "java-demo" is forbidden: User "system:serviceaccount:gitlab:default" cannot get resource "deployments" in API group "apps" in the namespace "gitlab"
Error from server (Forbidden): error when retrieving current configuration of:
Resource: "/v1, Resource=services", GroupVersionKind: "/v1, Kind=Service"
Name: "java-demo", Namespace: "gitlab"
Object: &{map["apiVersion":"v1" "kind":"Service" "metadata":map["annotations":map["kubectl.kubernetes.io/last-applied-configuration":""] "name":"java-demo" "namespace":"gitlab"] "spec":map["ports":[map["name":"java-demo" "port":'P' "targetPort":'\u1f90']] "selector":map["app":"java-demo"] "type":"ClusterIP"]]}
from server for: "deployment.yaml": services "java-demo" is forbidden: User "system:serviceaccount:gitlab:default" cannot get resource "services" in API group "" in the namespace "gitlab"
Cleaning up project directory and file based variables
00:00
ERROR: Job failed: command terminated with exit code 1
Am I missing some RBAC rules here?
thank you!
=== update 2022.06.04 =====
kubectl get role -n gitlab -o yaml
apiVersion: v1
items: []
kind: List
metadata:
resourceVersion: ""
=== update 2022.06.05 ===
Looking at the logic in https://gitlab.com/gitlab-org/charts/gitlab-runner/-/blob/main/templates/role.yaml, I modified values.yaml with
clusterWideAccess: false
and, now i get the role as:
k get role -n gitlab -o yaml
apiVersion: v1
items:
- apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
annotations:
meta.helm.sh/release-name: gitlab-runner
meta.helm.sh/release-namespace: gitlab
creationTimestamp: "2022-06-05T03:49:57Z"
labels:
app: gitlab-runner
app.kubernetes.io/managed-by: Helm
chart: gitlab-runner-0.41.0
heritage: Helm
release: gitlab-runner
name: gitlab-runner
namespace: gitlab
resourceVersion: "283754"
uid: 8040b295-c9fc-47cb-8c5c-74cbf6c4d8a7
rules:
- apiGroups:
- ""
resources:
- pods
verbs:
- list
- get
- watch
- create
- delete
- apiGroups:
- ""
resources:
- pods/exec
verbs:
- create
- apiGroups:
- ""
resources:
- pods/log
verbs:
- get
- apiGroups:
- ""
resources:
- pods/attach
verbs:
- list
- get
- create
- delete
- update
- apiGroups:
- ""
resources:
- secrets
verbs:
- list
- get
- create
- delete
- update
- apiGroups:
- ""
resources:
- configmaps
verbs:
- list
- get
- create
- delete
- update
- apiGroups:
- ""
resources:
- services
verbs:
- get
- list
- watch
- create
- update
- patch
- delete
- apiGroups:
- apps
resources:
- deployments
verbs:
- get
- list
- watch
- create
- update
- patch
- delete
kind: List
metadata:
resourceVersion: ""
service account and RoleBinding
k get sa -n gitlab
NAME SECRETS AGE
default 1 3d2h
gitlab-runner 1 2d2h
k get RoleBinding -n gitlab
NAME ROLE AGE
gitlab-runner Role/gitlab-runner 9h
however, the same error persists.
=== update 2022.06.06 ===
I applied the following to fix the issue for the moment
kubectl create rolebinding --namespace=gitlab gitlab-runner-4 --role=gitlab-runner --serviceaccount=gitlab:default

What kubernetes permissions does GitLab runner kubernetes executor need?

I've installed GitLab runner on a kubernetes cluster under a namespace gitlab-runner. Like so
# cat <<EOF | kubectl create -f -
{
"apiVersion": "v1",
"kind": "Namespace",
"metadata": {
"name": "gitlab-runner",
"labels": {
"name": "gitlab-runner"
}
}
}
# helm repo add gitlab https://charts.gitlab.io
# cat <<EOF|helm install --namespace gitlab-runner gitlab-runner -f - gitlab/gitlab-runner
gitlabUrl: https://gitlab.mycompany.com
runnerRegistrationToken: "c................Z"
The GitLab runner properly registers with the GitLab project but all jobs fail.
A quick look into the GitLab runner logs tells me that the service account used by the GitLab runner lack the proper permissions:
# kubectl logs --namespace gitlabrunner gitlab-runner-gitlab-runner-xxxxxxxxx
ERROR: Job failed (system failure): pods is forbidden: User "system:serviceaccount:gitlabrunner:default" cannot create resource "pods" in API group "" in the namespace "gitlab-runner" duration=42.095493ms job=37482 project=yyy runner=xxxxxxx
What permission does the gitlab runner kubernetes executor need?
I couldn't find in the GitLab runner documentation a list of permissions but I try adding permissions one by one and I compiled a list of the permission required for basic functioning.
The gitlab runner will use the service account system:serviceaccount:gitlab-runner:default so we need to create a role and assign that role to that service account.
# cat <<EOF | kubectl create -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: gitlab-runner
namespace: gitlab-runner
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["list", "get", "watch", "create", "delete"]
- apiGroups: [""]
resources: ["pods/exec"]
verbs: ["create"]
- apiGroups: [""]
resources: ["pods/log"]
verbs: ["get"]
# kubectl create rolebinding --namespace=gitlab-runner gitlab-runner-binding --role=gitlab-runne r --serviceaccount=gitlab-runner:default
With that role assigned to the service account, GitLab runner will be able to create, execute and delete the pod and also access the logs.
Unfortunately I couldn't find this in the official docs either just like #RubenLaguna stated. However, the default values.yaml of the kubernetes gitlab runner helm chart lets you define these RBAC rules nicely and does list some examples which I started with.
In my case I had to add a few and went with the following:
rbac:
create: true
rules:
- apiGroups: [""]
resources: ["pods", "secrets", "configmaps"]
verbs: ["get", "list", "watch", "create", "patch", "delete", "update"]
- apiGroups: [""]
resources: ["pods/exec", "pods/attach"]
verbs: ["create", "patch", "delete"]
- apiGroups: [""]
resources: ["pods/log"]
verbs: ["get"]
If you installed gitlab runners using helm chart then gitlab runners does not use the service account created by the helm chart instead it uses the default service account. There is a bug related to this. https://gitlab.com/gitlab-org/charts/gitlab-runner/-/issues/353 To work around this problem we created a ClusterRoleBinding as below.
kind: ClusterRoleBinding
metadata:
name: gitlab-runner-role-default
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: <helm-created-cluster-role-here>
subjects:
- kind: ServiceAccount
name: default
namespace: gitlab-runner

CephFS failing to mount on Kubernetes

I set up a Ceph cluster and mounted manually using the sudo mount -t command following the official documentation, and I checked the status of my Ceph cluster - no problems there. Now I am trying to mount my CephFS on Kubernetes but my pod is stuck in ContainerCreating when I run the kubectl create command because it is failing to mount. I looked at many related problems/solutions online but nothing works.
As reference, I am following this guide: https://medium.com/velotio-perspectives/an-innovators-guide-to-kubernetes-storage-using-ceph-a4b919f4e469
My setup consists of 5 AWS instances, and they are as follows:
Node 1: Ceph Mon
Node 2: OSD1 + MDS
Node 3: OSD2 + K8s Master
Node 4: OSD3 + K8s Worker1
Node 5: CephFS + K8s Worker2
Is it okay to stack K8s on top of the same instance as Ceph? I am pretty sure that is allowed, but if that is not allowed, please let me know.
In the describe pod logs, this is the error/warning:
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /root/userone/kubelet/pods/bbf28924-3639-11ea-879d-0a6b51accf30/volumes/kubernetes.io~cephfs/pvc-4777686c-3639-11ea-879d-0a6b51accf30 --scope -- mount -t ceph -o name=kubernetes-dynamic-user-4d05a2df-3639-11ea-b2d3-5a4147fda646,secret=AQC4whxeqQ9ZERADD2nUgxxOktLE1OIGXThBmw== 172.31.15.110:6789:/pvc-volumes/kubernetes/kubernetes-dynamic-pvc-4d05a269-3639-11ea-b2d3-5a4147fda646 /root/userone/kubelet/pods/bbf28924-3639-11ea-879d-0a6b51accf30/volumes/kubernetes.io~cephfs/pvc-4777686c-3639-11ea-879d-0a6b51accf30
Output: Running scope as unit run-2382233.scope.
couldn't finalize options: -34
These are my .yaml files:
Provisioner:
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: test-provisioner-dt
namespace: test-dt
rules:
- apiGroups: [""]
resources: ["persistentvolumes"]
verbs: ["get", "list", "watch", "create", "delete"]
- apiGroups: [""]
resources: ["persistentvolumeclaims"]
verbs: ["get", "list", "watch", "update", "create"]
- apiGroups: ["storage.k8s.io"]
resources: ["storageclasses"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["events"]
verbs: ["create", "update", "patch"]
- apiGroups: [""]
resources: ["services"]
resourceNames: ["kube-dns","coredns"]
verbs: ["list", "get"]
- apiGroups: [""]
resources: ["secrets"]
verbs: ["create", "get", "delete"]
- apiGroups: [""]
resources: ["endpoints"]
verbs: ["get", "list", "watch", "create", "update", "patch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: test-provisioner-dt
namespace: test-dt
subjects:
- kind: ServiceAccount
name: test-provisioner-dt
namespace: test-dt
roleRef:
kind: ClusterRole
name: test-provisioner-dt
apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: test-provisioner-dt
namespace: test-dt
rules:
- apiGroups: [""]
resources: ["secrets"]
verbs: ["create", "get", "delete"]
- apiGroups: [""]
resources: ["endpoints"]
verbs: ["get", "list", "watch", "create", "update", "patch"]
---
StorageClass:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: postgres-pv
namespace: test-dt
provisioner: ceph.com/cephfs
parameters:
monitors: 172.31.15.110:6789
adminId: admin
adminSecretName: ceph-secret-admin-dt
adminSecretNamespace: test-dt
claimRoot: /pvc-volumes
PVC:
apiVersion: v1
metadata:
name: postgres-pvc
namespace: test-dt
spec:
storageClassName: postgres-pv
accessModes:
- ReadWriteMany
resources:
requests:
storage: 2Gi
Output of kubectl get pv and kubectl get pvc show the volumes are bound and claimed, no errors.
Output of the provisioner pod logs all show success/no errors.
Please help!

Kubernetes Dashboard v1.8.3 deployment

I am just trying to deploy kubernetes Dashboard in a namespace called "test".
https://raw.githubusercontent.com/kubernetes/dashboard/v1.8.3/src/deploy/recommended/kubernetes-dashboard.yaml
I just replaced namespace from kube-system to test from the above yaml file and executed as below.
kubectl apply -f kubernetes-dashboard.yaml -n test
But, it is still trying to do something with namespace kube-system and getting the below error.
Image:
gcr.io/google_containers/kubernetes-dashboard-amd64:v1.8.3
Error:-
2018/05/31 16:56:55 Starting overwatch
2018/05/31 16:56:55 Using in-cluster config to connect to apiserver
2018/05/31 16:56:55 Using service account token for csrf signing
2018/05/31 16:56:55 No request provided. Skipping authorization
2018/05/31 16:56:55 Successful initial request to the apiserver, version: v1.10.2
2018/05/31 16:56:55 Generating JWE encryption key
2018/05/31 16:56:55 New synchronizer has been registered: kubernetes-dashboard-key-holder-kube-system. Starting
2018/05/31 16:56:55 Starting secret synchronizer for kubernetes-dashboard-key-holder in namespace kube-system
2018/05/31 16:56:55 Synchronizer kubernetes-dashboard-key-holder-kube-system exited with error: unexpected object: &Secret{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:,GenerateName:,Namespace:,SelfLink:,UID:,ResourceVersion:,Generation:0,CreationTimestamp:0001-01-01 00:00:00 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,},Data:map[string][]byte{},Type:,StringData:map[string]string{},}
2018/05/31 16:56:57 Restarting synchronizer: kubernetes-dashboard-key-holder-kube-system.
2018/05/31 16:56:57 Starting secret synchronizer for kubernetes-dashboard-key-holder in namespace kube-system
2018/05/31 16:56:57 Synchronizer kubernetes-dashboard-key-holder-kube-system exited with error: kubernetes-dashboard-key-holder-kube-system watch ended with timeout
2018/05/31 16:56:59 Storing encryption key in a secret
panic: secrets is forbidden: User "system:serviceaccount:test:dashboard" cannot create secrets in the namespace "kube-system"
goroutine 1 [running]:
github.com/kubernetes/dashboard/src/app/backend/auth/jwe.(*rsaKeyHolder).init(0xc420254e00)
/home/travis/build/kubernetes/dashboard/.tmp/backend/src/github.com/kubernetes/dashboard/src/app/backend/auth/jwe/keyholder.go:131 +0x2d3
github.com/kubernetes/dashboard/src/app/backend/auth/jwe.NewRSAKeyHolder(0x1a7ee00, 0xc42037a5a0, 0xc42037a5a0, 0x127b962)
/home/travis/build/kubernetes/dashboard/.tmp/backend/src/github.com/kubernetes/dashboard/src/app/backend/auth/jwe/keyholder.go:170 +0x83
main.initAuthManager(0x1a7e300, 0xc4201e2240, 0xc42066dc68, 0x1)
/home/travis/build/kubernetes/dashboard/.tmp/backend/src/github.com/kubernetes/dashboard/src/app/backend/dashboard.go:183 +0x12f
main.main()
/home/travis/build/kubernetes/dashboard/.tmp/backend/src/github.com/kubernetes/dashboard/src/app/backend/dashboard.go:101 +0x28c
I created Secret, Rolebinding, Serviceaccount, deployment, Service & Ingress in the namesapce "test". Removed namespace from the yaml file and supplied thru -n "test" while creating.
That happened because you created the ServiceAccount on a different namespace, namely test but as it says, it needs to be deployed in kube-system in order to be able to function.
You can find a nice walkthrough and possibly some clarifications here
However, if you still want to deploy on a different namespace, you would have to add the following role and rolebinding to your cluster:
# ------------------- Dashboard Role & Role Binding ------------------- #
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: kubernetes-dashboard-minimal
namespace: kube-system
rules:
# Allow Dashboard to create 'kubernetes-dashboard-key-holder' secret.
- apiGroups: [""]
resources: ["secrets"]
verbs: ["create"]
# Allow Dashboard to create 'kubernetes-dashboard-settings' config map.
- apiGroups: [""]
resources: ["configmaps"]
verbs: ["create"]
# Allow Dashboard to get, update and delete Dashboard exclusive secrets.
- apiGroups: [""]
resources: ["secrets"]
resourceNames: ["kubernetes-dashboard-key-holder"]
verbs: ["get", "update", "delete"]
# Allow Dashboard to get, update and delete Dashboard exclusive secrets.
- apiGroups: [""]
resources: ["secrets"]
resourceNames: ["kubernetes-dashboard-key-holder"]
verbs: ["get", "update", "delete"]
# Allow Dashboard to get and update 'kubernetes-dashboard-settings' config map.
- apiGroups: [""]
resources: ["configmaps"]
resourceNames: ["kubernetes-dashboard-settings"]
verbs: ["get", "update"]
# Allow Dashboard to get metrics from heapster.
- apiGroups: [""]
resources: ["services"]
resourceNames: ["heapster"]
verbs: ["proxy"]
- apiGroups: [""]
resources: ["services/proxy"]
resourceNames: ["heapster", "http:heapster:", "https:heapster:"]
verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: kubernetes-dashboard-minimal
namespace: kube-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: kubernetes-dashboard-minimal
subjects:
- kind: ServiceAccount
name: kubernetes-dashboard
namespace: test
---
I am afraid there is no other way around, you have to allow the service account to create secrets in kube-system namespace.

kubernetes RBAC role verbs to exec to pod

I my 1.9 cluster created this deployment role for the dev user. Deployment works as expected. Now I want to give exec and logs access to developer. What role I need to add for exec to the pod?
kind: Role
name: deployment-manager
rules:
- apiGroups: ["", "extensions", "apps"]
resources: ["deployments", "replicasets", "pods"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
Error message:
kubectl exec nginx -it -- sh
Error from server (Forbidden): pods "nginx" is forbidden: User "dev" cannot create pods/exec in the namespace "dev"
Thanks
SR
The RBAC docs say that
Most resources are represented by a string representation of their name, such as “pods”, just as it appears in the URL for the relevant API endpoint. However, some Kubernetes APIs involve a “subresource”, such as the logs for a pod. [...] To represent this in an RBAC role, use a slash to delimit the resource and subresource.
To allow a subject to read both pods and pod logs, and be able to exec into the pod, you would write:
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
namespace: default
name: pod-and-pod-logs-reader
rules:
- apiGroups: [""]
resources: ["pods", "pods/log"]
verbs: ["get", "list"]
- apiGroups: [""]
resources: ["pods/exec"]
verbs: ["create"]
Some client libraries may do an http GET to negotiate a websocket first, which would require the "get" verb. kubectl sends an http POST instead, that's why it requires the "create" verb in that case.