According to https://docs.gitlab.com/runner/executors/kubernetes.html#connecting-to-the-kubernetes-api, I can connect to another K8s cluster from my current Gitlab runner, but the thing is that they do not provide any information of what do you do or where do you need to specify these options.
EDIT: Finally managed to specify these options in the config.toml, but now any time my Gitlab-runner gets a job it fails as it follows:
ERROR: Job failed (system failure): User "system:serviceaccount:test-djg:default" cannot create secrets in project "gitlab" job=17304 project=3128 runner=c36ccf98
Any idea?
You get this error because executor pod doesn't have necessary permissions to run when RBAC is enabled in cluster. You should create a Role and RoleBinding and maybe a ServiceAccount based on your setup.
There is a pending merge request in gitlab-runner project that documents necessary permission and some other info about RBAC.
But for now if you want a quick setup it's possible to allow full access to namespace with following snippet. Note that you're gonna need to modifications to match you setup.
kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
namespace: gitlab
name: gitlab-admin
rules:
- apiGroups: [""] # The API group "" indicates the core API Group.
resources: ["*"]
verbs: ["*"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: gitlab-admin
namespace: gitlab
subjects:
- kind: ServiceAccount # May be "User", "Group" or "ServiceAccount"
name: default
namespace: gitlab
roleRef:
kind: Role
name: gitlab-admin
apiGroup: rbac.authorization.k8s.io
Related
I have a Kubernetes service running in namespace NA that is configured to run as a service account A. The service schedules a Kubernetes job in namespace NB. How do I make a job in NB act on behalf of service account A? I tried to specify the name of the service account for the job, but I get the following error:
Error creating: pods "pod_id_x is forbidden: error looking up service account NB/A: serviceaccount "A" not found
P.S. I am using Google Kubernetes Engine
AFAIK this can be done by granting Service Account [A] a rolebinding in namespace NB allowing it to deploy pods. You just need the proper role.
You can simply reference a ServiceAccount from another namespace in the RoleBinding:
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
name: pod-reader
namespace: ns2
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: pod-reader-from-ns1
namespace: ns2
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: pod-reader
subjects:
- kind: ServiceAccount
name: ns1-service-account
namespace: ns1
I have a Kubernetes cluster where my application is deployed.
there are some other users they should only be able to copy files into and from a pod. Using kubectl cp command.
This user context should not allow the user to do any other operations on the cluster other than kubectl cp.
kubectl cp internally uses exec. There is no way to provide permission to only copy but you can provide only exec permission.
Create a role with permission to pods/exec
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
namespace: default
name: pod-exec
rules:
- apiGroups: [""]
resources: ["pods/exec"]
verbs: ["create"]
Create a Rolebinding to assign the above role to a user.
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: pod-exec-binding
namespace: default
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: pod-exec
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: User
name: user
Rather than use kubectl cp, instead run a sidecar container with an sftp or rsync server. That will give you better control at all levels.
You can use opa and admission controller which only permit to run api manifest has a specific label like "cp" or "username" etc. and also benefits from gatekeeper
https://www.youtube.com/watch?v=ZJgaGJm9NJE&t=3040s
I know there are a lot of similar questions but none of them has a solution as far as I have browsed.
Coming to the issue, I have created a service account (using command), role (using .yaml file), role binding (using .yaml files). The role grants access only to the pods. But when I login into the dashboard (Token method) using the SA that the role is attached to, I'm able to view all the resources without any restrictions. Here are the file and commands used by me.
Role.yaml:
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
namespace: assembly-prod
name: testreadrole
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "watch", "list"]
RoleBinding.yaml
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: testrolebinding
namespace: assembly-prod
subjects:
- kind: ServiceAccount
name: testsa
apiGroup: ""
roleRef:
kind: Role
name: testreadrole
apiGroup: rbac.authorization.k8s.io
Command used to create service account:
kubectl create serviceaccount <saname> --namespace <namespacename>
UPDATE: I create a service account and did not attach any kind of role to it. When I tried to login with this SA, It let me through and I was able to perform all kinds activities including deleting "secrets". So by default all SA are assuming admin access and that is the reason why my above roles are not working. Is this behavior expected, If yes then how can I change it?
Try the below steps
# create service account
kubectl create serviceaccount pod-viewer
# Create cluster role/role
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: pod-viewer
rules:
- apiGroups: [""] # core API group
resources: ["pods", "namespaces"]
verbs: ["get", "watch", "list"]
---
# create cluster role binding
kubectl create clusterrolebinding pod-viewer \
--clusterrole=pod-viewer \
--serviceaccount=default:pod-viewer
# get service account secret
kubectl get secret | grep pod-viewer
pod-viewer-token-6fdcn kubernetes.io/service-account-token 3 2m58s
# get token
kubectl describe secret pod-viewer-token-6fdcn
Name: pod-viewer-token-6fdcn
Namespace: default
Labels: <none>
Annotations: kubernetes.io/service-account.name: pod-viewer
kubernetes.io/service-account.uid: bbfb3c4e-2254-11ea-a26c-0242ac110009
Type: kubernetes.io/service-account-token
Data
====
token: eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6InBvZC12aWV3ZXItdG9rZW4tNmZkY24iLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicG9kLXZpZXdlciIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6ImJiZmIzYzRlLTIyNTQtMTFlYS1hMjZjLTAyNDJhYzExMDAwOSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OnBvZC12aWV3ZXIifQ.Pgco_4UwTCiOfYYS4QLwqgWnG8nry6JxoGiJCDuO4ZVDWUOkGJ3w6-8K1gGRSzWFOSB8E0l2YSQR4PB9jlc_9GYCFQ0-XNgkuiZBPvsTmKXdDvCNFz7bmg_Cua7HnACkKDbISKKyK4HMH-ShgVXDoMG5KmQQ_TCWs2E_a88COGMA543QL_BxckFowQZk19Iq8yEgSEfI9m8qfz4n6G7dQu9IpUSmVNUVB5GaEsaCIg6h_AXxDds5Ot6ngWUawvhYrPRv79zVKfAxYKwetjC291-qiIM92XZ63-YJJ3xbxPAsnCEwL_hG3P95-CNzoxJHKEfs_qa7a4hfe0k6HtHTWA
ca.crt: 1025 bytes
namespace: 7 bytes
```
Login to dashboard using the above token. you should see only pods and namespaces
[![Refer the below link][1]][1]
[1]: https://i.stack.imgur.com/D9bDi.png
Okay I've found the solution for this. The major issue was I'm running my cluster on Azure AKS, which I should have mentioned in the question but did not. It was my mistake. In Azure AKS, if rbac is not enabled during cluster creation, then there is no use of roles and role-bindings at all. All request to the api-server will be treated as requests from Admin. This was confirmed by Azure support too. So that was the reason my cluster-role-binding and roles didn't apply.
I see that the .yamls you provided need some adjustments.
Role has wrong formatting after the rules part.
RoleBinding is missing namespace: after subjects:, and also is formatted wrongly.
Try something like this:
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
namespace: assembly-prod
name: testreadrole
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "watch", "list"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: testrolebinding
namespace: assembly-prod
subjects:
- kind: ServiceAccount
name: testsa
namespace: assembly-prod
roleRef:
kind: Role
name: testreadrole
apiGroup: rbac.authorization.k8s.io
There is a very useful guide about Non-Privileged RBAC User Administration in Kubernetes where you can find more detailed info regarding this particular topic.
I want to make a very simple example to learn how to use RBAC authorization in kubernetes. Therefore I use the example from the docs:
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
namespace: dev
name: dev-readpods-role
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "watch", "list"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: dev-tester-rolebinding
namespace: dev
subjects:
- kind: User
name: Tester
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: Role
name: dev-readpods-role
apiGroup: rbac.authorization.k8s.io
The role and the rolebinding are created.
When I log in with Tester and try
kubectl get pods -n dev
I get
Error from server (Forbidden): pods is forbidden: User "<url>:<port>/oidc/endpoint/OP#Tester" cannot list pods in the namespace "dev"
I read here (RBAC Error in Kubernetes) that the api-server have to be started with --authorization-mode=…,RBAC. How can I check this? I read somewhere else that if I run
kubectl api-versions | findstr rbac
and find entries RBAC should be activated. Is that true?
What am I doing wrong? Is there a good way to troubleshoot?
Thanks!
P.S. I'm running kubernetes inside IBM Cloud Private.
In ICP, it looks encouraging to use Teams (ICP's own term, I think). Try starting with it. But you need an LDAP server outside of ICP.
https://www.ibm.com/support/knowledgecenter/en/SSBS6K_2.1.0.3/user_management/admin.html
You would need to determine the invocation of the apiserver to see what --authorization-mode flag was passed to it. Normally this is contained in a systemd unit file or pod manifest. I'm not sure how IBM Cloud launches the apiserver
I am trying to create a script that records project resources every 15 minutes. How do I authenticate with Openshift API? Is there a token I can use that has read access on all namespaces? How do I create a service account that has access over all namespaces?
You'll need to create a ClusterRole that has read access to the resources and use ClusterRoleBinding to associate the ServiceAccount to that ClusterRole. Rough example, not tested but it should work:
# creates the service account "ns-reader"
apiVersion: v1
kind: ServiceAccount
metadata:
name: ns-reader
namespace: default
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
# "namespace" omitted since ClusterRoles are not namespaced
name: global-reader
rules:
- apiGroups: [""]
# add other rescources you wish to read
resources: ["pods", "secrets"]
verbs: ["get", "watch", "list"]
---
# This cluster role binding allows service account "ns-reader" to read pods in all available namespace
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: read-ns
subjects:
- kind: ServiceAccount
name: ns-reader
namespace: default
roleRef:
kind: ClusterRole
name: global-reader
apiGroup: rbac.authorization.k8s.io
When the ServiceAccount is setup, a number of secrets are created automatically associated with it. A couple of these secrets hold a token which can then be used when using the REST API directly or using oc. Use oc describe on the ServiceAccount to see the names of the Secret for the tokens. Then use oc describe on one of the Secrets to see the token.