Whats difference between "kubectl auth reconcile" and "kubectl apply" for working with RBAC? - kubernetes

I have some average yaml file defining some average role resource, all yaml should reflect my resource's desired state.
To get new average role into cluster I usually run kubectl apply -f my-new-role.yaml
but now I see this (recommended!?) alternative kubectl auth reconcile -f my-new-role.yaml
Ok, there may be RBAC relationships, ie Bindings, but shouldn't an apply do same thing?
Is there ever a case where one would update (cluster) roles but not want their related (cluster) bindings updated?

The kubectl auth reconcile command-line utility has been added in Kubernetes v1.8.
Properly applying RBAC permissions is a complex task because you need to compute logical covers operations between rule sets.
As you can see in the CHANGELOG-1.8.md:
Added RBAC reconcile commands with kubectl auth reconcile -f FILE. When passed a file which contains RBAC roles, rolebindings, clusterroles, or clusterrolebindings, this command computes covers and adds the missing rules. The logic required to properly apply RBAC permissions is more complicated than a JSON merge because you have to compute logical covers operations between rule sets. This means that we cannot use kubectl apply to update RBAC roles without risking breaking old clients, such as controllers.
The kubectl auth reconcile command will ignore any resources that are not Role, RoleBinding, ClusterRole, and ClusterRoleBinding objects, so you can safely run reconcile on the full set of manifests (see: Use 'kubectl auth reconcile' before 'kubectl apply')
I've created an example to demonstrate how useful the kubectl auth reconcile command is.
I have a simple secret-reader RoleBinding and I want to change a binding's roleRef (I want to change the Role that this binding refers to):
NOTE: A binding to a different role is a fundamentally different binding (see: A binding to a different role is a fundamentally different binding).
# BEFORE
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: secret-admin
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: secret-reader
subjects:
- kind: ServiceAccount
name: service-account-1
namespace: default
# AFTER
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: secret-admin
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: secret-creator
subjects:
- kind: ServiceAccount
name: service-account-1
namespace: default
As we know, roleRef is immutable, so it is not possible to update this secret-admin RoleBinding using kubectl apply:
$ kubectl apply -f secret-admin.yml
The RoleBinding "secret-admin" is invalid: roleRef: Invalid value: rbac.RoleRef{APIGroup:"rbac.authorization.k8s.io", Kind:"Role", Name:"secret-creator"}: cannot change roleRef
Instead, we can use kubectl auth reconcile. If a RoleBinding is updated to a new roleRef, the kubectl auth reconcile command handles a delete/recreate related objects for us.
$ kubectl auth reconcile -f secret-admin.yml
rolebinding.rbac.authorization.k8s.io/secret-admin reconciled
reconciliation required recreate
Additionally, you can use the --remove-extra-permissions and --remove-extra-subjects options.
Finally, we can check if everything has been successfully updated:
$ kubectl describe rolebinding secret-admin
Name: secret-admin
Labels: <none>
Annotations: <none>
Role:
Kind: Role
Name: secret-creator
Subjects:
Kind Name Namespace
---- ---- ---------
ServiceAccount service-account-1 default

Related

Kubernetes OIDC: No valid group mapping

I have the problem that I can log on to my dashboard via OIDC, but then the oidc group information is not mapped correctly and I cannot access the corresponding resources.
Basic setup
K8s version: 1.19.0
K8s setup: 1 master + 2 worker nodes
Based on Debian 10 VMs
CNI: Calico
Louketo Proxy as OIDC proxy
OIDC: Keycloak Server (Keycloak X [Quarkus])
Configurations
I have configured the K8s apiserver with these parameters.
kube-apiserver.yaml
- --oidc-issuer-url=https://test.test.com/auth/realms/Test
- --oidc-client-id=test
- --oidc-username-claim=preferred_username
- --oidc-username-prefix="oidc:"
- --oidc-groups-claim=groups
- --oidc-groups-prefix="oidc:"
ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: "test-cluster-admin"
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: "Test"
I used the following louketo parameters
Louketo Proxy
/usr/bin/louketo-proxy --discovery-url=$OIDC_DISCOVERY_URL --client-id=$OIDC_CLIENT_ID --client-secret=$OIDC_CLIENT_SECRET -listen=$OIDC_LISTEN_URL --encryption-key=$OIDC_ENCRYPTION_KEY --redirection-url=$OIDC_REDIRECTION_KEY --enable-refresh-tokens=true --upstream-url=$OIDC_UPSTREAM_URL --enable-metrics
I get the following error message inside the dashboard.
K8s error
replicasets.apps is forbidden: User "\"oidc:\"<user_name>" cannot list resource "replicasets" in API group "apps" in the namespace "default"
I hope you can help me with this problem, I already tried most of the manuals from the internet, but haven't found a solution yet.
PS: I have done the corresponding group mapping in the Keycloak server and also validated that the group entry is transferred.
If you are facing the same challenge as I did and you want to integrate Keycloak into your K8s cluster, share the dashboard and connect it to Keycloak, you can find my configuration below. Within my cluster I use the Louketo Proxy as interface between Kubernetes and Keycloak. The corresponding configuration of the deployment is not included in this post.
Keycloak
I want to start with the configuration of Keycloak. In the first step I created a corresponding client with the following settings.
After that I created the two group membership and audience (needed by the louketo proxy) mappers.
The exact settings of the mappers can be taken from the two images.
Group membership mapping
Audience mapping
Kubernetes
In the second step I had to update the api server manifest and create the RoleBinding and ClusterRoleBinding within the Kubernetes cluster.
Api server manifest (default path: /etc/kubernetes/manifests/kube-apiserver.yaml)
- --oidc-issuer-url=https://test.test.com/auth/realms/Test
- --oidc-client-id=test
- --oidc-username-claim=preferred_username
- --oidc-username-prefix="oidc:"
- --oidc-groups-claim=groups
- --oidc-groups-prefix="oidc:"
RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: "test"
namespace: "kubernetes-dashboard"
subjects:
- kind: User
name: "\"oidc:\"Test"
namespace: "kube-system"
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: "test"
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: "\"oidc:\"Test"
#Community I hope I can help you with this configuration. If you have any questions, feel free to ask me.
This is a community wiki answer aimed to approach the issue from the Kubernetes side. Any one familiar with the possible Keycloak group/role mapping solution feel free to edit it.
The error you see means that the service account for OIDC doesn't have the proper privileges to list replicasets in the default namespace. The easiest way out of it would be to simply setup the ServiceAccount, ClusterRole and ClusterRoleBinding from scratch and make sure it has the proper privileges. For example, you can create a clusterrolebinding with permissions “admin” by executing:
kubectl create clusterrolebinding OIDCrolebinding - -clusterrole=admin - - group=system:serviceaccounts:OIDC
The same can be done for the ClusterRole:
kubectl create clusterrole OIDC --verb=get,list,watch --resource=replicasets --namespace=default
More examples of how to use the kubectl create in this scenario can be found here.
Here you can find a whole official guide regarding the RBAC Authorization.
EDIT:
Also, please also check if your ClusterRoleBinding for the "\"oidc:\"<user_name>" is in the "default" namespace.

How to get kubernetes applications to change deploy configs

I have two applications running in K8. APP A has write access to a data store and APP B has read access.
APP A needs to be able to change APP B's running deployment.
How we currently do this is manually by kicking off a process in APP A which adds a new DB in the data store (say db bob). Then we do:
kubectl edit deploy A
And change an environment variable to bob. This starts a rolling restart of all the pods of APP B. We would like to automate this process.
Is there anyway to get APP A to change the deployment config of APP B in k8?
Firstly answering your main question:
Is there anyway to get a service to change the deployment config of another service in k8?
From my understanding you are calling it Service A and B for it's purpose in the real life, but to facilitate understanding I suggested an edit to call them APP A and APP B, because:
In Kubernetes, a Service is an abstraction which defines a logical set of Pods and a policy by which to access them (sometimes this pattern is called a micro-service).
So if in your question you meant:
"Is there anyway to get APP A to change the deployment config of APP B in k8?"
Then Yes, you can give a pod admin privileges to manage other components of the cluster using the kubectl set env command to change/add envs.
In order to achieve this, you will need:
A Service Account with needed permissions in the namespace.
NOTE: In my example below since I don't know if you are working with multiple namespaces I'm using a ClusterRole, granting cluster-admin to a specific user. If you use only 1 namespace for these apps, consider a Role instead.
A ClusterRoleBinding binding the permissions of the service account to a role of the Cluster.
The Kubectl client inside the pod (manually added or modifying the docker-image) on APP A
Steps to Reproduce:
Create a deployment to apply the cluster-admin privileges, I'm naming it manager-deploy.yaml:
apiVersion: apps/v1
kind: Deployment
metadata:
name: manager-deploy
labels:
app: manager
spec:
replicas: 1
selector:
matchLabels:
app: manager
template:
metadata:
labels:
app: manager
spec:
serviceAccountName: k8s-role
containers:
- name: manager
image: gcr.io/google-samples/node-hello:1.0
Create a deployment with a environment var, mocking your Service B. I'm naming it deploy-env.yaml:
apiVersion: apps/v1
kind: Deployment
metadata:
name: env-deploy
labels:
app: env-replace
spec:
replicas: 1
selector:
matchLabels:
app: env-replace
template:
metadata:
labels:
app: env-replace
spec:
serviceAccountName: k8s-role
containers:
- name: env-replace
image: gcr.io/google-samples/node-hello:1.0
env:
- name: DATASTORE_NAME
value: "john"
Create a ServiceAccount and a ClusterRoleBinding with cluster-admin privileges, I'm naming it service-account-for-pod.yaml (notice it's mentioned in manager-deploy.yaml:
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: k8s-role
subjects:
- kind: ServiceAccount
name: k8s-role
namespace: default
roleRef:
kind: ClusterRole
name: cluster-admin
apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: k8s-role
Apply the service-account-for-pod.yaml, deploy-env.yaml, manager-deploy.yamland list current environment variables from deploy-env pod:
$ kubectl apply -f manager-deploy.yaml
deployment.apps/manager-deploy created
$ kubectl apply -f deploy-env.yaml
deployment.apps/env-deploy created
$ kubectl apply -f service-account-for-pod.yaml
clusterrolebinding.rbac.authorization.k8s.io/k8s-role created
serviceaccount/k8s-role created
$ kubectl exec -it env-deploy-fbd95bb94-hcq75 -- printenv
DATASTORE_NAME=john
Shell into the manager pod, download the kubectl binary and apply the kubectl set env deployment/deployment_name VAR_NAME=VALUE:
$ kubectl exec -it manager-deploy-747c9d5bc8-p684s -- /bin/bash
root#manager-deploy-747c9d5bc8-p684s:/# curl -LO https://storage.googleapis.com/kubernetes-release/release/`curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt`/bin/linux/amd64/kubectl
root#manager-deploy-747c9d5bc8-p684s:/# chmod +x ./kubectl
root#manager-deploy-747c9d5bc8-p684s:/# mv ./kubectl /usr/local/bin/kubectl
root#manager-deploy-747c9d5bc8-p684s:/# kubectl set env deployment/env-deploy DATASTORE_NAME=bob
Verify the env var value on the pod (notice that the pod is recreated when deployment is modified:
$ kubectl exec -it env-deploy-7f565ffc4-t46zc -- printenv
DATASTORE_NAME=bob
Let me know in the comments if you have any doubt on how to apply this solution to your environment.
You could give service A access to your cluster (install kubectl and allow traffic from that NAT of service A to your cluster master) and with some cron jobs or jenkins / ssh or something that will execute your commands do it. You can also do kubectl patch or get the current config of second deployment kubectl get deployment <name> -o yaml --export > deployment.yaml and edit it there with some regex/awk/sed and then apply although the --export method is getting deprecated so you might aswell on service A download the GIT repo and apply the new config like that.
Thank you for the answers all (upvoted as they were both correct). I am just putting my own answer to document exactly what solved it for me.
In my case I just needed to make use of the patch url available on k8. That plus the this example worked.
All I needed to do was create a service account to restrict who can patch where. Restrict that account to Service A and use the java client in Service A to update the chart of Service B. After that the pods would roll and done.

How to secure kubernetes secrets?

I am trying to avoid kubernetes secrets view-able by any user.
I tried sealed secrets, but that is just hiding secrets to be stored in version control.
As soon as I apply that secret, I can see the secret using the below command.
kubectl get secret mysecret -o yaml
This above command is still showing base64 encoded form of secret.
How do I avoid someone seeing the secret ( even in base64 format) with the above simple command.
You can use Hashicrop Vault or kubernetes-external-secrets (https://github.com/godaddy/kubernetes-external-secrets).
Or if you just want to restrict only, then you should create a read-only user and restrict the access for the secret for the read-only user using role & role binding.
Then if anyone tries to describe secret then it will throw access denied error.
Sample code:
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: test-secrets
namespace: default
rules:
- apiGroups:
- ""
resources:
- pods
verbs:
- get
- list
- watch
- create
- delete
- apiGroups:
- ""
resources:
- pods/exec
verbs:
- create
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: test-secrets
namespace: default
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: test-secrets
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: User
name: demo
The above role has no access to secrets. Hence the demo user gets access denied.
There is no way to accomplish this with Kubernetes internal tools. You will always have to rely on a third-party tool.
Hashicorps Vault is one very often used solution, which is very powerful and supports some very nice features, like Dynamic Secrets or Envelope Encryption. But it can also get very complex in terms of configuration. So you need to decide for yourself what kind of solution you need.
I would recommend you using Sealed-Secrets. It encrypts your secrets and you can push the encrypted secrets safely in your repository. It has not such a big feature list, but it does exactly what you described.
You can Inject Hashicrop Vault secrets into Kubernetes pods via Init containers and keep them up to date with a sidecar container.
More details here https://www.hashicorp.com/blog/injecting-vault-secrets-into-kubernetes-pods-via-a-sidecar/

Kubernetes RBAC permissions - unknown 'clusterrole' flag when attempting to grant permissions?

I am using the Mirantis kubeadm-dind-cluster repository (https://github.com/Mirantis/kubeadm-dind-cluster) as my Kubernetes install; I came across this error when attempting to run a container -
panic: customresourcedefinitions.apiextensions.k8s.io is forbidden: User "system:serviceaccount:default:default" cannot create customresourcedefinitions.apiextensions.k8s.io at the cluster scope
So I attempted to add cluster-admin permissions to my account:
kubectl create clusterrolebinding serviceaccounts-cluster-admin --clusterrole=cluster-admin --group=system:serviceaccounts
And get the following error:
Error: unknown flag: --clusterrole
Why is this? How do I fix this or get around it? I'm not sure how to convert the command into a YAML file to "kubectl create -f" to but it seems like that might be the way to go.
All three nodes are on version 1.8.6.
What version of kubectl are you using? Be sure you are using a version that includes the kubectl create clusterrolebinding command
If your version of kubectl does not support that command, you can try creating it directly via a yaml file (though I'm not sure whether 1.5.x kubectl was happy submitting versions of API objects it didn't know about):
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: serviceaccounts-cluster-admin
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: system:serviceaccounts

Kubernetes log, User "system:serviceaccount:default:default" cannot get services in the namespace

Forbidden!Configured service account doesn't have access. Service account may have been revoked. User "system:serviceaccount:default:default" cannot get services in the namespace "mycomp-services-process"
For the above issue I have created "mycomp-service-process" namespace and checked the issue.
But it shows again message like this:
Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. User "system:serviceaccount:mycomp-services-process:default" cannot get services in the namespace "mycomp-services-process"
Creating a namespace won't, of course, solve the issue, as that is not the problem at all.
In the first error the issue is that serviceaccount default in default namespace can not get services because it does not have access to list/get services. So what you need to do is assign a role to that user using clusterrolebinding.
Following the set of minimum privileges, you can first create a role which has access to list services:
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
namespace: default
name: service-reader
rules:
- apiGroups: [""] # "" indicates the core API group
resources: ["services"]
verbs: ["get", "watch", "list"]
What above snippet does is create a clusterrole which can list, get and watch services. (You will have to create a yaml file and apply above specs)
Now we can use this clusterrole to create a clusterrolebinding:
kubectl create clusterrolebinding service-reader-pod \
--clusterrole=service-reader \
--serviceaccount=default:default
In above command the service-reader-pod is name of clusterrolebinding and it is assigning the service-reader clusterrole to default serviceaccount in default namespace. Similar steps can be followed for the second error you are facing.
In this case I created clusterrole and clusterrolebinding but you might want to create a role and rolebinding instead. You can check the documentation in detail here
This is only for non prod clusters
You should bind service account system:serviceaccount:default:default (which is the default account bound to Pod) with role cluster-admin, just create a yaml (named like fabric8-rbac.yaml) with following contents:
# NOTE: The service account `default:default` already exists in k8s cluster.
# You can create a new account following like this:
#---
#apiVersion: v1
#kind: ServiceAccount
#metadata:
# name: <new-account-name>
# namespace: <namespace>
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: fabric8-rbac
subjects:
- kind: ServiceAccount
# Reference to upper's `metadata.name`
name: default
# Reference to upper's `metadata.namespace`
namespace: default
roleRef:
kind: ClusterRole
name: cluster-admin
apiGroup: rbac.authorization.k8s.io
Then, apply it by running kubectl apply -f fabric8-rbac.yaml.
If you want unbind them, just run kubectl delete -f fabric8-rbac.yaml.
Just to add.
This can also occur when you are redeploying an existing application to the wrong Kubernetes cluster that are similar.
Ensure you check to be sure that the Kubernetes cluster you're deploying to is the correct cluster.