Where does GKE log RBAC permission events?
On Google Container Engine (GKE) clusters with kubernetes version v1.6 enable RBAC authorization per default. Apparently ABAC is enabled as fallback authorization as well in order to ease the transition of existing clusters to the new authorization scheme. The idea is that first RBAC is tried to authorize an action. If that fails, this should be logged somewhere and then ABAC is consulted to allow the action. This should enabled cluster admins to inspect the logs for missed RBAC permissions before finally switching off ABAC.
We have some clusters that disable GCP logging/monitoring, instead use an own ELK stack. Just to be sure I've created a test cluster with GCP's cloud logging and monitoring, but still can's find any RBAC events anywhere. The test pod is a prometheus server that discovers and scrapes other pods and nodes.
To make this more comprehensive. From Using RBAC Authorization:
When run with a log level of 2 or higher (--v=2), you can see RBAC denials in the apiserver log (prefixed with RBAC DENY:).
In GKE the apiservers logs can be accessed via HTTP like:
kubectl proxy &
curl -s http://localhost:8001/logs/kube-apiserver.log
The RBAC denials are logged to the master apiserver log.
Related
I am a junior developer currently running a service in a Kubernetes environment.
How can I check if a resource inside Kubernetes has been deleted for some reason?
As a simple example, if a deployment is deleted, I want to know which user deleted it.
Could you please tell me which log to look at.
And I would like to know how to collect these logs.
I don't have much experience yet, so I'm asking for help.
Also, if you have a reference or link, please share it. It will be very helpful to me.
Thank you:)
Start with enabling audit with lots of online resources about doing this.
If you are on AWS and using EKS I would suggest enabling "Amazon EKS control plane logging" By enabling it you can enable audit and diagnostic logs streaming in AWS cloudwatch logs, which are more easily accessible, and useful for audit and compliance requirements. Control plane logs make it easy for you to secure and run your clusters and make the entire system more audiatable.
As per AWS documentation:
Kubernetes API server component logs (api) – Your cluster's API server is the control plane component that exposes the Kubernetes API. For more information, see kube-apiserver in the Kubernetes documentation.
Audit (audit) – Kubernetes audit logs provide a record of the individual users, administrators, or system components that have affected your cluster. For more information, see Auditing in the Kubernetes documentation.
Authenticator (authenticator) – Authenticator logs are unique to Amazon EKS. These logs represent the control plane component that Amazon EKS uses for Kubernetes Role-Based Access Control (RBAC) authentication using IAM credentials. For more information, see Cluster authentication.
Controller manager (controllerManager) – The controller manager manages the core control loops that are shipped with Kubernetes. For more information, see kube-controller-manager in the Kubernetes documentation.
Scheduler (scheduler) – The scheduler component manages when and where to run pods in your cluster. For more information, see kube-scheduler in the Kubernetes documentation.
Reference: https://docs.aws.amazon.com/eks/latest/userguide/control-plane-logs.html
So there's this page about auditing-logs and I'm very confused about:
The k8s.io service is used for Kubernetes audit logs. These logs are generated by the Kubernetes API Server component and they contain information about actions performed using the Kubernetes API. For example, any changes you make on a Kubernetes resource by using the kubectl command are recorded by the k8s.io service. For more information, see Auditing in the Kubernetes documentation.
The container.googleapis.com service is used for GKE control plane audit logs. These logs are generated by the GKE internal components and they contain information about actions performed using the GKE API. For example, any changes you perform on a GKE cluster configuration using a gcloud command are recorded by the container.googleapis.com service.
which one shall I pick to get:
/var/log/kube-apiserver.log - API Server, responsible for serving the API
/var/log/kube-controller-manager.log - Controller that manages replication controllers
or these are all similar to EKS where audit logs means a separate thing?
Audit (audit) – Kubernetes audit logs provide a record of the individual users, administrators, or system components that have affected your cluster. For more information, see Auditing in the Kubernetes documentation.
If the cluster still exists, you should be able to do the following on GKE
kubectl proxy
curl http://localhost:8001/logs/kube-apiserver.log
AFAIK, there's no way to get server logs for clusters that have been deleted.
Logs for GKE control-plane components are available since November 29, 2022 for clusters with versions 1.22.0 and later.
You simply need to activate it on the clusters. Either
via CLI:
gcloud container clusters update [CLUSTER_NAME] \
--region=[REGION] \
--monitoring=SYSTEM,WORKLOAD,API_SERVER,SCHEDULER,CONTROLLER_MANAGER
via GCP web-console: Open the cluster-details, in the section "Features" edit the entry "Cloud Logging" and add the "Control Plane" components.
See documentation for details.
Note the notes in the solutions documentation, especially about reaching the logging.googleapis.com/write_requests quota (quick link).
You cannot. GKE does not make them available. Audit logs are different, those are a record of API actions.
Our infrastructure currently has 2 Kubernetes Cluster, with one Cluster (cluster-1) creating pods in another cluster (cluster-2). Since we are on kubernetes1.7.x, we are able to make this work.
However, with 1.8 Kubernetes added support for RBAC as a result of which we cannot create pods in the new cluster anymore.
We already added support for Service Accounts and made sure that RoleBindings are properly set-up. But the main issue is that the service-account is not propagated outside of the cluster (and rightly so). The user that cluster-2 receives the request is called 'client', so when we added a RoleBinding with 'client' as a User, everything worked.
This is most certainly not the correct solution, as now any cluster that talks to Kubernetes API server can create a pod.
Is there support for RBAC that works cross cluster? Or, is there a way to propagate the service info through to the cluster we want to create the pods in?
P.S.: Our Kubernetes cluster are currently on GKE. But, we would like this to work on all Kubernetes-engine.
Your cluster-1 SA uses a kubecfg (for cluster-2) which resolves to the user "client". The only way to solve this is to generate a kubecfg (for cluster-2) with an identity associated (cert/token) for your cluster-1 SA. Lot of ways to do that: https://kubernetes.io/docs/admin/authentication/
Simplest way is to create an identical SA in cluster-2 and use its token in the kubecfg in cluster-1. Give RBAC only to that SA.
I’m investigating this letsencrypt controller (https://github.com/tazjin/kubernetes-letsencrypt).
It requires pods have permission to make changes to records in Cloud DNS. I thought with the pods running on GKE I’d get that access with the default service account, but the requests are failing. What do I need to do do to allow the pods access to Cloud DNS?
The Google Cloud DNS API's changes.create call requires either the https://www.googleapis.com/auth/ndev.clouddns.readwrite or https://www.googleapis.com/auth/cloud-platform scope, neither of which are enabled by default on a GKE cluster.
You can add a new Node Pool to your cluster with the DNS scope by running:
gcloud container node-pools create np1 --cluster my-cluster --scopes https://www.googleapis.com/auth/ndev.clouddns.readwrite
Or, you can create a brand new cluster with the scopes you need, either by passing the --scopes flag to gcloud container clusters create, or in the New Cluster dialog in Cloud Console, click "More", and set the necessary scopes to "Enabled".
I would like to enable the ABAC mode for the Kubernetes Cluster I'm using in Google's Container Engine. (more specifically, I would like to restrict access to the API service for the default service account which is automatically assigned to all pods). However, since --authorization-mode=ABAC is a command line argument for kube-apiserver and since the API server is managed in Google Container Engine, I didn't find a way to enable authorization for my cluster.
Is there a way to enable ABAC mode on GCE?
I'm currently running Kubernetes v1.1.7 on server and nodes.
There is not a way to enable ABAC mode on Google Container Engine. If you need fine-grained control over the parameters passed to any of the master components you have to run Kubernetes on GCE instead.
In the meantime Google has added the possibility to use Role Based Access Control (RBAC) for a Kubernetes Cluster. It is enabled by default for all new Clusters running Kubernetes 1.6 or later: https://cloud.google.com/container-engine/docs/role-based-access-control