What options are available to me if I want to restrict the usage of Priorty Classes, if I'm running a managed Kubernetes service (AKS)?
The use-case here is that, I am as cluster admin want to restrict the usage of these. So that developers are not using those that are supposed to be used by critical components.
Multi-tenant cluster, semi 0-trust policy. This is something that could be "abused".
Is this something I can achieve with resource quotas even though I'm running Azure Kubernetes Service?
https://kubernetes.io/docs/concepts/policy/resource-quotas/#limit-priority-class-consumption-by-default
Cloud managed cluster does not allow you to customize the api-server. In this case, you can use opa gatekeeper or kyverno to write rules that reject un-necessary priority class settings.
Azure Kubernetes Service has a addon called azure-policy. When you enable this...you will get Gatekeper installed on your cluster. With this you can leverage admission control and validate or mutate requests to the API-server. The great benefit here is that you also have visual reporting of your compliance though Azure Policy in the portal.
# Existing Cluster
az aks enable-addons \
--addons azure-policy \
--name MyAKSCluster \
--resource-group MyResourceGroup
# New Cluster
az aks create \
--name MyAKSCluster \
--resource-group MyResourceGroup \
--enable-addons azure-policy
For your purpose you could now create an Azure Custom Policiy. There are also existing Policy Definitions or Initiative from Azure which you can leverage.
Related
I have installed Azure Workload Identity, e.g. like that:
az aks create -g myResourceGroup -n myAKSCluster --node-count 1 --enable-oidc-issuer --enable-workload-identity --generate-ssh-keys
This has installed a mutating webhook that is of version 0.15.0 in kube-system. Now when the new versions will come out, how do I keep it updated?
Does this happen automatically or I would need to uninstall/install again or do something like that?
yes, addons are maintained by Microsoft. Any update/upgrades will be rolled out automatically.
As mentioned here:
Add-ons are a fully supported way to provide extra capabilities for
your AKS cluster. Add-ons' installation, configuration, and lifecycle
is managed by AKS
Workload Identity is not even considered as an additional feature, but the same thing applies since it's a managed component of the cluster, and Microsoft is responsible for the lifecycle of it.
Generally, any out of box resource in the kube-system namespace is managed by Microsoft and will receive the updates automatically.
I was trying to upgrade my Azure AKS to protect against CVE-2021-25741.
I had seen that the following versions are affected:
I cannot upgrade my cluster to any of the versions above.
When I run the following command
az aks get-versions --location westeurope --output table
I get this:
So the question is how can I upgrade to a non-affected version.
you have to wait until the AKS Team is releasing a Kubernetes Version were this is fixed.
Until then, you can leverage Azure Policy & Gatekeeper to mitigate the attack scenario: https://sysdig.com/blog/cve-2021-25741-kubelet-falco/
Looking for the best way to integrate ACR with AKS for Producation environment, Seems there are multiple ways like, during installation, and after installation, using service principala,a nd using image pull secret etc..
So for our production environment looking for most recommended option, where the requirement as follows.
Is it mandatory to attach acr during aks creation itself
What will be the advantage if we are integrating ACR along with AKS instalation itself. (seems , we dont want to pass the image pull secret to the pod spec in that case and for other options we need to)
What is the another way to integrate ACR with AKS ( az aks update) command will help in this case? if yes, what will be the difference from the previous method where we integrated during AKS installation.
IF I want to setup a secodary AKS cluster in another region, but need to connect the ACR georeplicated instance of Primary instance of ACR , How i can get it done? In this case is it mandaory to attach tge ACR during AKS installation or later post installation also its good to go?
IMHO the best way is Azure RBAC. You dont need to attach the ACR while creating the AKS. You can leverage Azure RBAC and assign the Role "AcrPull" to the Kubelet identity of your nodepool. This can be done for every ACR you have:
export KUBE_ID=$(az aks show -g <resource group> -n <aks cluster name> --query identityProfile.kubeletidentity.objectId -o tsv)
export ACR_ID=$(az acr show -g <resource group> -n <acr name> --query id -o tsv)
az role assignment create --assignee $KUBE_ID --role "AcrPull" --scope $ACR_ID
Terraform:
resource "azurerm_role_assignment" "example" {
scope = azurerm_container_registry.acr.id
role_definition_name = "AcrPull"
principal_id = azurerm_kubernetes_cluster.aks.kubelet_identity[0].object_id
}
I am trying to follow this tutorial to backup a persistent volume in Azure AKS:
https://learn.microsoft.com/en-us/azure/aks/azure-disks-dynamic-pv
I can see the volumes doing
az aks get-credentials --resource-group MYRESOURCEGROUP --name HIDDEN --subscription MYSUBSCRIPTION
kubectl get pv
(Both disk and file, managed-premium and standard storage classes)
but then I do:
az disk list --resource-group MYRESOURCEGROUP --subscription MYSUBSCRIPTION
and I get an empty list, so I can't know the source full path to perform the snapshot.
Am I missing something?
Upgrade your az cli version.
I was getting this issue with az cli 2.0.75 returning an empty array for the disk list, with an AKS PV.
upgraded to az cli 2.9.1 and same command worked.
that happens because AKS is creating a service resource group with AKS resources, it is called something like MC_%AKS-name%_%AKS-resource-group-name%_%region% (not configurable at the time of writing). you should list disks in that resource group to view those.
How can Apache Airflow's KubernetesPodOperator pull docker images from a private repository?
The KubernetesPodOperator has an image_pull_secrets which you can pass a Secrets object to authenticate with the private repository. But the secrets object can only represent an environment variable, or a volume - neither of which fit my understanding of how Kubernetes uses secrets to authenticate with private repos.
Using kubectl you can create the required secret with something like
$ kubectl create secret docker-registry $SECRET_NAME \
--docker-server=https://${ACCOUNT}.dkr.ecr.${REGION}.amazonaws.com \
--docker-username=AWS \
--docker-password="${TOKEN}" \
--docker-email="${EMAIL}"
But how can you create the authentication secret in Airflow?
There is secret object with docker-registry type according to kubernetes documentation which can be used to authenticate to private repository.
As You mentioned in Your question; You can use kubectl to create secret of docker-registry type that you can then try to pass with image_pull_secrets.
However depending on platform You are using this might have limited or no use at all according to kubernetes documentation:
Configuring Nodes to Authenticate to a Private Registry
Note: If you are running on Google Kubernetes Engine, there will already be a .dockercfg on each node with credentials for Google Container Registry. You cannot use this approach.
Note: If you are running on AWS EC2 and are using the EC2 Container Registry (ECR), the kubelet on each node will manage and update the ECR login credentials. You cannot use this approach.
Note: This approach is suitable if you can control node configuration. It will not work reliably on GCE, and any other cloud provider that does automatic node replacement.
Note: Kubernetes as of now only supports the auths and HttpHeaders section of docker config. This means credential helpers (credHelpers or credsStore) are not supported.
Making this work on mentioned platforms is possible but it would require automated scripts and third party tools.
Like in Amazon ECR example: Amazon ECR Docker Credential Helper would be needed to periodically pull AWS credentials to docker registry configuration and then have another script to update kubernetes docker-registry secrets.
As for Airflow itself I don't think it has functionality to create its own docker-repository secrets.
You can request functionality like that in Apache Airflow JIRA.
P.S.
If You still have issues with Your K8s cluster you might want to create new question on stack addressing them.