I have a Kubernetes cluster. Inside my cluster is a Django application which needs to connect to my Kubernetes cluster on GKE. Upon my Django start up (inside my Dockerfile), I authenticate with Google Cloud by using:
gcloud auth activate-service-account $GKE_SERVICE_ACCOUNT_NAME --key-file=$GOOGLE_APPLICATION_CREDENTIALS
gcloud config set project $GKE_PROJECT_NAME
gcloud container clusters get-credentials $GKE_CLUSTER_NAME --zone $GKE_ZONE
I am not really sure if I need to do this everytime my Django container starts, and I am not sure I understand how authentication to Google Cloud works. Could I perhaps just generate my Kubeconfig file, store it somewhere safe and use it all the time instead of authenticating?
In other words, is a Kubeconfig file enough to connect to my GKE cluster?
If your service is running in a Pod inside the GKE cluster you want to connect to, use a Kubernetes service account to authenticate.
Create a Kubernetes service account and attach it to your Pod. If your Pod already has a Kubernetes service account, you may skip this step.
Use Kubernetes RBAC to grant the Kubernetes service account the correct permissions.
The following example grants edit permissions in the prod namespace:
kubectl create rolebinding yourserviceaccount \
--clusterrole=edit \
--serviceaccount=yournamespace:yourserviceaccount\
--namespace=prod
At runtime, when your service invokes kubectl, it automatically receives the credentials you configured.
You can also store the credentials as a secret and mount it on your pod so that it can read them from there
To use a Secret with your workloads, you can specify environment variables that reference the Secret's values, or mount a volume containing the Secret.
You can create a Secret using the command-line or a YAML file.
Here is an example using Command-line
kubectl create secret SECRET_TYPE SECRET_NAME DATA
SECRET_TYPE: the Secret type, which can be one of the following:
generic:Create a Secret from a local file, directory, or literal value.
docker-registry:Create a dockercfg Secret for use with a Docker registry. Used to authenticate against Docker registries.
tls:Create a TLS secret from the given public/private key pair. The public/private key pair must already exist. The public key certificate must be .PEM encoded and match the given private key.
For most Secrets, you use the generic type.
SECRET_NAME: the name of the Secret you are creating.
DATA: the data to add to the Secret, which can be one of the following:
A path to a directory containing one or more configuration files, indicated using the --from-file or --from-env-file flags.
Key-value pairs, each specified using --from-literal flags.
If you need more information about kubectl create you can check the reference documentation
Related
I have a container where I used a bitnami/kubectl image.
Now I want to run a few kubectl commands inside that container.
How kubectl container aware of my kubeconfig file?
I know that I can mount the local kubeconfig file into containers and use it.
But is there any other way possible to access kubeconfig without using it as a volume mount?
I went throug the documentation of RBAC in Kubernetes.
Does configure role and role-binding alone is enough to run kubectl apply and kubectl delete commands successfully even without mounting kubeconfig file?
It would be really helpful if someone helps me with this.
Thanks in advance!
Now I want to run a few kubectl commands inside that container.
Why do you need it inside the container?
kubectl is your CLI to "communicate" with the cluster, the commands are passed to the kube-api, parsed, and executed usually by Admission controller.
Not clear why you need to run kubectl commands inside the container, since kubectl use your kubeconfig file for the communication (it will read the certificate path to the certificate data) and will be able to connect to your cluster.
How to run K8S API in your container?
The appropriate solution is to run an API query inside your container.
Every pod stores internally the Token & ServiceAccount which will allow you to query the API
Use the following script I'm using to query the API
https://github.com/nirgeier/KubernetesLabs/blob/master/Labs/21-KubeAPI/api_query.sh
#!/bin/sh
#################################
## Access the internal K8S API ##
#################################
# Point to the internal API server hostname
API_SERVER_URL=https://kubernetes.default.svc
# Path to ServiceAccount token
# The service account is mapped by the K8S API server in the pods
SERVICE_ACCOUNT_FOLDER=/var/run/secrets/kubernetes.io/serviceaccount
# Read this Pod's namespace if required
# NAMESPACE=$(cat ${SERVICE_ACCOUNT_FOLDER}/namespace)
# Read the ServiceAccount bearer token
TOKEN=$(cat ${SERVICE_ACCOUNT_FOLDER}/token)
# Reference the internal certificate authority (CA)
CACERT=${SERVICE_ACCOUNT_FOLDER}/ca.crt
# Explore the API with TOKEN and the Certificate
curl -X GET \
--cacert ${CACERT} \
--header "Authorization: Bearer ${TOKEN}" \
${API_SERVER_URL}/api
You can use kubectl without your kubeconfig file. Your pod is launched with a service account. And all kubectl commands will be executed with the service account privileges. So you have to use rbac to grant access rights to that service account first.
I have a Kubernetes cluster in Digital Ocean, I want to pull the images from a private repository in GCP.
I tried to create a secret that make me able to to pull the images following this article https://blog.container-solutions.com/using-google-container-registry-with-kubernetes
Basically, these are the steps
In the GCP account, create a service account key, with a JSON credential
Execute
kubectl create secret docker-registry gcr-json-key \
--docker-server=gcr.io \
--docker-username=_json_key \
--docker-password="$(cat ~/json-key-file.json)" \
--docker-email=any#valid.email
In the deployment yaml reference the secret
imagePullSecrets:
- name: gcr-json-key
I don't understand why I am getting 403. If there are some restriccions to use the registry outside google cloud, or if I missed some configuration something.
Failed to pull image "gcr.io/myapp/backendnodeapi:latest": rpc error: code = Unknown desc = failed to pull and unpack image "gcr.io/myapp/backendnodeapi:latest": failed to resolve reference "gcr.io/myapp/backendnodeapi:latest": unexpected status code [manifests latest]: 403 Forbidden
Verify that you have enabled the Container Registry API, Installed Cloud SDK and Service account you are using for authentication has permissions to access Container Registry.
Docker requires privileged access to interact with registries. On Linux or Windows, add the user that you use to run Docker commands to the Docker security group.
This documentation has details on prerequisites for container registry.
Note:
Ensure that the version of kubectl is the latest version.
I tried replicating by following the document you provided and it worked at my end, So ensure that all the prerequisites are met.
That JSON string is not a password.
The documentation suggests to either activate the service account:
gcloud auth activate-service-account [USERNAME]#[PROJECT-ID].iam.gserviceaccount.com --key-file=~/service-account.json
Or add the configuration to $HOME/.docker/config.json
And then run docker-credential-gcr configure-docker.
Kubernetes seems to demand a service-account token secret
and this requires annotation kubernetes.io/service-account.name.
Also see Configure Service Accounts for Pods.
How can Apache Airflow's KubernetesPodOperator pull docker images from a private repository?
The KubernetesPodOperator has an image_pull_secrets which you can pass a Secrets object to authenticate with the private repository. But the secrets object can only represent an environment variable, or a volume - neither of which fit my understanding of how Kubernetes uses secrets to authenticate with private repos.
Using kubectl you can create the required secret with something like
$ kubectl create secret docker-registry $SECRET_NAME \
--docker-server=https://${ACCOUNT}.dkr.ecr.${REGION}.amazonaws.com \
--docker-username=AWS \
--docker-password="${TOKEN}" \
--docker-email="${EMAIL}"
But how can you create the authentication secret in Airflow?
There is secret object with docker-registry type according to kubernetes documentation which can be used to authenticate to private repository.
As You mentioned in Your question; You can use kubectl to create secret of docker-registry type that you can then try to pass with image_pull_secrets.
However depending on platform You are using this might have limited or no use at all according to kubernetes documentation:
Configuring Nodes to Authenticate to a Private Registry
Note: If you are running on Google Kubernetes Engine, there will already be a .dockercfg on each node with credentials for Google Container Registry. You cannot use this approach.
Note: If you are running on AWS EC2 and are using the EC2 Container Registry (ECR), the kubelet on each node will manage and update the ECR login credentials. You cannot use this approach.
Note: This approach is suitable if you can control node configuration. It will not work reliably on GCE, and any other cloud provider that does automatic node replacement.
Note: Kubernetes as of now only supports the auths and HttpHeaders section of docker config. This means credential helpers (credHelpers or credsStore) are not supported.
Making this work on mentioned platforms is possible but it would require automated scripts and third party tools.
Like in Amazon ECR example: Amazon ECR Docker Credential Helper would be needed to periodically pull AWS credentials to docker registry configuration and then have another script to update kubernetes docker-registry secrets.
As for Airflow itself I don't think it has functionality to create its own docker-repository secrets.
You can request functionality like that in Apache Airflow JIRA.
P.S.
If You still have issues with Your K8s cluster you might want to create new question on stack addressing them.
I want register kubernetes-elastic-agents with gocd-server. In the doc https://github.com/gocd/kubernetes-elastic-agents/blob/master/install.md
I need kubernetes security token and cluster ca certificate. My Kubernetes is running. How do I create a security token? Where can I find the cluster ca cert?
Jake
There are two answers:
The first is that it's very weird that one would need to manually input those things since they live in a well-known location on disk of any Pod (that isn't excluded via the automountServiceAccountToken field) as described in Accessing the API from a Pod
The second is that if you really do need a statically provisioned token belonging to a ServiceAccount, then you can either retrieve an existing token from the Secret that is created by default for every ServiceAccount, or create a second Secret as described in Manually create a service account API token
The CA cert you requested is present in every Pod in the cluster at the location mentioned in the first link, as well as in the ~/.kube/config of anyone who wishes to access the cluster. kubectl config view -o yaml will show it to you.
I have multiple kubernetes clusters running on GKE (let's say clusterA and clusterB)
I want to access both of those clusters from client-go in an app that is running in one of those clusters (e.g. access clusterB from an app that is running on clusterA)
I general for authenticating with kubernetes clusters from client-go I see that I have two options:
InCluster config
or from kube config file
So it is easy to access clusterA from clusterA but not clusterB from clusterA.
What are my options here? It seems that I just cannot pass GOOGLE_APPLICATION_CREDENTIALS and hope that client-go will take care of itself.
So my thinking:
create a dedicated IAM service account
create kube config with tokens for both clusters by doing gcloud container clusters get-credentials clusterA and gcloud container clusters get-credentials clusterB
use that kube config file in client-go via BuildConfigFromFlags on clusterA
Is this the correct approach, or is there a simpler way? I see that tokens have an expiration date?
Update:
It seems I can also use CLOUDSDK_CONTAINER_USE_CLIENT_CERTIFICATE=True gcloud beta container clusters get-credentials clusterB --zone. Which would add certificates to kube conf which I could use. But AFAIK those certificates cannot be revoked
client-go needs to know about:
cluster master’s IP address
cluster’s CA certificate
(If you're using GKE, you can see these info in $HOME/.kube/config, populated by gcloud container clusters get-credentials command).
I recommend you to either:
Have a kubeconfig file that contains these info for clusters A & B
Use GKE API to retrieve these info for clusters A & B (example here) (You'll need a service account to do this, explained below.)
Once you can create a *rest.Config object in client-go, client-go will use the auth plugin that's specified in the kubeconfig file (or its in-memory equivalent you constructed). In gcp auth plugin, it knows how to retrieve a token.
Then, Create a Cloud IAM Service Account and give it "Container Developer" role. Download its key.
Now, you have two options:
Option 1: your program uses gcloud
gcloud auth activate-service-account --key-file=key.json
KUBECONFIG=a.yaml gcloud container clusters get-credentials clusterA
KUBECONFIG=b.yaml gcloud container clusters get-credentials clusterB
Then create 2 different *rest.Client objects, one created from a.yaml, another from b.yaml in your program.
Now your program will rely on gcloud binary to retrieve token every time your token expires (every 1 hour).
Option 2: use GOOGLE_APPLICATION_CREDENTIALS
Don't install gcloud to your program’s environment.
Set your key.json to GOOGLE_APPLICATION_CREDENTIALS environment
variable for your program.
Figure out a way to get cluster IP/CA (explained above) so you can
construct two different *rest.Config objects for cluster A & B.
Now your program will use the specified key file to get an access_token
to Google API every time it expires (every 1h).
Hope this helps.
P.S. do not forget to import _ "k8s.io/client-go/plugin/pkg/client/auth/gcp" in your Go program. This loads the gcp auth plugin!