Use multiple contexts with same user-name in kubectl config - kubernetes

I want to use multiple clusters with my kubectl so I either put everything into one config or add one config file per cluster to the KUBECONFIG env variable. That's all fine.
My problem is now, that I've users with the same user-name for each cluster but they use different client-key-data for each cluster (context) but somehow the context uses that user-name so it's not clear which user belongs to which cluster.
Better give an example:
Cluster 1:
apiVersion: v1
kind: Config
clusters:
- cluster:
server: https://10.11.12.13:8888
name: team-cluster
contexts:
- context:
cluster: team-cluster
user: kubernetes-admin
name: kubernetes-admin#team-cluster
users:
- name: kubernetes-admin
user:
client-certificate-data: XXYYYZZZ
client-key-data: XXXYYYZZZ
Cluster 2:
apiVersion: v1
kind: Config
clusters:
- cluster:
server: https://10.11.12.14:8888
name: dev-cluster
contexts:
- context:
cluster: dev-cluster
user: kubernetes-admin
name: kubernetes-admin#dev-cluster
users:
- name: kubernetes-admin
user:
client-certificate-data: AABBCC
client-key-data: AABBCC
As you see, in both cluster there's a user with name kubernetes-admin but from the context it's not clear which of those. Maybe there's another way to give it a unique identifier that is used by the context.
Maybe the solution is obvious but I've not found any example for such a case. Thanks for any help.

I had same issue with my config and found out that name in users is not username used to log in - its just name used to identify user section in config. In Your case only cert key is used to know who You are. So You can use:
users:
- name: kubernetes-admin-1
user:
client-certificate-data: AABBCC
client-key-data: AABBCC
- name: kubernetes-admin-2
user:
client-certificate-data: XXYYYZZZ
client-key-data: XXXYYYZZZ
and refer to that in context just by key:
contexts:
- context:
cluster: dev-cluster
user: kubernetes-admin-1
Full config:
apiVersion: v1
kind: Config
clusters:
- cluster:
server: https://10.11.12.13:8888
name: team-cluster
- cluster:
server: https://10.11.12.14:8888
name: dev-cluster
contexts:
- context:
cluster: team-cluster
user: kubernetes-admin-1
name: kubernetes-admin#team-cluster
- context:
cluster: dev-cluster
user: kubernetes-admin-2
name: kubernetes-admin#dev-cluster
users:
- name: kubernetes-admin-1
user:
client-certificate-data: XXYYYZZZ
client-key-data: XXXYYYZZZ
- name: kubernetes-admin-2
user:
client-certificate-data: AABBCC
client-key-data: AABBCC
For auth methods that require username it's used something like this:
users:
- name: kubernetes-admin-with-password
user:
username: kubernetes-admin
password: mySecretPass
Using more than one kubeconfig is not much comfortable - you need to specify them for each command. You can have as much contexts and users if You want in one config and select right context (and save selected context as default).

If you have multiple kubeconfig files in the KUBECONFIG variable, then kubectl internally merges them before usage (see here). So, if you have two users with the same name in your kubeconfig files, they will probably override each other and you get either one or the other.
The solution is to either use different names for the users in the various kubeconfig files, or to explicitly specify one of the kubeconfig files, e.g. kubectl --kubeconfig dev-cluster.conf or having only a single kubeconfig file in the KUBECONFIG variable at a time.
In general, I would recommend the first approach and use a unique name for each different set of credentials (i.e. user) across your entire local configuration.

User in kubeconfig doesn't matter it is used as a reference for respective context. actually the common-name(CN) from client cert being used for authorization.
Working config: (i copied all clusters ca,certs and keys to $HOME/pki dir)
apiVersion: v1
kind: Config
current-context: "c1"
preferences: {}
clusters:
- cluster:
certificate-authority: ../pki/c1_ca.pem
server: https://20.100.1.10:6443
name: c1
- cluster:
certificate-authority: ../pki/c2_ca.pem
server: https://20.100.1.8:6443
name: c2
- cluster:
certificate-authority: ../pki/c3_ca.pem
server: https://20.100.1.11:6443
name: c3
contexts:
- context:
cluster: c1
namespace: default
user: c1
name: c1
- context:
cluster: c2
namespace: default
user: c2
name: c2
- context:
cluster: c3
namespace: default
user: c3
name: c3
users:
- name: c1
user:
client-certificate: ../pki/c1_client_cert.pem
client-key: ../pki/c1_client_key.pem
- name: c2
user:
client-certificate: ../pki/c2_client_cert.pem
client-key: ../pki/c2_client_key.pem
- name: c3
user:
client-certificate: ../pki/c3_client_cert.pem
client-key: ../pki/c3_client_key.pem

Related

kubernetes: change the current/default context via kubectl command

I am doing an exercise from KodeKoud which provide the CKAD certification training.
The exercise has a my-kube-config.yml file located under root/. The file content is below:
(I ommited some unrelated parts)
apiVersion: v1
kind: Config
clusters:
- name: production
cluster:
certificate-authority: /etc/kubernetes/pki/ca.crt
server: https://controlplane:6443
- name: development
cluster:
certificate-authority: /etc/kubernetes/pki/ca.crt
server: https://controlplane:6443
- name: test-cluster-1
cluster:
certificate-authority: /etc/kubernetes/pki/ca.crt
server: https://controlplane:6443
contexts:
- name: test-user#production
context:
cluster: production
user: test-user
- name: research
context:
cluster: test-cluster-1
user: dev-user
users:
- name: test-user
user:
client-certificate: /etc/kubernetes/pki/users/test-user/test-user.crt
client-key: /etc/kubernetes/pki/users/test-user/test-user.key
- name: dev-user
user:
client-certificate: /etc/kubernetes/pki/users/dev-user/developer-user.crt
client-key: /etc/kubernetes/pki/users/dev-user/dev-user.key
current-context: test-user#development
The exercise asking me to:
use the dev-user to access test-cluster-1. Set the current context
to the right one so I can do that.
Since I see in the config file, there is a context named research which meets the requirement, so I run the following command to change the current context to the required one:
kubectl config use-context research
but the console gives me error: error: no context exists with the name: "research".
Ok, I guessed maybe the name with value research is not acceptable, maybe I have to follow the convention of <user-name>#<cluster-name>? I am not sure , but I then tried the following:
I modified the name from research to dev-user#test-cluster-1, so that context part becomes:
- name: dev-user#test-cluster-1
context:
cluster: test-cluster-1
user: dev-user
after that I run command: kubectl config use-context dev-user#test-cluster-1, but I get error:
error: no context exists with the name: "dev-user#test-cluster-1"
Why? Based on the course material that is the way to chagne the default/current context. Is the course out-dated that I am using a deprecated one? What is the problem?
Your initial idea was correct. You would need to change the context to research which can be done using
kubectl config use-context research
But the command would not be applied to the correct config in this instance. You can see the difference by checking the current-context with and without a kubeconfig directed to the my-kube-config file.
kubectl config current-context
kubernetes-admin#kubernetes
kubectl config --kubeconfig=/root/my-kube-config current-context
test-user#development
So run the use-context command with the correct kubeconfig
kubectl config --kubeconfig=/root/my-kube-config use-context research
To able to change context, you have to edit $HOME/.kube/config file with your config data and merge with default one. I've tried to replicate your config file and it was possible to change the context, however your config file looks very strange.
See the lines from my console for your reference:
bazhikov#cloudshell:~ (nb-project-326714)$ kubectl config view
apiVersion: v1
clusters:
- cluster:
certificate-authority: /etc/kubernetes/pki/ca.crt
server: https://controlplane:6443
name: development
- cluster:
certificate-authority-data: DATA+OMITTED
server: https://35.246.22.167
name: gke_nb-project-326714_europe-west2_cluster-west2
...
...
...
- name: test-user
user:
client-certificate: /etc/kubernetes/pki/users/test-user/test-user.crt
client-key: /etc/kubernetes/pki/users/test-user/test-user.key
bazhikov#cloudshell:~ (nb-project-326714)$ kubectl config use-context research
Switched to context "research".
Copy your default config file prior editing if you don't want to ruin your cluster config :)

adding environment variable inside kubernetes config file

I am trying to set my company proxy inside my KUBECONFIG file hoping it would be picked up when i run kubectl from command line. I have tried many things but nothing helps so far. Here is the my config file.
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURqRENDQW5TZ0F3SUJBZ0lVRmdMaWRLWWlCcHlqbERBdEgvenRPRTE3ZTVBd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1hqRUxNQWtHQTFVRUJoTUNWVk14RGpBTUJnTlZCQWdUQlZSbGVHRnpNUTh3RFFZRFZRUUhFd1pCZFhOMAphVzR4RHpBTkJnTlZCQW9UQms5eVlXTnNaVEVNTUFvR0ExVUVDeE1EVDBSWU1ROHdEUVlEVlFRREV3WkxPRk1nClEwRXdIaGNOTWpBd05USXhNVGMxTlRBd1doY05NalV3TlRJd01UYzFOVEF3V2pCZU1Rc3dDUVlEVlFRR0V3SlYKVXpFT01Bd0dBMVVFQ0JNRlZHVjRZWE14RHpBTkJnTlZCQWNUQmtGMWMzUnBiakVQTUEwR0ExVUVDaE1HVDNKaApZMnhsTVF3d0NnWURWUVFMRXdOUFJGZ3hEekFOQmdOVkJBTVRCa3M0VXlCRFFUQ0NBU0l3RFFZSktvWklodmNOCkFRRUJCUUFEZ2dFUEFEQ0NBUW9DZ2dFQkFLYWFxcFBMVXJTcVh0N1d2Rm5zZEpnTHdQUWlaelhLci9JTzJDQVcKdklFa3VEOWt5VDlnNWQ5RzNwZFlkdW53THhLcG1DOE1VdWdBZmZ3VTFSNDNNSGNXK3MxTzFKS0dnd3hzMElyVApGRkZLZ0lEMTBDMXY3Wkp3amNPY0JvWXZXUTJ4TjN6czBITEt5cGMvY2Y5ZkpMTy9zWWJ4aXNQMDNZdHNGajMrClVUckNJS25XSWRyWlhqeEI5YVJKcmtXbVpKMTlIUG9oUE5TT2hYOTdLVDNJTnZIT1JFdldIbnZMVmN5VXlqWkUKQnBCcHNlc3N0aHcvNDBNOHRUSTJMdExFQzRKbE9NdXl6azB4Z0hJRGtKSzlCa214cVkwdkE4Y3RxVTEvbjRNeQpiRlRmV1poYmFwTW9FcGJINmZFRU9FalVoVmwzdjdTYWswMWRiek10L0RPdkd5OENBd0VBQWFOQ01FQXdEZ1lEClZSMFBBUUgvQkFRREFnRUdNQThHQTFVZEV3RUIvd1FGTUFNQkFmOHdIUVlEVlIwT0JCWUVGSWJKQmI3QTMzbEEKb1Z5aHBnZ0JsRDFJekhPRU1BMEdDU3FHU0liM0RRRUJDd1VBQTRJQkFRQ1MzRytEWTVKdTdIelFORjRUa2g1Mgp4cE1zdG84SzZBaGpWb2dqc21kWDQ5dlQ1WFdOU05GclJLeHdDcWIrU09DQ25hVTZvUFBPZVltSWI1YnBNcVZDCmsrYm9INUVXY2F1QzNWeWZCenppeTh0cktZdnZvam1PYTlBYkJnbHhNUVJ5VjNtQnJOZ0hGZktwaHV3N2FwZ0EKbWFrRWQwWjZTcXMzMSs0KzNGREJRL0Y4N0hpQ3hkbTZ4YmM0ayt3WFZPWFU3V3JEQlJ4cFRXT2J3bjNtWnRYeQpLYmw0UnBISGVOMnVkSFR2bE1rT3RCNHRlMGwrRURDbzFtbzZuZmJIM0w2QUJ5b3FyV0p1RzNCcWYzMWs1bEJhClNVcWdGaENxb3lDNnhWN09iUFdiN3BCSjF1UWdWck1DeFVTV3djVndrVkxKNTJTaWlUNWIyMy9LQWRrWFlGa0UKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
server: https://xxx.xxx.xx.xx:xxxx
name: cluster-ctdozjqhfqt
contexts:
- context:
cluster: cluster-ctdozjqhfqt
namespace: xxxxxxxxxx
user: user-ctdozjqhfqt
name: context-ctdozjqhfqt
current-context: context-ctdozjqhfqt
kind: Config
preferences: {}
users:
- name: user-ctdozjqhfqt
user:
exec:
apiVersion: client.authentication.k8s.io/v1beta1
args:
- ce
- cluster
- generate-token
- --cluster-id
- xxxxxxx
- --region
- xxxxxxxx
- --profile
- xxxxxxxx
command: xxxx
env:
- name: HTTP_PROXY
value: "http://xxxxxxx"
- name: HTTPS_PROXY
value: "http://xxxxx"
If I set the same environment variable on my terminal and run kubectl again, it just works. Am i missing something or doing something wrong in my config file?
I think your problem is the PR - kubernetes-proxy-kubeconfig.
This PR could help people accessing Cloud KubeContext that needs an explicit proxy, while local context like Minikube / docker-desktop / OnPremise should not use the explicit proxy (a very logic 502 bad gateway).
ADDITIONAL INFO:
Supporting streaming endpoints over socks5 proxy is more involved, however it seems like this would be useful for many people using standard http proxies.
Take a look: http-proxy-kubeconfig, kubernetes-proxy-setup-kubeconfig.

kubectl does not work with multiple clusters config

I have ~/.kube/config with following content:
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: REDACTED
server: https://REDACTED.yl4.us-east-1.eks.amazonaws.com
name: kubernetes-jenkins
- cluster:
certificate-authority-data: REDACTED
server: https://REDACTED.sk1.us-east-1.eks.amazonaws.com
name: kuberntes-dev
contexts:
- context:
cluster: kubernetes-dev
user: aws-dev
name: aws-dev
- context:
cluster: kubernetes-jenkins
user: aws-jenkins
name: aws-jenkins
current-context: aws-dev
kind: Config
preferences: {}
users:
- name: aws-dev
user:
exec:
apiVersion: client.authentication.k8s.io/v1alpha1
args:
- token
- -i
- EKS_DEV_CLUSTER
command: heptio-authenticator-aws
env: null
- name: aws-jenkins
user:
exec:
apiVersion: client.authentication.k8s.io/v1alpha1
args:
- token
- -i
- EKS_JENKINS_CLUSTER
command: heptio-authenticator-aws
env: null
But when I'm trying to kubectl cluster-info I get:
Kubernetes master is running at http://localhost:8080
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
The connection to the server localhost:8080 was refused - did you specify the right host or port?
As far as I understand something wrong in my kubeconfig, but I don't see what exactly.
Also I tried to find any related issues, but with no luck.
Could you suggest me something?
Thanks.
You need to choose the context that you'd like to use. More informantion on how use multiple clusters with multiple users here.
Essentially, you can view your current context (for the current cluster configured)
$ kubectl config current-context
To view, all the clusters configured:
$ kubectl config get-clusters
And to choose your cluster:
$ kubectl config use-context <cluster-name>
There are options to set different users per cluster in case you have them defined in your ~/kube/config file.
Your cluster name has a typo in it (name: kuberntes-dev) compared with the reference in the context (cluster: kubernetes-dev)

error: the server doesn't have resource type "svc"

Admins-MacBook-Pro:~ Harshin$ kubectl cluster-info
Kubernetes master is running at http://localhost:8080
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
error: the server doesn't have a resource type "services"
i am following this document
https://docs.aws.amazon.com/eks/latest/userguide/getting-started.html?refid=gs_card
while i am trying to test my configuration in step 11 of configure kubectl for amazon eks
apiVersion: v1
clusters:
- cluster:
server: ...
certificate-authority-data: ....
name: kubernetes
contexts:
- context:
cluster: kubernetes
user: aws
name: aws
current-context: aws
kind: Config
preferences: {}
users:
- name: aws
user:
exec:
apiVersion: client.authentication.k8s.io/v1alpha1
command: heptio-authenticator-aws
args:
- "token"
- "-i"
- "kunjeti"
# - "-r"
# - "<role-arn>"
# env:
# - name: AWS_PROFILE
# value: "<aws-profile>"
Change "name: kubernetes" to actual name of your cluster.
Here is what I did to work it through....
1.Enabled verbose to make sure config files are read properly.
kubectl get svc --v=10
2.Modified the file as below:
apiVersion: v1
clusters:
- cluster:
server: XXXXX
certificate-authority-data: XXXXX
name: abc-eks
contexts:
- context:
cluster: abc-eks
user: aws
name: aws
current-context: aws
kind: Config
preferences: {}
users:
- name: aws
user:
exec:
apiVersion: client.authentication.k8s.io/v1alpha1
command: aws-iam-authenticator
args:
- "token"
- "-i"
- "abc-eks"
# - "-r"
# - "<role-arn>"
env:
- name: AWS_PROFILE
value: "aws"
I have faced a similar issue, however this is not a direct solution but workaround. Use AWS cli commands to create cluster rather than console. As per documentation, the user or role which creates cluster will have master access.
aws eks create-cluster --name <cluster name> --role-arn <EKS Service Role> --resources-vpc-config subnetIds=<subnet ids>,securityGroupIds=<security group id>
Make sure that EKS Service Role has IAM access(I have given Full however AssumeRole will do I guess).
The EC2 machine Role should have eks:CreateCluster and IAM access. Worked for me :)
I had this issue and found it was caused default key setting in ~/.aws/credentials.
We have a few AWS accounts for different customers plus a sandbox account for our own testing and research. So our credentials file looks something like this:
[default]
aws_access_key_id = abc
aws_secret_access_key = xyz
region=us-east-1
[cpproto]
aws_access_key_id = abc
aws_secret_access_key = xyz
region=us-east-1
[sandbox]
aws_access_key_id = abc
aws_secret_access_key = xyz
region=us-east-1
I was messing around our sandbox account but the [default] section was pointing to another account.
Once I put the keys for sandbox into the default section the "kubectl get svc" command worked fine.
Seems we need a way to tell aws-iam-authenticator which keys to use same as --profile in the aws cli.
I guess you should uncomment "env" item and change your refer to ~/.aws/credentials
Because your aws_iam_authenticator requires exact AWS credentials.
Refer this document: https://docs.aws.amazon.com/eks/latest/userguide/create-kubeconfig.html
To have the AWS IAM Authenticator for Kubernetes always use a specific named AWS credential profile (instead of the default AWS credential provider chain), uncomment the env lines and substitute with the profile name to use.

Passing certificates/keys in kubeconfig throws an JSON error

Using Kubectl client (1.7.0) on Windows to connect to remote cluster.
The config file in Windows ( located in .kube) directory is configured as follows:
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: C:\Users\DK05478\.kube\ca.crt
server: https://10.99.70.153:6443
name: devo
contexts:
- context:
cluster: devo
user: admindevo
name: devo
current-context: devo
kind: Config
preferences: {}
users:
- name: admindevo
user:
client-certificate-data: C:\Users\DK05478\.kube\apiserver.crt
client-key-data: C:\Users\DK05478\.kube\apiserver.key
These certificate files I have downloaded from the remote system to my localhost. But this does not work. Throws the following error->
C:\Windows\kubernetes>kubectl version
Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.0", GitCommit:"d3ada0119e776222f11ec7945e6d860061339aad", GitTreeState:"clean", BuildDate:"2017-06-29T23:15:59Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"windows/amd64"}
error: Error loading config file "C:\Users\DK05478/.kube/config": [pos 107]: json: error decoding base64 binary 'C:\Users\DK05478\.kube\ca.crt': illegal base64 data at input byte 1
How can I fix this issue? whats that I am doing wrong ?
Remove the -data suffix from certificate-authority-data, client-certificate-data and client-key-data. Like what #sfgroups said, the xxx-data param is meant for a base64-encoded cert/key.
Once you do, your kubeconfig should look like this:
apiVersion: v1
clusters:
- cluster:
certificate-authority: C:\Users\DK05478\.kube\ca.crt
server: https://10.99.70.153:6443
name: devo
contexts:
- context:
cluster: devo
user: admindevo
name: devo
current-context: devo
kind: Config
preferences: {}
users:
- name: admindevo
user:
client-certificate: C:\Users\DK05478\.kube\apiserver.crt
client-key: C:\Users\DK05478\.kube\apiserver.key
certificate-authority-data: , client-certificate-data:, client-key-data: reference to the file. I think you need base64 encoded keys value here. you can look .kube/config file from your cluster master.
look at this page for base64 usage example https://kubernetes.io/docs/concepts/configuration/secret/