I've created a service account for CI purposes and am testing it out. Upon trying any kubectl command, I get the error:
error: You must be logged in to the server (Unauthorized)
Below is my .kube/config file
apiVersion: v1
clusters:
- cluster:
server: <redacted>
name: kubernetes
contexts:
- context:
cluster: kubernetes
user: bamboo
name: default
current-context: 'default'
kind: Config
preferences: {}
users:
- name: bamboo
user:
token: <redacted>
The service account exists and has a cluster role: edit and cluster role binding attached.
What am I doing wrong?
I reproduce the error if I copy the token directly without decoding. Then applied the following steps to decode and set the token and it is working as expected.
$ TOKENNAME=`kubectl -n <namespace> get serviceaccount/<serviceaccount-name> -o jsonpath='{.secrets[0].name}'`
$ TOKEN=`kubectl -n <namespace> get secret $TOKENNAME -o jsonpath='{.data.token}'| base64 --decode`
$ kubectl config set-credentials <service-account-name> --token=$TOKEN
So, I think it might be your case.
Related
I inherited a couple server running Kubernetes. And one of the things secops wants me to do is install an agent on the server. One of the first commands to run is
kubectl create secret generic
Running this, I am prompted for username and password. No one here knows what this is b/c the dev who set up the server is gone. So I don't know how to run this command and get passed the username/password. An obvious suggestion from someone else was using default user/pass but I can't even find that online. Found this to help get info on the server:
kubectl config view
Output of this command:
apiVersion: v1
clusters: null
contexts: null
current-context: ""
kind: Config
preferences: {}
users: null
Server:
CentOS Linux release 7.9.2009
Kernel - 5.17.2-1.el7.elrepo.x86_64
Any help is appreciated.
It is plausible that the kubeconfig file you are using is corrupt. You can reproduce similar symptoms(user/pass prompt) by editing the user name in your kubeconfig file. You need to find out(or create) the right kubeconfig file for the user. If you are an admin, you can find it at /etc/kubernetes/admin.conf in the master node.
Here are steps to reproduce the issue:
// This is my kubeconfig file, working fine
kubectl config view
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: DATA+OMITTED
server: https://127.0.0.1:6443
name: default
contexts:
- context:
cluster: default
user: default
name: default
current-context: default
kind: Config
preferences: {}
users:
- name: default
user:
client-certificate-data: REDACTED
client-key-data: REDACTED
// I searched for the user name
kubectl config view |grep 'user: default'
user: default
// corrupted the user name from default to default1
sed -i.bak 's/user: default/user: default1/g' ~/.kube/config
// now getting prompted for user/password
kubectl get pod --kubeconfig .kube/config
Please enter Username:
^C
//reverted the changes done earlier
sed -i 's/user: default1/user: default/g' ~/.kube/config
// commands working fine now
kubectl get pod --kubeconfig .kube/config
No resources found in default namespace.
I am using kubectl to connect remote kubernetes cluster(v1.15.2),I am copy config from remote server to local macOS:
scp -r root#ip:~/.kube/config ~/.kube
and change the url to https://kube-ctl.example.com,I exposed the api server to the internet:
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURvakNDQW9xZ0F3SUJBZ0lVU3FpUlZSU3FEOG1PemRCT1MyRzlJdGE0R2Nrd0RRWUpLb1pJaHZjTkFRRUwKQlFB92FERUxNQWtHQTFVRUJoTUNRMDR4RURBT0JnTlZCQWdUQjBKbGFVcHBibWN4RURBT0JnTlZCQWNUQjBKbAphVXBwYm1jeEREQUtCZ05WQkFvVEEyczRjekVTTUJBR0ExVUVDeE1KTkZCaGNtRmthV2R0TVJNd0VRWURWUVFECkV3cHJkV0psY201bGRHVnpNQ0FYR3RFNU1Ea3hNekUxTkRRd01Gb1lEekl4TVRrd09ESXdNVFUwTkRBd1dqQm8KTVFzd0NRWURWUVFHRXdKRFRqRVFNQTRHQTFVRUNCTUhRbVZwU21sdVp6RVFNQTRHQTFVRUJ4TUhRbVZwU21sdQpaekVNTUFvR0ExVUVDaE1EYXpoek1SSXdFQVlEVlFRTEV3azBVR0Z5WVdScFoyMHhFekFSQmdOVkJBTVRDbXQxClltVnlibVYwWlhNd2dnRWlNQTBHQ1NxR1NJYjNEUUVCQVFVQUE0SUJEd0F3Z2dFS0FvSUJBUUNzOGFFR2R2TUgKb0E1eTduTjVydnAvQkEyTVM1RG1TNWwzQ0p4S3VMOGJ1bkF3alF1c0lTalUxVWlqeVdGOW03VzA3elZJaVJpRwpiYzZGT3JkSEJ2QXgzazBpT2pPRlduTHp1UjdaSFhqQ3lTbDJRam9YN3gzL0l1MERQelNHTXJLSzZETGpTRTk4CkdadEpjUi9OSmpiVFFJc3FXbWFEdUIyc3dmcEc1ZmlZU1A1KzVpcld1TG1pYjVnWnJYeUJJNlJ0dVV4K1NvdW0KN3RDKzJaVG5QdFF0QnFUZHprT3p3THhwZ0Zhd1kvSU1mbHBsaUlMTElOamcwRktxM21NOFpUa0xvNXcvekVmUApHT25GNkNFWlR6bkdrTWc2aUVGenNBcDU5N2lMUDBNTkR4YUNjcTRhdTlMdnMzYkdOZmpqdDd3WkxIVklLa0lGCm44Mk92cExGaElq2kFnTUJBQUdqUWpCQU1BNEdBMVVkRHdFQi93UUVBd0lCQmpBUEJnTlZIUk1CQWY4RUJUQUQKQVFIL01CMEdBMVVkRGdRV0JCUm0yWHpJSHNmVzFjMEFGZU9SLy9Qakc4dWdzREFOQmdrcWhraUc5dzBCQVFzRgpBQU9DQVFFQW1mOUozN3RYTys1dWRmT2RLejdmNFdMZyswbVJUeTBRSEVIblk5VUNLQi9vN2hLUVJHRXI3VjNMCktUeGloVUhvbHY1QzVUdG8zbUZJY2FWZjlvZlp0VVpvcnpxSUFwNE9Od1JpSnQ1Yk94K1d6SW5qN2JHWkhnZjkKSk8rUmNxQnQrUWsrejhTMmJKRG04WFdvMW5WdjJRNU1pUndPdnRIbnRxd3MvTlJ2bHBGV25ISHBEVExjOU9kVwpoMllzWVpEMmV4d0FRVDkxSlExVjRCdklrZGFPeW9USHZ6U2oybThSTzh6b3JBd09kS1NTdG9TZXdpOEhMeGI2ClhmaTRFbjR4TEE3a3pmSHFvcDZiSFF1L3hCa0JzYi9hd29kdDJKc2FnOWFZekxEako3S1RNYlQyRW52MlllWnIKSUhBcjEyTGVCRGRHZVd1eldpZDlNWlZJbXJvVnNRPT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
server: https://k8s-ctl.example.com
name: kubernetes
contexts:
- context:
cluster: kubernetes
namespace: kube-system
user: admin
name: kubernetes
current-context: kubernetes
kind: Config
preferences: {}
users:
- name: admin
user:
when I get cluster pod info in my local Mac:
kubectl get pods --all-namespaces
give this error:
Unable to connect to the server: x509: certificate signed by unknown authority
when I access https://k8s-ctl.example.com in google chrome,the result is:
{
kind: "Status",
apiVersion: "v1",
metadata: { },
status: "Failure",
message: "Unauthorized",
reason: "Unauthorized",
code: 401
}
what should I do to make access remote k8s cluster sucess using kubectl client?
One way I have tried to using this .kube/config generate by command,but get the same result:
apiVersion: v1
clusters:
- cluster:
certificate-authority: ssl/ca.pem
server: https://k8s-ctl.example.com
name: default
contexts:
- context:
cluster: default
user: admin
name: default
current-context: default
kind: Config
preferences: {}
users:
- name: admin
user:
client-certificate: ssl/admin.pem
client-key: ssl/admin-key.pem
I've reproduced your problem and as you created your cluster following kubernetes-the-hard-way, you need to follow these steps to be able to access your cluster from a different console.
First you have to copy the following certificates created while you was bootstraping your cluster to ~/.kube/ directory in your local machine:
ca.pem
admin.pem
admin-key.pem
After copying these files to your local machine, execute the following commands:
kubectl config set-cluster kubernetes-the-hard-way \
--certificate-authority=~/.kube/ca.pem \
--embed-certs=true \
--server=https://${KUBERNETES_PUBLIC_ADDRESS}:6443
kubectl config set-credentials admin \
--client-certificate=~/.kube/admin.pem \
--client-key=~/.kube/admin-key.pem
kubectl config set-context kubernetes-the-hard-way \
--cluster=kubernetes-the-hard-way \
--user=admin
kubectl config use-context kubernetes-the-hard-way
Notice that you have to replace the ${KUBERNETES_PUBLIC_ADDRESS} variable with the remote address to your cluster.
When kubectl interacts with kube API server it will validate the kube API server certificate as well as send the certificate in client-certificate to the kube API server for mutual TLS authentication. I believe the problem is either of below.
the ca that you have used to generate the client-certificate is not the ca that has been used to startup the kube API server.
The ca in certificate-authority-data is not the ca used to generate kube API server certificate.
If you make sure that you are using same ca to generate all the certificates consistently across the board then it should work.
[xueke#master-01 admin]$ kubectl logs nginx-deployment-76bf4969df-999x8
Error from server (Forbidden): Forbidden (user=system:anonymous, verb=get, resource=nodes, subresource=proxy) ( pods/log nginx-deployment-76bf4969df-999x8)
[xueke#master-01 admin]$ kubectl config view
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: DATA+OMITTED
server: https://192.168.0.101:6443
name: kubernetes
contexts:
- context:
cluster: kubernetes
user: admin
name: kubernetes
current-context: kubernetes
kind: Config
preferences: {}
users:
- name: admin
user:
client-certificate-data: REDACTED
client-key-data: REDACTED
I specified the admin user here
How do I need to modify it?
The above error means your apiserver doesn't have the credentials (kubelet cert and key) to authenticate the kubelet's log/exec commands and hence the Forbidden error message.
You need to provide --kubelet-client-certificate=<path_to_cert> and --kubelet-client-key=<path_to_key> to your apiserver, this way apiserver authenticate the kubelet with the certficate and key pair.
For more information, have a look at:
https://kubernetes.io/docs/reference/access-authn-authz/kubelet-authn-authz/
In our case, the error stemmed from Azure services being downgraded because of a bug in DNS resolution, introduced in Ubuntu 18.04.
See Azure status and the technical thread. I ran this command to set a fallback DNS address in the nodes:
az vmss list-instances -g <resourcegroup> -n vmss --query "[].id" --output tsv \
| az vmss run-command invoke --scripts "echo FallbackDNS=168.63.129.16 >> /etc/systemd/resolved.conf; systemctl restart systemd-resolved.service" --command-id RunShellScript --ids #-
That's an RBAC error. The user had no permissions to see logs. If you have a user with cluster-admin permissions you can fix this error with
kubectl create clusterrolebinding the-boss --user system:anonymous --clusterrole cluster-admin
Note: Not a good idea to give an anonymous user cluster-admin role. Will fix the issue though.
I'm encountering some really weird behaviour while attempting to switch contexts using kubectl.
My config file declares two contexts; one points to an in-house cluster, while the other points to an Amazon EKS cluster.
apiVersion: v1
kind: Config
clusters:
- cluster:
certificate-authority-data: <..>
server: <..>
name: in-house
- cluster:
certificate-authority-data: <..>
server: <..>
name: eks
contexts:
- context:
cluster: in-house
user: divesh-in-house
name: in-house-context
- context:
cluster: eks
user: divesh-eks
name: eks-context
current-context: in-house-context
preferences: {}
users:
- name: divesh-eks
user:
exec:
apiVersion: client.authentication.k8s.io/v1alpha1
command: aws-iam-authenticator
args:
- "token"
- "-i"
- "eks"
env: null
- name: divesh-in-house
user:
client-certificate-data: <..>
client-key-data: <..>
I'm also using the aws-iam-authenticator to authenticate to the EKS cluster.
My problem is this - as long as I work with the in-house cluster, everything works fine. But, when I execute kubectl config use-context eks-context, I observe the following behaviour.
Any operation I try to perform on the cluster (say, kubectl get pods -n production) shows me a Please enter Username: prompt. I assumed the aws-iam-authenticator should have managed the authentication for me. I can confirm that running the authenticator manually (aws-iam-authenticator token -i eks) works fine for me.
Executing kubectl config view omits the divesh-eks user, so the output looks like
users:
- name: divesh-eks
user: {}
Switching back to the in-house cluster by xecuting kubectl config use-context in-house-context modifies my config file and deletes the divesh-eks-user, so the config file now contains
users:
- name: divesh-eks
user: {}
My colleagues don't seem to face this problem.
Thoughts?
The exec portion of that config was added in 1.10 (https://github.com/kubernetes/kubernetes/pull/59495)
If you use a version of kubectl prior to that version, it will not recognize the exec plugin (resulting in prompts for credentials), and if you use it to make kubeconfig changes, it will drop the exec field when it persists the changes
I see spinnaker using system:anonymous user to authenticate K8s. But I want a specific user(which I created already in K8s) to authenticate K8s. I used below kubeconfig to use user veeru
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: RETRACTED
server: https://xx.xx.xx.220:8443
name: xx-xx-xx-220:8443
contexts:
- context:
cluster: xx-xx-xx-220:8443
namespace: default
user: veeru/xx-xx-xx-220:8443
name: area-51/xx-xx-xx-220:8443/veeru
current-context: area-51/xx-xx-xx-220:8443/veeru
kind: Config
preferences: {}
users:
- name: veeru/xx-xx-xx-220:8443
user:
client-certificate-data: RETRACTED
client-key-data: RETRACTED
And I specified(like here) user in config(~/.hal/config) like below
kubernetes:
enabled: true
accounts:
- name: my-k8s-account
requiredGroupMembership: []
providerVersion: V1
dockerRegistries:
- accountName: my-docker-registry2
namespaces: []
configureImagePullSecrets: true
namespaces: ["area-51"]
user: veeru
omitNamespaces: []
kubeconfigFile: /home/ubuntu/.kube/config
oauthScopes: []
oAuthScopes: []
primaryAccount: my-k8s-account
But still spinnaker is using system:anonymous
2018-01-22 08:35:13.929 ERROR 4639 --- [pool-4-thread-1] c.n.s.c.o.DefaultOrchestrationProcessor : com.netflix.spinnaker.clouddriver.kubernetes.v1.deploy.exception.KubernetesOperationException: Get Service openshifttest-dev in area-51 for account my-k8s-account failed: User "system:anonymous" cannot get services in the namespace "area-51": User "system:anonymous" cannot get services in project "area-51"
Is there any way to specify user that spinnaker should use configured user other than system:anonymous
UPDATE-1
Followed: https://blog.spinnaker.io/spinnaker-kubernetes-rbac-c40f1f73c172
Got the secret from kubectl describe secret spinnaker-service-account-token-9sl6q and update in kubeconfig like below
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: REDACTED
server: https://xx.xx.xx.xx:6443
name: kubernetes
contexts:
- context:
cluster: kubernetes
namespace: webapp
user: kubernetes-admin
name: kubernetes-admin#kubernetes
- context:
cluster: kubernetes
user: spinnaker-service-account
name: spinnaker-context
current-context: spinnaker-context
kind: Config
preferences: {}
users:
- name: kubernetes-admin
user:
client-certificate-data: REDACTED
client-key-data: REDACTED
- name: spinnaker-service-account
user:
token: eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9....
Than I ran sudo hal deploy
....
! ERROR Unable to communicate with your Kubernetes cluster: Failure
executing: GET at: https://xx.xx.xx.xx:6443/api/v1/namespaces. Message:
Forbidden! User spinnaker-service-account doesn't have permission. namespaces is
forbidden: User "system:serviceaccount:default:spinnaker-service-account" cannot
list namespaces at the cluster scope..
? Unable to authenticate with your Kubernetes cluster. Try using
kubectl to verify your credentials.
....
I'm able run
$ kubectl get namespace webapp
NAME STATUS AGE
webapp Active 22m
I have specified webapp namespace and user as spinnaker-service-account in ~/.hal/config
I'm using GKE with basic authentication disabled. I have my spinnaker use a dedicated K8s service account that I created. In my ~/.kube/config I have tokens for each K8s cluster.
users:
- name: gke_operation-covfefe-1_asia-east1_testing-asia-east1
user:
token: token1
- name: gke_operation-covfefe-1_europe-west1_testing-europe-west1
user:
token: token2
- name: gke_operation-covfefe-1_us-central1_testing-us-central1
user:
token: token3
I got these tokens by running
kubectl get secret spinnaker-service-account -o json \
| jq -r .data.token \
| base64 -d
and then manually updating my ~/.kube/config file.
Make sure your service account has the required RBAC permissions. See blog post here.
Update:
Also make sure you give the service account the required RBAC permissions. See the "Role" section of the blog post above or the guide here. When you test the RBAC permissions with kubectl make sure you're using the same service account as the one Spinnaker is using.
Update 2
If you want spinnaker to act on all namespaces, use ClusterRole and ClusterRoleBinding in your RBAC. The blog post only uses Role and RoleBinding which restricts actions to a particular namespace(s). See this guide for the Cluster* way. Note the PR to fix a typo here.