Problem deploying cockroachdb-client-secure - kubernetes

Question

I am following this helm + secure - guide:
https://www.cockroachlabs.com/docs/stable/orchestrate-cockroachdb-with-kubernetes.html#helm
I deployed the cluster with this command: $ helm install my-release --values my-values.yaml cockroachdb/cockroachdb --namespace=thesis-crdb
This is how it looks: $ helm list --namespace=thesis-crdb
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
my-release thesis-crdb 1 2021-01-31 17:38:52.8102378 +0100 CET deployed cockroachdb-5.0.4 20.2.4
Here is how it looks using: $ kubectl get all --namespace=thesis-crdb
NAME READY STATUS RESTARTS AGE
pod/my-release-cockroachdb-0 1/1 Running 0 7m35s
pod/my-release-cockroachdb-1 1/1 Running 0 7m35s
pod/my-release-cockroachdb-2 1/1 Running 0 7m35s
pod/my-release-cockroachdb-init-fhzdn 0/1 Completed 0 7m35s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/my-release-cockroachdb ClusterIP None <none> 26257/TCP,8080/TCP 7m35s
service/my-release-cockroachdb-public ClusterIP 10.xx.xx.x <none> 26257/TCP,8080/TCP 7m35s
NAME READY AGE
statefulset.apps/my-release-cockroachdb 3/3 7m35s
NAME COMPLETIONS DURATION AGE
job.batch/my-release-cockroachdb-init 1/1 43s 7m36s
In the my-values.yaml-file I only changed the tls from false to true:
tls:
enabled: true
So far so good, but from here on the guide isn't really working for me anymore. I try as they say with getting the csr: kubectl get csr --namespace=thesis-crdb
No resources found
Ok, perhaps not needed. I carry on to deploy the client-secure
I download the file: https://raw.githubusercontent.com/cockroachdb/cockroach/master/cloud/kubernetes/client-secure.yaml
And changes the serviceAccountName: cockroachdb to serviceAccountName: my-release-cockroachdb.
I try to deploy it with $ kubectl create -f client-secure.yaml --namespace=thesis-crdb but it throws this error:
Error from server (Forbidden): error when creating "client-secure.yaml": pods "cockroachdb-client-secure" is forbidden: error looking up service account thesis-crdb/my-release-cockroachdb: serviceaccount "my-release-cockroachdb" not found
Anyone got an idea how to solve this? I'm fairly sure it's something with the namespace that is messing it up.
I have tried to put the namespace in the metadata-section
metadata:
namespace: thesis-crdb
And then try to deploy it with: kubectl create -f client-secure.yaml but to no avail:
Error from server (Forbidden): error when creating "client-secure.yaml": pods "cockroachdb-client-secure" is forbidden: error looking up service account thesis-crdb/my-release-cockroachdb: serviceaccount "my-release-cockroachdb" not found

Answer 1

You mention in question that you have changed serviceAccountName in YAML.
And changes the serviceAccountName: cockroachdb to serviceAccountName: my-release-cockroachdb.
So Root Cause of your issue is related with ServiceAccount misconfiguration.
Background
In your cluster you have something called ServiceAccount.
When you (a human) access the cluster (for example, using kubectl), you are authenticated by the apiserver as a particular User Account (currently this is usually admin, unless your cluster administrator has customized your cluster). Processes in containers inside pods can also contact the apiserver. When they do, they are authenticated as a particular Service Account (for example, default).
To ServiceAccount you also should configure RBAC which grants you permissions to create resources.
Role-based access control (RBAC) is a method of regulating access to computer or network resources based on the roles of individual users within your organization.
RBAC authorization uses the rbac.authorization.k8s.io API group to drive authorization decisions, allowing you to dynamically configure policies through the Kubernetes API.
If you don't have proper RBAC permissions you will not be able to create resources.
In Kubernetes you can find Role and ClusterRole. Role sets permissions within a particular namespace and ClusterRole sets permissions in whole cluster.
Besides that, you also need to bind roles using RoleBinding and ClusterRoleBinding.
In addition, if you would use Cloud environment, you would also need special rights in project. Your guide provides instructions to do it here.
Root cause
I've checked cockroachdb chart and it creates ServiceAccount, Role, ClusterRole, RoleBinding and ClusterRoleBinding for cockroachdb and prometheus. There is no configuration for my-release-cockroachdb.
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: cockroachdb
...
verbs:
- create
- get
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: cockroachdb
labels:
app: cockroachdb
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cockroachdb
...
In client-secure.yaml you change serviceAccountName to my-release-cockroachdb and Kubernetes cannot find that ServiceAccount as it was not created by cluster administrator or cockroachdb chart.
To list ServiceAccounts in default namespace you can use command $ kubectl get ServiceAccount, however if you would check all ServiceAccounts in cluster you should add -A to your command - $ kubectl get ServiceAccount -A.
Solution
Option 1 is to use existing ServiceAccount with proper permissions like SA created by cockroachdb chart which is cockroachdb, not my-release-cockroachdb.
Option 2 is to create ServiceAccount, Role/ClusterRole and RoleBinding/ClusterRoleBinding for my-release-cockroachdb.