Error installing TimescaleDB with K8S / Helm : MountVolume.SetUp failed for volume "certificate" : secret "timescaledb-certificate" not found - kubernetes

I just tried to install timescaleDB Single with Helm in minikube on Ubuntu 20.04.
After installing via:
helm install timescaledb timescaledb/timescaledb-single --namespace espace-client-v2
I got the message:
➜ ~ helm install timescaledb timescaledb/timescaledb-single --namespace espace-client-v2
NAME: timescaledb
LAST DEPLOYED: Fri Aug 7 17:17:59 2020
NAMESPACE: espace-client-v2
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
TimescaleDB can be accessed via port 5432 on the following DNS name from within your cluster:
timescaledb.espace-client-v2.svc.cluster.local
To get your password for superuser run:
# superuser password
PGPASSWORD_POSTGRES=$(kubectl get secret --namespace espace-client-v2 timescaledb-credentials -o jsonpath="{.data.PATRONI_SUPERUSER_PASSWORD}" | base64 --decode)
# admin password
PGPASSWORD_ADMIN=$(kubectl get secret --namespace espace-client-v2 timescaledb-credentials -o jsonpath="{.data.PATRONI_admin_PASSWORD}" | base64 --decode)
To connect to your database, chose one of these options:
1. Run a postgres pod and connect using the psql cli:
# login as superuser
kubectl run -i --tty --rm psql --image=postgres \
--env "PGPASSWORD=$PGPASSWORD_POSTGRES" \
--command -- psql -U postgres \
-h timescaledb.espace-client-v2.svc.cluster.local postgres
# login as admin
kubectl run -i --tty --rm psql --image=postgres \
--env "PGPASSWORD=$PGPASSWORD_ADMIN" \
--command -- psql -U admin \
-h timescaledb.espace-client-v2.svc.cluster.local postgres
2. Directly execute a psql session on the master node
MASTERPOD="$(kubectl get pod -o name --namespace espace-client-v2 -l release=timescaledb,role=master)"
kubectl exec -i --tty --namespace espace-client-v2 ${MASTERPOD} -- psql -U postgres
It seemed to have installed well.
But then, when executing:
PGPASSWORD_POSTGRES=$(kubectl get secret --namespace espace-client-v2 timescaledb-credentials -o jsonpath="{.data.PATRONI_SUPERUSER_PASSWORD}" | base64 --decode)
Error from server (NotFound): secrets "timescaledb-credentials" not found
After that, I realized pod has not even been created, and it gives me the following errors
MountVolume.SetUp failed for volume "certificate" : secret "timescaledb-certificate" not found
Unable to attach or mount volumes: unmounted volumes=[certificate], unattached volumes=[storage-volume wal-volume patroni-config timescaledb-scripts certificate socket-directory timescaledb-token-svqqf]: timed out waiting for the condition
What should I do ?

I could do it. If the page https://github.com/timescale/timescaledb-kubernetes doesn't give much details about installation process, you can go here:
https://github.com/timescale/timescaledb-kubernetes/tree/master/charts/timescaledb-single
I had to use kustomize to generate content:
./generate_kustomization.sh my-release
and then it generate several files:
credentials.conf kustomization.yaml pgbackrest.conf timescaledbMap.yaml tls.crt tls.key
then I did:
kubectl kustomize ./
which generated a k8s yml file, which I saved with the name timescaledbMap.yaml
Finally, I did:
kubectl apply -f timescaledbMap.yaml
Then it created all necesarry secrets, and I could install chart
. Hope it helps others.

Related

Kubernetes image from local registry

I want to deploy a pod using a Docker image which has been pushed to a private registry.
So far, I've used the following command to install the registry and push the image:
# Build the DockerImage file
DOCKER_IMAGE="truc/tf-http-server:0.1"
cd docker
docker build -t $DOCKER_IMAGE .
cd ..
# Install Registry V2
docker run -d -p 5000:5000 --restart=always --name registry registry:2
# Push image
docker tag $DOCKER_IMAGE localhost:5000/$DOCKER_IMAGE
docker push localhost:5000/$DOCKER_IMAGE
# Add to known repository
sudo bash -c 'cat << EOF > /etc/docker/daemon.json
{
"insecure-registries" : [ "192.168.1.37:5000" ]
}
EOF'
sudo systemctl daemon-reload
sudo systemctl restart docker
Pulling the image works directly from Docker:
$ sudo docker pull 192.168.1.37:5000/truc/tf-http-server:0.1
0.1: Pulling from truc/tf-http-server
Digest: sha256:b09c10375f1e90346f9b0c4bfb2bdfc7df919a4c89aaebfb433f2d845b37a960
Status: Downloaded newer image for 192.168.1.37:5000/truc/tf-http-server:0.1
192.168.1.37:5000/truc/tf-http-server:0.1
When I want to deploy the image from Kubernetes, I got the following error:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 29s default-scheduler Successfully assigned default/tf-http-server-nvl9v to worker01
Normal Pulling 16s (x2 over 29s) kubelet Pulling image "192.168.1.37:5000/truc/tf-http-server:0.1"
Warning Failed 16s (x2 over 29s) kubelet Failed to pull image "192.168.1.37:5000/truc/tf-http-server:0.1": rpc error: code = Unknown desc = failed to pull and unpack image "192.168.1.37:5000/truc/tf-http-server:0.1": failed to resolve reference "192.168.1.37:5000/truc/tf-http-server:0.1": failed to do request: Head "https://192.168.1.37:5000/v2/truc/tf-http-server/manifests/0.1": http: server gave HTTP response to HTTPS client
Warning Failed 16s (x2 over 29s) kubelet Error: ErrImagePull
Normal BackOff 3s (x2 over 28s) kubelet Back-off pulling image "192.168.1.37:5000/truc/tf-http-server:0.1"
Warning Failed 3s (x2 over 28s) kubelet Error: ImagePullBackOff
It seems like if the repository access was forbidden. Is there a way to make it reachable from Kubernetes ?
EDIT: To install Docker registy, run the following commands and follow the checked answer.
mkdir registry && cd registry && mkdir certs && cd certs
openssl genrsa 1024 > domain.key
chmod 400 domain.key
openssl req -new -x509 -nodes -sha1 -days 365 -key domain.key -out domain.crt -subj "/C=FR/ST=France/L=Lannion/O=TGI/CN=OrangeFactoryBox"
cd .. && mkdir auth
sudo apt-get install apache2-utils -y
htpasswd -Bbn username password > auth/htpasswd
cd ..
docker run -d \
--restart=always \
--name registry \
-v `pwd`/auth:/auth \
-v `pwd`/certs:/certs \
-v `pwd`/certs:/certs \
-e REGISTRY_AUTH=htpasswd \
-e REGISTRY_AUTH_HTPASSWD_REALM="Registry Realm" \
-e REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd \
-e REGISTRY_HTTP_ADDR=0.0.0.0:5000 \
-e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt \
-e REGISTRY_HTTP_TLS_KEY=/certs/domain.key \
-p 5000:5000 \
registry:2
sudo docker login -u username -p password localhost:5000
Assumption: The docker server where you tested it and the kubernetes
nodes are on the same private subnet 192.168.1.0/24
http: server gave HTTP response to HTTPS client
So, apparently your private docker registry is HTTP not HTTPS. Kuberentes prefers the registry to use a valid SSL certificate. On each node in your kubernetes cluster, you will need to explicitly tell the docker to treat this registry as an insecure registry. Following this change you will have to restart the docker service as well.
Kubernetes: Failed to pull image. Server gave HTTP response to HTTPS client.
{ "insecure-registries":["192.168.1.37:5000"] }
to the daemon.json file at /etc/docker.
You will also need to define the imagePullSecrets in your namespace and use it in your deployment/pod spec
First create the secret from your <path/to/.docker/config.json> using:
kubectl create secret generic regcred \
--from-file=.dockerconfigjson=<path/to/.docker/config.json> \
--type=kubernetes.io/dockerconfigjson
Then refer to this secret in your pod yaml
apiVersion: v1
kind: Pod
metadata:
name: private-reg
spec:
containers:
- name: private-reg-container
image: <your-private-image>
imagePullSecrets:
- name: regcred
Reference: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/

How to connect to OpenLDAP which created by official helm chart?

Using Helm 3 installed OpenLDAP:
helm install openldap stable/openldap
Got this message:
NAME: openldap
LAST DEPLOYED: Sun Apr 12 13:54:45 2020
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
OpenLDAP has been installed. You can access the server from within the k8s cluster using:
openldap.default.svc.cluster.local:389
You can access the LDAP adminPassword and configPassword using:
kubectl get secret --namespace default openldap -o jsonpath="{.data.LDAP_ADMIN_PASSWORD}" | base64 --decode; echo
kubectl get secret --namespace default openldap -o jsonpath="{.data.LDAP_CONFIG_PASSWORD}" | base64 --decode; echo
You can access the LDAP service, from within the cluster (or with kubectl port-forward) with a command like (replace password and domain):
ldapsearch -x -H ldap://openldap.default.svc.cluster.local:389 -b dc=example,dc=org -D "cn=admin,dc=example,dc=org" -w $LDAP_ADMIN_PASSWORD
Test server health using Helm test:
helm test openldap
You can also consider installing the helm chart for phpldapadmin to manage this instance of OpenLDAP, or install Apache Directory Studio, and connect using kubectl port-forward.
However I can't use this command to search content on ldap server in the k8s cluster:
export LDAP_ADMIN_PASSWORD=[REAL_PASSWORD_GET_ABOVE]
ldapsearch -x -H ldap://openldap.default.svc.cluster.local:389 -b dc=example,dc=org -D "cn=admin,dc=example,dc=org" -w $LDAP_ADMIN_PASSWORD
Got error
ldap_sasl_bind(SIMPLE): Can't contact LDAP server (-1)
I also login to the pod to run
kubectl exec -it openldap -- /bin/bash
# export LDAP_ADMIN_PASSWORD=[REAL_PASSWORD_GET_ABOVE]
# ldapsearch -x -H ldap://openldap.default.svc.cluster.local:389 -b dc=example,dc=org -D "cn=admin,dc=example,dc=org" -w $LDAP_ADMIN_PASSWORD
The same.
As it's stated in the notes:
NOTES:
OpenLDAP has been installed. You can access the server from within the k8s cluster using:
openldap.default.svc.cluster.local:389
You can access the LDAP adminPassword and configPassword using:
kubectl get secret --namespace default openldap -o jsonpath="{.data.LDAP_ADMIN_PASSWORD}" | base64 --decode; echo
kubectl get secret --namespace default openldap -o jsonpath="{.data.LDAP_CONFIG_PASSWORD}" | base64 --decode; echo
You can access the LDAP service, from within the cluster (or with kubectl port-forward) with a command like (replace password and domain):
ldapsearch -x -H ldap://openldap.default.svc.cluster.local:389 -b dc=example,dc=org -D "cn=admin,dc=example,dc=org" -w $LDAP_ADMIN_PASSWORD
Test server health using Helm test:
helm test openldap
You can also consider installing the helm chart for phpldapadmin to manage this instance of OpenLDAP, or install Apache Directory Studio, and connect using kubectl port-forward.
You can do:
$ kubectl port-forward services/openldap 3389:389
Forwarding from 127.0.0.1:3389 -> 389
Forwarding from [::1]:3389 -> 389
Handling connection for 3389
From another shell, outside the Kubernetes cluster:
$ kubectl get secret --namespace default openldap -o jsonpath="{.data.LDAP_ADMIN_PASSWORD}" | base64 --decode; echo
l3dkQByvzKKboCWQRyyQl96ulnGLScIx
$ ldapsearch -x -H ldap://localhost:3389 -b dc=example,dc=org -D "cn=admin,dc=example,dc=org" -w l3dkQByvzKKboCWQRyyQl96ulnGLScIx
Also it was already mentioned in a comment by #Totem

Can't connect to postgres installed from stable/postgresql helm chart

I'm trying to install postgresql via helm. I'm not overriding any settings, but when I try to connect, I get a "password authentication failed" error:
$ helm install goatsnap-postgres stable/postgresql
NAME: goatsnap-postgres
LAST DEPLOYED: Mon Jan 27 12:44:22 2020
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
** Please be patient while the chart is being deployed **
PostgreSQL can be accessed via port 5432 on the following DNS name from within your cluster:
goatsnap-postgres-postgresql.default.svc.cluster.local - Read/Write connection
To get the password for "postgres" run:
export POSTGRES_PASSWORD=$(kubectl get secret --namespace default goatsnap-postgres-postgresql -o jsonpath="{.data.postgresql-password}" | base64 --decode)
To connect to your database run the following command:
kubectl run goatsnap-postgres-postgresql-client --rm --tty -i --restart='Never' --namespace default --image docker.io/bitnami/postgresql:11.6.0-debian-9-r0 --env="PGPASSWORD=$POSTGRES_PASSWORD" --command -- psql --host goatsnap-postgres-postgresql -U postgres -d postgres -p 5432
To connect to your database from outside the cluster execute the following commands:
kubectl port-forward --namespace default svc/goatsnap-postgres-postgresql 5432:5432 &
PGPASSWORD="$POSTGRES_PASSWORD" psql --host 127.0.0.1 -U postgres -d postgres -p 5432
$ kubectl get secret --namespace default goatsnap-postgres-postgresql -o jsonpath="{.data.postgresql-password}" | base64 --decode
DCsSy0s8hM
$ kubectl run goatsnap-postgres-postgresql-client --rm --tty -i --restart='Never' --namespace default --image docker.io/bitnami/postgresql:11.6.0-debian-9-r0 --env="PGPASSWORD=DCsSy0s8hM" --command -- psql --host goatsnap-postgres-postgresql -U postgres -d postgres -p 5432
psql: FATAL: password authentication failed for user "postgres"
pod "goatsnap-postgres-postgresql-client" deleted
pod default/goatsnap-postgres-postgresql-client terminated (Error)
I've tried a few other things, all of which get the same error:
Run kubectl run [...] bash, launch psql, type the password at the prompt
Run kubectl port-forward [...], launch psql locally, type the password
Un/re-install the chart a few times
Use helm install --set postgresqlPassword=[...], use the explicitly-set password
I'm using OSX 10.15.2, k8s 1.15.5 (via Docker Desktop 2.2.0.0), helm 3.0.0, postgres chart postgresql-7.7.2, postgres 11.6.0
I think I had it working before (although I don't see evidence in my scrollback), and I think I updated Docker Desktop since then, and I think I saw something about a K8s update in the notes for the Docker update. So if I'm remembering all those things correctly, maybe it's related to the k8s update?
Whoops, never mind—I found the resolution in an issue on helm/charts. The password ends up in a persistent volume, so if you uninstall and reinstall the chart, the new version retains the password from the old one, not the value from kubectl get secret. I uninstalled the chart, deleted the old PVC and PV, and reinstalled, and now I'm able to connect.

How to reset grafana's admin password (installed by helm)

My password once worked, but I don't remember if I changed it or not.
However, I can't reset it.
I tried with no success:
kubectl get secret --namespace default grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo
> DpveUuOyxNrandompasswordYuB5Fs2cEKKOmG <-- does not work (anymore?)
PS: I did not set any admin email for web-based reset
Ok found.
Best way is to run grafana-cli inside grafana's pod.
kubectl exec --namespace default -it $(kubectl get pods --namespace default -l "app=grafana,release=grafana" -o jsonpath="{.items[0].metadata.name}") grafana-cli admin reset-admin-password yourNewPasswordHere
INFO[01-21|10:24:17] Connecting to DB logger=sqlstore dbtype=sqlite3
INFO[01-21|10:24:17] Starting DB migration logger=migrator
Admin password changed successfully ✔
Okay, try this.
kubectl get pod -n monitoring
kubectl exec -it grafana-00000000aa-lpwkk -n monitoring -- sh
grafana-cli admin reset-admin-password NEWPASSWORD
If you installed it via kube-prometheus-stack helm chart then the admin password is stored in a secret named kube-prometheus-stack-grafana. You need to set it there are restart the Grafana pod.
Alternatively, you can just decode the password and use:
kubectl get secrets/kube-prometheus-stack-grafana -o json | jq '.data | map_values(#base64d)'

Failing to connect to the postgress statefulset database installed from stable/postgresql helm chart

Do you guys have played with the stable/postgresql helm chart?
I successfully install the release using this command within a GKE context
$helm install --name pg-set -f ./values-production.yaml stable/postgresql --set postgresqlDatabase=nixmind-db
It went fine, but I can't further connect to my cluster for ckeck/tests..., cause it fails connecting to the database due to password.
I tried all the following way but got the same error :
$export POSTGRES_PASSWORD=$(kubectl get secret --namespace default pg-set-postgresql -o jsonpath="{.data.postgresql-password}" | base64 --decode)
$kubectl run pg-set-postgresql-client --rm --tty -i --restart='Never' --namespace default --image docker.io/bitnami/postgresql:10.7.0 --env="PGPASSWORD=$POSTGRES_PASSWORD" --command -- psql --host pg-set-postgresql -U postgres -d nixmind-db
psql: FATAL: password authentication failed for user "postgres"
pod "pg-set-postgresql-client" deleted
pod default/pg-set-postgresql-client terminated (Error)
$export POSTGRES_PASSWORD=$(kubectl get secret --namespace default pg-set-postgresql -o jsonpath="{.data.postgresql-password}" | base64 --decode)
$kubectl run pg-set-postgresql-client --rm --tty -i --restart='Never' --namespace default --image docker.io/bitnami/postgresql:10.7.0 --env="POSTGRESS_PASSWORD=$POSTGRES_PASSWORD" --command -- psql --host pg-set-postgresql -U postgres -d nixmind-db
psql: FATAL: password authentication failed for user "postgres"
pod "pg-set-postgresql-client" deleted
pod default/pg-set-postgresql-client terminated (Error)
$export POSTGRES_PASSWORD=$(kubectl get secret --namespace default pg-set-postgresql -o jsonpath="{.data.postgresql-password}" | base64 --decode)
$kubectl run pg-set-postgresql-client --tty -i --restart='Never' --namespace default --image docker.io/bitnami/postgresql:10.7.0 --env="POSTGRESQL_PASSWORD=$POSTGRES_PASSWORD" --command -- psql --host pg-set-postgresql -U postgres -d nixmind-db
If you don't see a command prompt, try pressing enter.
psql: fe_sendauth: no password supplied
pod default/pg-set-postgresql-client terminated (Error)
$export POSTGRES_PASSWORD=$(kubectl get secret --namespace default pg-set-postgresql -o jsonpath="{.data.postgresql-password}" | base64 --decode)
$kubectl run pg-set-postgresql-client --tty -i --restart='Never' --namespace default --image docker.io/bitnami/postgresql:10.7.0 --env="POSTGRESQL_PASSWORD=$POSTGRES_PASSWORD" --command -- psql --host pg-set-postgresql -U postgres -d nixmind-db
If you don't see a command prompt, try pressing enter.
psql: fe_sendauth: no password supplied
pod default/pg-set-postgresql-client terminated (Error)
Thy to connect localy to the client and run the command
$kubectl run pg-set-postgresql-client --tty -i --restart='Never' --namespace default --image docker.io/bitnami/postgresql:10.7.0 --env="POSTGRES_PASSWORD=$POSTGRES_PASSWORD" --command -- /bin/bash
If you don't see a command prompt, try pressing enter.
I have no name!#pg-set-postgresql-client:/$ env
.............................................................................
POSTGRESQL_PASSWORD=m7RWxRvpSk
POSTGRESQL_USERNAME=postgres
POSTGRESQL_NUM_SYNCHRONOUS_REPLICAS=0
POSTGRESQL_INITDB_ARGS=
KUBERNETES_SERVICE_HOST=10.63.240.1
NAMI_VERSION=1.0.0-1
.............................................................................
I have no name!#pg-set-postgresql-client:/$ psql --host pg-set-postgresql -U postgres -d nixmind-db
Password for user postgres:
psql: FATAL: password authentication failed for user "postgres"
It seems to be a problem being submitted several time, but without real solution...
How do you guys test this release after installation?
What am I missing? Is the password retrieved from the cluster secrets the one configured on the master database?
May be this command psql --host pg-set-postgresql -U postgres -d nixmind-db tries to access the slave and not master really?
Why the psql client doesn't even read the password from environment variables as expected?
It's all OK.
I was characters issue with my terminal, the password retrieved from cluster credentials is the good one, and the database is accessible.