Mount host dir for Postgres on Minikube - permissions issue - postgresql

I'm trying to setup PostgreSQL on Minikube with data path being my host folder mounted on Minikube (I'd like to keep my data on host).
With the kubernetes object created (below) I get permission error, the same one as here How to solve permission trouble when running Postgresql from minikube? although the question mentioned doesn't answer the issue. It advises to mount minikube's VM dir instead.
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: postgres
labels:
app: postgres
spec:
replicas: 1
selector:
matchLabels:
app: postgres
template:
metadata:
labels:
app: postgres
spec:
containers:
- name: postgres
image: postgres
volumeMounts:
- mountPath: /var/lib/postgresql/data
name: storage
env:
- name: POSTGRES_PASSWORD
value: user
- name: POSTGRES_USER
value: pass
- name: POSTGRES_DB
value: k8s
volumes:
- name: storage
hostPath:
path: /data/postgres
Is there any other way to do that other than building own image on top of Postgres and playing with the permissions somehow? I'm on macOS with Minikube 0.30.0 and I'm experiencing that with both Virtualbox and hyperkit drivers for Minikube.

Look at these lines from here : hostPath
the files or directories created on the underlying hosts are only writable by root. You either need to run your process as root in a privileged Container or modify the file permissions on the host to be able to write to a hostPath volume
So, either you have to run as root or you have to change the file permission of /data/postgres directory.
However, you can run your Postgres container as root without rebuilding docker image.
You have to add following to your container:
securityContext:
runAsUser: 0
Your yaml should look like this:
apiVersion: apps/v1
kind: Deployment
metadata:
name: postgres
labels:
app: postgres
spec:
replicas: 1
selector:
matchLabels:
app: postgres
template:
metadata:
labels:
app: postgres
spec:
containers:
- name: postgres
image: postgres
volumeMounts:
- mountPath: /var/lib/postgresql/data
name: storage
env:
- name: POSTGRES_PASSWORD
value: user
- name: POSTGRES_USER
value: pass
- name: POSTGRES_DB
value: k8s
securityContext:
runAsUser: 0
volumes:
- name: storage
hostPath:
path: /data/postgres

Related

Accessing Postgresql data of Kubernetes cluster

I have kubernetes cluster with two replicas of a PostgreSQL database in it, and I wanted to see the values stored in the database.
When I exec myself into one of the two postgres pod (kubectl exec --stdin --tty [postgres_pod] -- /bin/bash) and check the database from within, I have only a partial part of the DB. The rest of the DB data is on the other Postgres pod, and I don't see any directory created by the persistent volumes with all the database stored.
So in short I create 4 tables; in one postgres pod I have 4 tables but 2 are empty, in the other postgres pod there are 3 tables and the tables that were empty in the first pod, here are filled with data.
Why the pods don't have the same data in it?
How can I access and download the entire database?
PS. I deploy the cluster using HELM in minikube.
Here are the YAML files:
---
apiVersion: v1
kind: ConfigMap
metadata:
name: postgres-config
labels:
app: postgres
data:
POSTGRES_DB: database-pg
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
PGDATA: /data/pgdata
---
kind: PersistentVolume
apiVersion: v1
metadata:
name: postgres-pv-volume
labels:
type: local
app: postgres
spec:
storageClassName: manual
capacity:
storage: 1Gi
accessModes:
- ReadWriteMany
hostPath:
path: "/mnt/data"
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: postgres-pv-claim
spec:
storageClassName: manual
accessModes:
- ReadWriteMany
resources:
requests:
storage: 1Gi
---
apiVersion: v1
kind: Service
metadata:
name: postgres
labels:
app: postgres
spec:
ports:
- name: postgres
port: 5432
nodePort: 30432
type: NodePort
selector:
app: postgres
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: postgres
spec:
serviceName: postgres-service
selector:
matchLabels:
app: postgres
replicas: 2
template:
metadata:
labels:
app: postgres
spec:
containers:
- name: postgres
image: postgres:13.2
volumeMounts:
- name: postgres-disk
mountPath: /data
# Config from ConfigMap
envFrom:
- configMapRef:
name: postgres-config
volumeClaimTemplates:
- metadata:
name: postgres-disk
spec:
accessModes: ["ReadWriteOnce"]
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: postgres
labels:
app: postgres
spec:
selector:
matchLabels:
app: postgres
replicas: 2
template:
metadata:
labels:
app: postgres
spec:
containers:
- name: postgres
image: postgres:13.2
imagePullPolicy: IfNotPresent
envFrom:
- configMapRef:
name: postgres-config
volumeMounts:
- mountPath: /var/lib/postgresql/data
name: postgredb
volumes:
- name: postgredb
persistentVolumeClaim:
claimName: postgres-pv-claim
---
I found a solution to my problem of downloading the volume directory, however when I run multiple replicasets of postgres, the tables of the DB are still scattered between the pods.
Here's what I did to download the postgres volume:
First of all, minikube supports some specific directories for volume appear:
minikube is configured to persist files stored under the following directories, which are made in the Minikube VM (or on your localhost if running on bare metal). You may lose data from other directories on reboots.
/data
/var/lib/minikube
/var/lib/docker
/tmp/hostpath_pv
/tmp/hostpath-provisioner
So I've changed the mount path to be under the /data directory. This made the database volume visible.
After this I ssh'ed into minikube and copied the database volume to a new directory (I used /home/docker as the user of minikube is docker).
sudo cp -R /data/pgdata /home/docker
The volume pgdata was still owned by root (access denied error) so I changed it to be owned by docker. For this I also set a new password which I knew:
sudo passwd docker # change password for docker user
sudo chown -R docker: /home/docker/pgdata # change owner from root to docker
Then you can exit and copy the directory into you local machine:
exit
scp -r $(minikube ssh-key) docker#$(minikube ip):/home/docker/pgdata [your_local_path].
NOTE
Mario's advice, which is to use pgdump is probably a better solution to copy a database. I still wanted to download the volume directory to see if it has the full database, when the pods have only a part of all the tables. In the end it turned out it doesn't.

Postgres on Azure kubernetes volume permission error

I'm trying to deploy Postgresql on Azure Kubernetes with data persistency. So I'm using PVC.
I searched lots of posts on here, most of them offered yaml files like below, but it's giving the error below;
chmod: changing permissions of '/var/lib/postgresql/data/pgdata': Operation not permitted
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".
Data page checksums are disabled.
initdb: error: could not change permissions of directory "/var/lib/postgresql/data/pgdata": Operation not permitted
fixing permissions on existing directory /var/lib/postgresql/data/pgdata ...
deployment yaml file is below;
apiVersion: apps/v1
kind: Deployment
metadata:
name: postgresql
spec:
replicas: 1
selector:
matchLabels:
app: postgresql
template:
metadata:
labels:
app: postgresql
spec:
containers:
- name: postgresql
image: postgres:13.2
securityContext:
runAsUser: 999
imagePullPolicy: "IfNotPresent"
ports:
- containerPort: 5432
envFrom:
- secretRef:
name: postgresql-secret
volumeMounts:
- mountPath: /var/lib/postgresql/data
name: postgredb-kap
volumes:
- name: postgredb-kap
persistentVolumeClaim:
claimName: postgresql-pvc
Secret yaml is below;
apiVersion: v1
kind: Secret
metadata:
name: postgresql-secret
type: Opaque
data:
POSTGRES_DB: a2V5sd4=
POSTGRES_USER: cG9zdGdyZXNhZG1pbg==
POSTGRES_PASSWORD: c234Rw==
PGDATA: L3Za234dGF0YQ==
pvc and sc yaml files are below:
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: postgresql-pvc
labels:
app: postgresql
spec:
storageClassName: postgresql-sc
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi
---
allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: postgresql-sc
mountOptions:
- dir_mode=0777
- file_mode=0777
- uid=1000
- gid=1000
parameters:
skuName: Standard_LRS
provisioner: kubernetes.io/azure-file
reclaimPolicy: Retain
So when I use the mountpath like "- mountPath: /var/lib/postgresql/", it's working. I can reach the DB and it's good. But when I delete the pod and recreating, there is no DB! So no data persistency.
Can you please help, what am I missing here?
Thanks!
One thing you could try is to change uid=1000,gid=1000 in mount options to 999 since this is the uid of postgres user in postgres conatiner (I didn't test this).
Another solution that will for certain solve this issue involves init conatainers.
Postgres container requires to start as root to be able to chown pgdata dir since its mounted as root dir. After it does this, it drops root permisions and runs as postgres user.
But you can use init container (running as root) to chmod the volume dir so that you can run main container as non-root.
Here is an example:
initContainers:
- name: init
image: alpine
command: ["sh", "-c", "chown 999:999 /var/lib/postgresql/data"]
volumeMounts:
- mountPath: /var/lib/postgresql/data
name: postgredb-kap
Based on the helpful answer from Matt. For bitnami postgresql the initContainer also works but with a slightly different configuration:
initContainers:
- name: init
image: alpine
command: ["sh", "-c", "chown 1001:1001 /bitnami/postgresql"]
volumeMounts:
- mountPath: /bitnami/postgresql
name: postgres-volume

Change Postgres config using Kubernetes configMap

I'm trying to deploy postgres using Kubernetes. This step works well.
But now I need to customised the pg_hba.conf file. So the idea is to use a configMap to mount the personal file.
Here are my deployment file:
apiVersion: v1
kind: ConfigMap
metadata:
name: pg-hba-configmap
data:
pg_hba.conf: |+
# TYPE DATABASE USER ADDRESS METHOD
# Some personal settings here
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: db
spec:
selector:
matchLabels:
app: db
serviceName: db
replicas: 1
template:
metadata:
labels:
app: db
spec:
nodeSelector:
db: 'true'
containers:
- name: db
image: kartoza/postgis:10.0-2.4
volumeMounts:
- name: data-volume
mountPath: /var/lib/postgresql
- name: pg-hba-config-volume
mountPath: /var/lib/postgresql/10/main/pg_hba.conf
subPath: pg_hba.conf
volumes:
- name: data-volume
hostPath:
path: /data/db
- name: pg-hba-config-volume
configMap:
name: pg-hba-configmap
But this doesn't work. The db is in Error state. In logs, I can see this:
chown: changing ownership of '/var/lib/postgresql/10/main/pg_hba.conf': Read-only file system
Also, on the server where is mounted the database, I only have this:
root#server:/data/db/10/main# ls -l
total 0
-rwxr-xr-x 1 root root 0 Oct 8 08:49 pg_hba.conf
I should have all the database here. It's like if all the folder was overwritten.
What am I doing wrong here and how can I mount my customised config ?
Thanks

chmod: changing permissions of '/var/lib/postgresql/data': Operation not permitted

Hi I have set up an small NFS server at home using my raspberry pi.
An I want to set that as the default storage for all of my kubernetes containers.
However I keep on getting this chmod: changing permissions of '/var/lib/postgresql/data': Operation not permitted
here is my config.
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: pg-ss
spec:
replicas: 1
selector:
matchLabels:
app: postgres
template:
metadata:
labels:
app: postgres
spec:
containers:
- name: postgres
image: postgres:9.6
volumeMounts:
- name: pv-data
mountPath: /var/lib/postgresql/data
env:
- name: POSTGRES_USER
value: postgres
- name: POSTGRES_PASSWORD
value: postgres
ports:
- containerPort: 5432
name: postgredb
volumes:
- name: pv-data
nfs:
path: /mnt/infra-data/pg
server: 192.168.1.150
readOnly: false
I'm wondering what would be the cause of this. and how can i solve it.
Thanks,

Kubernetes PostgreSQL: How do I store config files elsewhere than the data directory?

Currently I'm trying to create a PostgreSQL Deployment for replication, using the postgres:latest image.
The config files are located by default within the data directory /var/lib/postgresql/data. For replication to work I need the data directory to be empty, but that means I have to keep the config files elsewhere.
Referring to the PostgreSQL Documentation:
If you wish to keep the configuration files elsewhere than the data directory, the postgres -D command-line option or PGDATA environment variable must point to the directory containing the configuration files, and the data_directory parameter must be set in postgresql.conf (or on the command line) to show where the data directory is actually located. Notice that data_directory overrides -D and PGDATA for the location of the data directory, but not for the location of the configuration files.
In a physical machine setup, we can manually move the files and set the location of data-directory in the postgresql.conf file. However in Kubernetes it is not so straight-forward.
I tried to use volumeMount with subPath to mount the config files in another location, then use command to change the new location of postgresql.conf.
Sample .yaml file:
apiVersion: v1
kind: ConfigMap
metadata:
name: pg-replica
labels:
app: postgres
name: pg-replica
data:
POSTGRES_DB: postgres
POSTGRES_USER: postgres
POSTGRES_PASSWORD: mypassword
pg_hba.conf: |
# Contents
postgresql.conf: |
data_directory = /var/lib/postgresql/data/data-directory
recovery.conf: |
# Contents
---
apiVersion: v1
kind: Service
metadata:
name: pg-replica
labels:
app: postgres
name: pg-replica
spec:
type: NodePort
ports:
- nodePort: 31000
port: 5432
selector:
app: postgres
name: pg-replica
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: pg-replica
spec:
selector:
matchLabels:
app: postgres
name: pg-replica
replicas: 1
template:
metadata:
labels:
app: postgres
name: pg-replica
spec:
containers:
- name: pg-replica
image: postgres:latest
imagePullPolicy: "IfNotPresent"
ports:
- containerPort: 5432
envFrom:
- configMapRef:
name: pg-replica
volumeMounts:
- name: pg-replica
mountPath: /var/lib/postgresql/data
- name: replica-config
mountPath: /var/lib/postgresql/postgresql.conf
subPath: postgresql.conf
- name: replica-config
mountPath: /var/lib/postgresql/pg_hba.conf
subPath: pg_hba.conf
- name: replica-config
mountPath: /var/lib/postgresql/recovery.conf
subPath: recovery.conf
command:
- "/bin/bash"
- "postgres -c config_file=/var/lib/postgresql/postgresql.conf"
volumes:
- name: pg-replica
persistentVolumeClaim:
claimName: pv-replica-claim
- name: replica-config
configMap:
name: pg-replica
The returned message was as following:
/bin/bash: postgres -c config_file=/var/lib/postgresql/postgresql.conf: No such file or directory
What is wrong with this configuration? And what steps am I missing to make it work?
Edit:
When using the volumeMount field, the directory is overwritten (all other files were removed) despite I specified the exact file to mount on with subPath. What could be the cause for this?
I realized there were a few mistakes here after posting this question...
I used PostgreSQL 11 for replication prior so I assumed they worked the same way (which of course is wrong, there are some changes). The recovery.conf is omitted from PostgreSQL 12 and it gave this error message FATAL: XX000: using recovery command file "recovery.conf" is not supported when I had it. so I had to remove it from my ConfigMap.
I confused about Docker's Entrypoint & Command to Kubernetes' Command & Args. After being corrected by my senior that Kubernetes Command will override the Docker Entrypoint, I'm going to need and use only Args afterwards.
The following are the changes I made to my ConfigMap and Deployment.
apiVersion: v1
kind: ConfigMap
metadata:
name: pg-replica
labels:
app: postgres
name: pg-replica
data:
POSTGRES_DB: postgres
POSTGRES_USER: postgres
POSTGRES_PASSWORD: mypassword
pg_hba.conf: |
# Contents
postgresql.conf: |
data_directory = '/var/lib/postgresql/data'
# the contents from recovery.conf are intergrated into postgresql.conf
primary_conninfo = # host address and authentication credentials
promote_trigger_file = # trigger file path
extra.sh: |
#!/bin/sh
postgres -D /var/lib/postgresql
---
apiVersion: v1
kind: Service
metadata:
name: pg-replica
labels:
app: postgres
name: pg-replica
spec:
type: NodePort
ports:
- nodePort: 31000
port: 5432
selector:
app: postgres
name: pg-replica
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: pg-replica
spec:
selector:
matchLabels:
app: postgres
name: pg-replica
replicas: 1
template:
metadata:
labels:
app: postgres
name: pg-replica
spec:
containers:
- name: pg-replica
image: postgres:latest
imagePullPolicy: "IfNotPresent"
ports:
- containerPort: 5432
envFrom:
- configMapRef:
name: pg-replica
volumeMounts:
- name: pg-replica
mountPath: /var/lib/postgresql/data
- name: replica-config
mountPath: /var/lib/postgresql/postgresql.conf
subPath: postgresql.conf
- name: replica-config
mountPath: /var/lib/postgresql/pg_hba.conf
subPath: pg_hba.conf
- name: replica-config
mountPath: /docker-entrypoint-initdb.d/extra.sh
subPath: extra.sh
args:
- "-c"
- "config_file=/var/lib/postgresql/postgresql.conf"
- "-c"
- "hba_file=/var/lib/postgresql/pg_hba.conf"
volumes:
- name: pg-replica
persistentVolumeClaim:
claimName: pv-replica-claim
- name: replica-config
configMap:
name: pg-replica
The arguments in Args will set the location of the .conf files to where I specified.
For further steps in Replication:
After the pod is up, I manually ran the pod's shell with kubectl exec.
I removed all the files from the data-directory for step 3 (to copy files from master pod).
rm -rf /var/lib/postgresql/data/*
Use pg_basebackup to backup data from master node.
pg_basebackup -h <host IP> --port=<port number used> -D /var/lib/postgresql/data -P -U replica -R -X stream
And that's it. Now I managed to have my pg-replica pod replicating my master pod.
As mentioned in comments, I really encorage you to use the Postgres Helm chart to setup yout environment.
The way you solved the issue could work, but if the pod died for some reason, all work you have done will be lost and you'll need to reconfigure everything again.
Here you can found all information about how to create a postgres deployment with high availability and replication.
To install HELM you can follow this guide.