Postgres on Docker exits immediately and deletes data in external filesystem - postgresql

I'm following instructions in this article to pull a Postgres Docker image and run it.
$ mkdir -p $HOME/docker/volumes/postgres
$
$ docker pull postgres:9.5
$
$ docker run --name pg-docker -e POSTGRES_PASSWORD=postgres -d -p 5432:5432 -v $HOME/docker/volumes/postgres:/var/lib/postgresql/data postgres:9.5
I see Postgres files and data getting created in the path I mounted with -v above, but the container exits soon after starting, and the files vanish.
My questions:
Why does the container exit at all?
Why does it delete all the files? The whole purpose of -v is to
preserve data outside the container.
Update 1:
Here are the logs from the container that I was able to get when the container was briefly up (after the container exited, I only got an error message Error: No such container: postgres:9.5 but no logs):
$ docker logs pg-docker
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".
Data page checksums are disabled.
fixing permissions on existing directory /var/lib/postgresql/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default timezone ... Etc/UTC
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
Update 2:
Running docker without -d produced more output, which has the reason for this situation:
creating template1 database in /var/lib/postgresql/data/base/1 ...
LOG: could not link file "pg_xlog/xlogtemp.36" to "pg_xlog/000000010000000000000001": Operation not supported
FATAL: could not open file "pg_xlog/000000010000000000000001": No such file or directory
child process exited with exit code 1
initdb: removing contents of data directory "/var/lib/postgresql/data"
Does it matter that I'm on a Windows system inside a Git Bash shell?

Related

Deploying Postgres database on azure Container Instance?

I am trying to deploy PostgresDatabase on azure container instance.
To deploy on docker using bind mount(since Azure container Instance only support bind mount) i am using the below command, and it is deployed on docker.
docker run -d -p 5434:5432 --name some-postgres -e POSTGRES_PASSWORD=mysecretpassword -e PGDATA=/var/lib/postgresql/data/pgdata -v /home/ubuntu/volum:/var/lib/postgresql/data postgres
If i do something similar for deploying on Azure container Instance
az container create \
--resource-group $ACI_PERS_RESOURCE_GROUP \
--name postgreariesdb25-1 \
--location eastus \
--image postgres \
--dns-name-label $ACI_DNS_LABEL \
--environment-variables POSTGRES_PASSWORD=mysecretpassword PGDATA=/var/lib/postgresql/data/pgdata \
--ports 5432 \
--azure-file-volume-account-name $ACI_PERS_STORAGE_ACCOUNT_NAME \
--azure-file-volume-account-key $STORAGE_KEY \
--azure-file-volume-share-name $ACI_PERS_SHARE_NAME \
--azure-file-volume-mount-path /var/lib/postgresql/data
I am getting the below message inside logs of Azure Container
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".
Data page checksums are disabled.
fixing permissions on existing directory /var/lib/postgresql/data/pgdata ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 20
selecting default shared_buffers ... 400kB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
2020-11-24 05:23:39.218 UTC [85] FATAL: data directory "/var/lib/postgresql/data/pgdata" has wrong ownership
2020-11-24 05:23:39.218 UTC [85] HINT: The server must be started by the user that owns the data directory.
child process exited with exit code 1
initdb: removing contents of data directory "/var/lib/postgresql/data/pgdata"
running bootstrap script ...
Volume Mount is required to have data in case of container restart.
This is a known error for mounting Azure File Share to Azure Container Instance. Currently, it does not support to change the ownership of the mount point. If you do not want to use other services, then you need to create a script to move the data to the mount point and the mount point should be a new folder that does not exist in the image. For you, the mount point /var/lib/postgresql/data exists in the image and contains the files that Postgresql depends on, then this point cannot be the mount point.

Postgresql in a Docker Container on Windows: How to persist data to a local windows folder

I'm trying to run postgres in a docker container on windows. I also want keep the data in a windows folder, so I tried this:
mkdir c:\pgdata
PS > docker run --name postgres -v c:\pgdata:/var/lib/postgresql/data -d postgres
d12af76bed7f8078babc0b6d35710dfc02b12d650904ed53ca95bb99984e9b36
This appeared to work, but the container is not running and the log tells a different story:
2019-07-24 23:19:20.861 UTC [77] FATAL: data directory "/var/lib/postgresql/data" has wrong ownership
2019-07-24 23:19:20.861 UTC [77] HINT: The server must be started by the user that owns the data directory.
child process exited with exit code 1
If I remove the volume option, it starts up fine, but then I don't get my database files persisted where I want them. What am I doing wrong here?
You did nothing wrong, just have a look for the full log:
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".
Data page checksums are disabled.
fixing permissions on existing directory /var/lib/postgresql/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 20
selecting default shared_buffers ... 400kB
selecting default timezone ... Etc/UTC
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
2019-07-25 01:28:18.301 UTC [77] FATAL: data directory "/var/lib/postgresql/data" has wrong ownership
2019-07-25 01:28:18.301 UTC [77] HINT: The server must be started by the user that owns the data directory.
child process exited with exit code 1
initdb: removing contents of data directory "/var/lib/postgresql/data"
running bootstrap script ...
From above, you can see fixing permissions on existing directory /var/lib/postgresql/data ... ok which is executed in docker-entrypoint.sh to change the ownership from root to postgres, but unfortunately, this just works on linux host not on windows host.
Then, why not work on windows, see this discussion, mainly because current implementation was based on CIFS/Samba which make docker cannot improve it.
So, I guess you have no chance to persist data to windows if you insist to use bind mount.
But, if you not insist, a closer solution maybe use Named Volumes like next:
PS C:\> docker volume create my-vol
my-vol
PS C:\> docker run --name postgres -v my-vol:/var/lib/postgresql/data -d postgres
079d4b5b3f73bc0c4c586cdfee3fdefc8a27cdcd409e857de985bead254cd23f
PS C:\> docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
079d4b5b3f73 postgres "docker-entrypoint.s…" 5 seconds ago Up 2 seconds 5432/tcp postgres
PS C:\> docker volume inspect my-vol
[
{
"CreatedAt": "2019-07-25T01:43:01Z",
"Driver": "local",
"Labels": {},
"Mountpoint": "/var/lib/docker/volumes/my-vol/_data",
"Name": "my-vol",
"Options": {},
"Scope": "local"
}
]
Finally, the data will persist in /var/lib/docker/volumes/my-vol/_data, but the limit is this folder is not on windows, it's on hyper-v machine as you may know docker for windows use hyper-v to simulate linux kernel.
But it may still meet your requirement, because even you remove current container, next time if you use same volume name(here is my-vol) to mount, the data will still be in the new container, the named volume will not be delete even container deleted, it will be persist in hyper-v virtual machine.
in windows, once you installed docker, you also got docker-compose.exe command. so let's use it for your postgres:
step1. create an folder in your host(windows: D:\workspace\docker_folder\postgres9.5)
step2. make a docker-compose.yml and paste its content as below:
version: '3'
services:
postgres9.5:
container_name: "postgres9.5"
image: postgres:9.5
# notice here, D:\workspace should be written as: /d/workspace
volumes:
- /d/workspace/docker_folder/postgres9.5:/var/lib/postgresql/data
command: 'postgres'
ports:
- "5432:5432"
stdin_open: true
tty: true
environment:
- POSTGRES_DB=postgres
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=88888888
step3. run docker-compose up, then you will see the files in your D:\workspace\docker_folder, as below:

pg_upgrade tool failed: invalid "unknown" user columns

Postgresql update from 9.6 to 10.4 (on Fedora 28) has me stuck: one table in one database has a column of data type "unknown". I would gladly remove the column, but since I cannot get postgresql service to start (because "An old version of the database format was found"), I have no access to the database. In more detail:
postgresql-setup --upgrade fails.
/var/lib/pgsql/upgrade_postgresql.log attributes this failure to column with data type "unknown": "...Checking for invalid 'unknown' user columns: fatal .... check tables_using_unknown.txt". And "tables_using_unknown.txt" specifies one column in one table that I wish I could drop, but can't, because I can't get the server to start:
systemctl start postgresql.service fails, and
systemctl status postgresql.service complains about the "old version of the database"
I have found no obvious way to install postgresql 9.6 on Fedora 28.
Is there a way to drop the column without a running server? Or at least produce a dump of the database? Or can I force the upgrade tool to drop columns with data type "unknown"? Or is there any other obvious solution that I'm missing?
Here's what finally worked for me:
I used a docker container (on the same machine) with postgres 9.6 to access the "old" database directory,
converted the problematic column from "unknown" to "text" in the container,
dumped the relevant database to a file on the container's host, and then
loaded the dumped db into the postgres 10.4 environment.
Not pretty, but worked. In more detail:
I copied postgresql's data directory (/var/lib/pgsql/data/ in Fedora) -- containing the database that could not be converted -- to a new, empty directory /home/hj/pg-problem/.
I created a Dockerfile (text file) called "Docker-pg-problem" reading
FROM postgres:9.6
# my databases need German locale;
# if you just need en_US, comment the next two lines out.
RUN localedef -i de_DE -c -f UTF-8 -A /usr/share/locale/locale.alias de_DE.UTF-8
ENV LANG de_DE.utf8
and saved it as the only file in the new, empty folder /home/hj/pg-problem/docker/.
I started the docker daemon and ran a container that uses the data from my copy of the problematic data (in /home/hj/pg-problem/data/) as data directory for the postgres 9.6 server in the container. (NB: the "docker build" command in line three needs a working internet connection, takes a while, and should finish saying "Successfully built").
root#host: cd /home/hj/pg-problem/docker
root#host: service docker start
root#host: docker build -t hj/failed-update -f Dockerfile .
root#host: docker run -it --rm -p 5431:5432 -v /home/hj/pg-problem/data:/var/lib/postgresql/data:z --name failed-update -e POSTGRES_PASSWORD=secret hj/failed-update
Then, I opened a terminal in the container to fix the database:
hj#host: docker exec -it failed-update bash
Inside the container, I fixed and dumped the database:
root#container: su postgres
postgres#container: psql <DB-name>
postgres#container: alter table <Table-name> alter column <Col-Name> type text;
postgres#container: \q
postgres#container: dump_db <DB-name> /var/lib/postgresql/data/dbREPAIRED.sql
I dumped the db right into the data directory so I could easily access the dumped file from the docker host.
On the docker host, the dumped database was, obviously, in /home/hj/pg-problem/data/dbREPAIRED.sql, and from there I could load it into postgresql 10:
postgres#host: createdb <DB-name>
postgres#host: psql <DB-name> < /home/hj/pg-problem/data/dbREPAIRED.sql
Since I was on a laptop with limited disk space, I deleted the docker stuff:
root#host: docker rm $(docker ps -a -q)
root#host: docker rmi $(docker images -q)

Postgres DB not starting in Docker - Mac

I already using Postgres DB in my development and at the last time when I'm push my Spring Micro-services the Postgres DB is not starting, following is the docker output from kitematic
The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".
2017-10-17T08:37:47.562145630Z
Data page checksums are disabled.
2017-10-17T08:37:47.562162938Z
fixing permissions on existing directory /var/lib/postgresql/data ... ok
initdb: could not create directory "/var/lib/postgresql/data/pg_xlog": No space left on device
initdb: removing contents of data directory "/var/lib/postgresql/data"
Any one have a idea on this? I couldn't find a solution
Hi if you have good amount of space in your machine ,then this problem is because of dangling images and dangling volume remove those using
docker volume rm $(docker volume ls -f dangling=true -q) or docker volume prune ,for images
docker rmi $(docker images -f dangling=true -q)
you can also do docker system prune to clean the docker completely
if you still face this problem the only way to do is reset the docker complete just like a new one

Run postgresql docker image with persistent data returns permission error

I've been running my docker images upon my Vagrant machine (the box is ubuntu 14.04) without any big issues. But the following error is racking my brains. I wish you people can help me.
When I run this:
$ docker run -it -v /vagrant/postgresql/data:/var/lib/postgresql/data postgres:9.2
I get this error
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".
fixing permissions on existing directory /var/lib/postgresql/data ... ok
initdb: could not create directory "/var/lib/postgresql/data/pg_xlog": Permission denied
initdb: removing contents of data directory "/var/lib/postgresql/data"
vagrant#legionat:/vagrant/sonarqube$ docker run -it -v /vagrant/postgresql/data:/var/lib/postgresql/data postgres:9.2
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".
fixing permissions on existing directory /var/lib/postgresql/data ... ok
initdb: could not create directory "/var/lib/postgresql/data/pg_xlog": Permission denied
initdb: removing contents of data directory "/var/lib/postgresql/data"
I've tried open all the permissions of /vagrant/postgresql without success. Maybe this is a problem of the official docker image.
EDIT:
I've just noticed that that is a lot of people facing the same problem as me: https://github.com/docker-library/postgres/issues/26
And, as someone asked about this in the comments, here it goes:
$ ls -l /vagrant/postgresql/data
total 0
If you are just concerned about persisting the data, I would recommend using a data volume instead of a host volume.
Run docker volume create --name pgdata
Then connect it to your container with:
docker run --rm --name pg -v pgdata:/var/lib/PostgreSQL/data postgres:9.2
Even after that container is gone, you can start a new one connected to the volume and your data will be there.
Just make sure your Vagrant user has the permission to access this directory:
ls -ld /vagrant/postgresql/data
And as deinspanjer said. you can use named volume for persisting the data