Volume mount in docker for postgres data - postgresql

I am trying to insert data into postgres using docker.
I have a folder in my code named data which has insert commands and has one file named init.sql.
I want to insert the data from init.sql present in folder data to tables present in docker.
version: '3.1'
services:
postgres:
image: postgres:11.6-alpine
restart: always
ports:
- "5432:5432"
environment:
POSTGRES_PORT: 5432
volumes:
- ./tables:/docker-entrypoint-initdb.d/
- ./data:/var/lib/postgresql/data
volumes:
data: {}
I am trying this but I get the error:
initdb: directory "/var/lib/postgresql/data" exists but is not empty
I think I am not using the correct use case, I am new to docker compose.
But is there any way, my use case can get satisfied?

This is caused by an improper usage of the volumes syntax for your named volume.
In order to mount a named volume you have to just use its name like this:
volumes:
- data:/var/lib/postgresql/data
If your syntax begins with a . then it will be a bind mount from your host.
volumes:
- ./data:/var/lib/postgresql/data
The above code is mounting the host folder data relative to where you your docker-compose.yml is located.
This docker-compose.yml should do what you expect.
version: '3.1'
services:
postgres:
image: postgres:11.6-alpine
restart: always
ports:
- "5432:5432"
environment:
POSTGRES_PORT: 5432
volumes:
- ./tables:/docker-entrypoint-initdb.d/
- data:/var/lib/postgresql/data
volumes:
data:
If for some reason your volume has been created already, with an empty or no database, your first step should be running:
docker-compose down --volumes
From the documentation:
-v, --volumes Remove named volumes declared in the `volumes`
section of the Compose file and anonymous volumes
attached to containers.
From: https://docs.docker.com/compose/reference/down/

Related

Moved docker-compose.yml creates a new postgres database

I have set up a Postgres database on docker on ubuntu with the docker-compose.yml just for that database within the folder ~/postgres and I'd run docker-compose up -d to run my database from within the ~/postgres folder.
Here is my docker-compose.yml:
version: '3'
services:
database:
image: "postgres"
ports:
- "0.0.0.0:5432:5432"
env_file:
- database.env
volumes:
- database-data:/var/lib/postgresql/data/
volumes:
database-data:
This database is set up and working perfectly, so I decided to set up my web application as well and, because the docker-compose.yml file was inside that folder, I moved it outside to ~/ so I could use it for my web app as well.
This is what the docker-compose.yml in ~/ looks like:
version: "3"
services:
database:
image: "postgres"
ports:
- "0.0.0.0:5432:5432"
env_file:
- postgres/database.env
volumes:
- database-data:/var/lib/postgresql/data/
webapp:
image: webapp/site
build:
context: ./retro-search-engine
dockerfile: Dockerfile
args:
buildno: 1
links:
- "database:db"
ports:
- "0.0.0.0:8000:80"
volumes:
- webapp:/var/www
environment:
db_host: db
db_username: xxxx
db_password: xxxx
db_database: xxxx
db_port: 5432
volumes:
database-data:
webapp:
As you can see, the database docker configuration is basically the same, the only thing that changes is the path to the database.env file since it's still in the previous folder.
So, the problem here is that when I run docker-compose up -d from ~/, everything starts normally but when I access the database, all of my tables are gone.
If I go back to ~/postgres and do docker-compose up -d in that folder (with the previous docker-compose.yml) and connect to the db, I can access my tables.
So what I think is happening is that it's either creating a new container or somehow the folder where the data is stored is relative to the docker-compose.yml file and it's creating a new database because it can't find the old files.
I have no idea how to solve this issue, I have googled around and couldn't find anything so I decided to ask here before I dump my whole db and restore it into a new container, which I don't want to do because it's a 16gb database and it's gonna take forever.
Does anyone have any idea how I can use my new docker-compose.yml with the data from my database?
Thanks in advance.
First:
Replace : postgres/database.env by ./postgres/database.env
Use docker compose up --build :
it will rebuild the image (usefull if you made some change to your dockerfile). try to avoid to use -d when developing, you'll avoid to have tons of container running.
Second:
I suggest you to follow the following reco, It will resolve your problem and it will be cleaner if you want to use a pipeline CI/CD and to create more "autonomous" image and container on demand.
rootfolder
|-docker-compose.yaml
|-postgres/
| |--All_other_files_for_the_postgres_docker_image
|-webapp/
|-- Dockerfile
|-- All_other_files_for_the_webapp_docker_image
bellow you will find my "correction" :
version: "3"
services:
database:
image: "postgres"
container_name: "my_postgres_container"
ports:
- "0.0.0.0:5432:5432"
env_file:
- ./postgres/database.env
volumes:
- database-data:/var/lib/postgresql/data/
webapp:
image: webapp/site
container_name: "my_webapp_container"
build:
context: ./retro-search-engine
dockerfile: Dockerfile
links:
- "database:db"
ports:
- "0.0.0.0:8000:80"
volumes:
- webapp:/var/www
environment:
db_host: db
db_username: xxxx
db_password: xxxx
db_database: xxxx
db_port: 5432
volumes:
database-data:
webapp:
If you want to use an existing postgres image that is already present (to see if an image already existe you can do : docker image | grep postgres)
then you can do directly in your docker-compose :
image: "<your_image_name>"

Docker-Compose MariaDB creates my volume but still uses container id volume

I can't seem to get docker/Maria to use my named docker volume. The host docker volumes directory is empty. But, there is a new container id right next to my named volume that looks like it has all of the MariaDB parts in it. The question is why?
My docker compose file:
version: "3.7"
#
# [Volumes]
#
volumes:
data-mysql:
#
# [Services]
#
services:
mariadb:
volumes:
- data-mysql:/var/lib/mysql
image: linuxserver/mariadb
container_name: mariadb
environment:
- PUID=1000
- GUID=1000
- MYSQL_ROOT_PASSWORD=<snipped>
- TZ=Etc/UTC
ports:
- 3306:3306
restart: unless-stopped
I've tried moving the volume part before and after the services line with no difference. When I do a docker-compose up, it does say it's creating the volume: mariadb_data-mysql, but when I shut down docker, there is nothing in the folder.
Thanks for any insight!
Nick
The data folder for MARIADB image you are using (linuxserver/mariadb) is /config/databases/ and not /var/lib/mysql. Replace this in your docker-compose.yml and it will work.
Also, the order in your docker-compose.yml does not matter: docker-compose will compile it and order everything alphabetically anyway before processing.

How to migrate the volumes of a Docker-Compose Postgres to another host?

I have a Docker Compose file with some services. One of them is the database which I would like to back up the volumes and migrate all the data to another machine.
My docker-compose.yml looks like this
version: '3'
services:
service1:
...
serviceN:
db:
image: postgres:11
ports:
- 5432:5432
networks:
- postgresnet
environment:
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres
volumes:
- postgresql:/var/lib/postgresql
- postgresql_data:/var/lib/postgresql/data
volumes:
postgresql:
postgresql_data:
networks:
postgresnet:
driver: bridge
How could I backup the data of postgresql and postgresql_data volunes and migrate them to another machine?
Easiest way is to share external volumes between your docker-compose files.
First create volume
docker volume create shared-data
Next modify you yml:
...
volumes:
postgresql:
postgresql_data:
external:
name: shared-data
...
Now your postgresql_data is mapped to external volume and everything you save there could be visible from outside. Just create same configuration in another docker-compose.yml and enjoy.

How do I back up docker volume for postgres?

Here is my docker compose file
version: '2'
services:
postgres9:
image: postgres:9.4
expose:
- 5432
volumes:
- data:/var/lib/postgresql/data
volumes:
data: {}
I would like to back up data .
Is there a command for that in docker? I am using mac for my local and linux for my server
Volumes are not anything special in case of Docker. They are simple directories/files, when you are using volumes in compose file docker will create a directory and mount that inside of your container when you run the container. You can see that by:
docker inspect <container name/id>
in the output you will find information about volumes, you can see there the directory on the host OS
To backup your volume you can simply compress and store the directory using tar. To do that you need to know the path of the directory, you can either mount a directory from host OS like:
version: '2'
services:
postgres9:
image: postgres:9.4
expose:
- 5432
volumes:
- /var/lib/postgresql/data:/var/lib/postgresql/data
and you can backup /var/lib/postgresql/data from the host OS either by mounting it another container or from the host OS directly
OR there is another way, you can mount the same volume in another container in readonly mode and backup the directory:
version: '2'
services:
postgres9:
image: postgres:9.4
expose:
- 5432
volumes:
- data:/var/lib/postgresql/data
backuptool:
image: busybox:latest
volumes:
- data:/data:ro
volumes:
data: {}
you can then tar and upload the backup of /data from the backuptool container

Persisting database using docker volumes

I'm trying to persist postgres data in a docker container so that as soon as you docker-compose down and docker-compose up -d you don't lose data from your previous session. I haven't been able to make it do much of anything - pulling the container down and back up again routinely deletes the data.
Here's my current docker-compose.yml:
version: '2'
services:
api:
build: .
ports:
- '8245:8245'
volumes:
- .:/home/app/api
- /home/app/api/node_modules
- /home/app/api/public/src/bower_components
links:
- db
db:
build: ./database
env_file: .env
ports:
- '8246:5432'
volumes_from:
- dbdata
dbdata:
image: "postgres:9.5.2"
volumes:
- /var/lib/postgresql/data
Help?
According to the Dockment of Docker Compose, when you write something like:
volumes:
- /var/lib/postgresql/data
It creates a new docker volume and map it to /var/lib/postgresql/data inside the container.
Therefore, each time you run docker-compose up and docker-compose down, it creates new volume. You can confirm the behavior with docker volume ls.
To avoid it, you have two options:
(A) Map host directory into container
You can map directory of host into container using <HOST_PATH>:<CONTAINER_PATH>.
volumes:
- /path/to/your/host/directory:/var/lib/postgresql/data
The data of postgresql will be saved into /path/to/your/host/directory of the container host.
(B) Use external container
docker-compose has an option of external container.
When it is set to true, it won't always create volume.
Here's an example.
version: '2'
services:
dbdata:
image: postgres:9.5.2
volumes:
- mypostgresdb:/var/lib/postgresql/data
volumes:
mypostgresdb:
external: true
With external: true, docker-compose won't create the mypostgredb volume, so you have to create it by your own using following command:
docker volume create --name=mypostgredb
The data of postgresql will be saved into docker volume named mypostgredb. Read reference for more detail.