In the past, I created a docker-compose.yml that has a postgres database with adminer, and I persisted the data to a local directory. I have just tried to do that again, but I receive an error saying:
initdb: error: directory "/var/lib/postgresql/data" exists but is not empty
initdb: hint: If you want to create a new database system, either remove or empty the directory "/var/lib/postgresql/data" or run initdb with an argument other than "/var/lib/postgresql/data".
So this says that there is something already in the container's data storage directory despite it being a fresh container. But because I didn't get this before it seems like there was a change with the docker image, although I am just guessing from the message. How can I resolve this now? I would just like to have a docker postgres running that will persist whatever data I add to it. Here is my compose file:
version: '3.9'
services:
db:
image: postgres
restart: always
ports:
- 5432:5432
environment:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: pgpassword
POSTGRES_DB: my_db
healthcheck:
test: ["CMD", "pg_isready", "-q", "-d", "my_db", "-U"]
timeout: 45s
interval: 10s
retries: 10
volumes:
- ./db/pgdata:/var/lib/postgresql/data/pgdata
adminer:
image: adminer
restart: always
ports:
- 8080:8080
Related
I created my app with .yml
services:
db:
image: postgres:11-alpine
environment:
POSTGRES_DB: wiki
POSTGRES_PASSWORD: quantoxrocks
POSTGRES_USER: wikijs
logging:
driver: "none"
restart: unless-stopped
volumes:
- db-data:/var/lib/postgresql/data
wiki:
image: ghcr.io/requarks/wiki:2
depends_on:
- db
environment:
DB_TYPE: postgres
DB_HOST: db
DB_PORT: 5432
DB_USER: wikijs
DB_PASS: quantoxrocks
DB_NAME: wiki
restart: unless-stopped
ports:
- "3000:3000"
webserver:
image: nginx:alpine
restart: unless-stopped
tty: true
ports:
- "443:443"
- "80:80"
volumes:
- ./nginx/conf.d:/etc/nginx/conf.d
- ./ssl:/etc/nginx/ssl
volumes:
db-data:
I logged in my db container and want to create database. I have tried at least 10 times and I am sure that password is from the above docker-compose.yml file. It does not work.
docker exec -it wiki_db_1 sh
Next
psql -h wiki_db_1 -U wikijs
Password for user wikijs:
psql: FATAL: password authentication failed for user "wikijs"
Why? How can I check any further logs?
The environment variables for Postgres are only used if there is no database present already when the container starts.
You have a volume mapping of /var/lib/postgresql/data and it's likely that you already have a database there, which was created with different values from the environment variable values.
If you don't have any important data in the existing database, you can delete the volume and Postgres will create a new database with the correct username/password.
I am testing some stuff where I have to init my Postgres DB DDL into airflow Postgres DB when I compose-up it should automatically init for one time as it will be cached afterward as airflow DB works usually. Thanks
As requested in the last comment: Adding your own database to the Airflow Docker-compose file:
Put this piece of code as a service somewhere amongst the other services:
mypostgres:
image: postgres:13
environment:
POSTGRES_USER: mydbuser
POSTGRES_PASSWORD: securepassword
POSTGRES_DB: mydb
volumes:
- ./database:/var/lib/postgresql/data
- ./init-database.sh:/docker-entrypoint-initdb.d/init-database.sh
restart: always
Make sure you have a database-directory and a init-database.sh file in the current directory (otherwise the volume mappings fail)
I have found a solution that works and init your scripts when you docker composed up.
pro TIP:
If you want to add more files and you have already init the airflow DB or your DB what you can do is docker-compose down --volume what this will do will automatically remove all the data in the data directory. and for init to work Postgres data directory have to be empty
postgres:
image: postgres:13
environment:
POSTGRES_USER: airflow
POSTGRES_PASSWORD: airflow
POSTGRES_DB: airflow
ports:
- "5432:5432"
volumes:
- postgres-db-volume:/var/lib/postgresql/data
- /path/to/my/host/folder/filename.sql:/docker-entrypoint-initdb.d/filename.sql
healthcheck:
test: ["CMD", "pg_isready", "-U", "airflow"]
interval: 5s
retries: 5
restart: always
volumes:
postgres-db-volume:
Dear stackoverflowers,
I use docker-compose to run the dockerized postgresql server and dockerized pgadmin4 webserver.
When i try to restore a dump via the web interface it shows me an empty folder with the path "/" for source location of the dump.
Now my question, is it in general possible to restore a dump via dockerized pgadmin and if, what path from which container (postgres or pgadmin) do i have to mount as volume to provide the dump to be restored?
version: '3.8'
services:
db:
container_name: pg_container
image: postgres:12.10
restart: always
environment:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: ***
POSTGRES_DB: postgres
ports:
- "5432:5432"
pgadmin:
container_name: pgadmin4_container
image: dpage/pgadmin4
restart: always
environment:
PGADMIN_DEFAULT_EMAIL: admin#admin.com
PGADMIN_DEFAULT_PASSWORD: ***
ports:
- "5050:80"
With kind regards
starguy
I'm in the same situation as you are. In the web pgadmin, you have an option to upload a .sql file just by clicking at the ... button.
I have set up a Postgres database on docker on ubuntu with the docker-compose.yml just for that database within the folder ~/postgres and I'd run docker-compose up -d to run my database from within the ~/postgres folder.
Here is my docker-compose.yml:
version: '3'
services:
database:
image: "postgres"
ports:
- "0.0.0.0:5432:5432"
env_file:
- database.env
volumes:
- database-data:/var/lib/postgresql/data/
volumes:
database-data:
This database is set up and working perfectly, so I decided to set up my web application as well and, because the docker-compose.yml file was inside that folder, I moved it outside to ~/ so I could use it for my web app as well.
This is what the docker-compose.yml in ~/ looks like:
version: "3"
services:
database:
image: "postgres"
ports:
- "0.0.0.0:5432:5432"
env_file:
- postgres/database.env
volumes:
- database-data:/var/lib/postgresql/data/
webapp:
image: webapp/site
build:
context: ./retro-search-engine
dockerfile: Dockerfile
args:
buildno: 1
links:
- "database:db"
ports:
- "0.0.0.0:8000:80"
volumes:
- webapp:/var/www
environment:
db_host: db
db_username: xxxx
db_password: xxxx
db_database: xxxx
db_port: 5432
volumes:
database-data:
webapp:
As you can see, the database docker configuration is basically the same, the only thing that changes is the path to the database.env file since it's still in the previous folder.
So, the problem here is that when I run docker-compose up -d from ~/, everything starts normally but when I access the database, all of my tables are gone.
If I go back to ~/postgres and do docker-compose up -d in that folder (with the previous docker-compose.yml) and connect to the db, I can access my tables.
So what I think is happening is that it's either creating a new container or somehow the folder where the data is stored is relative to the docker-compose.yml file and it's creating a new database because it can't find the old files.
I have no idea how to solve this issue, I have googled around and couldn't find anything so I decided to ask here before I dump my whole db and restore it into a new container, which I don't want to do because it's a 16gb database and it's gonna take forever.
Does anyone have any idea how I can use my new docker-compose.yml with the data from my database?
Thanks in advance.
First:
Replace : postgres/database.env by ./postgres/database.env
Use docker compose up --build :
it will rebuild the image (usefull if you made some change to your dockerfile). try to avoid to use -d when developing, you'll avoid to have tons of container running.
Second:
I suggest you to follow the following reco, It will resolve your problem and it will be cleaner if you want to use a pipeline CI/CD and to create more "autonomous" image and container on demand.
rootfolder
|-docker-compose.yaml
|-postgres/
| |--All_other_files_for_the_postgres_docker_image
|-webapp/
|-- Dockerfile
|-- All_other_files_for_the_webapp_docker_image
bellow you will find my "correction" :
version: "3"
services:
database:
image: "postgres"
container_name: "my_postgres_container"
ports:
- "0.0.0.0:5432:5432"
env_file:
- ./postgres/database.env
volumes:
- database-data:/var/lib/postgresql/data/
webapp:
image: webapp/site
container_name: "my_webapp_container"
build:
context: ./retro-search-engine
dockerfile: Dockerfile
links:
- "database:db"
ports:
- "0.0.0.0:8000:80"
volumes:
- webapp:/var/www
environment:
db_host: db
db_username: xxxx
db_password: xxxx
db_database: xxxx
db_port: 5432
volumes:
database-data:
webapp:
If you want to use an existing postgres image that is already present (to see if an image already existe you can do : docker image | grep postgres)
then you can do directly in your docker-compose :
image: "<your_image_name>"
I have a docker-compose.yaml that spins up an express server and postgres database in separate containers. The postgres container maps a volume to a db folder, which contains database seeding scripts. Locally, docker-compose runs fine and I can execute integration tests against the networked containers. However, the same scripts fail when running on Jenkins, with the following error:
ERROR: for db Cannot start service db: error while creating mount
source path
'/var/jenkins_home/workspace/project_name/db':
mkdir /var/jenkins_home: read-only file system
docker-compose.yaml
...
db:
image: postgres:10.5
restart: always
networks:
- cloud
environment:
POSTGRES_USER: someuser
POSTGRES_PASSWORD: somepassword
POSTGRES_DB: somedb
ports:
- 5432:5432
volumes:
- ./db:/docker-entrypoint-initdb.d
healthcheck:
test: ["CMD-SHELL", "pg_isready -U someuser -d somedb"]
interval: 10s
timeout: 5s
retries: 5
I have read in several places that volumes require absolute paths (not relative), but I tried harcoding /var/jenkins_home/workspace/project_name/db to the left side of the volume config to no avail.