I'm currently using Docker Compose to setup Flyway to handle table creation and migration.
When all my docker images run, Flyway tells me it successfully validated my migrations but when I check the wallet database and list tables, no relations exist.
Here is my docker compose file:
version: '3'
services:
web:
build:
dockerfile: ./Dockerfile
container_name: wallet-api-web
depends_on:
- flyway
ports:
- "8080:8080"
db:
build:
dockerfile: ./utils/db.Dockerfile
container_name: postgres-db
environment:
POSTGRES_PASSWORD: testuser1234
POSTGRES_USER: test_user
POSTGRES_DB: wallet
image: postgres:13
ports:
- "5432:5432"
restart: always
flyway:
command: -url=jdbc:postgresql://db:5432/wallet -schemas=wallet -user=test_user -password=testuser1234 -connectRetries=60 migrate
container_name: flyway-migrations
depends_on:
- db
image: flyway/flyway
volumes:
- ./migration:/flyway/sql
Here is the dockerfile used to build web.
FROM node:15
ENV NODE_ENV=production
WORKDIR /app
COPY ["package.json", "package-lock.json*", "./"]
RUN npm install --production
COPY . .
RUN npm run build
ENTRYPOINT npm run start
Here is the Dockerfile used to build the database image.
FROM postgres:latest
ENV POSTGRES_DB wallet
COPY ../init.sql /docker-entrypoint-initdb.d/
Here is the init file used by the postgres image.
CREATE USER test_user;
CREATE DATABASE wallet;
GRANT ALL PRIVILEGES ON DATABASE wallet TO test_user;
Here is my migration file:
CREATE TABLE wallet.transaction_events (
transaction_version int,
transaction_type VARCHAR(200),
coins int,
transaction_id VARCHAR(500),
wallet_id VARCHAR(500) PRIMARY KEY
)
This is my current file structure (directories use - and files use +)
base directory
- migration
- sql
+ V1__transaction_events.sql
- utils
- db.Dockerfile
+ Dockerfile
+ init.sql
+ docker-compose.yml
+ package-lock.json
+ package.json
Any assistance would be greatly appreciated as i'm new to docker and flyway and this has been stumping me for awhile. If any follow up information is required ill be sticking around so feel free to ask. Thank you again.
Related
I have my NestJS application that use PrismaORM for connection to PostgreSQL. But building of docker file crashes after executing npx prisma migrate dev --name init with error Can't reach database server at postgres:5432
My docker-compose.yml
version: "3.8"
services:
api:
build:
dockerfile: Dockerfile
context: .
depends_on:
- postgres
env_file:
- ./.env
ports:
- "8080:5000"
postgres:
image: postgres:10.4
ports:
- "5432:5432"
environment:
POSTGRES_USER: user
POSTGRES_PASSWORD: password
POSTGRES_DB: db
volumes:
- ./postgres-data:/var/lib/postgresql/data
env_file:
- ./.env
**My Dockerfile
**
FROM node:16
WORKDIR /qmessanger/src/server
COPY package*.json ./
COPY prisma ./prisma/
COPY .env ./
COPY . .
RUN npm install
RUN npm run build
RUN npx prisma generate
RUN npx prisma migrate dev --name init
EXPOSE 8080
CMD [ "node", "dist/main" ]
My .env config
DATABASE_URL="postgresql://user:password#postgres:5432/db"
POSTGRES_HOST=postgres
POSTGRES_PORT=5432
POSTGRES_USER=user
POSTGRES_PASSWORD=password
POSTGRES_DB=db
My prisma config
generator client {
provider = "prisma-client-js"
}
datasource db {
provider = "postgresql"
url = env("DATABASE_URL")
}
The issue is that you're trying to run the migration before the postgres service is running. You would need to run the migration as part of your startup command or entrypoint.
Here is an example of a similar problem in Django and they used an entrypoint script to run the migration:
How do you perform Django database migrations when using Docker-Compose?
Kindly ask you to help with docker and Postgres.
I have a local Postgres database and a project on NestJS.
I killed 5432 port.
My Dockerfile
FROM node:16.13.1
WORKDIR /app
COPY package.json ./
COPY yarn.lock ./
RUN yarn install
COPY . .
COPY ./dist ./dist
CMD ["yarn", "start:dev"]
My docker-compose.yml
version: '3.0'
services:
main:
container_name: main
build:
context: .
env_file:
- .env
volumes:
- .:/app
- /app/node_modules
ports:
- 4000:4000
- 9229:9229
command: yarn start:dev
depends_on:
- postgres
restart: always
postgres:
container_name: postgres
image: postgres:12
env_file:
- .env
environment:
PG_DATA: /var/lib/postgresql/data
POSTGRES_HOST_AUTH_METHOD: 'trust'
ports:
- 5432:5432
volumes:
- pgdata:/var/lib/postgresql/data
restart: always
volumes:
pgdata:
.env
DB_TYPE=postgres
DB_HOST=postgres
DB_PORT=5432
DB_USERNAME=hleb
DB_NAME=artwine
DB_PASSWORD=Mypassword
running sudo docker-compose build - NO ERRORS
running sudo docker-compose up --force-recreate - ERROR
ERROR [ExceptionHandler] role "hleb" does not exist.
I've tried multiple suggestions from existing issues but nothing helped.
What am I doing wrong?
Thanks!
Do not use sudo - unless you have to.
Use the latest Postgres release if possible.
The Postgresql Docker Image provides some environment variables, that will help you bootstrapping your database.
Be aware:
The PostgreSQL image uses several environment variables which are easy to miss. The only variable required is POSTGRES_PASSWORD, the rest are optional.
Warning: the Docker specific variables will only have an effect if you start the container with a data directory that is empty; any pre-existing database will be left untouched on container startup.
When you do not provide the POSTGRES_USER environment variable in the docker-compose.yml file, it will default to postgres.
Your .env file used for Docker Compose does not contain the docker specific environment variables.
So amending/extending it to:
POSTGRES_USER=hleb
POSTGRES_DB=artwine
POSTGRES_PASSWORD=Mypassword
should do the trick. You will have to re-create the volume (delete it) to make this work, if the data directory already exists.
I am trying to create table 'User' in my database with a docker container with PostgreSQL and pgAdmin on docker-compose up.
I am following this StackOveflow article here.
DockerFile
FROM postgres:latest
COPY schema.sql /docker-entrypoint-initdb.d/
docker-compose.yaml
version: "3.8"
services:
db:
build: .
restart: always
environment:
POSTGRES_DB: xxx
POSTGRES_USER: admin
POSTGRES_PASSWORD: xxx
PGDATA: /var/lib/postgresql/data
ports:
- "5432:5432"
volumes:
- db-data:/var/lib/postgresql/data
pgadmin:
image: dpage/pgadmin4:latest
restart: always
environment:
PGADMIN_DEFAULT_EMAIL: admin#xxx.com
PGADMIN_DEFAULT_PASSWORD: xxx
PGADMIN_LISTEN_PORT: 80
ports:
- "8080:80"
volumes:
- pgadmin-data:/var/lib/pgadmin
links:
- "db:pgsql-server"
volumes:
db-data:
pgadmin-data:
schema.sql
create Table Users (
id SERIAL,
uuid VARCHAR(100) NOT NULL
)
The container and admin console starts up fine. However, the table is not generated and cannot be found in pgAdmin.
I exec-ed into the container and the file schema.sql is well copied into /docker-entrypoint-initdb.d/ directory.
In the PostgreSQL Docker repository, the documentation mentions that my data directory has to be empty or the files in the init folder will not be run. Could this be my issue?
How can I create my User table in my dockerized PostgreSQL when I run the container?
This is my docker-compose file. Is there any easy way to get a postgres extension installed? I'm trying to install pg_trgm.
Edit: I now have two dockerfiles and an install script. It doesn't seem to be working when I run docker-compose up build
Internal server error: pq: operator does not exist: character varying % unknown
services:
db:
build:
context: .
dockerfile: db/Dockerfile
image: postgres
ports:
- "5432:5432"
environment:
- POSTGRES_USER=x
- POSTGRES_PASSWORD=x
- POSTGRES_DB=x
api:
build:
context: .
args:
app_env: ${APP_ENV}
volumes:
- .:/go/src/x/y/z
ports:
- "8080:8080"
db/Dockerfile:
FROM postgres
COPY db/install-extensions.sql /docker-entrypoint-initdb.d
db/install-extensions.sql
CREATE EXTENSION IF NOT EXISTS pg_trgm;
Try this
FROM postgres
COPY ./install-extensions.sql /docker-entrypoint-initdb.d
and remove db from your file.
OR you can write
version: "3.1"
services:
db:
image: postgres:9.6
restart: always
environment:
POSTGRES_PASSWORD: unit
POSTGRES_USER: unit
POSTGRES_DB: unit
ports:
- 5432:5432
volumes:
- ./scripts:/docker-entrypoint-initdb.d
then
create directory scripts
put your .sql or .sh file
remove created containers docker-compose rm -v
start docker docker-compose up --build
in logs you must see something like this:
created_extension
I'm not sure precisely why but in order to get this working I had to use a shell script:
#!/usr/bin/env bash
echo "enabling pg_trgm on database $POSTGRES_DB"
psql -U $POSTGRES_USER --dbname="$POSTGRES_DB" <<-'EOSQL'
create extension if not exists pg_trgm;
EOSQL
echo "finished with exit code $?"
possibly because I was overriding the default database and user name.
I have a problem migrating using Knex js inside my docker-compose container.
the problem is that npm run db (knex migrate:rollback && knex migrate:latest && knex seed:run) would run right before the database is even created. Is there anyway to say that I would only like to run npm run db after the database has been created?
NOTE : if I do this npm commands on the docker terminal after it has been built everything works fine. just fyi
here is my docker-compose.yml
version: '3.6'
services:
#Backend api
server:
container_name: server
build: ./
command: npm run db
working_dir: /user/src/server
ports:
- "5000:5000"
volumes:
- ./:/user/src/server
environment:
POSTGRES_URI: postgres://test:test#192.168.99.100:5432/interapp
links:
- postgres
# PostgreSQL database
postgres:
environment:
POSTGRES_USER: test
POSTGRES_PASSWORD: test
POSTGRES_DB: interapp
POSTGRES_HOST: postgres
image: postgres
ports:
- "5432:5432"
and here is my Dockerfile
FROM node:10.14.0
WORKDIR /user/src/server
COPY ./ ./
RUN npm install
CMD ["/bin/bash"]
on the docker-compose.yml file, using sh (bash) for a contained environment context for your command to run in. ie. sh -c 'npm run db'
your docker-compose file would now be
secondly, use the depends_on step to wait for the database to start
services:
#Backend api
server:
container_name: server
build: ./
command: sh -c 'npm run db'
working_dir: /user/src/server
depends_on:
-postgres
ports:
- "5000:5000"
volumes:
- ./:/user/src/server
environment:
POSTGRES_URI: postgres://test:test#192.168.99.100:5432/interapp
links:
- postgres
Simply adding depends_on to server service should do the trick here.
services:
server:
depends_on:
- postgres
...
This will cause docker-compose to start postgres container before the server container. It will not however wait for postgres to be ready. In this case it shouldn't be problem, because postgres starts really quickly.
If you want something more solid, or depends_on doesn't do the trick, you can add entrypoint wrapping script to your container. See https://docs.docker.com/compose/startup-order/, where you can read more about it. There are also links to tools, so you don't have to write your own script from scratch.