Extend docker postgres image to create extra database - postgresql

I've already seen this topic : https://stackoverflow.com/a/26599273/2323245
But I have the following problem :
postgres_1 | FATAL: role "docker" does not exist
app_1 | Error: Could not establish a connection with the database
This is my docker-compose.yml file
version: "2"
services:
app:
build:
context: .
dockerfile: Dockerfiles/app.dockerfile
links:
- postgres
depends_on:
- postgres
ports:
- "8080:8080"
environment:
PORT: 8080
networks:
- neo_dev
restart: always
postgres:
build:
context: .
dockerfile: Dockerfiles/postgres.dockerfile
networks:
- neo_dev
volumes:
- postgresql:/var/lib/postgresql
# This needs explicit mapping due to https://github.com/docker-library/postgres/blob/4e48e3228a30763913ece952c611e5e9b95c8759/Dockerfile.template#L52
- postgresql_data:/var/lib/postgresql/data
ports:
- "5430:5432" #in case you already have postgres running in your host machine
networks:
neo_dev:
driver: bridge
volumes:
postgresql:
postgresql_data:
My postgres.dockerfile is
FROM library/postgres
MAINTAINER Samir Bouaked "sbouaked#neocasesoftware.com"
ADD Dockerfiles/init.sql /docker-entrypoint-initdb.d/
And this is my init.sql file
CREATE USER docker with password 'docker';
CREATE DATABASE docker;
GRANT ALL PRIVILEGES ON DATABASE docker TO docker;
In my golang app, i'm trying to reach the database with
router.GET("/db", func(c *gin.Context) {
db, err := sql.Open("postgres", "host=postgres port=5432 user=docker dbname=docker password=docker sslmode=disable")
if err != nil {
log.Fatal("Error: The data source arguments are not valid - " + err.Error())
}
err = db.Ping()
if err != nil {
log.Println("Error: Could not establish a connection with the database")
c.String(http.StatusOK, "hotstName : %s\nDbStatus : %s", os.Getenv("HOSTNAME"), err)
} else {
c.String(http.StatusOK, "\nhotstName : %s\nDbStatus : connection ok !\n", os.Getenv("HOSTNAME"))
}
defer db.Close()
})
I tried different solutions like passing directly in the dockerfile env variables like that
ENV POSTGRES_USER docker
ENV POSTGRES_PASSWORD docker
ENV POSTGRES_DB docker
But have always the same result...
For more information you can check this repo https://github.com/neocase/docker-go-starter-kit
I don't understand the problem

It was a problem of docker volume.
I have to docker volume rm postgresvolume and do again a docker-compose up --build
It wasn't a problem of environment variables, but juste the fact that the init.sql was in cash and not run at the begining. It's due to some test I make a the start of the project

If you want to create the DB by passing ENV variables using docker-compose you can specify something like
postgres:
container_name: postgres
image: library/postgres
ports:
- "5432:5432"
environment:
- POSTGRES_DATABASE=docker
- POSTGRES_USER=docker
- POSTGRES_PASSWORD=docker
- POSTGRES_ADMIN_PASSWORD=docker
in your compose YML file.
This should create you a postgres container with an DB docker with user/pwd docker, no need for your own dockerfile anymore (so you can use any postgres image that supports this behavior)
After running compose (and if it doesn't work as you expect) you should have a look at the logs if your postgres container is working at all, or if there is an issue with your app.

Related

Moved docker-compose.yml creates a new postgres database

I have set up a Postgres database on docker on ubuntu with the docker-compose.yml just for that database within the folder ~/postgres and I'd run docker-compose up -d to run my database from within the ~/postgres folder.
Here is my docker-compose.yml:
version: '3'
services:
database:
image: "postgres"
ports:
- "0.0.0.0:5432:5432"
env_file:
- database.env
volumes:
- database-data:/var/lib/postgresql/data/
volumes:
database-data:
This database is set up and working perfectly, so I decided to set up my web application as well and, because the docker-compose.yml file was inside that folder, I moved it outside to ~/ so I could use it for my web app as well.
This is what the docker-compose.yml in ~/ looks like:
version: "3"
services:
database:
image: "postgres"
ports:
- "0.0.0.0:5432:5432"
env_file:
- postgres/database.env
volumes:
- database-data:/var/lib/postgresql/data/
webapp:
image: webapp/site
build:
context: ./retro-search-engine
dockerfile: Dockerfile
args:
buildno: 1
links:
- "database:db"
ports:
- "0.0.0.0:8000:80"
volumes:
- webapp:/var/www
environment:
db_host: db
db_username: xxxx
db_password: xxxx
db_database: xxxx
db_port: 5432
volumes:
database-data:
webapp:
As you can see, the database docker configuration is basically the same, the only thing that changes is the path to the database.env file since it's still in the previous folder.
So, the problem here is that when I run docker-compose up -d from ~/, everything starts normally but when I access the database, all of my tables are gone.
If I go back to ~/postgres and do docker-compose up -d in that folder (with the previous docker-compose.yml) and connect to the db, I can access my tables.
So what I think is happening is that it's either creating a new container or somehow the folder where the data is stored is relative to the docker-compose.yml file and it's creating a new database because it can't find the old files.
I have no idea how to solve this issue, I have googled around and couldn't find anything so I decided to ask here before I dump my whole db and restore it into a new container, which I don't want to do because it's a 16gb database and it's gonna take forever.
Does anyone have any idea how I can use my new docker-compose.yml with the data from my database?
Thanks in advance.
First:
Replace : postgres/database.env by ./postgres/database.env
Use docker compose up --build :
it will rebuild the image (usefull if you made some change to your dockerfile). try to avoid to use -d when developing, you'll avoid to have tons of container running.
Second:
I suggest you to follow the following reco, It will resolve your problem and it will be cleaner if you want to use a pipeline CI/CD and to create more "autonomous" image and container on demand.
rootfolder
|-docker-compose.yaml
|-postgres/
| |--All_other_files_for_the_postgres_docker_image
|-webapp/
|-- Dockerfile
|-- All_other_files_for_the_webapp_docker_image
bellow you will find my "correction" :
version: "3"
services:
database:
image: "postgres"
container_name: "my_postgres_container"
ports:
- "0.0.0.0:5432:5432"
env_file:
- ./postgres/database.env
volumes:
- database-data:/var/lib/postgresql/data/
webapp:
image: webapp/site
container_name: "my_webapp_container"
build:
context: ./retro-search-engine
dockerfile: Dockerfile
links:
- "database:db"
ports:
- "0.0.0.0:8000:80"
volumes:
- webapp:/var/www
environment:
db_host: db
db_username: xxxx
db_password: xxxx
db_database: xxxx
db_port: 5432
volumes:
database-data:
webapp:
If you want to use an existing postgres image that is already present (to see if an image already existe you can do : docker image | grep postgres)
then you can do directly in your docker-compose :
image: "<your_image_name>"

How to make postgres listen on the container's new exposed port (not 5432)?

I try to initialize a database with Go.
I use port 5433 at postgres:alpine because 5432 is already taken by another microservice app.
func Init() {
DB, err = gorm.Open(postgres.New(postgres.Config{
DSN: "host=url_db user=gorm password=gorm dbname=gorm port=5433 sslmode=disable TimeZone=Asia/Tokyo",
}), &gorm.Config{})
if err != nil {
panic(err)
}
autoMigration()
}
url_db:
build:
context: ./api/services/url/db
dockerfile: Dockerfile
container_name: "url_db"
environment:
POSTGRES_USER: gorm
POSTGRES_PASSWORD: gorm
POSTGRES_DB: gorm
POSTGRES_HOST: url_db
ports:
- 5433:5433
You can confirm that only 5432 is exposed here.
I tried to expose 5433 by creating a new Dockerfile like this.
FROM postgres:alpine
EXPOSE 5433
But I got this error.
failed to initialize database, got error failed to connect to `host=url_db user=gorm database=gorm`: dial error (dial tcp 172.19.0.3:5433: connect: connection refused)
This comment:
Simply exposing the port on the docker image won't do anything unless postgres is actually configured to listen on that port. – super 5 mins ago
that teaches me the title(How can I expose a new port(not 5432) at postgres:alpine image?) is not the point, so I updated the title.
How to make postgres listen on the container's new exposed port (not 5432)?
You have multiple options:
Option 1: Define own postgresql.conf
url_db:
build:
context: ./api/services/url/db
dockerfile: Dockerfile
container_name: "url_db"
command: postgres -c "config_file=/etc/postgresql/postgresql.conf"
environment:
POSTGRES_USER: gorm
POSTGRES_PASSWORD: gorm
POSTGRES_DB: gorm
POSTGRES_HOST: url_db
ports:
- 5433:5433
volumes:
- /path/to/config:/etc/postgresql/postgresql.conf
Postgres has an example config at /usr/share/postgresql/postgresql.conf.sample within the container.
To get the config run:
docker run -i --rm postgres cat /usr/share/postgresql/postgresql.conf.sample > my-postgres.conf
Option 1: Overwrite the RUN command
url_db:
build:
context: ./api/services/url/db
dockerfile: Dockerfile
container_name: "url_db"
command: postgres -c port=5433
environment:
POSTGRES_USER: gorm
POSTGRES_PASSWORD: gorm
POSTGRES_DB: gorm
POSTGRES_HOST: url_db
ports:
- 5433:5433
You can have multiple containers that are internally listening on the same port, so long as they're mapped to different ports on the host (if they're published at all). In your example, you can set
url_db:
image: postgres:latest
environment:
POSTGRES_USER: gorm
et: cetera
ports:
- 5433:5432
Connections from outside Docker reach the remapped port, on <host ip>:5433. Connections between Docker containers use the standard service port, on url_db:5432. These connections ignore (and don't require) ports:.
"Expose" in modern Docker means almost nothing; it is most valuable as documentation in an image showing what port(s) the service normally uses. You can in theory ask Compose to expose: additional ports without modifying the image, but there's no practical effect from doing so.
For Docker Compose assuming we want to change port to 5433
An alternative to doing this is to do the following in your docker-compose.yml file
you can mount the postgres data into a volume in your directory in this case ./db
postgres:
image: postgres:10.14-alpine
environment:
POSTGRES_DB: iam
ports:
- 5433:5433
volumes:
- ./db:/var/lib/postgresql/data
Find the postgresql.conf file, then search the port
change the port to
port = 5433 # (change requires restart)

server and postgresql cannot connect when both running on Docker `getaddrinfo ENOTFOUND`

after creating docker containers with docker compose file (below), I call
$ docker run myApp
However, I get
Error: getaddrinfo ENOTFOUND main_db
this only happens when both server and postgresql are in docker containers (I am able to connect to postgresql on localhost)
I'm running a NestJS app using TypeOrm to connect to a postgresql server
inside the app.module.ts where it boots up the connection my config should match my docker postgresql config. the host points to the container I created on docker main_db and I declared this as a dependency of my server, the main service. Everything should be on the same network webnet.:
TypeOrmModule.forRoot({
type: 'postgres',
host: 'main_db',
port: +process.env.POSTGRES_PORT,
username: process.env.POSTGRES_USER,
password: process.env.POSTGRES_PASSWORD,
database: process.env.POSTGRES_DB,
autoLoadEntities: true,
synchronize: true,
logging: dbLogging,
}),
docker-compose.yml
version: '3.7'
services:
main:
container_name: main
build:
context: .
target: development
volumes:
- .:/usr/src/app
- /usr/src/app/node_modules
ports:
- ${SERVER_PORT}:${SERVER_PORT}
- 9229:9229
command: npm run start:dev
env_file:
- .env
networks:
- webnet
depends_on:
- main_db
main_db:
container_name: main_db
image: postgres:12
restart: always
networks:
- webnet
environment:
POSTGRES_DB: ${POSTGRES_DB}
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
POSTGRES_USER: ${POSTGRES_USER}
PG_DATA: /var/lib/postgresql/data
ports:
- '${POSTGRES_PORT}:${POSTGRES_PORT}'
volumes:
- pgdata:/var/lib/postgresql/data
networks:
webnet:
volumes:
pgdata:
.env file
POSTGRES_PORT=5432
POSTGRES_USER=test
POSTGRES_PASSWORD=test
POSTGRES_DB=test
SERVER_PORT=3001
When creating a Dockerfile and a docker-compose.yml file and you call docker run myApp for the app defined inside of the Dockerfile instead of calling docker-compose up, you will see the app running; however, it will not start the containers defined in the docker-compose file. In the case of the NestJS server, it was running the server, but could not find the container with the database, since this container was not being spun up. Although the distinction between the app setup in the Dockerfile and the definition of the containers in the docker-compose.yml was clear, I didn't realize that the docker command didn't reference the docker-compose.yml. Thus, posting here in case anyone else has a similar confusion.

docker faild to restore postgres backup

I'm trying to create and restore postgres backup using docker.
the docker failed to do it and gives me the following error:
/usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
dockerfile:
FROM postgres:11
ENV POSTGRES_USER postgres
ENV POSTGRES_PASSWORD postgres
ENV POSTGRES_DB dbName
COPY backup.backup /
COPY initdb.sh /docker-entrypoint-initdb.d
initdb.sh:
pg_restore --username=postgres --create --exit-on-error --verbose --dbname=dbName backup.backup
docker-compose.yml:
version: '2'
services:
db:
image: postgres:11
expose:
- "5432"
ports:
- "15432:5432"
volumes:
- dock-volume:/var/lib/postgresql/data
environment:
- POSTGRES_PASSWORD= postgres
- POSTGRES_DB= dbName
volumes:
dock-volume:
I tried to add the environment variables to docker-compose.yml but it doesnt help..
You should not create a Dockerfile for the Postgres because you already have the definition in your Dockercompose file. The variables that you define under environment are visible inside Postgres. If you want to have a backup and make sure that is running when you initialize you can do:
volumes:
- ~/Downloads/data/my_buckup.psql:/docker-entrypoint-initdb.d/stage.sql
Then when Postgres initialize this script will be run. You can see the documentation here under the section How to extend this image
I hope it makes sense

Gorm + Docker error while connection

I trying connection to my postgresql database with Docker:
package main
import (
"fmt"
"log"
"github.com/jinzhu/gorm"
_ "github.com/jinzhu/gorm/dialects/postgres"
)
type Product struct {
gorm.Model
Code string
Price uint
}
var db *gorm.DB
func init() {
connection := fmt.Sprintf("host=db sslmode=disable user=dnz-dev password=dnz-dev")
db, err := gorm.Open("postgres", connection)
if err != nil {
log.Fatalln(err)
}
defer db.Close()
}
func main() {
// Migrate the schema
db.AutoMigrate(&Product{})
}
and docker-compose
version: "3.3"
services:
db:
build: ./dnz-db
container_name: dnz-database
ports:
- "6000:5432"
volumes:
- ./dnz-db/data:/var/lib/postgresql/data
environment:
- POSTGRES_USER=dnz-dev
- POSTGRES_PASSWORD=dnz-dev
api:
build: ./dnz-api
container_name: dnz-api
volumes:
- ./dnz-api:/go/src/app
ports:
- "5000:3000"
depends_on:
- db
I run docker-compose up --build and I get this error:
Attaching to dnz-database, dnz-api dnz-api | 2017/11/05 10:23:46 dial
tcp 172.21.0.2:5432: getsockopt: connection refused dnz-api | exit
status 1 dnz-api exited with code 1
What I doing wrong?
You are unable to connect to your psql container because you haven't linked it. Take a look at Docker-Compose documentation on links.
Also, I'm assuming you aren't scheduling your container startup order. Psql container must start before golang's one. Take a look at Docker Compose documentation on startup order. You can use wait-for-it to achieve such objective ast stated in the docs. Just wget it and save it in your project.
I don't know the contents of your Dockerfile but I'll assume it's something like:
FROM golang:1.9
RUN mkdir -p /go/src/github.com/pavel/gorm-psql
WORKDIR /go/src/github.com/pavel/gorm-psql
ADD . /go/src/github.com/pavel/gorm-psql
RUN go get -v
So, your docker-compose.yml should be edited to first run wait-for-it.sh and link psql container to your app with something like:
version: '3.3'
services:
db:
image: postgres
environment:
POSTGRES_DB: dnz-dev
POSTGRES_USER: dnz-dev
POSTGRES_PASSWORD: dnz-dev
ports:
- 6000:5432
api:
build: .
command: ["./wait-for-it.sh", "db:6000", "--", "go", "run", "main.go"]
volumes:
- .:/go/src/github.com/pavel/gorm-psql
ports:
- "5000:3080"
depends_on:
- db
links:
- db
If your main has another name just change it. Edit your volumes to point whatever path you need. I've set a standard one with /go/src/github.com/pavel/gorm-psql. I'm assuming you created a db named dnz-dev, if the name is different just edit it.
$ go env:
. . .
GOPATH="/home/pavel/go"
GOROOT="/usr/lib/go"
. . .
Just run docker-compose up and it should work just fine. I'm relying on postgresql and golang latest images.