I am trying to integrate bitbucket pipeline with my python package that use Postgres database.
To achieve this I am using Postgres service but I am not able to find any way in bitbucket-pipelines.yml to populate database schema.
Below is my bitbucket-pipeline.yml and now I am getting error "bash: psql: command not found"
image: python:2.7.13
definitions:
services:
postgres:
image: postgres
pipelines:
default:
- step:
caches:
- pip
script:
- python setup.py sdist
services:
- postgres
branches:
master:
- step:
name: Run unit/integration tests
deployment: test
caches:
- pip
script:
- sudo apt-get update && sudo apt-get install -y postgresql-client
- psql -c 'drop database if exists testdb;' -U postgres
- psql -c 'create database testdb;' -U postgres
- python setup.py sdist
- python -m unittest discover tests/
This worked for me (I had to remove the sudos for before the apt-get)
image: atlassian/default-image:2
clone:
depth: 5 # include the last five commits
definitions:
services:
postgres:
image: postgres
environment:
POSTGRES_DB: test_annotation
POSTGRES_USER: user
POSTGRES_PASSWORD: password
pipelines:
default:
- step:
caches:
- node
script:
- apt-get update && apt-get install -y postgresql-client
- PGPASSWORD=password psql -h localhost -p 5432 -U user test_annotation;
- chmod 755 ./scripts/create-test-database.sh
- ./scripts/create-test-database.sh
services:
- postgres
Make sure services is intented correctly, otherwise, the db won't start.
Julien
Related
I am trying to use psql in a github action but am seeing the following error:
psql: error: could not connect to server: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
where my github action yml file is shown below (the run_all_tests.sh file just calls a subprocess that tries to run the command psql). Does anyone know why this could be happening?
name: Python application
on:
push:
branches: [master]
pull_request:
branches: [master]
jobs:
build:
runs-on: ubuntu-latest
# Service containers to run with `container-job`
services:
# Label used to access the service container
postgres:
# Docker Hub image
image: postgres
# Provide the password for postgres
env:
POSTGRES_PASSWORD: postgres
# Set health checks to wait until postgres has started
ports:
- 5432:5432
options: >-
--health-cmd pg_isready
--health-interval 10s
--health-timeout 5s
--health-retries 5
steps:
- name: Copy the code
uses: actions/checkout#v2
- name: Set up Python 3.8
uses: actions/setup-python#v2
with:
python-version: 3.8
- name: Install dependencies
run: |
python3 setup.py install
- name: Test with unittest
run: |
cd backend/py
source run_all_tests.sh
env:
# The hostname used to communicate with the PostgreSQL service container
POSTGRES_HOST: postgres
# The default PostgreSQL port
POSTGRES_PORT: 5432
Since I was having the same issue, I tried a different approach that worked for me.
In the first place, I run the job within a container:
jobs:
build:
container: gradle:jdk11
That won't make the psql command available so you need to add a run step to install it. The particular installation method may differ depending on the Docker image you choose:
jobs:
build:
container: gradle:jdk11
...
steps:
- run: |
apt-get update
apt-get install --yes --no-install-recommends postgresql-client
Please note you may have different steps above or below.
Now it's time to execute all these SQL you need. The most important thing here: database hostname is postgres which is the id of the service container.
jobs:
build:
container: gradle:jdk11
...
steps:
- run: |
apt-get update
apt-get install --yes --no-install-recommends postgresql-client
- run: |
psql -h postgres -U postgres -c 'CREATE DATABASE ...'
psql -h postgres -U postgres -c 'CREATE ROLE ...'
Since the job is running directly on a runner machine (not within a docker container), you need to connect to "localhost" instead of "postgres". It should work if you change POSTGRES_HOST: postgres to POSTGRES_HOST: localhost.
This is described in detail in the docs: https://docs.github.com/en/actions/using-containerized-services/creating-postgresql-service-containers
I tried installing pg_cron on Postgres running inside a Docker container but getting this error could not access file "pg_cron": No such file or directory. Any ideas on how to resolve?
Based on https://stackoverflow.com/a/51797554, I tried the following:
docker-compose.yml
version: '3.7'
services:
pg:
container_name: pg-container
image: postgres:11.5
environment:
POSTGRES_DB: "pgdb"
POSTGRES_USER: "pguser"
POSTGRES_PASSWORD: "pgpass"
volumes:
- ./:/docker-entrypoint-initdb.d
- pgstorage
ports:
- "5432:5432"
volumes:
pgstorage:
002-setup.sh
#!/bin/sh
# Remove last line "shared_preload_libraries='citus'"
sed -i '$ d' ${PGDATA}/postgresql.conf
cat <<EOT >> ${PGDATA}/postgresql.conf
shared_preload_libraries='pg_cron'
cron.database_name='${POSTGRES_DB:-postgres}'
EOT
# Required to load pg_cron
pg_ctl restart
003-main.sql
CREATE EXTENSION pg_cron;
From what I can see you are not installing pg_cron anywhere. Since it is not packaged with the default Postgres Docker image you will have to care of that.
For example by extending the Image and using a build entry in your docker-compose.yml.
# Dockerfile relative to docker-compose.yml
FROM postgres:11.5
RUN apt-get update && apt-get -y install git build-essential postgresql-server-dev-11
RUN git clone https://github.com/citusdata/pg_cron.git
RUN cd pg_cron && make && make install
version: '3.7'
services:
pg:
container_name: pg-container
build: .
environment:
POSTGRES_DB: "pgdb"
POSTGRES_USER: "pguser"
POSTGRES_PASSWORD: "pgpass"
volumes:
- ./:/docker-entrypoint-initdb.d
ports:
- "5432:5432"
This worked for me - it probably needs some more optimization.
The proposed solution didn't work with a newly created container for me. So, I did it like this:
Docker file
FROM postgres:11.5
RUN apt-get update && apt-get -y install git build-essential postgresql-server-dev-11
RUN git clone https://github.com/citusdata/pg_cron.git
RUN cd pg_cron && make && make install
RUN cd / && \
rm -rf /pg_cron && \
apt-get remove -y git build-essential postgresql-server-dev-11 && \
apt-get autoremove --purge -y && \
apt-get clean && \
apt-get purge
COPY init-db /docker-entrypoint-initdb.d
init-db/pg-cron.sh
#!/usr/bin/env bash
# use same db as the one from env
dbname="$POSTGRES_DB"
# create custom config
customconf=/var/lib/postgresql/data/custom-conf.conf
echo "" > $customconf
echo "shared_preload_libraries = 'pg_cron'" >> $customconf
echo "cron.database_name = '$dbname'" >> $customconf
chown postgres $customconf
chgrp postgres $customconf
# include custom config from main config
conf=/var/lib/postgresql/data/postgresql.conf
found=$(grep "include = '$customconf'" $conf)
if [ -z "$found" ]; then
echo "include = '$customconf'" >> $conf
fi
Also, you can place other init files into init-db directory.
Docker compose file
version: '3.7'
services:
postgres:
container_name: your-container
build: .
environment:
POSTGRES_DB: "your_db"
POSTGRES_USER: "your_user"
POSTGRES_PASSWORD: "your_user"
volumes:
- pgdata:/var/lib/postgresql/data
ports:
- "5432:5432"
volumes:
pgdata:
driver: local
For those who are looking for a ready image, please try the following:
docker pull ramazanpolat/postgres_cron:11
Trying to dockerize an application and in my application, i have the following
docker-compose.yml
version: '3.4'
services:
app:
build:
context: .
dockerfile: Dockerfile
depends_on:
- database
ports:
- "3000:3000"
volumes:
- .:/app
- gem_cache:/usr/local/bundle/gems
env_file: .env
environment:
RAILS_ENV: development
database:
image: postgres:10.12
volumes:
- ./init.sql/:/docker-entrypoint-initdb.d/init.sql
- db_data:/var/lib/postgresql/data
volumes:
gem_cache:
db_data:
In my init.sql file
CREATE USER user1 WITH PASSWORD 'password';
ALTER USER user1 WITH SUPERUSER;
i have already run chmod +x init.sql
In my .env file i have the following
DATABASE_NAME=tools_development
DATABASE_USER=user1
DATABASE_PASSWORD=password
DATABASE_HOST=database
And this is my Dockerfile
FROM ruby:2.7.0
ENV BUNDLER_VERSION=2.1.4
RUN apt-get -y update --fix-missing
RUN apt-get install -y bash git build-essential nodejs libxml2-dev openssh-server libssl-dev libreadline-dev zlib1g-dev postgresql-client libcurl4-openssl-dev libxml2-dev libpq-dev tzdata
RUN gem install bundler -v 2.1.4
WORKDIR /app
COPY Gemfile Gemfile.lock ./
RUN bundle check || bundle install
COPY . ./
ENTRYPOINT ["./entrypoint.sh"]
But each time I run docker-compose run --build and try to run my application. I get error:
could not translate host name "database" to address: Name or service not known
I have tried everything possible but still the same error.
Does anyone have any idea on how to fix this issue?
I know the issue is happening because the postgres initialize scripts are not running. I have seen a lot of options online and i have tried everything but I am still facing the same error.
Any help is appreciated
Thanks
From the postgres docker image documentation you can see that the POSTGRES_USER and POSTGRES_PASSWORD are the necessary environment variables to setup the postgres container
You could add these environment variables to your .env file. So the file will be as follow:
.env
DATABASE_NAME=tools_development
DATABASE_USER=user1
DATABASE_PASSWORD=password
DATABASE_HOST=database
POSTGRES_USER=user1
POSTGRES_PASSWORD=password
POSTGRES_DB=tools_development
These environment variables will be used from the postgres container to init the DB and assign the user, so you can get rid of the init.sql file
After that you need to add the reference of the .env file in the database(postgres:10.12) service.
So your docker compose file should be as follow:
docker-compose.yml
...
database:
image: postgres:10.12
volumes:
- db_data:/var/lib/postgresql/data
env_file: .env
...
So I've setup a bitbucket-pipelines.yml for my python app. It needs a postgres database so I've followed the tutorial here (https://confluence.atlassian.com/bitbucket/test-with-databases-in-bitbucket-pipelines-856697462.html) which has led me to the following config:
image: node
pipelines:
default:
- step:
script:
- npm install
- npm test
services:
- postgres
definitions:
services:
postgres:
image: postgres
environment:
POSTGRES_DB: 'pipelines'
POSTGRES_USER: 'test_user'
POSTGRES_PASSWORD: 'test_user_password'
I need some specific extensions in my db, how can I add these. I tried to add an extra in the script that installs them but at that point postgres doesn't seem to be up and running.
This worked for me, without the need to build my own image (I add 2 extensions to postgres):
image: node:8.11.1 # or any image you need
clone:
depth: 1 # include the last commit
definitions:
services:
postgres:
image: postgres
environment:
POSTGRES_DB: test
POSTGRES_USER: postgres
POSTGRES_PASSWORD: your_password
pipelines:
default:
- step:
caches:
- node
script:
- npm install
- apt-get update
- apt-get -y install postgresql-client
- ./bin/utilities/wait-for-it.sh -h localhost -p 5432 -t 30
- PGPASSWORD=your_password psql -h localhost -p 5432 -c "create extension if not exists \"uuid-ossp\"; create extension if not exists pg_trgm;" -U postgres test;
- npm test test/drivers/* test/helpers/* test/models/*
services:
- postgres
wait-for-it.sh makes sure that postgres is ready to run, you can find it here: https://github.com/vishnubob/wait-for-it
Then, I run psql to create the extensions in the test database. Here note the variable PGPASSWORD I set before running that and that I use the -h and -p parameters to connect to the running postgres instance (otherwise it will try to do it with unix sockets, which doesn't seem to work).
You need to create your own image based on postgres, then push it to repository and use in pipeline
definitions:
services:
postgres:
image: your_custom_image_based_on_postgres
environment:
POSTGRES_DB: 'pipelines'
POSTGRES_USER: 'test_user'
POSTGRES_PASSWORD: 'test_user_password'
You can also find image that suit your requirements in https://hub.docker.com/
I have am trying to follow this tutorial and set up a postgresql container.
I have the following script:
#!/bin/bash
# wait-for-postgres.sh
set -e
host="$1"
shift
cmd="$#"
until psql -h "$host" -U "postgres" -c '\l'; do
>&2 echo "Postgres is unavailable - sleeping"
sleep 1
done
>&2 echo "Postgres is up - executing command"
exec $cmd
And the following docker-compose.yml:
version: '2'
services:
server:
build: .
ports:
- 3030:3030
depends_on:
- database
command: ["./setup/wait-for-postgres.sh", "localhost:5432", "--", "node", "src"]
database:
image: postgres
environment:
- "POSTGRES_USER=postgres"
- "POSTGRES_PASSWORD=postgres"
- "POSTGRES_DB=tide_server"
ports:
- 5432:5432
The problem is that when I run docker-compose up I get the following error:
server_1 | Postgres is unavailable - sleeping
server_1 | psql: could not translate host name "192.168.64.2:5432" to address: Name or servi
ce not known
server_1 | Postgres is unavailable - sleeping
server_1 | psql: could not translate host name "192.168.64.2:5432" to address: Name or servi
ce not known
server_1 | Postgres is unavailable - sleeping
server_1 | psql: could not translate host name "192.168.64.2:5432" to address: Name or servi
ce not known
Now I have tried setting the host as database, localhost, 0.0.0.0, and even the containers IP but nothing works, I have no idea what it should be or how to debug it, I am not 100% sure how docker-compose links the containers.
do not use depends_on. try it with "links"
version: '2'
services:
server:
build: .
ports:
- 3030:3030
links:
- database
#environment could be usefull too
environment:
DATABASE_HOST: database
command: ["./setup/wait-for-postgres.sh", "localhost:5432", "--", "node", "src"]
database:
image: postgres
environment:
- "POSTGRES_USER=postgres"
- "POSTGRES_PASSWORD=postgres"
- "POSTGRES_DB=tide_server"
ports:
- 5432:5432
for more informations https://docs.docker.com/compose/compose-file/#links
May be an old thread to answer but I have been using depends_on with the following docker-compose file
version: '3.4'
volumes:
postgres_data:
driver: local
services:
postgres:
image: postgres
volumes:
- ./postgres_data:/var/lib/postgresql:rw
- ./deployments:/opt/jboss/wildfly/standalone/deployments:rw
environment:
POSTGRES_DB: keycloak
POSTGRES_USER: keycloak
POSTGRES_PASSWORD: password
ports:
- 5432:5432
keycloak:
image: jboss/keycloak
environment:
POSTGRES_ADDR: postgres
POSTGRES_DATABASE: keycloak
POSTGRES_USER: keycloak
POSTGRES_PASSWORD: password
KEYCLOAK_USER: admin
KEYCLOAK_PASSWORD: Pa55w0rd
ports:
- 8080:8080
- 9990:9990
depends_on:
- postgres
The tutorial skips over a few things, and is confusing in that it mentions the wait-for-it.sh script, but then shows a much simplified version that doesn't work if you pass hostname:port as one argument to it.
I had a crack at getting this to work and both for future me and others I will add the steps below. I did this on MacOS, and have both docker and docker-compose installed as well as nodejs.
I don't have your node app handy so I used the one as described here https://nodejs.org/de/docs/guides/nodejs-docker-webapp/
I have the following directory structure:
/src/package.json
/src/server.js
/.pgpass
/docker-compose.yml
/Dockerfile
/wait-for-postgres.sh
The contents of these files is listed below.
Steps
From the ./src directory run $ npm install (creates package-lock.json)
Fix pgpass permissions with $ chmod 600 .pgpass
Make the script executable $ chmod +x wait-for-postgres.sh
From the root directory $ docker-compose up
It will pull the postgres image and build the node app container.
When that's done it will wait for postgres and when postgres is up you'll see it ready.
Files
The src files are exactly as per the node js dockerize link above
/src/package.json
{
"name": "docker_web_app",
"version": "1.0.0",
"description": "Node.js on Docker",
"author": "First Last <first.last#example.com>",
"main": "server.js",
"scripts": {
"start": "node server.js"
},
"dependencies": {
"express": "^4.16.1"
}
}
/src/server.js
'use strict';
const express = require('express');
// Constants
const PORT = 8080;
const HOST = '0.0.0.0';
// App
const app = express();
app.get('/', (req, res) => {
res.send('Hello world\n');
});
app.listen(PORT, HOST);
console.log(`Running on http://${HOST}:${PORT}`);
.pgpass
This uses the username:password postgres:postgres and is purely for development demo purposes. In the wild you will use some other method of secrets management and never ever commit a pgpass file to version control
#host:port:db:user:pass
db:5432:*:postgres:postgres
docker-compose.yml
I have added the wait-for-postgres.sh script as a managed volume, in the original question it was bundling it in with the app src which was weird.
I have also mounted the .pgpass file in the root user's home directory, which psql will look in for auto-password completion. If you don't have some method of supplying this then you'll get an error:
psql: fe_sendauth: no password supplied
Notice the command for the server container is referring to database which is a valid docker-compose internal dns name for the postgres container.
version: '2'
services:
server:
build: .
ports:
- 3030:3030
depends_on:
- database
volumes:
- ./wait-for-postgres.sh:/usr/app/setup/wait-for-postgres.sh
- ./.pgpass:/Users/root/.pgpass
command: ["/usr/app/setup/wait-for-postgres.sh", "database", "--", "node", "src"]
database:
image: postgres
environment:
- "POSTGRES_USER=postgres"
- "POSTGRES_PASSWORD=postgres"
- "POSTGRES_DB=tide_server"
ports:
- 5432:5432
Dockerfile
I have modified this from the node js tutorial, pinning it to the Debian "buster" version and also installing psql which it needs for that script.
FROM node:10-buster
RUN apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys B97B0AFCAA1A47F044F244A07FCC7D46ACCC4CF8
RUN echo "deb http://apt.postgresql.org/pub/repos/apt/ buster-pgdg main" > /etc/apt/sources.list.d/pgdg.list && \
wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | apt-key add -
RUN apt-get -y update - && \
apt-get -y install libpq-dev && \
apt-get -y install postgresql-client-11
# Create app directory
WORKDIR /usr/src/app
# Install app dependencies
# A wildcard is used to ensure both package.json AND package-lock.json are copied
# where available (npm#5+)
COPY package*.json ./
RUN npm install
# If you are building your code for production
# RUN npm ci --only=production
# Bundle app source
COPY . .
EXPOSE 8080
CMD [ "node", "server.js" ]
wait-for-postgres.sh
I have modified the script very slightly because I ran the "shellcheck" linter and it complained about a few things. I realise this script is from the docker tutorial page.
#!/bin/bash
# wait-for-postgres.sh
set -e
host="$1"
shift
cmd="$*"
export PGPASSFILE=./pgpass
until psql -h "$host" -U "postgres" -c '\l'; do
>&2 echo "Postgres is unavailable - sleeping"
sleep 1
done
>&2 echo "Postgres is up - executing command"
exec "$cmd"
The problem here is the host itself.
psql -h **"$host"** -U "<USER>" -c '\l'
You are passing a wrong HOSTNAME "localhost:5432" / "192.168.64.2:5432"
What I did is setup a ~/.pgpass that has
localhost:5432:DB:USER:PASSWORD
and instead of passing "localhost:5432", omit the port. Just use "localhost"
This works for me ...