I've being trying to setup gitlab CI with my django project. The project uses postgis extension. After all this setup I still get the error that postgis.control file could not be found
$ export PGPASSWORD=$POSTGRES_PASSWORD
$ psql -c "CREATE EXTENSION IF NOT EXISTS postgis;" -d $POSTGRES_DB -U $POSTGRES_USER -h "postgres"
ERROR: could not open extension control file "/usr/share/postgresql/11/extension/postgis.control": No such file or directory
ERROR: Job failed: exit code 1
Here is my .gitlab-ci.yml file
image: python:3.6
stages:
- test
services:
- mdillon/postgis
- postgres
variables:
POSTGRES_DB: my_db
POSTGRES_USER: my_user
POSTGRES_PASSWORD: ""
TESTFOLDER: "myapp/apps/api myapp/apps/logger"
DATABASE_URL: "postgres://my_user:#mdillon-postgis/my_db"
test:
stage: test
image: mdillon/postgis
before_script:
- apt-get update -qy
- export PGPASSWORD=$POSTGRES_PASSWORD
- psql -c "CREATE EXTENSION IF NOT EXISTS postgis;" -d $POSTGRES_DB -U $POSTGRES_USER -h "postgres"
- psql -c "CREATE EXTENSION IF NOT EXISTS postgis_topology;" -d $POSTGRES_DB -U $POSTGRES_USER -h "postgres"
- apt-get install -y openjdk-8-jre-headless libjpeg-dev zlib1g-dev software-properties-common ghostscript libxslt1-dev binutils libproj-dev libgdal-dev gdal-bin memcached libmemcached-dev
- export DEBIAN_FRONTEND=noninteractive;
- pip install --upgrade pip
- pip install -r requirements/base.pip
- pip install flake8
script:
- python manage.py test $TESTFOLDER --noinput --settings=myapp.settings.gitlab_ci --parallel 4 --verbosity=2
only:
- master
In my case, I discovered that it was the host I was using to connect to the database that was the cause of the problem.
After reading through the GitLab documentation, I discovered GitLab uses the name of the service as the host for the connection. So in my case, when connecting from my Python application, I used mdillon-postgis as my host.
You can find more details here https://docs.gitlab.com/ee/ci/docker/using_docker_images.html#accessing-the-services
You probably don't have the postgis installed in the database server. you need ro run: sudo apt-get install postgis
Related
I'm trying to run a pipeline on gitlab of a python webapp made with Django, that uses a postgre database. After installing postgre, the psql command gives the error:
psql: error: could not connect to server: No such file or directory
Here's (part of) my .gitlab-ci.yml file:
image: python:latest
# Install postgreSQL service on container
services:
- postgres:12.2-alpine
# Change pip's cache directory to be inside the project directory
variables:
PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip"
DJANGO_SETTINGS_MODULE: "my_app.settings"
POSTGRES_DB: $POSTGRES_DB
POSTGRES_USER: $POSTGRES_USER
POSTGRES_PASSWORD: $POSTGRES_PASSWORD
POSTGRES_HOST_AUTH_METHOD: trust
# Let's cache also the packages
# Install packages in a virtualenv and cache it as well
cache:
paths:
- .cache/pip
- venv/
before_script:
- pip install virtualenv --upgrade pip
- virtualenv venv
- source venv/bin/activate
- apt-get update
#- apt-get install -y postgresql postgresql-client libpq-dev # postgre db requirements
stages: # List of stages for jobs, and their order of execution
- build
- verify
- unit_test
- integration_test
- package
- release
- deploy
build:
stage: build
script:
- pip install -r requirements.txt
- echo "Build stage finished"
verify:
stage: verify
script:
- prospector -X ./my_app # static code analysis
- bandit -r ./my_app # static code analysis pt. 2
- echo "Verify stage finished"
unit_test:
stage: unit_test
script:
- echo "Running unit_test 1"
- pytest ./my_app/unit_test.py #running unit_test
- echo "Creating db"
- apt-get install -y postgresql postgresql-client libpq-dev # postgre db
- psql -U postgres
- psql -d "CREATE USER $POSTGRES_USER WITH PASSWORD $POSTGRES_PASSWORD CREATEDB;"
- psql -d "CREATE DATABASE $POSTGRES_DB OWNER $POSTGRES_USER;"
- echo "Unit testing stage finished"
How can I make psql work on gitlab CI/CD pipeline?
You're on the right track with the "services" keyword, which will cause a postgres database to run on the host "postgres" (the DNS of the service is based on the name of the container unless you specify an "alias" with the service).
Your issue is that psql attempts to connect to localhost unless you specify otherwise, so your psql -U postgres attempts to connect on localhost. Try using psql -U postgres -p 5432 -h postgres instead.
gitlab-ci.yaml file:
liquibase:
stage: liquibase
image: openjdk:8-jre-alpine
services:
- postgres
script:
- INIT_PATH='pwd'
- apk upgrade
- apk add bash
- apk add postgresql
- cd migrations
- mkdir /liquibase
- mkdir /Downloads
- cd /Downloads
- wget "https://github.com/liquibase/liquibase/releases/download/liquibase-parent-3.7.0-bin.zip"
- wget "https://repo1.maven.org/maven2/org/postgresql/postgresql/42.2.8/postgresql-42.2.8.jar"
- unzip liquibase-3.7.0-bin.zip -d /liquibase -q
- cd ../../liquibase
- export PATH=$PATH:/liquibase
- psql -h "postgres" -U "$POSTGRES_USER" -d "$POSTGRES_DB" -l
- liquibase --changeLogFile=$INIT_PATH/migrations/baseline_postgres.xml --url="jdbc:postgresql://postgres:5432/custom_baseline" --username $POSTGRES_USER" -d "$POSTGRES_DB" -l
- cd ../../..
- pg_dump -h "postgres" -U "$POSTGRES_USER" -d "POSTGRES_DB" > baseline_schema.sql
This stage in my gitlab-ci pipeline (kubernetes executor) returns the following error:
pg_dump: server version: 13.1 (debian 13.1.1.pddg100+1); pg_dump version 11:10
pg_dump: aborting because of server version mismatch
I have tried adding symbolic links as other posts have suggested but I haven't succeeded. Any suggestions on resolving the pg_dump error for this stage in my Gitlab-CI pipeline?
You are using pg_dump from the wrong PostgreSQL version. Change the PATHenvironment variable or use an absolute path.
In my docker project with postgres 9.6 I need to add support for uploading of big sql dumps.
I added pg_dump by adding in my web/Dockerfile.yml file line:
postgresql-client-common \
as it contains pg_dump.
After building the app I enter bash and
# uname -a
Linux ff5146f21dbd 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1 05:24:09 UTC 2019 x86_64 GNU/Linux
root#ff5146f21dbd:/var/www/lprods_docker_root# pg_dump
Warning: No existing local cluster is suitable as a default target. Please see man pg_wrapper(1) how to specify one.
Error: You must install at least one postgresql-client-<version> package
root#ff5146f21dbd:/var/www/lprods_docker_root# locate -i pg_dump
Last command outputs nothing
web/Dockerfile.yml contains :
FROM php:7.1-apache
RUN apt-get update && \
apt-get install -y \
python \
libfreetype6-dev \
libwebp-dev \
libjpeg62-turbo-dev \
libpng-dev \
libzip-dev \
nano \
git-core \
curl \
build-essential \
openssl \
libssl-dev \
libgmp-dev \
libldap2-dev \
libpq-dev \
netcat \
postgresql-client-common \
locate \
&& git clone https://github.com/nodejs/node.git \
&& cd node \
&& git checkout v12.0.0 \
&& ./configure \
&& make \
&& make install
RUN npm install cross-env
RUN docker-php-ext-configure gd --with-freetype-dir=/usr/include/ --with-webp-dir=/usr/include/ --with-jpeg-dir=/usr/include/
RUN docker-php-ext-install gd pgsql pdo_pgsql zip gmp bcmath pcntl ldap sysvmsg exif \
&& a2enmod rewrite
COPY virtualhost.conf /etc/apache2/sites-enabled/000-default.conf
and docker-compose.yml:
version: '3'
services:
web:
build:
context: ./web # directory of web/Dockerfile.yml
dockerfile: Dockerfile.yml
environment:
- APACHE_RUN_USER=#1000
# - APACHE_RUN_USER=www-data
container_name: lprods_web
volumes:
- ${APP_PATH_HOST}:${APP_PTH_CONTAINER}
ports:
- "8086:80"
working_dir: ${APP_PTH_CONTAINER}
db:
image: postgres:9.6.10-alpine
container_name: lprods_db
ports:
- '5433:5432'
restart: always
environment:
POSTGRES_USER: 'postgres'
POSTGRES_PASSWORD: '1'
POSTGRES_DB: 'wprods'
volumes:
- ./init:/docker-entrypoint-initdb.d/
phppgadmin:
image: dockage/phppgadmin:latest
environment:
- PHP_PG_ADMIN_SERVER_HOST=db
- PHP_PG_ADMIN_SERVER_PORT=5432
- PHP_PG_ADMIN_SERVER_DEFAULT_DB=postgres
container_name: lprods_phppgadmin
restart: always
ports:
- "8087:80"
- "443:443"
links:
- db
composer:
image: composer:1.6
container_name: lprods_composer
volumes:
- ${APP_PATH_HOST}:${APP_PTH_CONTAINER}
working_dir: ${APP_PTH_CONTAINER}
command: composer install --ignore-platform-reqs
Why error and how to fix it?
MODIFIED :
I have
# echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
But checking version I got error :
# /usr/bin/pg_dump -v
Warning: No existing local cluster is suitable as a default target. Please see man pg_wrapper(1) how to specify one.
Error: You must install at least one postgresql-client-<version> package
The same error I got when running psql command(actually I need it):
/usr/bin/psql -h localhost -d wprods -U postgres -W -f "/var/www/lprods_docker_root/wprods_2017_10_23.sql"
Error: You must install at least one postgresql-client-<version> package
What is wrong with my configuration ?
MODIFIED 2:
In my docker-compose.yml I have :
db:
image: postgres:9.6.10-alpine
...
volumes:
- ./init:/docker-entrypoint-initdb.d/
Is it?
command :
locate -i postgresql.conf
outputs nothing
find /usr -name "postgresql.conf"
Also outputs nothing.
Also I have :
# ps -ef | grep postgres
root 28 21 0 08:40 pts/0 00:00:00 grep postgres
What did I miss ?
postgresql-client-common is a Ubuntu-specific package of scripts that allows users to work with multiple versions of Postgres (or multiple database clusters). It provies several scripts and symlinks that essentially override the functions of "real" postgres utilities (like pg_dump):
root#foo:/usr/bin# ls -al | grep pg_dump
lrwxrwxrwx 1 root root 37 Feb 8 2018 pg_dump -> ../share/postgresql-common/pg_wrapper
lrwxrwxrwx 1 root root 37 Feb 8 2018 pg_dumpall -> ../share/postgresql-common/pg_wrapper
You may need to also install postgresql-client-10 to make pg_dump work.
Note also that you have only installed the client tools. If you need to get a fully-functional PostgreSQL database up and running, I believe you will also need to install the postgresql-10 package
When I run the command psql --version within the railsApp container, I get 9.4.12 and when I run the same within the postgres container, I get 9.6.2. How can I get the versions to match?
I am getting the following error when I try to do a migration on Rails App which does a pg_dump sql import.
pg_dump: server version: 9.6.2; pg_dump version: 9.4.12
pg_dump: aborting because of server version mismatch
rails aborted!
Here's my Docker-compose.yml file:
version: "2.1"
services:
railsApp:
build:
context: ./
ports:
- "3000:3000"
links:
- postgres
volumes:
- .:/app
postgres:
image: postgres:9.6
ports:
- "5432:5432"
volumes:
- ./.postgres:/var/lib/postgresql
The Dockerfile:
FROM ruby:2.3.3
# setup /app as our working directory
RUN mkdir /app
WORKDIR /app
# Replace shell with bash so we can source files
RUN rm /bin/sh && ln -s /bin/bash /bin/sh
# Set debconf to run non-interactively
RUN echo 'debconf debconf/frontend select Noninteractive' | debconf-set-selections
# Install base dependencies
FROM ruby:2.3.3
# setup /app as our working directory
RUN mkdir /app
WORKDIR /app
# Replace shell with bash so we can source files
RUN rm /bin/sh && ln -s /bin/bash /bin/sh
# Set debconf to run non-interactively
RUN echo 'debconf debconf/frontend select Noninteractive' | debconf-set-selections
# Install base dependencies
RUN apt-get update && apt-get install -y -q --no-install-recommends \
apt-transport-https \
build-essential \
ca-certificates \
curl \
git \
libssl-dev \
python \
rsync \
software-properties-common \
wget \
postgresql-client \
graphicsmagick \
&& rm -rf /var/lib/apt/lists/*
# Install node and npm with nvm
RUN curl https://raw.githubusercontent.com/creationix/nvm/v0.33.0/install.sh | bash
ENV NVM_DIR=/root/.nvm
ENV NODE_VERSION v7.2.1
ENV NODE_PATH $NVM_DIR/versions/node/$NODE_VERSION
ENV PATH $NODE_PATH/bin:./node_modules/.bin:$PATH
RUN source $NVM_DIR/nvm.sh \
&& nvm install $NODE_VERSION \
&& nvm alias default $NODE_VERSION \
&& nvm use default
# Install our ruby dependencies
ADD Gemfile Gemfile.lock /app/
RUN bundle install
# copy the rest of our code over
ADD . /app
ENV RAILS_ENV development
ENV SECRET_KEY_BASE a6bdc5f788624f00b68ff82456d94bf81bb50c2e114b2be19af2e6a9b76f9307b11d05af4093395b0471c4141b3cd638356f888e90080f8ae60710f992beba8f
# Expose port 3000 to the Docker host, so we can access it from the outside.
EXPOSE 3000
# Set the default command to run our server on port 3000
CMD ["rails", "server", "-p", "3000", "-b", "0.0.0.0"]
The same issue for me, I used an alternative way to take a dump,
First I access the terminal and run the pd_dump inside docker and copied the file from docker to host.
Below are the commands
docker exec -it <container-id> /bin/bash # accessing docker terminal
pg_dump > ~/dump # taking dump inside the docker
docker cp <container-id>:/root/dump ~/dump #copying the dump files to host
Hope the above solution helps.
The easiest approach is to use the correct postgres version in the docker-compose. Change:
postgres:
image: postgres:9.6
To:
postgres:
image: postgres:9.4.2
All available versions here.
How can i enable the Postgis extension in a Dockerfile? The Postgres installation is already working
this is the command in Ubuntu:
sudo -u postgres psql -c "CREATE EXTENSION postgis;"
My Dockerfile:
# Set the base image to Ubuntu
FROM ubuntu:14.04
# Update the repository sources list
RUN apt-get update -y
################## BEGIN INSTALLATION ######################
# Postgres with Postgis
# Install wget
RUN apt-get install wget -y
# Setup Postgres repository
RUN wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -
# Add Postgres repository
RUN sh -c "echo "deb http://apt.postgresql.org/pub/repos/apt/ trusty-pgdg main" >> /etc/apt/sources.list.d/postgresql.list"
# Update repository
RUN apt-get update -y
# Install Postgres with Postgis
RUN apt-get install postgresql-9.3-postgis-2.1 -y
# Change rights for start Postgresql
RUN chmod +x /etc/init.d/postgresql
# Start Postgresql
CMD service postgresql start && tail -F /var/lib/postgresql/data/serverlog
The solution for enable an extension
# Enable Postgis
RUN service postgresql start \
&& sudo -u postgres psql -c "CREATE EXTENSION postgis;"
RUN service postgresql stop
If you see other Dockerfiles using that psql -c command, like apache/marmotta Dockerfile, you will see lines like:
RUN service postgresql start \
&& psql --command "CREATE USER $DB_USER WITH PASSWORD '$DB_PASS';" \
&& psql --command "CREATE DATABASE $DB_NAME WITH OWNER $DB_USER;"
USER root
RUN service postgresql stop
In other words, you need to make sure the postgresql service is running for those commands to succeed.