I have some issues with the TravisCI platform when I try to test the code when Pull Requesting some changes. It works fine so far, but now I get this warning in the build-run log when I run the travis_apt_get_update command:
The PostgreSQL version 9.4 is obsolete, but the server or client packages
are still installed. Please install the latest packages (postgresql-13 and
postgresql-client-13) and upgrade the existing clusters with
pg_upgradecluster (see manpage).
I have added the postgresql-13 and postgresql-client-13 packages, here is the .travis.yml file:
language: perl
perl:
- "5.30"
dist: xenial
env:
- HOST_URL="localhost"
cache:
directories:
- $HOME/path/to/local
services:
- postgresql
addons:
postgresql: 13
apt:
packages:
- postgresql-13
- postgresql-client-13
- libpq-dev
- build-essential
- libssl-dev
- zlib1g-dev
- clang-tidy
env:
global:
- PGPORT=5433
before_install:
- wget http://mirrors.kernel.org/ubuntu/pool/universe/a/astyle/astyle_3.1-1ubuntu2_amd64.deb
- sudo dpkg -i astyle_3.1-1ubuntu2_amd64.deb
- sudo chmod -R 777 /var/log/
Now, in the log it says that I have to upgrade the clusters with pg_upgradecluster, but I really don't know what that means.
This example (.travis.yml) shows how to run Postgres13 within your Ubuntu jobs. You can configure/add additional configuration as your setup and see if that helps.
---
dist: focal
language: ruby
addons:
postgresql: '13'
apt:
packages:
- postgresql-13
env:
global:
- PGUSER=postgres
- PGPORT=5432
- PGHOST=localhost
before_install:
- sudo sed -i -e '/local.*peer/s/postgres/all/' -e 's/peer\|md5/trust/g' /etc/postgresql/*/main/pg_hba.conf
- sudo service postgresql restart
- sleep 1
- postgres --version
script:
- psql -c 'create database travis_ci_test;' -U postgres
Let me know if you have further questions.
Thanks.
Related
I'm trying to connect my Symfony 6 project with the MongoDB database on the Docker environment but when I launch the following command
composer require doctrine/mongodb-odm
This message appears
Your requirements could not be resolved to an installable set of packages.
Problem 1
- doctrine/mongodb-odm[1.0.0, ..., 1.0.8] require symfony/console ~2.3|~3.0 -> found symfony/console[v2.3.0, ..., v2.8.52, v3.0.0, ..., v3.4.47] but it conflicts with your root composer.json require (6.1.*).
- doctrine/mongodb-odm[1.1.0, ..., 1.3.7] require php ^5.6 || ^7.0 -> your php version (8.1.12) does not satisfy that requirement.
- Root composer.json requires doctrine/mongodb-odm ^1.0 -> satisfiable by doctrine/mongodb-odm[1.0.0, ..., 1.3.7].
You can also try re-running composer require with an explicit version constraint, e.g. "composer require doctrine/mongodb-odm:*" to figure out if any version is installable, or "composer require doctrine/mongodb-odm:^2.1" if you know which you need.
Installation failed, reverting ./composer.json and ./composer.lock to their original content.
my docker-compose file
version: '3.8'
services:
mongo:
image: mongo
restart: always
environment:
MONGO_INITDB_ROOT_USERNAME: root
MONGO_INITDB_ROOT_PASSWORD: example
mongo-express:
image: mongo-express
restart: always
ports:
- 8081:8081
environment:
ME_CONFIG_MONGODB_ADMINUSERNAME: root
ME_CONFIG_MONGODB_ADMINPASSWORD: example
ME_CONFIG_MONGODB_URL: mongodb://root:example#mongo:27017/
www:
build: php
container_name: www-docker-env
ports:
- '80:80'
volumes:
- ./php/vhosts:/etc/apache2/sites-enabled
- ./:/var/www
-
type: 'bind'
source: '../project/'
target: '/var/www/'
consistency: 'delegated'
restart: always
networks:
- dev
networks:
dev:
volumes:
mongo-data:
the Dockerfile file
FROM php:8.1-apache
ARG user
ARG uid
RUN echo 'ServerName localhost' >> /etc/apache2/apache2.conf
RUN apt-get update \
&& apt-get install -y --no-install-recommends locales apt-utils git libicu-dev g++ libpng-dev libxml2-dev libzip-dev libonig-dev libxslt-dev \
&& apt install -y unzip;
RUN echo "en_US.UTF-8 UTF-8" > /etc/locale.gen && \
echo "fr_FR.UTF-8 UTF_8" >> /etc/locale.gen && \
locale-gen
RUN docker-php-ext-configure intl
RUN docker-php-ext-install pdo pdo_mysql gd opcache intl zip calendar dom mbstring zip gd xsl
RUN pecl install apcu && docker-php-ext-enable apcu
# Clear cache
RUN apt-get clean && rm -rf /var/lib/apt/lists/*
RUN curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --filename=composer
WORKDIR /var/www/
USER $user
Is there any point that I did not notice in the above files, and is there any solution to the following problem?
If you want to use PHP 8.1, upgrade mongodb-odm to version 2.3. This is the first version to support PHP 8 as outlined in this statement:
We have released a new minor version 2.3 of Doctrine MongoDB ODM, the first version with support for using PHP 8 Attributes as a new driver for mapping documents and several other changes. See all changes and contributors in the Changelog on GitHub.
Otherwise, downgrade to PHP 7.0 as indicated in the error message.
I had the following gitlab-ci.yml in my python-package repository:
image: python:latest
unit-test:
stage: test
tags:
- docker
script:
- pip install tox
- tox
formatting-check:
stage: test
tags:
- docker
script:
- pip install black
- black --check .
using this tox.ini file:
[tox]
envlist = my_env
[testenv]
deps =
-rrequirements.txt
commands =
python -m pytest tests -s
This did work as I wanted it to.
However, then I added tests to test my code against a local Postgresql database using https://pypi.org/project/pytest-postgresql/. For this, I had to install PostgreSQL(apt -y install postgresql postgresql-contrib libpq5).
When I added this to my gitlab-ci.yml:
image: python:latest
unit-test:
stage: test
tags:
- docker
script:
- apt -y install postgresql postgresql-contrib libpq5
- pip install tox
- tox
formatting-check:
stage: test
tags:
- docker
script:
- pip install black
- black --check .
I got the error from tox, that some module in Postgres (pg_ctl) wouldn't allow being run as the root. Log here: https://pastebin.com/fMu1JY5L
So, I must execute tox as a user, not the root.
My first idea was to create a new user (useradd) and then switch to that user, but su requires inputting a password.
From a quick google search I found out the easiest solution to creating a new user is to create a new Docker Image using Docker-in-Docker.
So, as of now I have this configuration:
gitlab-ci.yml:
image: docker:19.03.12
services:
- docker:19.03.12-dind
stages:
- build
- test
variables:
DOCKER_HOST: tcp://docker:2375
DOCKER_TLS_CERTDIR: ""
CONTAINER_TEST_IMAGE: $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG
before_script:
- docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
- docker info
docker-build:
stage: build
script:
- docker build --pull -t $CONTAINER_TEST_IMAGE .
- docker push $CONTAINER_TEST_IMAGE
formatting-check:
stage: test
script:
- docker pull $CONTAINER_TEST_IMAGE
- docker run $CONTAINER_TEST_IMAGE black --check .
unit-test:
stage: test
script:
- docker pull $CONTAINER_TEST_IMAGE
- docker run $CONTAINER_TEST_IMAGE tox
Dockerfile:
FROM python:latest
RUN apt update
RUN apt -y install postgresql postgresql-contrib libpq5
RUN useradd -m exec_user
USER exec_user
ENV PATH "$PATH:/home/exec_user/.local/bin"
RUN pip install black tox
(I had to add ENV PATH "$PATH:/home/exec_user/.local/bin" because pip would cry about it not being in the Path)
tox.ini:
[tox]
envlist = my_env
[testenv]
deps =
-rrequirements.txt
commands =
python -m pytest tests -s
The job docker-build completes — the other two fail.
As for formatting-check:
$ docker run $CONTAINER_TEST_IMAGE black --check .
ERROR: Job failed: execution took longer than 1h0m0s seconds
The black command usually executes extremely fast (<1s).
As for unit-test:
/bin/sh: eval: line 120: tox: not found
Cleaning up file based variables
00:01
ERROR: Job failed: exit code 127
I have also found, that replacing docker run $CONTAINER_TEST_IMAGE tox with docker run $CONTAINER_TEST_IMAGE python3 -m tox doesn't work. Here, python3 isn't found (which seems odd given that the base image is python:latest).
If you have any idea how to solve this issue, let me know :D
My first idea was to create a new user (useradd) and then switch to that user, but su requires inputting a password.
This should work for your use case. Running su as root will not require a password. You could also use sudo -u postgres tox (must apt install sudo first).
As a basic working example using su (as seen here - job) using the postgres user, which is created automatically when postgres is installed.
myjob:
image: python:3.9-slim
script:
- apt update && apt install -y --no-install-recommends libpq-dev postgresql-client postgresql build-essential
- pip install psycopg2 psycopg pytest pytest-postgresql
- su postgres -c pytest
# or in your case, you might use: su postgres -c tox
Alternatively, you might consider just using GitLab's services feature to run your postgres server if that's the only obstacle in your way. You can pass --postgresql-host and --postgresql-password to pytest to tell the extension to use the services.
I'm trying to run a pipeline on gitlab of a python webapp made with Django, that uses a postgre database. After installing postgre, the psql command gives the error:
psql: error: could not connect to server: No such file or directory
Here's (part of) my .gitlab-ci.yml file:
image: python:latest
# Install postgreSQL service on container
services:
- postgres:12.2-alpine
# Change pip's cache directory to be inside the project directory
variables:
PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip"
DJANGO_SETTINGS_MODULE: "my_app.settings"
POSTGRES_DB: $POSTGRES_DB
POSTGRES_USER: $POSTGRES_USER
POSTGRES_PASSWORD: $POSTGRES_PASSWORD
POSTGRES_HOST_AUTH_METHOD: trust
# Let's cache also the packages
# Install packages in a virtualenv and cache it as well
cache:
paths:
- .cache/pip
- venv/
before_script:
- pip install virtualenv --upgrade pip
- virtualenv venv
- source venv/bin/activate
- apt-get update
#- apt-get install -y postgresql postgresql-client libpq-dev # postgre db requirements
stages: # List of stages for jobs, and their order of execution
- build
- verify
- unit_test
- integration_test
- package
- release
- deploy
build:
stage: build
script:
- pip install -r requirements.txt
- echo "Build stage finished"
verify:
stage: verify
script:
- prospector -X ./my_app # static code analysis
- bandit -r ./my_app # static code analysis pt. 2
- echo "Verify stage finished"
unit_test:
stage: unit_test
script:
- echo "Running unit_test 1"
- pytest ./my_app/unit_test.py #running unit_test
- echo "Creating db"
- apt-get install -y postgresql postgresql-client libpq-dev # postgre db
- psql -U postgres
- psql -d "CREATE USER $POSTGRES_USER WITH PASSWORD $POSTGRES_PASSWORD CREATEDB;"
- psql -d "CREATE DATABASE $POSTGRES_DB OWNER $POSTGRES_USER;"
- echo "Unit testing stage finished"
How can I make psql work on gitlab CI/CD pipeline?
You're on the right track with the "services" keyword, which will cause a postgres database to run on the host "postgres" (the DNS of the service is based on the name of the container unless you specify an "alias" with the service).
Your issue is that psql attempts to connect to localhost unless you specify otherwise, so your psql -U postgres attempts to connect on localhost. Try using psql -U postgres -p 5432 -h postgres instead.
I have the following Dockerfile for a django app:
FROM python:3.6
RUN mkdir /server
WORKDIR /server
RUN apt-get update && apt-get upgrade -y && apt-get install -y \
libsqlite3-dev
RUN pip install -U pip setuptools
RUN pip install --upgrade pip
COPY requirements.txt ./
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
WORKDIR /server/django
ENTRYPOINT ["/bin/bash", "../docker-entrypoint-server"]
relevant docker-compose related to it:
version: '3'
services:
server:
build: .
container_name: server
environment:
SERVER_ENV: ${SERVER_ENV}
DB_AUTH_SOURCE: ${DB_AUTH_SOURCE}
DB_NAME: ${DB_NAME}
DB_HOST: ${DB_HOST}
DB_USER: ${DB_USER}
DB_PASSWORD: ${DB_PASSWORD}
networks:
- app
ports:
- 8081:8000
volumes:
- .:/server
command: /bin/bash
tty: true
stdin_open: true
they work like a charm on linux/mac, but not on Windows 10. When the build reaches the COPY instruction, all it copies are the directory and only the first nested directory within it with no content attached to it.
tried to check for the shared C:\ option on docker, didnt work.
tried on powershell with admin rights and nothing.
What could be the possible causes? Why it works on two host OS and not on Win 10?
edit 1: versions
Windows: Windows 10 Education, 1803
Linux: Ubuntu 18.03 LTS
Mac: High Sierra 10.03
Docker: latest version on all OSs
edit 2: the solution
Turns out that unsharing and sharing, pointed out by JDPeckham, revealed the problem: a firewall misconfig caused by an antivirus that was controlling these configs.
This article: https://success.docker.com/article/error-a-firewall-is-blocking-file-sharing-between-windows-and-the-containers is very helpful for troubleshooting those.
So I have at one point had this going via Beanstalk, using Amazon Instance (2013.09) ami-35792c5c . At the time this ebextension scripts worked great when placed in the root of your repo in .ebextensions/
00_repo.config
packages:
rpm:
pgdg-redhat93-9.3-1: 'http://yum.postgresql.org/9.3/redhat/rhel-6-x86_64/pgdg-redhat93-9.3-1.noarch.rpm'
remi: 'http://rpms.famillecollet.com/enterprise/remi-release-6.rpm'
files:
"/etc/yum.repos.d/pgdg-93-redhat.repo":
mode: "000644"
owner: root
group: root
content: |
[pgdg93]
name=PostgreSQL 9.3 $releasever - $basearch
baseurl=http://yum.postgresql.org/9.3/redhat/rhel-6-$basearch
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-PGDG-93
[pgdg93-source]
name=PostgreSQL 9.3 $releasever - $basearch - Source
failovermethod=priority
baseurl=http://yum.postgresql.org/srpms/9.3/redhat/rhel-6-$basearch
enabled=0
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-PGDG-93
commands:
epel_repo:
command: yum-config-manager -y --enable epel
remi_repo:
command: yum-config-manager -y --enable remi
01_app.config
packages:
yum:
libtiff-devel: ''
libjpeg-devel: ''
libzip-devel: ''
freetype-devel: ''
postgresql-devel: ''
gdal: ''
gdal-python: ''
geos: ''
proj: ''
libmemcached: ''
libmemcached-devel: ''
cyrus-sasl-devel: ''
zlib-devel: ''
container_commands:
01_collectstatic:
command: 'PYTHONPATH=.:..:../lib cd site/kpmkhv && ./manage.py collectstatic -c --noinput && cd ../..'
leader_only: true
02_syncdb:
command: 'PYTHONPATH=.:..:../lib cd site/kpmkhv && ./manage.py syncdb --noinput && cd ../..'
leader_only: true
03_migrate:
command: 'PYTHONPATH=.:..:../lib cd site/kpmkhv && ./manage.py migrate --noinput && cd ../..'
leader_only: true
option_settings:
- namespace: aws:elasticbeanstalk:container:python
option_name: WSGIPath
value: site/kpmkhv/wsgi.py
- namespace: aws:elasticbeanstalk:container:python:staticfiles
option_name: /static/
value: site/kpmkhv/static/
- option_name: DJANGO_SETTINGS_MODULE
value: settings_prod
So now when I use the same instance and launch my environment, I get this error regarding a dependency.
Error: Package: gdal-libs-1.9.2-5.rhel6.x86_64 (pgdg93)
Requires: libpoppler.so.5()(64bit)
You could try using --skip-broken to work around the problem
You could try running: rpm -Va --nofiles --nodigest
Looks like the same repo is now returning a newer version of poppler, it was 12.x and now its 22.x and gdal needs the old version.
I also tested this out on an EC2 Instance and got the same error. But then I ran into this document from amazon on locking an AMI to its original repository version.
So adding this to User Options via the EC2 Console when you launch fixes the problem on EC2:
#cloud-config
repo_releasever: 2014.03
What is the best way to always have this option when your Beanstalk launches the EC2 Instance on your behalf? I read about cloud-init and perhaps a script deployed via ebextensions would be the best bet?
Any insight on this is appreciated, thanks.
My solution to a 2017.03 image was to:
commands:
01_yum_update:
command: sudo yum -y update
02_epel_repo:
command: sudo yum-config-manager -y --enable epel
03_install_gdal_packages:
command: sudo yum -y install gdal gdal-devel
files:
"/etc/httpd/conf.d/wsgihacks.conf":
mode: "000644"
owner: root
group: root
content: |
WSGIPassAuthorization On
packages:
yum:
git: []
postgresql95-devel: []
gettext: []
libjpeg-turbo-devel: []
libffi-devel: []
So I now have a working ebextensions workflow on 2013.09 stack ami ami-35792c5c. For the 2014.09 stack see the other solution. The solution below works with postgis by installing the needed gdal component, 00_repo.config needs to look like this:
files:
"/etc/yum.repos.d/pgdg-93-redhat.repo":
mode: "000644"
owner: root
group: root
content: |
[pgdg93]
name=PostgreSQL 9.3 $releasever - $basearch
baseurl=http://yum.postgresql.org/9.3/redhat/rhel-6-$basearch
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-PGDG-93
[pgdg93-source]
name=PostgreSQL 9.3 $releasever - $basearch - Source
failovermethod=priority
baseurl=http://yum.postgresql.org/srpms/9.3/redhat/rhel-6-$basearch
enabled=0
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-PGDG-93
commands:
epel_repo:
command: yum-config-manager -y --enable epel
remi_repo:
command: yum-config-manager -y --enable remi
packages:
rpm:
pgdg-redhat93-9.3-1: 'http://yum.postgresql.org/9.3/redhat/rhel-6-x86_64/pgdg-redhat93-9.3-1.noarch.rpm'
remi: 'http://rpms.famillecollet.com/enterprise/remi-release-6.rpm'
qt4-devel: 'http://mirror.centos.org/centos/6/os/x86_64/Packages/qt-4.6.2-28.el6_5.x86_64.rpm'
The 2nd extension stays in tact. This works on Amazon Instance (2013.09) ami-35792c5c, I haven't tried the newer containers yet with it.
Alternative 1:
If you want less dependencies on repos / rpms from the ebextensions file, you could upload all required rpms to S3 and update the ebextensions 'packages' to point to your s3 rpms. Setup your S3 bucket for public get access using CORS headers.
Alternative 2:
Create a custom AMI where you compile all dependencies from source. This way there will be no rpm conflicts and you don't have to temper with the default repos supplied by the AMI you are customizing. See this answer:
Configuring Amazon Elastic Beanstalk with PostGIS
Also check out the tool I made:
https://github.com/radlws/django-awseb-tasks
I had a similar issue on the 2014.09 stack, ami id ami-246ed34c and got around the problem like this:
I found and downloaded the required dependencies online here and here. These are the versions that work:
lib64jpeg8-8b-5.1.mga1.x86_64.rpm
lib64poppler5-0.12.4-2.1mdv2010.1.x86_64.rpm
Zipped both .rpm files
Uploaded the .zip files to S3 and make sure the server can access them.
Added to ebextensions/00_sources.config the following code:
/etc/mylibs/jpeg8/ : https://s3.amazonaws.com/path-to-first-rpm-file.rpm.zip
/etc/mylibs/poppler/ : https://s3.amazonaws.com/path-to-second-rpm-file.rpm.zip
Installed via yum in ebextensions/01_packages.config. I had to use these commands to be able to install them.
commands:
jpeg8_repo:
command: yum -y install /etc/mylibs/jpeg8/lib64jpeg8-8b-5.1.mga1.x86_64.rpm
ignoreErrors: true
poppler_repo:
command: yum -y install /etc/mylibs/poppler/lib64poppler5-0.12.4-2.1mdv2010.1.x86_64.rpm
ignoreErrors: true
I was finally able to get this to work on the 2015.03 image and PostgreSQL 9.4 with a suggestion I found on the PostgreSQL yum mailing list (http://www.postgresql.org/message-id/1429688221.3808.17.camel#gunduz.org)
Specifically, I installed libpoppler straight from a CentOS 6 mirror. I also used Mike Edward's suggestion of the Amazon Linux specific pgdg rpm. In whole, it ended up being a rather simple solution, which looked like:
yum -y install http://yum.postgresql.org/9.4/redhat/rhel-6-x86_64/pgdg-ami201503-94-9.4-1.noarch.rpm
yum --enablerepo=epel -y install postgis2_94 http://centos-distro.cavecreek.net/centos/6.6/os/x86_64/Packages/poppler-0.12.4-3.el6_0.1.x86_64.rpm
For the 2015.03 image, I had good luck using this value for the pgdg repo:
pgdg-redhat93-9.3-1: 'http://yum.postgresql.org/9.3/redhat/rhel-6-x86_64/pgdg-ami201503-93-9.3-1.noarch.rpm'
With that, I was able to install gdal without needing to bring in libpoppler, etc., as separate rpms from S3.