I am working with my repo to build a test app for swift with circleCI and postgres but when it comes to testing I can't seem to grasp how to connect the two images in the testing phase.
I am running
circleci local execute --job build
Which should build both the swift and postgres images. I give them both the same env variables I give in the application. However I get this error when trying to run it. In my experience when trying to set up the two docker containers with compose this error was showing up when my api could not connect to the db container over the network.
Test Case 'AppTests.RemoveUserTest' started at 2019-04-09 19:46:15.380
Fatal error: 'try!' expression unexpectedly raised an error: NIO.ChannelError.connectFailed(NIO.NIOConnectionError(host: "db", port: 5432, dnsAError: Optional(NIO.SocketAddressError.unknown(host: "db", port: 5432)), dnsAAAAError: Optional(NIO.SocketAddressError.unknown(host: "db", port: 5432)), connectionErrors: [])): file /home/buildnode/jenkins/workspace/oss-swift-4.2-package-linux-ubuntu-16_04/swift/stdlib/public/core/ErrorType.swift, line 184
I know it says it failed because of a try statement but that try statement is failing because it's requesting actions from Postgres which is not there. Any ideas?
My current config.yml for circleci
version: 2
jobs:
build:
docker:
- image: swift:4.2
environment:
POSTGRES_USER: test
POSTGRES_PASSWORD: test
POSTGRES_DB: test
DB_HOSTNAME: db
PORT: 5432
- image: postgres:11.2-alpine
environment:
POSTGRES_USER: test
POSTGRES_PASSWORD: test
POSTGRES_DB: test
steps:
- checkout
- run: apt-get update -qq
- run: apt-get install -yq libssl-dev pkg-config wget
- run: apt-get install -y postgresql-client || true
- run:
name: install dockerize
command: wget https://github.com/jwilder/dockerize/releases/download/$DOCKERIZE_VERSION/dockerize-linux-amd64-$DOCKERIZE_VERSION.tar.gz && tar -C /usr/local/bin -xzvf dockerize-linux-amd64-$DOCKERIZE_VERSION.tar.gz && rm dockerize-linux-amd64-$DOCKERIZE_VERSION.tar.gz
environment:
DOCKERIZE_VERSION: v0.3.0
- run:
name: Wait for db
command: dockerize -wait tcp://localhost:5432 -timeout 1m
- run:
name: Compile code
command: swift build
- run:
name: Run unit tests
command: swift test
release:
docker:
- image: swift:4.2
steps:
- checkout
- run:
name: Compile code with optimizations
command: swift build -c release
push-to-docker-hub:
docker:
- image: docker:latest
steps:
- checkout
- setup_remote_docker
- run:
name: Install dependencies
command: |
apk add --update --no-cache curl jq python py-pip
- run:
name: Build Docker Image
command: |
docker build -t api .
docker tag api <>/repo:latest
docker tag api <>/repo:$CIRCLE_SHA1
docker login -u $DOCKER_USER -p $DOCKER_PASS
docker push <>/repo:latest
docker push <>/repo:$CIRCLE_SHA1
# - persist_to_workspace:
# root: ./
# paths:
# - k8s-*.yml
workflows:
version: 2
tests:
jobs:
- build
- push-to-docker-hub:
requires:
- build
context: dockerhub
filters:
branches:
only: master
#- linux-release
You're setting the hostname for the database to db, but not defining that anywhere. You need to name your Docker container to match the DB_HOSTNAME environment variable like so https://github.com/vapor/postgresql/blob/master/circle.yml#L8
Related
I am trying to use psql in a github action but am seeing the following error:
psql: error: could not connect to server: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
where my github action yml file is shown below (the run_all_tests.sh file just calls a subprocess that tries to run the command psql). Does anyone know why this could be happening?
name: Python application
on:
push:
branches: [master]
pull_request:
branches: [master]
jobs:
build:
runs-on: ubuntu-latest
# Service containers to run with `container-job`
services:
# Label used to access the service container
postgres:
# Docker Hub image
image: postgres
# Provide the password for postgres
env:
POSTGRES_PASSWORD: postgres
# Set health checks to wait until postgres has started
ports:
- 5432:5432
options: >-
--health-cmd pg_isready
--health-interval 10s
--health-timeout 5s
--health-retries 5
steps:
- name: Copy the code
uses: actions/checkout#v2
- name: Set up Python 3.8
uses: actions/setup-python#v2
with:
python-version: 3.8
- name: Install dependencies
run: |
python3 setup.py install
- name: Test with unittest
run: |
cd backend/py
source run_all_tests.sh
env:
# The hostname used to communicate with the PostgreSQL service container
POSTGRES_HOST: postgres
# The default PostgreSQL port
POSTGRES_PORT: 5432
Since I was having the same issue, I tried a different approach that worked for me.
In the first place, I run the job within a container:
jobs:
build:
container: gradle:jdk11
That won't make the psql command available so you need to add a run step to install it. The particular installation method may differ depending on the Docker image you choose:
jobs:
build:
container: gradle:jdk11
...
steps:
- run: |
apt-get update
apt-get install --yes --no-install-recommends postgresql-client
Please note you may have different steps above or below.
Now it's time to execute all these SQL you need. The most important thing here: database hostname is postgres which is the id of the service container.
jobs:
build:
container: gradle:jdk11
...
steps:
- run: |
apt-get update
apt-get install --yes --no-install-recommends postgresql-client
- run: |
psql -h postgres -U postgres -c 'CREATE DATABASE ...'
psql -h postgres -U postgres -c 'CREATE ROLE ...'
Since the job is running directly on a runner machine (not within a docker container), you need to connect to "localhost" instead of "postgres". It should work if you change POSTGRES_HOST: postgres to POSTGRES_HOST: localhost.
This is described in detail in the docs: https://docs.github.com/en/actions/using-containerized-services/creating-postgresql-service-containers
In my Circle build, I'm trying to use a Postgres container for testing. The test DB will be created automatically, but Python's not finding the DB. Here's my config.yml:
version: 2.1
orbs:
python: circleci/python#0.2.1
jobs:
build:
executor: python/default
docker:
- image: circleci/python:3.8.0
environment:
- ENV: CIRCLE
- image: circleci/postgres:9.6
environment:
POSTGRES_USER: circleci
POSTGRES_DB: circle_test
POSTGRES_HOST_AUTH_METHOD: trust
steps:
- checkout
- python/load-cache
- run:
command: |
python3 -m venv venv
. venv/bin/activate
pip3 install -r requirements.txt
- python/save-cache
- run:
name: Running tests
command: |
. venv/bin/activate
python3 ./api/manage.py test
- store_artifacts:
path: test-reports/
destination: python_app
workflows:
main:
jobs:
- build
It seems like everything's fine until the tests start:
psycopg2.OperationalError: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
There's no such folder:
ls -a /var/run/
. .. exim4 lock utmp
It looks like psycopg2 defaults to using the UNIX socket, so you'll need to specify the db host as localhost in your connect() call.
Also an aside, but it looks like you're specifying an additional but unneeded Python image. The executor for the job is already set to be python/default so it doesn't seem like the next image circleci/python:3.8.0 is doing anything based on the config you've pasted in.
Some changes in the PGHOST and DATABASE_URL config solved it:
jobs:
build:
executor: python/default
docker:
- image: circleci/python:3.8.0
environment:
ENV: CIRCLE
DATABASE_URL: postgresql://circleci#localhost/circle_test
- image: circleci/postgres:9.6
environment:
PGHOST: localhost
PGUSER: circleci
POSTGRES_USER: circleci
POSTGRES_DB: circle_test
POSTGRES_HOST_AUTH_METHOD: trust
steps:
- checkout
- python/load-cache
- run:
name: Wait for db to run
command: dockerize -wait tcp://localhost:5432 -timeout 1m
- run:
command: |
python3 -m venv venv
. venv/bin/activate
pip3 install -r requirements.txt
- python/save-cache
- run:
name: Running tests
command: |
. venv/bin/activate
cat /etc/hosts
python3 ./django/gbookapi/manage.py test gbook.tests
- store_test_results:
path: test-reports.xml
- store_artifacts:
path: test-reports/
destination: python_app
workflows:
main:
jobs:
- build
I'm trying to set up a CI/CD pipeline in GitHub Actions for my Elixir project.
I can fetch dependencies, compile them, check formatting, credo... But when the tests starts, I'm not able to reach the PostgreSQL service declared on the YAML.
How can I link both containers? (Elixir and PostgreSQL)
According to the logs shown on GitHub Actions, both containers are on the same Docker network, so they should be reachable from each other using their network aliases. However, when I try to connect to the postgres one, it says NXDOMAIN. Also the ping doesn't work, as expected.
The content of my workflow:
name: Elixir CI
on: push
jobs:
build:
runs-on: ubuntu-18.04
container:
image: elixir:1.9.1
services:
postgres:
image: postgres
ports:
- 5432:5432
env:
POSTGRES_USER: my_app
POSTGRES_PASSWORD: my_app
POSTGRES_DB: my_app_test
steps:
- uses: actions/checkout#v1
- name: Install Dependencies
env:
MIX_ENV: test
run: |
cp config/test.secret.ci.exs config/test.secret.exs
mix local.rebar --force
mix local.hex --force
apt-get update -qqq && apt-get install make gcc -y -qqq
mix deps.get
- name: Compile
env:
MIX_ENV: test
run: mix compile --warnings-as-errors
- name: Run formatter
env:
MIX_ENV: test
run: mix format --check-formatted
- name: Run Credo
env:
MIX_ENV: test
run: mix credo
- name: Run Tests
env:
MIX_ENV: test
run: mix test
Also, on Elixir I have set up the test task to connect to postgres:5432, but it says the host does not exist.
According to some tutorials and examples I found on the Internet, this configurations looks like valid, but nothing I could do made it work.
You need to pass the name of the service ("postgres") as POSTGRES_HOST to the application and set the port POSTGRES_PORT: ${{ job.services.postgres.ports[5432] }} (spaces matter.)
Github CI dynamically routes port and host to it.
I wrote a blog post on the subject a couple of days ago.
I am trying to deploy a Flack app/service which is built into a docker container to Gitlab CI. I am able to get everything working via docker-compose except when I try to run tests against the postgres database I am getting the below error:
Is the server running on host "events_db" (172.19.0.2) and accepting
TCP/IP connections on port 5432?
Presumably this is because the containers can't see each other. I've tried many different methods. But below is my latest. I have attempted to have docker-compose spin up both containers (just like it does on local), run the postgres db as a git lab service, run from a python image instead of a docker image, use a docker.prod.yml where I remove the volumes and variables.
Nothing is working. I've checked just about every link that shows up on google when you look for 'gitlab ci docker flask postgres' and I believe that I am massively misunderstanding the implementation.
I do have gitlab runner up and going.
.gitlab-ci.yml
image: docker:latest
services:
- docker:dind
- postgres:latest
stages:
- test
variables:
POSTGRES_DB: events_test
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
DATABASE_URL: postgres://postgres#postgres:5432/events_test
FLASK_ENV: development
APP_SETTINGS: app.config.TestingConfig
DOCKER_COMPOSE_VERSION: 1.23.2
before_script:
#- rm /usr/local/bin/docker-compose
- apk add --no-cache py-pip python-dev libffi-dev openssl-dev gcc libc-dev make
- pip install docker-compose
#- mv docker-compose /usr/local/bin
- docker-compose up -d --build
test:
stage: test
#coverage: '/TOTAL.+ ([0-9]{1,3}%)/'
script:
- docker-compose exec -T events python manage.py test
after_script:
- docker-compose down
docker-compose.yml
version: '3.3'
services:
events:
build:
context: ./services/events
dockerfile: Dockerfile
volumes:
- './services/events:/usr/src/app'
ports:
- 5001:5000
environment:
- FLASK_ENV=development
- APP_SETTINGS=app.config.DevelopmentConfig
- DATABASE_URL=postgres://postgres:postgres#events_db:5432/events_dev # new
- DATABASE_TEST_URL=postgres://postgres:postgres#events_db:5432/events_test # new
events_db:
build:
context: ./services/events/app/db
dockerfile: Dockerfile
ports:
- 5435:5432
environment:
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres
What is the executor type of your Gitlab Runner?
If you're using the Kubernetes executor, add this variable:
DOCKER_HOST: tcp://localhost:2375/
For non-Kubernetes executors, we use tcp://docker:2375/
DOCKER_HOST: tcp://docker:2375/
Also, the Gitlab Runner should be in "privileged" mode.
More info:
https://docs.gitlab.com/ee/ci/docker/using_docker_build.html#help-and-feedback
Hope that helps!
I'm trying to setup Gitlab Continuous Integration to run some test on an app that uses the MERN stack, I need to configure a mongodb so the test can run but I can't
This is my .gitlab-ci.yml file
image: node:latest
stages:
- build
- test
cache:
paths:
- node_modules/
- packages/back-end/node_modules
- packages/front-end/node_modules
install_dependencies:
stage: build
script:
- npm install
- npm run lerna bootstrap
artifacts:
paths:
- node_modules/
- packages/back-end/node_modules
- packages/front-end/node_modules
testing:
image: docker:latest
stage: test
services:
- docker:dind
- mongo:latest
variables:
PORT: 3001
URL_DB: "mongodb://mongodb#mongodb/todos"
script:
- docker run -d --network host --name todos -e MONGO_INITDB_ROOT_USERNAME=test -e MONGO_INITDB_ROOT_PASSWORD=test -p 27017:27017 -v "/usr/db:/data/db" mongo:latest
- cd packages/back-end
- npm start
- npm test
I have this error on the test stage:
Skipping Git submodules setup
Checking cache for default...
No URL provided, cache will not be downloaded from shared cache server. Instead a local version of cache will be extracted.
Successfully extracted cache
Downloading artifacts for install_dependencies (28022)...
Downloading artifacts from coordinator... ok id=28022 responseStatus=200 OK token=jy6ye2Q1
docker run -d --network host --name todos -e MONGO_INITDB_ROOT_USERNAME=test -e MONGO_INITDB_ROOT_PASSWORD=test -p 27017:27017 -v "/usr/db:/data/db" mongo:latest
docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?.
See 'docker run --help'.
ERROR: Job failed: command terminated with exit code 125