How to access a service in Github Actions CI/CD? - github

I'm trying to set up a CI/CD pipeline in GitHub Actions for my Elixir project.
I can fetch dependencies, compile them, check formatting, credo... But when the tests starts, I'm not able to reach the PostgreSQL service declared on the YAML.
How can I link both containers? (Elixir and PostgreSQL)
According to the logs shown on GitHub Actions, both containers are on the same Docker network, so they should be reachable from each other using their network aliases. However, when I try to connect to the postgres one, it says NXDOMAIN. Also the ping doesn't work, as expected.
The content of my workflow:
name: Elixir CI
on: push
jobs:
build:
runs-on: ubuntu-18.04
container:
image: elixir:1.9.1
services:
postgres:
image: postgres
ports:
- 5432:5432
env:
POSTGRES_USER: my_app
POSTGRES_PASSWORD: my_app
POSTGRES_DB: my_app_test
steps:
- uses: actions/checkout#v1
- name: Install Dependencies
env:
MIX_ENV: test
run: |
cp config/test.secret.ci.exs config/test.secret.exs
mix local.rebar --force
mix local.hex --force
apt-get update -qqq && apt-get install make gcc -y -qqq
mix deps.get
- name: Compile
env:
MIX_ENV: test
run: mix compile --warnings-as-errors
- name: Run formatter
env:
MIX_ENV: test
run: mix format --check-formatted
- name: Run Credo
env:
MIX_ENV: test
run: mix credo
- name: Run Tests
env:
MIX_ENV: test
run: mix test
Also, on Elixir I have set up the test task to connect to postgres:5432, but it says the host does not exist.
According to some tutorials and examples I found on the Internet, this configurations looks like valid, but nothing I could do made it work.

You need to pass the name of the service ("postgres") as POSTGRES_HOST to the application and set the port POSTGRES_PORT: ${{ job.services.postgres.ports[5432] }} (spaces matter.)
Github CI dynamically routes port and host to it.
I wrote a blog post on the subject a couple of days ago.

Related

Why can my Github Action step not connect to Docker container?

In Github Actions I want to test my frontend through Cypress. I run my Jekyll application in a Docker container and then run the tests in the next step. However, when connecting the Cypress tests to the Docker container I get a 'Connection refused' error. Does anyone know how I can access my container after running docker-compose?
This is my Github Actions yml file:
name: Cypress Tests
on: [push]
jobs:
cypress-run:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
- name: Cypress run
uses: cypress-io/github-action#v4.2.0
with:
start: docker-compose up -d
This is my docker-compose file:
version: '2'
services:
jekyll:
image: jekyll/jekyll:latest
command: jekyll serve --watch --force_polling --verbose
ports:
- 4000:4000
volumes:
- .:/srv/jekyll
So apparently although my Jekyll container started perfectly fine, the actual Jekyll server crashed on startup. I needed to specify the UID and GID of Github actions to my Jekyll environment in order for it to start correctly.
This is my new docker-compose file:
version: '2'
services:
jekyll:
image: jekyll/jekyll:latest
command: jekyll serve --watch --force_polling --verbose
hostname: localhost
environment:
JEKYLL_UID: 1001
JEKYLL_GID: 1001
ports:
- 4000:4000
volumes:
- .:/srv/jekyll

Github Actions to Connect Postgres service with custom container image

In my Django project, I have a CI workflow for running tests, which requires a Postgres service. Recently a new app introduced heavier packages such as pandas, matplotlib, pytorch and so on and this increased the run-tests job time from 2 to 12 minutes which is absurd. Also in my project, I have a base Docker image with Python and these packages that are heavier to speed up the build of the images. So I was thinking to use this same image in the workflow when running the steps because the packages would be loaded already.
Unfortunately, all goes well until it reaches the step to actually run the tests because it seems that the postgres service is not connected with the container and I get the following error:
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "localhost" (127.0.0.1) and accepting
TCP/IP connections on port 5432?
This is my workflow right now. Any ideas on what I am doing wrong?
name: server-ci
on:
pull_request:
types: [opened]
env:
DJANGO_SETTINGS_MODULE: settings_test
jobs:
run-tests:
name: Run tests
runs-on: ubuntu-latest
container:
image: myimage/django-server:base
credentials:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_PASSWORD }}
ports:
- 8000:8000
services:
postgres:
image: postgres
env:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: admin
POSTGRES_DB: mydb
ports:
- 5432:5432
options: --health-cmd pg_isready --health-interval 10s --health-timeout 5s --health-retries 5
env:
POSTGRES_HOST: localhost
POSTGRES_PORT: 5432
POSTGRES_PASSWORD: admin
POSTGRES_USER: postgres
steps:
- name: Checkout repository
uses: actions/checkout#v2
- name: Cache dependencies
uses: actions/cache#v2
with:
path: /opt/venv
key: /opt/venv-${{ hashFiles('**/requirements.txt') }}
- name: Install dependencies
run: |
python -m pip install --upgrade pip
python -m pip install -r requirements.txt
if: steps.cache.outputs.cache-hit != 'true'
- name: Run tests
run: |
./manage.py test --parallel --verbosity=2
It turns out that the workflow is now running in a container of its own, next to the postgres container. So the port mapping to the runner VM doesn’t do anything any more (because it affects the host, not Docker containers on it).
The job and service containers get attached to the same Docker network, so all I need to do is change POSTGRES_HOST to postgres (the name of the service container) and Docker’s DNS should do the rest.
Credits: https://github.community/t/connect-postgres-service-with-custom-container-image/189994/2?u=everspader

Using psql in a github action

I am trying to use psql in a github action but am seeing the following error:
psql: error: could not connect to server: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
where my github action yml file is shown below (the run_all_tests.sh file just calls a subprocess that tries to run the command psql). Does anyone know why this could be happening?
name: Python application
on:
push:
branches: [master]
pull_request:
branches: [master]
jobs:
build:
runs-on: ubuntu-latest
# Service containers to run with `container-job`
services:
# Label used to access the service container
postgres:
# Docker Hub image
image: postgres
# Provide the password for postgres
env:
POSTGRES_PASSWORD: postgres
# Set health checks to wait until postgres has started
ports:
- 5432:5432
options: >-
--health-cmd pg_isready
--health-interval 10s
--health-timeout 5s
--health-retries 5
steps:
- name: Copy the code
uses: actions/checkout#v2
- name: Set up Python 3.8
uses: actions/setup-python#v2
with:
python-version: 3.8
- name: Install dependencies
run: |
python3 setup.py install
- name: Test with unittest
run: |
cd backend/py
source run_all_tests.sh
env:
# The hostname used to communicate with the PostgreSQL service container
POSTGRES_HOST: postgres
# The default PostgreSQL port
POSTGRES_PORT: 5432
Since I was having the same issue, I tried a different approach that worked for me.
In the first place, I run the job within a container:
jobs:
build:
container: gradle:jdk11
That won't make the psql command available so you need to add a run step to install it. The particular installation method may differ depending on the Docker image you choose:
jobs:
build:
container: gradle:jdk11
...
steps:
- run: |
apt-get update
apt-get install --yes --no-install-recommends postgresql-client
Please note you may have different steps above or below.
Now it's time to execute all these SQL you need. The most important thing here: database hostname is postgres which is the id of the service container.
jobs:
build:
container: gradle:jdk11
...
steps:
- run: |
apt-get update
apt-get install --yes --no-install-recommends postgresql-client
- run: |
psql -h postgres -U postgres -c 'CREATE DATABASE ...'
psql -h postgres -U postgres -c 'CREATE ROLE ...'
Since the job is running directly on a runner machine (not within a docker container), you need to connect to "localhost" instead of "postgres". It should work if you change POSTGRES_HOST: postgres to POSTGRES_HOST: localhost.
This is described in detail in the docs: https://docs.github.com/en/actions/using-containerized-services/creating-postgresql-service-containers

Running localhost server for Unit Tests executed through GitHub Actions

I have the following GitHub workflow for building my project
name: build
on: [push]
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
- name: set up JDK 1.8
uses: actions/setup-java#v1
with:
java-version: 1.8
- name: Build with Maven
run: mvn clean compile test
The build works just fine.
However the project JUnit test require that a localhost server is listening on port 4444 ... and I get the following error:
Connection refused: localhost/127.0.0.1:4444
The server is spun up before each JUnit test and is part of the tests suite.
How do I tell the docker container that network connections are allowed on this port?
Or are there any open ports by default?
I share my solution. Hopefully it will help.
The Dockerfile for the test server listening on port (in my case 8080). As the comments mention above, you need to expose your port
FROM golang:1.15.2-alpine
# Setup you server
# This container exposes port 8080 to the outside world
EXPOSE 8080
# Run the executable
CMD ["./main"]
The docker-compose file (this is not a must, but in my case I had to start multiple depending containers). Again make sure the port is defined
main-server:
build: ./
container_name: main-server
image: artofimagination/main-server
ports:
- 8080:8080
The github action. Just run the docker-compose command for your server. Then the application that needs to connect to the port. In this case pytest tries to send requests to main-server through port 8080
It is worth to mention that in my example pytest accessing 127.0.0.1:8080
- name: Check out code into the Go module directory
uses: actions/checkout#v2
- name: Start test server
run: docker-compose up -d main-server
- name: Run functional test
run: pip3 install -r test/requirements.txt && pytest -v test
Good luck and I hope this helps.

CircleCI Swift with Postgres connection issues

I am working with my repo to build a test app for swift with circleCI and postgres but when it comes to testing I can't seem to grasp how to connect the two images in the testing phase.
I am running
circleci local execute --job build
Which should build both the swift and postgres images. I give them both the same env variables I give in the application. However I get this error when trying to run it. In my experience when trying to set up the two docker containers with compose this error was showing up when my api could not connect to the db container over the network.
Test Case 'AppTests.RemoveUserTest' started at 2019-04-09 19:46:15.380
Fatal error: 'try!' expression unexpectedly raised an error: NIO.ChannelError.connectFailed(NIO.NIOConnectionError(host: "db", port: 5432, dnsAError: Optional(NIO.SocketAddressError.unknown(host: "db", port: 5432)), dnsAAAAError: Optional(NIO.SocketAddressError.unknown(host: "db", port: 5432)), connectionErrors: [])): file /home/buildnode/jenkins/workspace/oss-swift-4.2-package-linux-ubuntu-16_04/swift/stdlib/public/core/ErrorType.swift, line 184
I know it says it failed because of a try statement but that try statement is failing because it's requesting actions from Postgres which is not there. Any ideas?
My current config.yml for circleci
version: 2
jobs:
build:
docker:
- image: swift:4.2
environment:
POSTGRES_USER: test
POSTGRES_PASSWORD: test
POSTGRES_DB: test
DB_HOSTNAME: db
PORT: 5432
- image: postgres:11.2-alpine
environment:
POSTGRES_USER: test
POSTGRES_PASSWORD: test
POSTGRES_DB: test
steps:
- checkout
- run: apt-get update -qq
- run: apt-get install -yq libssl-dev pkg-config wget
- run: apt-get install -y postgresql-client || true
- run:
name: install dockerize
command: wget https://github.com/jwilder/dockerize/releases/download/$DOCKERIZE_VERSION/dockerize-linux-amd64-$DOCKERIZE_VERSION.tar.gz && tar -C /usr/local/bin -xzvf dockerize-linux-amd64-$DOCKERIZE_VERSION.tar.gz && rm dockerize-linux-amd64-$DOCKERIZE_VERSION.tar.gz
environment:
DOCKERIZE_VERSION: v0.3.0
- run:
name: Wait for db
command: dockerize -wait tcp://localhost:5432 -timeout 1m
- run:
name: Compile code
command: swift build
- run:
name: Run unit tests
command: swift test
release:
docker:
- image: swift:4.2
steps:
- checkout
- run:
name: Compile code with optimizations
command: swift build -c release
push-to-docker-hub:
docker:
- image: docker:latest
steps:
- checkout
- setup_remote_docker
- run:
name: Install dependencies
command: |
apk add --update --no-cache curl jq python py-pip
- run:
name: Build Docker Image
command: |
docker build -t api .
docker tag api <>/repo:latest
docker tag api <>/repo:$CIRCLE_SHA1
docker login -u $DOCKER_USER -p $DOCKER_PASS
docker push <>/repo:latest
docker push <>/repo:$CIRCLE_SHA1
# - persist_to_workspace:
# root: ./
# paths:
# - k8s-*.yml
workflows:
version: 2
tests:
jobs:
- build
- push-to-docker-hub:
requires:
- build
context: dockerhub
filters:
branches:
only: master
#- linux-release
You're setting the hostname for the database to db, but not defining that anywhere. You need to name your Docker container to match the DB_HOSTNAME environment variable like so https://github.com/vapor/postgresql/blob/master/circle.yml#L8