using ddev in Github actions/workflows - github

Question

I have acceptance tests set up with ddev. They get run locally on ddev composer cookieman:test. I would like to use the same setup with Github actions.
Did anybody have any luck with ddev in Github actions/workflow? I am getting until here where ddev's healthcheck fails:
...
Creating ddev-router ... done
Failed to start extension-cookieman-master: ddev-router failed to become ready: logOutput=2019/11/15 02:24:19 [emerg] 1630#1630: no servers are inside upstream in /etc/nginx/conf.d/default.conf:89
nginx: [emerg] no servers are inside upstream in /etc/nginx/conf.d/default.conf:89
nginx: configuration file /etc/nginx/nginx.conf test failed
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
curl: (22) The requested URL returned error: 404 Not Found
ddev-router healthcheck endpoint not responding
, err=container /ddev-router unhealthy: 2019/11/15 02:24:19 [emerg] 1630#1630: no servers are inside upstream in /etc/nginx/conf.d/default.conf:89
nginx: [emerg] no servers are inside upstream in /etc/nginx/conf.d/default.conf:89
nginx: configuration file /etc/nginx/nginx.conf test failed
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
curl: (22) The requested URL returned error: 404 Not Found
ddev-router healthcheck endpoint not responding
##[error]Process completed with exit code 1.
.github/workflows/tests.yml:
name: Tests
on: [push, pull_request]
jobs:
tests-via-ddev:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v1
- run: export DEBIAN_FRONTEND=noninteractive
# update docker
- run: sudo -E apt-get purge -y docker docker-engine docker.io containerd runc nginx
- run: curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
- run: sudo -E add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
- run: sudo -E apt-get update
- run: sudo -E apt-get -y -o Dpkg::Options::="--force-confnew" install docker-ce
# install linuxbrew
- run: sh -c "$(curl -fsSL https://raw.githubusercontent.com/Linuxbrew/install/master/install.sh)"
- run: echo "::add-path::/home/linuxbrew/.linuxbrew/bin"
# install ddev + docker-compose
- run: brew tap drud/ddev && brew install ddev docker-compose
# Start ddev
- run: ddev start || exit 0
# Debug
- run: ls -als .ddev/
- run: curl 127.0.0.1 || exit 0
- run: curl 127.0.0.1/healthcheck || exit 0
- run: docker ps || exit 0
# we want Clover coverage
- run: ddev exec enable_xdebug
# Run tests
- run: ddev composer cookieman:test
I tried
using Ubuntu 16.04
fully upgrading all packages on Ubuntu 16.04/18.04
configuring ddev like that:
run: ddev config global --router-bind-all-interfaces=true
run: ddev config global --omit-containers=dba,ddev-ssh-agent
changing to unprivileged router ports (settings router_http_port, router_https_port in config.yaml)
If I force it to continue with ddev start || exit 0 I can see containers up and running:
- run: docker ps || exit 0
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c36601a06fd6 drud/ddev-router:v1.11.0 "/app/docker-entrypo…" 27 seconds ago Up 24 seconds (unhealthy) 0.0.0.0:4430->4430/tcp, 0.0.0.0:4444->4444/tcp, 0.0.0.0:8025->8025/tcp, 80/tcp, 0.0.0.0:8080->8080/tcp ddev-router
18152602a054 drud/ddev-webserver:v1.11.0-built "/start.sh" 30 seconds ago Up 28 seconds (healthy) 8025/tcp, 127.0.0.1:32770->80/tcp, 127.0.0.1:32769->443/tcp ddev-extension-cookieman-master-web
33aca55715f2 selenium/standalone-chrome:3.12 "/opt/bin/entry_poin…" 32 seconds ago Up 30 seconds 4444/tcp ddev-extension-cookieman-master-chrome
6c852ae62974 drud/ddev-dbserver:v1.11.0-10.2-built "/docker-entrypoint.…" 32 seconds ago Up 30 seconds (healthy) 127.0.0.1:32768->3306/tcp ddev-extension-cookieman-master-db
curl 127.0.0.1 yields the default nginx start page (while I would expect '503: No ddev back-end site available')
curl 127.0.0.1/healthcheck yields a 404
So far my conclusion is: ddev-router is reachable but its nginx does not have the appropriate configuration (no servers are inside upstream in /etc/nginx/conf.d/default.conf). Thus ddev only runs the pre-start hook form config.yaml. post-start is not reached.
You can see the output of the last runs here https://github.com/dmind-gmbh/extension-cookieman/actions?query=branch%3Afeat%2Facceptance-tests
EDIT/AMEND:
This is the (mal-)generated /etc/nginx/conf.d/default.conf from ddev-router:
# If we receive X-Forwarded-Proto, pass it through; otherwise, pass along the
# scheme used to connect to this server
map $http_x_forwarded_proto $proxy_x_forwarded_proto {
default $http_x_forwarded_proto;
'' $scheme;
}
# If we receive X-Forwarded-Port, pass it through; otherwise, pass along the
# server port the client connected to
map $http_x_forwarded_port $proxy_x_forwarded_port {
default $http_x_forwarded_port;
'' $server_port;
}
# If we receive Upgrade, set Connection to "upgrade"; otherwise, delete any
# Connection header that may have been passed to this server
map $http_upgrade $proxy_connection {
default upgrade;
'' close;
}
# Apply fix for very long server names
server_names_hash_bucket_size 128;
# Default dhparam
# ssl_dhparam /etc/nginx/dhparam/dhparam.pem;
# Set appropriate X-Forwarded-Ssl header
map $scheme $proxy_x_forwarded_ssl {
default off;
https on;
}
rfay mentioned a miscommunication between the ddev-router and the underlying docker daemon via sockets.

Answer 1

EDIT:
I put my findings into a Github action that can be included in other projects, too: https://github.com/marketplace/actions/setup-ddev
I came to the conclusion that the problem is with docker-gen.
In the first line of the template (https://github.com/drud/ddev/blob/master/containers/ddev-router/nginx.tmpl or also jwilder's https://github.com/jwilder/nginx-proxy/blob/master/nginx.tmpl) the .Docker.CurrentContainerID is empty which seemed to happen to some people in some contexts https://github.com/jwilder/docker-gen/issues/196#issuecomment-225412753.
The suggested removal of '-only-exposed' did not work for me. I changed the template a bit instead to not rely on the container and that was it.
:)
This is still a bit dirty and only a Proof-of-concpet:
this is the changed template https://github.com/jonaseberle/github-action-setup-ddev/blob/master/.ddev/patches/ddev-router/nginx.tmpl (compare the upstream {} section where I removed the check if containers are on the same network with the router)
In the workflow I do
ddev start || exit 0 # this will fail and also not execute any post-start hooks
docker cp nginx-debug.tmpl ddev-router:/app/nginx-debug.tmpl
docker exec ddev-router sh -c "docker-gen -only-exposed -notify 'sleep 1 && nginx -s reload' /app/nginx-debug.tmpl /etc/nginx/conf.d/default.conf"
... ddev is now up and healthy
Not decided how to move on from here. Maybe #rfay would have an idea how to change the nginx-template. Or I will use a custom Dockerfile for the ddev-router with a docker-compose.ddev-router.yaml to change the file just for the Github actions run...
EDIT/AMEND:
The shorter and tested version of this is:
ddev start || docker cp .ddev/patches/ddev-router/nginx.tmpl ddev-router:/app/nginx.tmpl
ddev start - this triggers a container restart and thus a docker-gen run