AWS Build Project service docker-compose build command not working - docker-compose

I want to build docker images and push into ECR for that I have written below buildspect.yml file and build my project using AWS build project service.
My buildspec.yml file is as below:
version: 0.2
phases:
install:
runtime-versions:
docker: 18
commands:
- nohup /usr/local/bin/dockerd --host=unix:///var/run/docker.sock --host=tcp://localhost:2375 --storage-driver=overlay2&
- timeout 15 sh -c "until docker info; do echo .; sleep 1; done"
pre_build:
commands:
- $(aws ecr get-login --no-include-email --region ${AWS_DEFAULT_REGION})
- REPOSITORY_URI_SERVER=<accountnumber>.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com/${IMAGE_NAME}
build:
commands:
- docker-compose build
post_build:
commands:
- docker-compose push
While I do build using AWS build project service then I will get an error like:
ERROR: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
errors pretty printing info
[Container] 2020/07/10 01:57:15 Command did not exit successfully timeout 15 sh -c "until docker info; do echo .; sleep 1; done" exit status 124
[Container] 2020/07/10 01:57:15 Phase complete: INSTALL State: FAILED
[Container] 2020/07/10 01:57:15 Phase context status code: COMMAND_EXECUTION_ERROR Message: Error while executing command: timeout 15 sh -c "until docker info; do echo .; sleep 1; done". Reason: exit status 124

I have tried to write buildspect.yml file as below
I have added an artifact tag in my file that is the place where my project build is a store.
post_build:
commands:
- docker-compose push
- printf '[{"name":"<name of container>","imageUri":"<path of your image"}]' > imagedefinitions.json
- cat imagedefinitions.json
artifacts:
files: imagedefinitions.json
I got success in it.

Related

docker-compose not sending signals to trap

I have a simple docker-compose.yml
version: '3.8'
services:
port-forwarder:
image: test
container_name: port-forwarder
volumes:
- "/path_to_scripts/port_forwarder.sh:/port_forwarder.sh"
command: "/port_forwarder.sh"
port_forwarder.sh
#!/usr/bin/env bash
trap 'echo exiting ; exit 0' SIGTERM TERM SIGINT INT EXIT WINCH SIGWINCH
tail -f /dev/null
Dockerfile
FROM alpine:3.12.0
RUN apk add --no-cache bash
If I build the image and run it using
docker build -t test .
docker run --name test -it --rm -v /path_to_scripts/port_forwarder.sh:/port_forwarder.sh test /port_forwarder.sh
and then hit ctrl-c, it catches the SIGTERM and writes "exiting" on the console. But when I run it using 'docker-compose up' and hit ctrl-c it hangs and is killed after 10 seconds. I am using docker on OSX. What could be the reason and how can I catch the termination signal from docker-compose?

Simple example to build a Dockerfile in Azure

Does anyone have a working azure-pipelines.yaml file that utilizes containers only to build Dockerfiles which I have in my repo.
resources:
containers:
- container: docker-image
image: docker:18.09.6
jobs:
- job: Build
container: docker-image
steps:
- script: |
echo hello from Linux
Docker --version
The above ends up with the error:
##[section]Starting: Initialize containers
##[command]/usr/bin/docker version --format '{{.Server.APIVersion}}'
'1.40'
Docker daemon API version: '1.40'
##[command]/usr/bin/docker version --format '{{.Client.APIVersion}}'
'1.40'
Docker client API version: '1.40'
##[command]/usr/bin/docker ps --all --quiet --no-trunc --filter "label=f38b39"
##[command]/usr/bin/docker network prune --force --filter "label=f38b39"
##[command]/usr/bin/docker network create --label f38b39 vsts_network_b78f6272ef304388ac849f3f37071eea
fda578a22eeb5aae1fcd41a4b816a279d20cfa552dbe3ad366fa2a009ab35313
##[command]/usr/bin/docker pull docker:18.09.6
18.09.6: Pulling from library/docker
e7c96db7181b: Already exists
5297bd381816: Pulling fs layer
3a664477889c: Pulling fs layer
a9b893dcc701: Pulling fs layer
48bf7c1cb0dd: Pulling fs layer
555b6ea27ad2: Pulling fs layer
48bf7c1cb0dd: Waiting
555b6ea27ad2: Waiting
3a664477889c: Verifying Checksum
3a664477889c: Download complete
5297bd381816: Download complete
48bf7c1cb0dd: Verifying Checksum
48bf7c1cb0dd: Download complete
5297bd381816: Pull complete
555b6ea27ad2: Verifying Checksum
555b6ea27ad2: Download complete
a9b893dcc701: Verifying Checksum
a9b893dcc701: Download complete
3a664477889c: Pull complete
a9b893dcc701: Pull complete
48bf7c1cb0dd: Pull complete
555b6ea27ad2: Pull complete
Digest: sha256:bf929409251faa1d1fcf63af68d02bb942054cfbe14e6ad9dca0dc4a01cbffad
Status: Downloaded newer image for docker:18.09.6
docker.io/library/docker:18.09.6
##[command]/usr/bin/docker inspect --format="{{index .Config.Labels \"com.azure.dev.pipelines.agent.handler.node.path\"}}" docker:18.09.6
##[command]/usr/bin/docker create --name docker-image_docker18096_794554 --label f38b39 --network vsts_network_b78f6272ef304388ac849f3f37071eea -v "/var/run/docker.sock":"/var/run/docker.sock" -v "/home/vsts/work/1":"/__w/1" -v "/home/vsts/work/_temp":"/__w/_temp" -v "/opt/hostedtoolcache":"/__t" -v "/home/vsts/work/_tasks":"/__w/_tasks" -v "/home/vsts/agents/2.150.3/externals":"/__a/externals":ro -v "/home/vsts/work/.taskkey":"/__w/.taskkey" docker:18.09.6 "/__a/externals/node/bin/node" -e "setInterval(function(){}, 24 * 60 * 60 * 1000);"
b8856a42131206302453903ae8fbc85957368b2e377ee5a9d1b44627130006b0
##[command]/usr/bin/docker start b8856a42131206302453903ae8fbc85957368b2e377ee5a9d1b44627130006b0
b8856a42131206302453903ae8fbc85957368b2e377ee5a9d1b44627130006b0
##[command]/usr/bin/docker ps --all --filter id=b8856a42131206302453903ae8fbc85957368b2e377ee5a9d1b44627130006b0 --filter status=running --no-trunc --format "{{.ID}} {{.Status}}"
b8856a42131206302453903ae8fbc85957368b2e377ee5a9d1b44627130006b0 Up Less than a second
##[command]/usr/bin/docker exec b8856a42131206302453903ae8fbc85957368b2e377ee5a9d1b44627130006b0 sh -c "command -v bash"
OCI runtime exec failed: exec failed: cannot exec a container that has stopped: unknown
##[error]Docker exec fail with exit code 126
##[section]Finishing: Initialize containers
that's because that container doesnt run endlessly if you just start it, it exits, change docker:18.09.6 to ubuntu or something that doesnt exit immediately and it will work.

OWASP/ZAP dangling when trying to scan

I am trying out OWASP/ZAP to see if it is something we can use for our project, but I cannot make it work I don't know what I am doing wrong and the documentation really does not help. What I am trying is to run a scan on my api running in a docker container locally on my windows machine so I run the command:
docker run -v $(pwd):/zap/wrk/:rw -t owasp/zap2docker-stable zap-baseline.py -t http://172.21.0.2:8080/swagger.json -g gen.conf -r testreport.html the ip 172.21.0.2 is the IPAddress of my api container even tried with localhost and 127.0.0.1
but it just hangs in the following log message:
_XSERVTransmkdir: ERROR: euid != 0,directory /tmp/.X11-unix will not be created.
Feb 14, 2019 1:43:31 PM java.util.prefs.FileSystemPreferences$1 run
INFO: Created user preferences directory.
Nothing happens and my zap docker container is in a unhealthy state, after some time it just crashes and ends up with a bunch of NullPointerExceptions. Is zap docker only working for linux, something specifically I need to do when running it on a windows machine? I don't get why this is not working even when I am following specifically the guideline in https://github.com/zaproxy/zaproxy/wiki/Docker
Edit 1
My latest try where I am trying to target my host ip address directly and the port that I am exposing my api to gives me the following error:
_XSERVTransmkdir: ERROR: euid != 0,directory /tmp/.X11-unix will not be created.
Feb 14, 2019 2:12:07 PM java.util.prefs.FileSystemPreferences$1 run
INFO: Created user preferences directory.
Total of 3 URLs
ERROR Permission denied
2019-02-14 14:12:57,116 I/O error(13): Permission denied
Traceback (most recent call last):
File "/zap/zap-baseline.py", line 347, in main
with open(base_dir + generate, 'w') as f:
IOError: [Errno 13] Permission denied: '/zap/wrk/gen.conf'
Found Java version 1.8.0_151
Available memory: 3928 MB
Setting jvm heap size: -Xmx982m
213 [main] INFO org.zaproxy.zap.DaemonBootstrap
When you run docker with: docker run -v $(pwd):/zap/wrk/:rw ...
you are mapping the /zap/wrk/ directory in the docker image to the current working directory (cwd) of the machine in which you are running docker.
I think the problem is that your current user doesn't have write access to the cwd.
Try below command, hope it resolves issue.
$docker run --user $(id -u):$(id -g) -v $(pwd):/zap/wrk/:rw --rm -t owasp/zap2docker-stable zap-baseline.py -t https://your_url -g gen.conf -r testreport.html
The key error here is:
IOError: [Errno 13] Permission denied: '/zap/wrk/gen.conf'
This means that the script cannot write to the gen.conf file that you have mounted on /zap/wrk
Do you have write access to the cwd when its not mounted?
The reason for that is, if you use -r parameter, zap will attempt to generate the file report.html at location /zap/wrk/. In order to make this work, we have to mount a directory to this location /zap/wrk.
But when you do so, it is important that the zap container is able to perform the write operations on the mounted directory.
So, below is the working solution using gitlab ci yml. I started with this approach of using image: owasp/zap2docker-stable however then had to go to the vanilla docker commands to execute it.
test_site:
stage: test
image: docker:latest
script:
# The folder zap-reports created locally will be mounted to owasp/zap2docker docker container,
# On execution it will generate the reports in this folder. Current user is passed so reports can be generated"
- mkdir zap-reports
- cd zap-reports
- docker pull owasp/zap2docker-stable:latest || echo
- docker run --name zap-container --rm -v $(pwd):/zap/wrk -u $(id -u ${USER}):$(id -g ${USER}) owasp/zap2docker-stable zap-baseline.py -t "https://example.com" -r report.html
artifacts:
when: always
paths:
- zap-reports
allow_failure: true
So the trick in the above code is
Mount local directory zap-reports to /zap/wrk as in $(pwd):/zap/wrk
Pass the current user and group on the host machine to the docker container so the process is using the same user / group. This allows writing of reports on the directory mounted from local host. This is done by -u $(id -u ${USER}):$(id -g ${USER})
Below is the working code with image: owasp/zap2docker-stable
test_site:
variables:
GIT_STRATEGY: none
stage: test
image:
name: owasp/zap2docker-stable:latest
before_script:
- mkdir -p /zap/wrk
script:
- zap-baseline.py -t "https://example.com" -g gen.conf -I -r testreport.html
- cp /zap/wrk/testreport.html testreport.html
artifacts:
when: always
paths:
- zap.out
- testreport.html

Docker in Docker Executor in Gitlab-Runner does not work (Cannot connect to the docker deamon)

So i recently tried Docker and Gitlab Runner but it seems i cant get it to work.
This is the log i have:
Running with gitlab-runner 10.0.2 (a9a76a50)
on my-docker (c588e5e2)
Using Docker executor with image docker:git ...
Starting service docker:dind ...
Pulling docker image docker:dind ...
Using docker image docker:dind ID=sha256:b9145b364a203c0afc538ca615b3470e41729edfb7338017f5d4eeb5b13b2d90 for docker service...
Waiting for services to be up and running...
Using docker image sha256:7961fbf38d6f827265aed22fe41a1db889c54913283b678a8623efdda9573977 for predefined container...
Pulling docker image docker:git ...
Using docker image docker:git ID=sha256:5917639be9495ab183f357e8bafafea82449f0c4b12b745eef8bd23d474220ca for build container...
Running on runner-c588e5e2-project-1-concurrent-0 via gitlabServer...
Cloning repository...
Cloning into '<Project name>'...
Checking out ed0ce69e as master...
Skipping Git submodules setup
$ # Auto DevOps variables and functions # collapsed multi-line command
$ setup_docker
$ build
Building Heroku-based application using gliderlabs/herokuish docker image...
**docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?.**
See 'docker run --help'.
ERROR: Job failed: exit code 125
What could be the error? Docker itsself is running as it seems, but the docker inside does not seem to work.
This is my .toml file:
[[runners]]
name = my name
url = my url
token = my token
executor = "docker"
[runners.docker]
tls_verify = false
image = "docker:latest"
privileged = true
disable_cache = false
volumes = ["/cache"]
shm_size = 0
[runners.cache]
Thanks in advance for help!
Edit: Thats what "docker ps" gave as output:
ED STATUS PORTS NAMES
e66e844481b7 7961fbf38d6f "gitlab-runner-ser..." 2 sec onds ago Up Less than a second runner-73520410-proje ct-1-concurrent-0-docker-0-wait-for-service
4f659dba7bac b9145b364a20 "dockerd-entrypoin..." 2 sec onds ago Up 1 second 2375/tcp runner-73520410-proje ct-1-concurrent-0-docker-0
73776d4638b9 gitlab/gitlab-runner:latest "/usr/bin/dumb-ini..." 19 mi nutes ago Up 19 minutes gitlab-runner
Edit 2: My gitlab-ci.yaml
#ruby 2.2
rspec:ruby2.2:
image: ruby:2.2
script:
- bundle exec rspec spec
tags:
- ruby
except:
- tags
#ruby 2.1
rspec:ruby2.1:
image: ruby:2.1
script:
- bundle exec rspec spec
tags:
- ruby
except:
- tags
.go: &go_definition
before_script:
- apt-get update -qq && apt-get install -y ruby
- ruby -v
script:
- go version
- which go
- bin/compile
- support/go-test
- support/go-format check
go:1.8:
<<: *go_definition
image: golang:1.8
codeclimate:
before_script: []
image: docker:latest
variables:
DOCKER_DRIVER: overlay
services:
- docker:dind
script:
- docker pull codeclimate/codeclimate
- docker run --env CODECLIMATE_CODE="$PWD" --volume "$PWD":/code --volume /var/run/docker.sock:/var/run/docker.sock --volume /tmp/cc:/tmp/cc codeclimate/codeclimate analyze -f json > codeclimate.json
artifacts:
paths: [codeclimate.json]
This is my registration command, I think you are missing to pass the privileged during registration and also make sure the gitlab-runner user is part of the docker group:
gitlab-runner register \
--template-config /tmp/gitlab-config.toml \
--config /etc/gitlab-runner/config.toml \
--non-interactive \
--url "$gitlab_url" \
--registration-token "$runner_registration_token" \
--name "$runner_name" \
--tag-list "$runner_tags" \
--run-untagged="$runner_run_untagged" \
--locked="$runner_locked" \
--access-level="$runner_access" \
--maximum-timeout="$maximum_timeout" \
--executor "docker" \
--docker-privileged \
--docker-volumes "/cache" \
--docker-volumes "/certs/client" \
--docker-image "$runner_image"
sudo usermod -aG docker gitlab-runner
# concurrent global can't be setup in registration
# See: https://gitlab.com/gitlab-org/gitlab/-/issues/332497
sed -i "s/concurrent.*/concurrent = $concurrent/" /etc/gitlab-runner/config.toml
# prometheus port for GitLab Runner is 9252 as defined here https://github.com/prometheus/prometheus/wiki/Default-port-allocations
echo -e "listen_address = \":9252\"\n$(cat /etc/gitlab-runner/config.toml)" > /etc/gitlab-runner/config.toml

gitlab-ci-multi-runner 9.2.0: ERROR: Job failed: exit code 252

I am running my tests inside a docker image on a docker gitlab executor. The logs are looking like this:
Running with gitlab-ci-multi-runner 9.2.0
.
.
<after_script called>
ERROR: Job failed: exit code 252
The last thing I am doing in my job script (shell) is a call to mongo eval
My after_script for this job just has a docker rm -f imagename like so:
after_script:
- docker rm -f imagename
First of all, I cannot find any concrete reference to this error code #. I don't know if it is a mongo eval error code or gitlab ci. My best guess is gitlab ci because that script works fine if run directly on the build machine. Also, the after_script is executed
I added the following in the beginning of my .gitlab-ci.yml:
variables:
CI_DEBUG_TRACE: "true"
But all I am seeing is:
+ docker rm -f imagename
imagename
+ exit 0
ERROR: Job failed: exit code 252
Any help or any clue is appreciated!
My mongo eval connection was failing because my mongo command was incorrect. I
Left the docker container running by using && tail -f /dev/null in my docker CMD.
I shelled in with $docker exec -it
Ran my script and executed the mongo eval section step by step again
Getting a connection fail for the wrong mongo eval javascript threw me off :/