Does anyone have a working azure-pipelines.yaml file that utilizes containers only to build Dockerfiles which I have in my repo.
resources:
containers:
- container: docker-image
image: docker:18.09.6
jobs:
- job: Build
container: docker-image
steps:
- script: |
echo hello from Linux
Docker --version
The above ends up with the error:
##[section]Starting: Initialize containers
##[command]/usr/bin/docker version --format '{{.Server.APIVersion}}'
'1.40'
Docker daemon API version: '1.40'
##[command]/usr/bin/docker version --format '{{.Client.APIVersion}}'
'1.40'
Docker client API version: '1.40'
##[command]/usr/bin/docker ps --all --quiet --no-trunc --filter "label=f38b39"
##[command]/usr/bin/docker network prune --force --filter "label=f38b39"
##[command]/usr/bin/docker network create --label f38b39 vsts_network_b78f6272ef304388ac849f3f37071eea
fda578a22eeb5aae1fcd41a4b816a279d20cfa552dbe3ad366fa2a009ab35313
##[command]/usr/bin/docker pull docker:18.09.6
18.09.6: Pulling from library/docker
e7c96db7181b: Already exists
5297bd381816: Pulling fs layer
3a664477889c: Pulling fs layer
a9b893dcc701: Pulling fs layer
48bf7c1cb0dd: Pulling fs layer
555b6ea27ad2: Pulling fs layer
48bf7c1cb0dd: Waiting
555b6ea27ad2: Waiting
3a664477889c: Verifying Checksum
3a664477889c: Download complete
5297bd381816: Download complete
48bf7c1cb0dd: Verifying Checksum
48bf7c1cb0dd: Download complete
5297bd381816: Pull complete
555b6ea27ad2: Verifying Checksum
555b6ea27ad2: Download complete
a9b893dcc701: Verifying Checksum
a9b893dcc701: Download complete
3a664477889c: Pull complete
a9b893dcc701: Pull complete
48bf7c1cb0dd: Pull complete
555b6ea27ad2: Pull complete
Digest: sha256:bf929409251faa1d1fcf63af68d02bb942054cfbe14e6ad9dca0dc4a01cbffad
Status: Downloaded newer image for docker:18.09.6
docker.io/library/docker:18.09.6
##[command]/usr/bin/docker inspect --format="{{index .Config.Labels \"com.azure.dev.pipelines.agent.handler.node.path\"}}" docker:18.09.6
##[command]/usr/bin/docker create --name docker-image_docker18096_794554 --label f38b39 --network vsts_network_b78f6272ef304388ac849f3f37071eea -v "/var/run/docker.sock":"/var/run/docker.sock" -v "/home/vsts/work/1":"/__w/1" -v "/home/vsts/work/_temp":"/__w/_temp" -v "/opt/hostedtoolcache":"/__t" -v "/home/vsts/work/_tasks":"/__w/_tasks" -v "/home/vsts/agents/2.150.3/externals":"/__a/externals":ro -v "/home/vsts/work/.taskkey":"/__w/.taskkey" docker:18.09.6 "/__a/externals/node/bin/node" -e "setInterval(function(){}, 24 * 60 * 60 * 1000);"
b8856a42131206302453903ae8fbc85957368b2e377ee5a9d1b44627130006b0
##[command]/usr/bin/docker start b8856a42131206302453903ae8fbc85957368b2e377ee5a9d1b44627130006b0
b8856a42131206302453903ae8fbc85957368b2e377ee5a9d1b44627130006b0
##[command]/usr/bin/docker ps --all --filter id=b8856a42131206302453903ae8fbc85957368b2e377ee5a9d1b44627130006b0 --filter status=running --no-trunc --format "{{.ID}} {{.Status}}"
b8856a42131206302453903ae8fbc85957368b2e377ee5a9d1b44627130006b0 Up Less than a second
##[command]/usr/bin/docker exec b8856a42131206302453903ae8fbc85957368b2e377ee5a9d1b44627130006b0 sh -c "command -v bash"
OCI runtime exec failed: exec failed: cannot exec a container that has stopped: unknown
##[error]Docker exec fail with exit code 126
##[section]Finishing: Initialize containers
that's because that container doesnt run endlessly if you just start it, it exits, change docker:18.09.6 to ubuntu or something that doesnt exit immediately and it will work.
Related
I have a docker image, already made, that another pipeline uses for build jobs. That image already has a user:1000 with sudo (paswordless) permissions and a home directory. This was done to make manual use of the container more useful... there are applications in the image that prefer to run under a non-root user.
The pipeline using this image finds the existing user (great!) but then tries to give the user sudo permissions that it already has and this breaks the flow...
--<yaml pipeline code>--
container:
image: acr.url/foo/bar:v1
endpoint: <svc-connection>
--<pipeline run>--
...
/usr/bin/docker network create --label dc4b27 vsts_network_6b3e...
/usr/bin/docker inspect --format="{{index .Config.Labels \"com.azure.dev.pipelines.agent.handler.node.path\"}}" ***/foo/bar:v1
/usr/bin/docker create --name 9479... --label dc4b27 --network vsts_network_6b3ee... -v "/var/run/docker.sock":"/var/run/docker.sock" -v "/opt/azagent/_work/9":"/__w/9" -v "/opt/azagent/_work/_temp":"/__w/_temp" -v "/opt/azagent/_work/_tasks":"/__w/_tasks" -v "/opt/azagent/_work/_tool":"/__t" -v "/opt/azagent/externals":"/__a/externals":ro -v "/opt/azagent/_work/.taskkey":"/__w/.taskkey" ***/foo/bar:v1 "/__a/externals/node/bin/node" -e "setInterval(function(){}, 24 * 60 * 60 * 1000);"
9056...
/usr/bin/docker start 9056...
9056...
/usr/bin/docker ps --all --filter id=9056... --filter status=running --no-trunc --format "{{.ID}} {{.Status}}"
9056... Up Less than a second
/usr/bin/docker exec 9056... sh -c "command -v bash"
/bin/bash
whoami
devops
id -u devops
1000
Try to create a user with UID '1000' inside the container.
/usr/bin/docker exec 9056... bash -c "getent passwd 1000 | cut -d: -f1 "
/usr/bin/docker exec 9056... id -u viv
1000
Grant user 'viv' SUDO privilege and allow it run any command without authentication.
/usr/bin/docker exec 9056... groupadd azure_pipelines_sudo
groupadd: Permission denied.
groupadd: cannot lock /etc/group; try again later.
##[error]Docker exec fail with exit code 10
Finishing: Initialize containers
I am OK working with user:1000 in the container as the azure agent runs on the host VM under user:1000('devops') and so the id's match inside and outside of the container, getting around a shortcoming of the docker volume mount system.
The question is: Is there a pipeline yaml method or control parameter to tell the run not to try and setup sudo permissions on the discovered user account (uid:1000) in the container?
I am getting around this issue right now by adding options: --user 0 to the container: section in the yaml script but I would prefer not to do that...
Thx.
I want to build docker images and push into ECR for that I have written below buildspect.yml file and build my project using AWS build project service.
My buildspec.yml file is as below:
version: 0.2
phases:
install:
runtime-versions:
docker: 18
commands:
- nohup /usr/local/bin/dockerd --host=unix:///var/run/docker.sock --host=tcp://localhost:2375 --storage-driver=overlay2&
- timeout 15 sh -c "until docker info; do echo .; sleep 1; done"
pre_build:
commands:
- $(aws ecr get-login --no-include-email --region ${AWS_DEFAULT_REGION})
- REPOSITORY_URI_SERVER=<accountnumber>.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com/${IMAGE_NAME}
build:
commands:
- docker-compose build
post_build:
commands:
- docker-compose push
While I do build using AWS build project service then I will get an error like:
ERROR: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
errors pretty printing info
[Container] 2020/07/10 01:57:15 Command did not exit successfully timeout 15 sh -c "until docker info; do echo .; sleep 1; done" exit status 124
[Container] 2020/07/10 01:57:15 Phase complete: INSTALL State: FAILED
[Container] 2020/07/10 01:57:15 Phase context status code: COMMAND_EXECUTION_ERROR Message: Error while executing command: timeout 15 sh -c "until docker info; do echo .; sleep 1; done". Reason: exit status 124
I have tried to write buildspect.yml file as below
I have added an artifact tag in my file that is the place where my project build is a store.
post_build:
commands:
- docker-compose push
- printf '[{"name":"<name of container>","imageUri":"<path of your image"}]' > imagedefinitions.json
- cat imagedefinitions.json
artifacts:
files: imagedefinitions.json
I got success in it.
I am trying out OWASP/ZAP to see if it is something we can use for our project, but I cannot make it work I don't know what I am doing wrong and the documentation really does not help. What I am trying is to run a scan on my api running in a docker container locally on my windows machine so I run the command:
docker run -v $(pwd):/zap/wrk/:rw -t owasp/zap2docker-stable zap-baseline.py -t http://172.21.0.2:8080/swagger.json -g gen.conf -r testreport.html the ip 172.21.0.2 is the IPAddress of my api container even tried with localhost and 127.0.0.1
but it just hangs in the following log message:
_XSERVTransmkdir: ERROR: euid != 0,directory /tmp/.X11-unix will not be created.
Feb 14, 2019 1:43:31 PM java.util.prefs.FileSystemPreferences$1 run
INFO: Created user preferences directory.
Nothing happens and my zap docker container is in a unhealthy state, after some time it just crashes and ends up with a bunch of NullPointerExceptions. Is zap docker only working for linux, something specifically I need to do when running it on a windows machine? I don't get why this is not working even when I am following specifically the guideline in https://github.com/zaproxy/zaproxy/wiki/Docker
Edit 1
My latest try where I am trying to target my host ip address directly and the port that I am exposing my api to gives me the following error:
_XSERVTransmkdir: ERROR: euid != 0,directory /tmp/.X11-unix will not be created.
Feb 14, 2019 2:12:07 PM java.util.prefs.FileSystemPreferences$1 run
INFO: Created user preferences directory.
Total of 3 URLs
ERROR Permission denied
2019-02-14 14:12:57,116 I/O error(13): Permission denied
Traceback (most recent call last):
File "/zap/zap-baseline.py", line 347, in main
with open(base_dir + generate, 'w') as f:
IOError: [Errno 13] Permission denied: '/zap/wrk/gen.conf'
Found Java version 1.8.0_151
Available memory: 3928 MB
Setting jvm heap size: -Xmx982m
213 [main] INFO org.zaproxy.zap.DaemonBootstrap
When you run docker with: docker run -v $(pwd):/zap/wrk/:rw ...
you are mapping the /zap/wrk/ directory in the docker image to the current working directory (cwd) of the machine in which you are running docker.
I think the problem is that your current user doesn't have write access to the cwd.
Try below command, hope it resolves issue.
$docker run --user $(id -u):$(id -g) -v $(pwd):/zap/wrk/:rw --rm -t owasp/zap2docker-stable zap-baseline.py -t https://your_url -g gen.conf -r testreport.html
The key error here is:
IOError: [Errno 13] Permission denied: '/zap/wrk/gen.conf'
This means that the script cannot write to the gen.conf file that you have mounted on /zap/wrk
Do you have write access to the cwd when its not mounted?
The reason for that is, if you use -r parameter, zap will attempt to generate the file report.html at location /zap/wrk/. In order to make this work, we have to mount a directory to this location /zap/wrk.
But when you do so, it is important that the zap container is able to perform the write operations on the mounted directory.
So, below is the working solution using gitlab ci yml. I started with this approach of using image: owasp/zap2docker-stable however then had to go to the vanilla docker commands to execute it.
test_site:
stage: test
image: docker:latest
script:
# The folder zap-reports created locally will be mounted to owasp/zap2docker docker container,
# On execution it will generate the reports in this folder. Current user is passed so reports can be generated"
- mkdir zap-reports
- cd zap-reports
- docker pull owasp/zap2docker-stable:latest || echo
- docker run --name zap-container --rm -v $(pwd):/zap/wrk -u $(id -u ${USER}):$(id -g ${USER}) owasp/zap2docker-stable zap-baseline.py -t "https://example.com" -r report.html
artifacts:
when: always
paths:
- zap-reports
allow_failure: true
So the trick in the above code is
Mount local directory zap-reports to /zap/wrk as in $(pwd):/zap/wrk
Pass the current user and group on the host machine to the docker container so the process is using the same user / group. This allows writing of reports on the directory mounted from local host. This is done by -u $(id -u ${USER}):$(id -g ${USER})
Below is the working code with image: owasp/zap2docker-stable
test_site:
variables:
GIT_STRATEGY: none
stage: test
image:
name: owasp/zap2docker-stable:latest
before_script:
- mkdir -p /zap/wrk
script:
- zap-baseline.py -t "https://example.com" -g gen.conf -I -r testreport.html
- cp /zap/wrk/testreport.html testreport.html
artifacts:
when: always
paths:
- zap.out
- testreport.html
So i recently tried Docker and Gitlab Runner but it seems i cant get it to work.
This is the log i have:
Running with gitlab-runner 10.0.2 (a9a76a50)
on my-docker (c588e5e2)
Using Docker executor with image docker:git ...
Starting service docker:dind ...
Pulling docker image docker:dind ...
Using docker image docker:dind ID=sha256:b9145b364a203c0afc538ca615b3470e41729edfb7338017f5d4eeb5b13b2d90 for docker service...
Waiting for services to be up and running...
Using docker image sha256:7961fbf38d6f827265aed22fe41a1db889c54913283b678a8623efdda9573977 for predefined container...
Pulling docker image docker:git ...
Using docker image docker:git ID=sha256:5917639be9495ab183f357e8bafafea82449f0c4b12b745eef8bd23d474220ca for build container...
Running on runner-c588e5e2-project-1-concurrent-0 via gitlabServer...
Cloning repository...
Cloning into '<Project name>'...
Checking out ed0ce69e as master...
Skipping Git submodules setup
$ # Auto DevOps variables and functions # collapsed multi-line command
$ setup_docker
$ build
Building Heroku-based application using gliderlabs/herokuish docker image...
**docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?.**
See 'docker run --help'.
ERROR: Job failed: exit code 125
What could be the error? Docker itsself is running as it seems, but the docker inside does not seem to work.
This is my .toml file:
[[runners]]
name = my name
url = my url
token = my token
executor = "docker"
[runners.docker]
tls_verify = false
image = "docker:latest"
privileged = true
disable_cache = false
volumes = ["/cache"]
shm_size = 0
[runners.cache]
Thanks in advance for help!
Edit: Thats what "docker ps" gave as output:
ED STATUS PORTS NAMES
e66e844481b7 7961fbf38d6f "gitlab-runner-ser..." 2 sec onds ago Up Less than a second runner-73520410-proje ct-1-concurrent-0-docker-0-wait-for-service
4f659dba7bac b9145b364a20 "dockerd-entrypoin..." 2 sec onds ago Up 1 second 2375/tcp runner-73520410-proje ct-1-concurrent-0-docker-0
73776d4638b9 gitlab/gitlab-runner:latest "/usr/bin/dumb-ini..." 19 mi nutes ago Up 19 minutes gitlab-runner
Edit 2: My gitlab-ci.yaml
#ruby 2.2
rspec:ruby2.2:
image: ruby:2.2
script:
- bundle exec rspec spec
tags:
- ruby
except:
- tags
#ruby 2.1
rspec:ruby2.1:
image: ruby:2.1
script:
- bundle exec rspec spec
tags:
- ruby
except:
- tags
.go: &go_definition
before_script:
- apt-get update -qq && apt-get install -y ruby
- ruby -v
script:
- go version
- which go
- bin/compile
- support/go-test
- support/go-format check
go:1.8:
<<: *go_definition
image: golang:1.8
codeclimate:
before_script: []
image: docker:latest
variables:
DOCKER_DRIVER: overlay
services:
- docker:dind
script:
- docker pull codeclimate/codeclimate
- docker run --env CODECLIMATE_CODE="$PWD" --volume "$PWD":/code --volume /var/run/docker.sock:/var/run/docker.sock --volume /tmp/cc:/tmp/cc codeclimate/codeclimate analyze -f json > codeclimate.json
artifacts:
paths: [codeclimate.json]
This is my registration command, I think you are missing to pass the privileged during registration and also make sure the gitlab-runner user is part of the docker group:
gitlab-runner register \
--template-config /tmp/gitlab-config.toml \
--config /etc/gitlab-runner/config.toml \
--non-interactive \
--url "$gitlab_url" \
--registration-token "$runner_registration_token" \
--name "$runner_name" \
--tag-list "$runner_tags" \
--run-untagged="$runner_run_untagged" \
--locked="$runner_locked" \
--access-level="$runner_access" \
--maximum-timeout="$maximum_timeout" \
--executor "docker" \
--docker-privileged \
--docker-volumes "/cache" \
--docker-volumes "/certs/client" \
--docker-image "$runner_image"
sudo usermod -aG docker gitlab-runner
# concurrent global can't be setup in registration
# See: https://gitlab.com/gitlab-org/gitlab/-/issues/332497
sed -i "s/concurrent.*/concurrent = $concurrent/" /etc/gitlab-runner/config.toml
# prometheus port for GitLab Runner is 9252 as defined here https://github.com/prometheus/prometheus/wiki/Default-port-allocations
echo -e "listen_address = \":9252\"\n$(cat /etc/gitlab-runner/config.toml)" > /etc/gitlab-runner/config.toml
I try to configure continuous testing/integration with odoo and postgres docker container.
But I stuck on a problem, Gitlab CI can't do any operations on docker postgres.
My goal is to put a database template into a postgres container after run it and before testing.
I tried to use ssh executor after shell executor, but I always stuck on the same problem.
Notice that all commands here can be complete on the runner without problems, I test it.
I wrote this yml files:
variables:
# Configure postgres service (https://hub.docker.com/_/postgres/)
POSTGRES_DB: db
POSTGRES_USER: odoo
POSTGRES_PASSWORD: odoo
before_script:
# Pull container version
- docker pull postgres:9.5
- docker pull odoo:8.0
after_script:
# Remove all used container
- docker stop $(docker ps -a -q) && docker rm $(docker ps -aq)
stages:
- prepare
job1:
stage: prepare
# prepare postgres db
script:
#Launch postgres container
- docker run -d -e POSTGRES_USER=$POSTGRES_USER -e POSTGRES_PASSWORD=$POSTGRES_PASSWORD --name db postgres:9.5
# Copy and restore db template
- docker cp /home/myuser/odoov8_test.dump db:/home
- docker exec -i db su -s /bin/sh - postgres -c "createdb odoov8_test && pg_restore -d odoov8_test --no-owner --verbose /home/odoov8_test.dump"
# launch odoo with own addons folder (/own/path/to/addons:/mnt/extra-addons) testdatabase (-d) module to install without space with all dependances (-i) test unable (--test-enable) and stop after init (--stop-after-init)
- docker run -v $CI_PROJECT_DIR:/mnt/extra-addons -p 8069:8069 --name odoo --link db:db -t odoo:8.0 -- -d odoov8_test.dump -i crm,sale --test-enable --stop-after-init
I got this result:
[0KRunning with gitlab-ci-multi-runner 1.11.2 (0489844)
on Test docker odoo (7fafb15a)
[0;m[0KUsing Shell executor...
[0;mRunning on debian-8-clean...
[32;1mFetching changes...[0;m
HEAD est maintenant à 7d196ea Update .gitlab-ci.yml
Depuis https://myserver.com/APSARL/addons-ext
7d196ea..47591ac master -> origin/master
[32;1mChecking out 47591ac6 as master...[0;m
[32;1mSkipping Git submodules setup[0;m
[32;1m$ docker pull postgres:9.5[0;m
9.5: Pulling from library/postgres
Digest: sha256:751bebbc12716d7d9818678e91cbec8138e52dc4a084f0e81c58cd8b419930e5
Status: Image is up to date for postgres:9.5
[32;1m$ docker pull odoo:8.0[0;m
8.0: Pulling from library/odoo
Digest: sha256:9deda039e0df28aaf515001dd1606ab74a16ed25504447edc2912bca9019cd43
Status: Image is up to date for odoo:8.0
[32;1m$ docker run -d -e POSTGRES_USER=$POSTGRES_USER -e POSTGRES_PASSWORD=$POSTGRES_PASSWORD --name db postgres:9.5[0;m
60a0c75fd55e953e6a25a3cc0f13093ec2f1ee96bfb8384ac19d00f740dd1d4e
[32;1m$ docker cp /home/myuser/odoov8_test.dump db:/home
[0;m[32;1m$ docker exec -i db su -s /bin/sh - postgres -c "createdb odoov8_test && pg_restore -d odoov8_test --no-owner --verbose /home/odoov8_test.dump"
[0;m
createdb: could not connect to database template1: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
[32;1mRunning after script...[0;m
[32;1m$ docker stop $(docker ps -a -q) && docker rm $(docker ps -aq)[0;m
60a0c75fd55e
60a0c75fd55e
[31;1mERROR: Job failed: exit status 1
[0;m