I am using Concourse-CI in combination with a private Docker registry and everything works fine. However, I want to run a task as an image I provide via the registry. To clarify: I don't want to run the image within the task, the task source should be my image. Unfortunately I wasn't able to find an example on here or on the Concourse-CI docs.
My resource:
resources:
- name: my-image
type: registry-image
source:
repository: ((registry-url))/my-image
username: ...
password: ...
ca_certs:
- ((registry-cert))
So, if I'm correct, the task/config/source cannot take a resource but an anonymous-resource where I would provide a docker.io link.
I am very appreciative for some help. :)
Edit: OK, so my first mistake was to only look at the Task schema, I can configure an image (https://concourse-ci.org/jobs.html#schema.step.task-step.image) but when I do:
- task: test
image: my-image
config:
platform: linux
inputs:
run:
...
I get this error: find or create container on worker 4c38517c9713: no image plugin configured.
Ok,
so the answer was to make the image privileged for some reason...
Related
I'm having some issues when trying to use Hashicorp vault template (kubernetes with Google Kubernetes Engine) with to.be.continuous.
Actually when I use it with Google Docker Kaniko layer I got an error message: ... wget: bad address 'vault-secrets-provider'.
It seems that Kaniko doesn't recognize the vault-secrets-provider layer. Would you please help me with this? Or perhaps, where I can ask for some help?
This is a summary of .gitlab-ci.yml
# Kubernetes template
- project: 'to-be-continuous/kubernetes'
ref: '2.0.4'
file: '/templates/gitlab-ci-k8s.yml'
- project: "to-be-continuous/kubernetes"
ref: "2.0.4"
file: "templates/gitlab-ci-k8s-vault.yml"
...
K8S_DEFAULT_KUBE_CONFIG: "#url#http://vault-secrets-provider/api/secrets/noprod?field=kube_config"
VAULT_BASE_URL: "http://myvault.myserver.com/v1"
Error Message:
[ERROR] Failed getting secret K8S_DEFAULT_KUBE_CONFIG:
... wget: bad address 'vault-secrets-provider'
I tried many times directly without Vault layer and Kaniko works ok, I mean without Vault secrets.
How I can accomplish this? I tried modifying the kaniko template but without success.
I will appreciate any help with this.
To fix your issue, first upgrade the docker template to its latest version (2.3.0 at the time this response was written).
Then depending on your case you have 2 options:
Docker needs to handle some of your secrets managed by Vault: then you shall also activate the Vault variant for Docker,
Docker doesn't needs to handle any secret managed by Vault: don't use the Vault variant for Docker, you'll have a warning message from Docker not being able to decode the secret (basically the same as the one you had, but not failing the build),
You shall simply use it in your .gitlab-ci.yml file:
include:
# Docker template
- project: 'to-be-continuous/docker'
ref: '2.3.0'
file: '/templates/gitlab-ci-docker.yml'
# Vault variant for Docker (depending on your above case)
- project: 'to-be-continuous/docker'
ref: '2.3.0'
file: '/templates/gitlab-ci-docker-vault.yml'
# Kubernetes template
- project: 'to-be-continuous/kubernetes'
ref: '2.0.4'
file: '/templates/gitlab-ci-k8s.yml'
- project: "to-be-continuous/kubernetes"
ref: "2.0.4"
file: "/templates/gitlab-ci-k8s-vault.yml"
K8S_DEFAULT_KUBE_CONFIG: "#url#http://vault-secrets-provider/api/secrets/noprod?field=kube_config"
VAULT_BASE_URL: "http://myvault.myserver.com/v1"
I am learning to use Azure Pipelines for CI/CD. I read the official document and found the Docker Compose task has an action called Lock services. I have no idea what this action actually do and what it means by locking the images.
Can anyone explain it to me, or provide me some examples on when and how to use it?
We have public the source code of this task, so you can check this page to analyze what the exactly action this command do.
For image, there has 2 different identifies: tag and digest. Now, let's assume one scenario:
Most of time, a tagged image in Container Registry is mutable, so with appropriate permissions you or anyone can update/push/delete an image with the same tag to that registry. However, when you deploy a image to production env, you could not sure whether the image with one specific tag does not been overwritten and it is the one you want to deployed.
At this time, digest would be a best choice for you.
Because digest is a SHA256 calculated from the image and identifies it uniquely. Once there has any changes to your image, the corresponding SHA256 value will be changes also.
Explanation of this action:
Check this code line(defined here). It's work logic is read out the image(s) used in the docker-compose.yml file, pull image(s) and generate a digest for them. Next a new docker-compose.yml file is automatically generated, which the image will be specified with digest in this new docker-compose.yml file.
Sample:
The task definition i used:
- task: DockerCompose#0
displayName: 'Lock services'
inputs:
containerregistrytype: 'Container Registry'
dockerRegistryEndpoint: {service connection name}
dockerComposeFile: 'Docker/docker-compose.yml'
action: 'Lock services'
removeBuildOptions: true
The docker-compose.yml:
version: '3'
services:
web:
image: xxxx/testwebapp
ports:
- "1983:80"
newsfeed:
image: xxx/merlin
redis:
image: redis
See the build log of this task:
And the contents of new docker-compose.yml which generated.
(List them by using cat xxx command):
Now, when you deploy the images to production, just use the new docker-compose.yml file the task generated automatically. This can guarantee the deployed image is the version you built at the beginning, even if someone overwrites this image later.
I am trying to create a GCS bucket using Deployment Manager using the following resource config:
resources:
- type: storage.v1.bucket
name: upload-bucket
properties:
project: <project-id>
name: <unique-bucket-name>
However, I get the following error:
- code: RESOURCE_ERROR
location: /deployments/the-bucket/resources/upload-bucket
message: '{"ResourceType":"storage.v1.bucket","ResourceErrorCode":"403","ResourceErrorMessage":{"code":403,"errors":[{"domain":"global","message":"205531008256#cloudservices.gserviceaccount.com
does not have storage.buckets.get access to upload-bucket.","reason":"forbidden"}],"message":"205531008256#cloudservices.gserviceaccount.com
does not have storage.buckets.get access to upload-bucket.","statusMessage":"Forbidden","requestPath":"https://www.googleapis.com/storage/v1/b/upload-bucket","httpMethod":"GET","suggestion":"Consider
granting permissions to 205531008256#cloudservices.gserviceaccount.com"}}'
The role of 205531008256#cloudservices.gserviceaccount.com is Project Editor by default (which surely has enough permissions?), however I've also tried adding Storage Admin and Project Owner - neither seems to help.
My 2 questions are:
Why it is trying to use this service account?
How can I get Deployment Manager to be able to create a bucket?
Thanks
I ran into the exact same problem. Allow me to restate Andres S's answer more clearly.
When you wrote
resources:
- type: storage.v1.bucket
name: upload-bucket
properties:
project: <project-id>
name: <unique-bucket-name>
you probably intended create a bucket called <unique-bucket-name> and figured that upload-bucket would just be a name to refer to this bucket in the Deployment Manager. What GCP actually did was attempt to use upload-bucket as the actual bucket name. As far as I can tell, <unique-bucket-name> is never used. This caused a problem, since someone else already owns the bucket upload-bucket.
Try this code, I think you are specifying the name twice.
resources:
- type: storage.v1.bucket
name: <unique-bucket-name>
properties:
project: <project-id>
I recently run into similar issue, where Deployment Manager failed to create the bucket.
I have verified that:
the permissions are not an issue as the same deployment contained other bucket that was created.
the bucket name is not an issue as I was able to create the bucket manually.
After some googling I found there is other way to create the bucket. Instead of using type: storage.v1.bucket you can also use type: gcp-types/storage-v1:buckets.
So my final solution was to create the bucket like this:
- name: images-bucket
type: gcp-types/storage-v1:buckets
properties:
name: images-my-project-name
location: "eu"
When i developing pipeline i can't understand the difference between "Resource Types" and "Resource".
According to documentation Resource type is there only to provide the type of the resource and check for the tags. Like in example bellow:
---
resource_types:
- name: rss
type: docker-image
source:
repository: suhlig/concourse-rss-resource
tag: latest
resources:
- name: booklit-releases
type: rss
source:
url: http://www.qwantz.com/rssfeed.php
jobs:
- name: announce
plan:
- get: booklit-releases
trigger: true
Why do we need both of them? isn't it enough just to use resources?
I'm also new to this project. Please correct me if I am wrong.
I think in the term of the container:
A resource type is an image and we need to config the repository and tag in its source so that the concourse can locate/download it.
A resource is a container which is an instance of that image and can be used in the jobs when the pipeline is running. Its source that we configure is the common parameters which will be passed on the stdin to the check, in and out scripts when the resource is configured in a get or put step.
I think it's a little similar to the comparison between the class and object.
I'm pretty impressed by the power and simplicity of Concourse. Since my pipelines keep growing I decided to move the tasks to separate files. One of the tasks use a custom Docker image from our own private registry. So, in that task file I have:
image_resource:
type: docker-image
source:
repository: docker.mycomp.com:443/app-builder
tag: latest
username: {{dckr-user}}
password: {{dckr-pass}}
When I do a set-pipeline, I pass the --load-from-vars argument to load credentials etc from a seperate file.
Now here's my problem: I notice that the vars in my pipeline files are replaced with the actual correct values, but once the task runs, the afore mentioned {{dckr-user}} and {{dckr-pass}} are not replaced.
How do I achieve this?
In addition to what was provided in this answer
If specifically you are looking to use private images in a task, you can do the following in your pipeline.yml:
resources:
- name: some-private-image
type: docker
params:
repository: ...
username: {{my-username}}
password: {{my-password}}
jobs:
- name: foo
plan:
- get: some-private-image
- task: some-task
image: some-private-image
Because this is your pipeline, you can use --load-vars-from, which will first get your image as a resource and then use it for the subsequent task.
You can also see this article on pre-fetching ruby gems in test containers on Concourse
The only downside to this is you cannot use this technique when running a fly execute.
As of concourse v3.3.0, you can set up Credential Management in order to use variables from one of the supported credential managers which are currently Vault, Credhub, Amazon SSM, and Amazon Secrets Manager. So you don't have to separate your task files partially in the pipeline.yml anymore. The values you set in the Vault will be also accessible from the task.yml files.
And since v3.2.0 {{foo}} is deprecated in favor of ((foo)).
Using the Credential Manager you can parameterize:
source under resources in a pipeline
source under resource_types in a pipeline
webhook_token under resources in a pipeline
image_resource.source under image_resource in a task config
params in a pipeline
params in a task config
For setting up vault with concourse you can refer to:
https://concourse-ci.org/creds.html
You can always define tasks in a pipeline.yml...
For example:
jobs:
- name: dotpersecond
plan:
- task: dotpersecond
config:
image_resource:
type: docker-image
source:
repository: docker.mycomp.com:443/app-builder
tag: latest
username: {{dckr-user}}
password: {{dckr-pass}}
run:
path: sh
args:
- "-c"
- |
for i in `seq 1000`; do echo hi; sleep 2; done