I have a repository that build and deploys two different components - a frontend and a backend. Each of these have a specific set of steps that need to be executed for the CICD. Is there a way to run a selective set of steps based on which component has actually changed. For e.g. let us say all my frontend is under frontend/ and all my backend is under backend/. Is there a way to run a selective set of steps when there are changes only in the frontend ?
The closest approach would be to adopt branch naming conventions that separate frontend and backend test builds.
For example, you could manage all frontend work with a frontend- prefix and all backend work with a backend- prefix. The codeship-steps.yml would then be implemented as:
- name: Frontend tests
service: your-app
type: serial
tag: ^frontend-
steps:
- service: your-app
command: ./run-frontend-test.sh
- [other step commands...]
- name: Backend tests
service: your-app
type: serial
tag: ^backend-
steps:
- service: your-app
command: ./run-backend-test.sh
- [other step commands...]
See here for more.
Related
I have one repo it has 3 yaml files for frontend, backend & admin frontend. Developer push on backend works on every time.
My 3 yamls running but frontend & admin frontend folder code have no changes in that time i need only backend yaml only run
remaining two yaml simply do nothing,
For my scenario if user pushes to backend code only backend ci only i need to run
How to configure that?
You can achieve what you want by using the paths subtype in each of your workflow triggers.
I recommend to check the Github official documentation for more details.
In your case, supposing that you have 3 folders following the structure below:
repository
|__ backend
|__ frontend
|__ admin_fronted
For each workflow, you could user the following implementation:
name: Backend
on:
push:
paths:
- 'backend/**'
jobs:
[ ... ]
name: Frontend
on:
push:
paths:
- 'frontend/**'
jobs:
[ ... ]
name: Admin Frontend
on:
push:
paths:
- 'admin_frontend/**'
jobs:
[ ... ]
That way, each push will trigger a workflow only if at least a file in the specific path has been updated.
Note that there is also a paths-ignore subtype when you can use the opposite behavior if needed.
I am using Concourse-CI in combination with a private Docker registry and everything works fine. However, I want to run a task as an image I provide via the registry. To clarify: I don't want to run the image within the task, the task source should be my image. Unfortunately I wasn't able to find an example on here or on the Concourse-CI docs.
My resource:
resources:
- name: my-image
type: registry-image
source:
repository: ((registry-url))/my-image
username: ...
password: ...
ca_certs:
- ((registry-cert))
So, if I'm correct, the task/config/source cannot take a resource but an anonymous-resource where I would provide a docker.io link.
I am very appreciative for some help. :)
Edit: OK, so my first mistake was to only look at the Task schema, I can configure an image (https://concourse-ci.org/jobs.html#schema.step.task-step.image) but when I do:
- task: test
image: my-image
config:
platform: linux
inputs:
run:
...
I get this error: find or create container on worker 4c38517c9713: no image plugin configured.
Ok,
so the answer was to make the image privileged for some reason...
I found out about a really nice GitHub Actions Feature called Environments. Using the appropriate syntax a Environment could also be created inside a GitHub Action workflow.yml like this:
environment:
name: test_environment
url: https://your-apps-url-here.com
As the docs state thats a valid way to create GitHub Action Environments:
Running a workflow that references an environment that does not exist
will create an environment with the referenced name.
But inside my current GitHub Action workflow is there a way I dynamically set the url based on a deployment step output? I have a dynamic URL resulting from the deployment process to AWS which I can't define up-front.
The job workflow docs tell us that there's also a way of using expressions inside the url field:
environment:
name: test_environment
url: ${{ steps.step_name.outputs.url_output }}
Now imagine a ci.yml workflow file that uses AWS CLI to deploy a static website to S3, where we used a tool like Pulumi to dynamically create a S3 Bucket inside our AWS account. We can read the dynamically created S3 url using the following command pulumi stack output bucketName. The deploy step inside the ci.yml could then look like this:
- name: Deploy Nuxt.js generated static site to S3 Bucket via AWS CLI
id: aws-sync
run: |
aws s3 sync ../dist/ s3://$(pulumi stack output bucketName) --acl public-read
echo "::set-output name=s3_url::http://$(pulumi stack output bucketUrl)"
working-directory: ./deployment
There are 2 crucial points here: First we should use id inside the deployment step to define a step name we could easily access via step_name inside our environment:url. Second we need to define a step output using echo "::set-output name=s3_url::http://$(pulumi stack output bucketUrl)". In this example I create a variable s3_url. You could replace the pulumi stack output bucketUrl with any other command you'd like or tool you use, which responds with your dynamic environment url.
Be also sure to add a http:// or https:// in order to prevent an error message like this:
Environment URL 'microservice-ui-nuxt-js-hosting-bucket-bc75fce.s3-website.eu-central-1.amazonaws.com' is not a valid http(s) URL, so it will not be shown as a link in the workflow graph.
Now the environment definition at the top of our ci.yml can access the s3_url output variable from our deployment step like this:
jobs:
ci:
runs-on: ubuntu-latest
environment:
name: microservice-ui-nuxt-js-deployment
url: ${{ steps.aws-sync.outputs.s3_url }}
steps:
- name: Checkout
...
Using steps.aws-sync we reference the deployment step directly, since we defined it with the id. The appended .outputs.s3_url then directly references the variable containing our S3 url. If you defined everything correctly the GitHub Actions UI will render the environment URL directly below the finished job:
Here's also a fully working workflow embedded inside a example project.
I built out my dev environment manually, I wanted to focus on logic and skip the learning curve on serverless, but before deploying to prod I want to standardize and parameterize my stack.
setting up my dynamodb tables has been straight forward, but I'm running into snags with deploying a new api-gateway.
I've been using aws codebuild to package layers for lambda functions and an s3 bucket to store my lambda code.
Let's take my dev-rest-auth api (custom authentication) as an example.
I have several resources for login/out, passwords and registration; most are protected by a custom authorizer (login/logout aren't) and all have cors policies. I'm using a custom domain account-api.dev.example.com I use several dynamodb tables for housing user data (let's avoid security discussions please, I'm not storing raw passwords and am encrypting using leading industry standards) and temporary codes (password reset & account verification).
to test serverless implementation I'd like to build a yaml file that recreates my existing infrastructure - so the first question is -- is that possible? Can I parameterize the deployment of an API gateway, with custom authorizer, custom domain, and several lambdas?
Next question is how?
Organizationally I'm breaking out my yml files by resource:
I have several dynamodb yml files that look like this:
Resources:
UserTable:
Type: AWS::DynamoDB::Table
DeletionPolicy: Retain
Properties:
TableName: ${self:custom.resource-prefix}-UserTable-${self:custom.stage}
AttributeDefinitions:
- AttributeName: email
AttributeType: S
KeySchema:
- AttributeName: email
KeyType: HASH
# Set the capacity to auto-scale
BillingMode: PAY_PER_REQUEST
This was a much earlier attempt (several months ago, from googling, but I don't remember where I found it or what it does) of standing up an API gateway:
Resources:
SharedGW:
Type: AWS::ApiGateway::RestApi
Properties:
Name: SharedGW
Outputs:
apiGatewayRestApiId:
Value:
Ref: SharedGW
Export:
Name: SharedGW-restApiId
apiGatewayRestApiRootResourceId:
Value:
Fn::GetAtt:
- SharedGW
- RootResourceId
Export:
Name: SharedGW-rootResourceId
I pull everything together in a serverless.yml file that references the resource files like this:
...
resources:
# S3 Bucket
- ${file(resources/s3/s3-static-host.yml)}
- ${file(resources/s3/s3-CodeBuildResults.yml)}
# DynamoDB
- ${file(resources/dynamodb/dynamodb-mealtable.yml)}
- ${file(resources/dynamodb/dynamodb-ziptable.yml)}
- ${file(resources/dynamodb/dynamodb-usertable.yml)}
- ${file(resources/dynamodb/dynamodb-passwordresettable.yml)}
- ${file(resources/dynamodb/dynamodb-accountregistrationtable.yml)}
- ${file(resources/dynamodb/dynamodb-restaurant_table.yml)}
# DNS Records (Route 53)
# TODO: Determine why DNS hangs
# - ${file(resources/route_53/dev_dns.yml)}
# Gateways
- ${file(resources/api_gateway/local_rest_auth.yml)}
# - ${file(resources/api_gateway/rest_auth.yml)}
...
I've seen several examples of connecting a lambda to a gateway, but it's not clear where the gateway is being created), it's also not clear how the lambda is being created/if I'd be able to reference layers/function code in s3.
I've seen some tutorials for doing this with aws amplify via the cli, but my dream-state would be that I could effectively create a new aws account, deploy this serverless and have my site up and running automatically - with just a little route 53 work to point to a new domain.
I'm pretty impressed by the power and simplicity of Concourse. Since my pipelines keep growing I decided to move the tasks to separate files. One of the tasks use a custom Docker image from our own private registry. So, in that task file I have:
image_resource:
type: docker-image
source:
repository: docker.mycomp.com:443/app-builder
tag: latest
username: {{dckr-user}}
password: {{dckr-pass}}
When I do a set-pipeline, I pass the --load-from-vars argument to load credentials etc from a seperate file.
Now here's my problem: I notice that the vars in my pipeline files are replaced with the actual correct values, but once the task runs, the afore mentioned {{dckr-user}} and {{dckr-pass}} are not replaced.
How do I achieve this?
In addition to what was provided in this answer
If specifically you are looking to use private images in a task, you can do the following in your pipeline.yml:
resources:
- name: some-private-image
type: docker
params:
repository: ...
username: {{my-username}}
password: {{my-password}}
jobs:
- name: foo
plan:
- get: some-private-image
- task: some-task
image: some-private-image
Because this is your pipeline, you can use --load-vars-from, which will first get your image as a resource and then use it for the subsequent task.
You can also see this article on pre-fetching ruby gems in test containers on Concourse
The only downside to this is you cannot use this technique when running a fly execute.
As of concourse v3.3.0, you can set up Credential Management in order to use variables from one of the supported credential managers which are currently Vault, Credhub, Amazon SSM, and Amazon Secrets Manager. So you don't have to separate your task files partially in the pipeline.yml anymore. The values you set in the Vault will be also accessible from the task.yml files.
And since v3.2.0 {{foo}} is deprecated in favor of ((foo)).
Using the Credential Manager you can parameterize:
source under resources in a pipeline
source under resource_types in a pipeline
webhook_token under resources in a pipeline
image_resource.source under image_resource in a task config
params in a pipeline
params in a task config
For setting up vault with concourse you can refer to:
https://concourse-ci.org/creds.html
You can always define tasks in a pipeline.yml...
For example:
jobs:
- name: dotpersecond
plan:
- task: dotpersecond
config:
image_resource:
type: docker-image
source:
repository: docker.mycomp.com:443/app-builder
tag: latest
username: {{dckr-user}}
password: {{dckr-pass}}
run:
path: sh
args:
- "-c"
- |
for i in `seq 1000`; do echo hi; sleep 2; done