Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed last month.
Improve this question
We are having AzureCR as our container registry and our Azure Devops build pipelines having docker image build and push tasks to create various application specific custom images over the dockerhubs base images.
We need to have all these custom images and the dockerhub- base public images scanned using the Jfrog Xray before the custom images pushed to the ACR and other deployment taks.
How the Jfrog xray tool can be integrated with Azure Pipeline yaml file to scan the newly built custom images just after the maven build & docker image build tasks and before the image push to ACR .
Is there any way to integrate Azure Devops and jfrog Xray together to scan these custom images as part of Azure Pipeline build just before the push to ACR ?
Tried pipeline
parameters:
imageName: ''
includeLatestTag: false
buildContext: '$(System.DefaultWorkingDirectory)/release/target/docker'
publishDocker: ''
steps:
- task: Docker#1
inputs:
azureSubscriptionEndpoint: 'mysub'
azureContainerRegistry: $(containerRegistry)
command: build
includeLatestTag: ${{ parameters.includeLatestTag }}
dockerFile: '${{ parameters.buildContext }}/Dockerfile'
useDefaultContext: false
buildContext: ${{ parameters.buildContext }}
imageName: ${{ parameters.imageName }}
arguments: $(buildArgs)
name: Build_Docker_Image
displayName: 'Build Docker image'
- task: JFrogDocker#1
inputs:
command: 'Scan'
xrayConnection: 'jfrog xray token'
watchesSource: 'none'
licenses: true
allowFailBuild: true
threads: '3'
skipLogin: false
- task: Docker#1
inputs:
azureSubscriptionEndpoint: 'mysub'
azureContainerRegistry: $(containerRegistry)
command: push
includeLatestTag: ${{ parameters.includeLatestTag }}
dockerFile: '${{ parameters.buildContext }}/Dockerfile'
useDefaultContext: false
buildContext: ${{ parameters.buildContext }}
imageName: ${{ parameters.imageName }}
name: Push_Docker_Image
displayName: 'Push Docker image'
I tried to add the below task in between Dicker image build and push tasks . But not getting any option scan them . Any guidance?
The new JFrog extension, JFrog Azure DevOps Extension, has the JFrog Docker task that allows scanning local docker images (as well as pulling and pushing them from/to Artifactory).
By adding the Xray scan task following the instructions here, we can have the build task wait for the Xray scan to complete. However, it is necessary for the build to publish the build information first to the Artifactory in order to have the Xray processing initiated.
So, my proposal here is to have the build promotion enabled against the target repository to push the images, when the build scan stage is completed.
Related
The Goal
I have a WPF application that is to be updated from a sFTP server. Azure Devops is being used as the CI/CD and pushed out to the sFTP server.
Question
I wish to build a .appinstaller from what I gather when creating a .appinstaller for msix you require to have a public facing URI. I do not wish to do so i want it to be stored on the sFTP server and then downloaded it from there then installed locally.
Any tips?
This is a partial yaml file used to build
- task: MsixPackaging#1
inputs:
outputPath: '$(Build.ArtifactStagingDirectory)/$(Build.BuildNumber).msix'
solution: 'app.sln'
clean: true
generateBundle: true
buildConfiguration: 'release'
msbuildLocation: 'C:\Download'
buildForX86: false
updateAppVersion: false
appPackageDistributionMode: 'SideloadOnly'
msbuildLocationMethod: 'version'
msbuildVersion: 'latest'
msbuildArchitecture: 'x64'
- task: AppInstallerFile#1
displayName: 'Create AppInstaller'
inputs:
package: '$(Build.ArtifactStagingDirectory)/$(Build.BuildNumber).msix'
outputPath: '$(Build.ArtifactStagingDirectory)/$(Build.BuildNumber).appinstaller'
uri: '\\PC\Download'
mainItemUri: '\\PC\Download'
showPromptWhenUpdating: true
updateBlocksActivation: true
Since you are using sftp server, then the port related to SSH on that server should be already turned on.
You can use SSH protocol to handle the file you want to upload/push.
You need first create a SSH service connection in DevOps project settings.
And then push the file based on the task CopyFilesOverSSH#0:
trigger:
- none
pool: VMAS #This is the agent pool on my side, you can set up your own agent pool.
# vmImage: ubuntu-latest
steps:
#Put your logic here.
- task: CopyFilesOverSSH#0
inputs:
sshEndpoint: 'SSH_To_Remote_VM'
contents: 'test.appinstaller'
readyTimeout: '20000'
It works fine on my side:
Also you can use the task SSH#0 to run scripts on remote machine.
In the settings I've enabled Github Pages:
I have a Github Action which builds and deploy the page to the branch gh-pages.
name: Continuous Deployment
on:
push:
branches:
- master
schedule:
- cron: '0 0 * * *'
jobs:
build-and-deploy:
name: Build and deploy to Github Pages
runs-on: ubuntu-latest
steps:
- name: Checkout repo
uses: actions/checkout#v3
- name: Use nodejs
uses: actions/setup-node#v3
with:
node-version: '16.x'
- name: Get yarn cache directory path
id: yarn-cache-dir-path
run: echo "::set-output name=dir::$(yarn cache dir)"
- name: Activate dependency cache
uses: actions/cache#v3
id: yarn-cache # use this to check for `cache-hit` (`steps.yarn-cache.outputs.cache-hit != 'true'`)
with:
path: ${{ steps.yarn-cache-dir-path.outputs.dir }}
key: ${{ runner.os }}-yarn-${{ hashFiles('**/yarn.lock') }}
restore-keys: |
${{ runner.os }}-yarn-
- name: Install dependencies
run: yarn install --frozen-lockfile
- name: Caching Gatsby
id: gatsby-cache-build
uses: actions/cache#v3
with:
path: |
public
.cache
key: ${{ runner.os }}-gatsby-build-cache-${{ github.run_id }}
restore-keys: |
${{ runner.os }}-gatsby-build-cache-
- name: Build website
run: yarn build:with-prefix
env:
PATH_PREFIX: '/xyz'
SITE_URL: 'https://xyz.github.io/xyz'
CI: true
- name: Deploy to GitHub Pages
uses: JamesIves/github-pages-deploy-action#v4.3.3
with:
branch: gh-pages
folder: public
clean: true
Now there is another Github Action which seem to deploy my page to Github Actions (using Jakyll):
Now I have two questions, which I couldn't answer by searching the internet:
Do I need this other action pages-build-deployment?
If not, how can I disable it?
If yes, for what it's needed? Am I doing the same work twice?
Do I need this other action pages-build-deployment?
This is provided by GitHub automatically.
As discussed in JamesIves/github-pages-deploy-action #1073:
I think there's a lot of scope for confusion for users of this action, when pages build and deployment will also be running on every push and pushing pages assets.
I think it's impossible to really say right now what that looks like until GitHub announces what these actions intend to do in the long run.
As the post suggests, this is a necessary step that occurs after the push gets made, this was already occurring behind the scenes it just wasn't made visible.
So you can ignore it (and there is no obvious way to disable it).
As for doing the same work twice, the same post adds:
What used to happen is, behind the scenes, we used the github-pages[bot] to pull that branch down and deploy the content to the github-pages environment we create for you automatically.
Now this step happens transparently with this new Actions workflow.
The end goal we’re working on is, if you’d prefer to deploy to the pages environment directly (without committing the content to a gh-pages branch), you’ll be able to do that in your own workflow. This will remove the need for the second workflow that we trigger when you commit to the pages branch.
Context: Dec. 2021 "GitHub Pages: using GitHub Actions for builds and deployments for public repositories".
The initial benefit of this change is enabling you to see your build logs and any errors that may occur which has been a long standing issue for Pages users.
However, in the future this will enable us to give you the ability to fully customize your pages build and deployment workflow to use any static site generator you want without having to push the build output to a special branch of the repository.
I have one action (a yaml file) for deploying a docker image to Google Cloud Run.
I would like to receive Slack or Email messages informing the build and push results.
How could the message action be triggered after build action is completed?
Is it possible to get the result of the build action?
There are 2 options of doing this:
Use a second job inside the same workflow.yml together with the needs keyword
Create a separate notify.yml workflow that uses the workflow_run event as a trigger
1. Same workflow, separate job with needs keyword
In your workflow.yml file you simply define two jobs like this (leveraging the needs: build configuration in the second job):
name: CI build and notify
on: [push]
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
- name: Deploy Docker image to Google Cloud Run
run: ...
notify:
needs: build
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
- name: Notify Slack and send eMail
run: ...
As the docs state, the second notify job will only start if the first build job succeeded:
Identifies any jobs that must complete successfully before this job
will run.
Here's a screenshot of how this approach can look like practically from my own project (I have a second publish-snapshot job instead of your notify job - but the concept stays the same):
There's also a way to always let the notify job run, even if the build job failed. You have to enhance the needs with a if: always() configuration then.
2. Separate workflow, using the workflow_run event as a trigger
Using the workflow_run event as a trigger we end up having 2 separate GitHub Actions workflow yaml files:
build.yml
name: CI build
on: [push]
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
- name: Deploy Docker image to Google Cloud Run
run: ...
notify.yml
name: CI notify
# Only trigger, when the build workflow succeeded
on:
workflow_run:
workflows: ["CI build"]
types:
- completed
jobs:
notify:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
- name: Notify Slack and send eMail
run: ...
A crucial point here is that the name: CI build definition of the first yaml file must exactly match the workflow_run: workflows: ["CI build"] definition in the second yaml file. Another point is that this approach needs to be done on the default branch (which is mostly main or master) as the docs state:
Note: This event will only trigger a workflow run if the workflow file
is on the default branch.
Here's also a full example project using the 1st option if you're interested.
First, you are mixing terms here. According to GitHub Actions documentation a single YAML file is called a workflow (not an action) and consists of jobs. Jobs contain a sequence of steps (including actions) that are executed one after another. A particular workflow execution is called a run. Having that in mind lets go the questions.
How could the message workflow be triggered after build workflow is completed?
You can use GitHub API to trigger a webhook event called repository_dispatch (only for the base branch) or workflow_dispatch. This can be easily done using a dedicated Repository Dispach action in your build workflow.
Is it possible to get the result of the build workflow?
Yes, the result of the workflow run can be obtained using given GitHub API
But if you only want to send the build result notification of the currently executed workflow you don't need to create a separate workflow and trigger it from the parent. You can use dedicated Slack actions or e-mail actions.
you can try in step 2 the following directive:
needs: step-1-job-name
just after job name
The best method i've found for reusable workflows in the same repository is https://docs.github.com/en/actions/using-workflows/reusing-workflows
In the workflow you want to re-use (let's call it wf1):
# wf1.yml
on:
workflow_call:
In the workflow you want to call wf1 from (let's call this wf2)
# wf2.yml
jobs:
call-wf1:
uses: ./.github/workflows/wf1.yml
secrets: inherit
In order to get data from one workflow to another, check out https://docs.github.com/en/actions/using-workflows/reusing-workflows#using-outputs-from-a-reusable-workflow.
First, set up your reusable workflow to output the data:
name: Reusable workflow
on:
workflow_call:
# Map the workflow outputs to job outputs
outputs:
firstword:
description: "The first output string"
value: ${{ jobs.example_job.outputs.output1 }}
secondword:
description: "The second output string"
value: ${{ jobs.example_job.outputs.output2 }}
jobs:
example_job:
name: Generate output
runs-on: ubuntu-latest
# Map the job outputs to step outputs
outputs:
output1: ${{ steps.step1.outputs.firstword }}
output2: ${{ steps.step2.outputs.secondword }}
steps:
- id: step1
run: echo "::set-output name=firstword::hello"
- id: step2
run: echo "::set-output name=secondword::world"
Now use it in the calling workflow:
name: Call a reusable workflow and use its outputs
on:
workflow_dispatch:
jobs:
job1:
# local repo
# uses: .github/workflows/called-workflow.yml#v1
# other repo
uses: octo-org/example-repo/.github/workflows/called-workflow.yml#v1
job2:
runs-on: ubuntu-latest
needs: job1
steps:
- run: echo ${{ needs.job1.outputs.firstword }} ${{ needs.job1.outputs.secondword }}
How could the message action be triggered after build action is completed?
This should now (August 2020) be possible with "GitHub Actions improvements for fork and pull request workflows"
Another frequently-requested feature for Actions is a way to trigger one workflow based on the completion of another workflow.
For example, you may want to take the results of a CI workflow and run some further analysis.
The new workflow_run event enables you to trigger a new workflow when one or more workflows are requested or completed.
Runs triggered by the workflow_run event always use the default branch for the repository, and have access to a read/write token as well as secrets.
As an example, as a maintainer you could set up a workflow that takes the artifacts generated by the pull request workflow, do some analysis, and post comments back to the pull request.
This event is also available as a web hook.
I have a GitHub repository with a .NET Core 3.0 website solution in it. In Azure DevOps, I went through the wizard to create a new pipeline linked to that repository using the ASP.NET Core template on the Configure step of the wizard. This is what my YAML looks like:
# ASP.NET Core
# Build and test ASP.NET Core projects targeting .NET Core.
# Add steps that run tests, create a NuGet package, deploy, and more:
# https://learn.microsoft.com/azure/devops/pipelines/languages/dotnet-core
trigger:
- develop
pool:
vmImage: 'ubuntu-latest'
variables:
buildConfiguration: 'Release'
steps:
- script: dotnet build --configuration $(buildConfiguration)
displayName: 'dotnet build $(buildConfiguration)'
When I try to manually run the pipeline to test it, this is the output I get everytime:
##[warning]There was a failure in sending the provision message: Unexpected response code from remote provider NotFound
,##[warning]There was a failure in sending the provision message: Unexpected response code from remote provider NotFound
,##[warning]There was a failure in sending the provision message: Unexpected response code from remote provider NotFound
,##[warning]There was a failure in sending the provision message: Unexpected response code from remote provider NotFound
,##[warning]There was a failure in sending the provision message: Unexpected response code from remote provider NotFound
,##[error]Provisioning request delayed or failed to send 5 time(s). This is over the limit of 3 time(s).
Pool: Azure Pipelines
Image: ubuntu-latest
Started: Yesterday at 10:04 PM
Duration: 10h 54m 5s
Job preparation parameters
ContinueOnError: False
TimeoutInMinutes: 60
CancelTimeoutInMinutes: 5
Expand:
MaxConcurrency: 0
########## System Pipeline Decorator(s) ##########
Begin evaluating template 'system-pre-steps.yml'
Evaluating: eq('true', variables['system.debugContext'])
Expanded: eq('true', Null)
Result: False
Evaluating: resources['repositories']['self']
Expanded: Object
Result: True
Evaluating: not(containsValue(job['steps']['*']['task']['id'], '6d15af64-176c-496d-b583-fd2ae21d4df4'))
Expanded: not(containsValue(Object, '6d15af64-176c-496d-b583-fd2ae21d4df4'))
Result: True
Evaluating: resources['repositories']['self']['checkoutOptions']
Result: Object
Finished evaluating template 'system-pre-steps.yml'
********************************************************************************
Template and static variable resolution complete. Final runtime YAML document:
steps:
- task: 6d15af64-176c-496d-b583-fd2ae21d4df4#1
inputs:
repository: self
I thought maybe ubuntu-latest was maybe no longer a valid vmImage, so I tried changing it to ubuntu-18.04 and got the same result. The Micosoft-hosted agents documentation says either should be valid.
Do I have something wrong with my yaml file? I have setup pipelines before with the old no-yaml interface with no issues, so I am a little confused.
i think nowadays it should looks like this:
trigger:
- develop
jobs:
- job: buildjob
variables:
buildConfiguration: 'Release'
pool:
vmImage: 'ubuntu-latest'
steps:
- script: dotnet build --configuration $(buildConfiguration)
displayName: 'dotnet build $(buildConfiguration)'
although this says you can omit jobs if you only have a single job, but I dont see anything wrong with your yaml other than the fact you use steps directly (which, again, should be fine)
Looks like nothing wrong with your yaml file and format.
Since you are using a GitHub repository with a .NET Core 3.0 website. Please pay attention when you create pipeline, make sure you have selected GitHub not Azure Repos Git.
Also as you have mentioned I have setup pipelines before with the old no-yaml interface with no issues, you could setup your pipelines with classic editor first.
There is also a view yaml option.
You could follow that format and content to create a yaml template, which may do the trick.
I was looking through my account and noticed my Agent Pool settings looked a little suspect on the project. It showed I had 11 available agents that were online even though I was using the free plan on a private repository so it should only have one.
I ended up deleting my Azure DevOps organization and creating a new one. Now the YAML configuration I initially posted works fine.
I'm pretty impressed by the power and simplicity of Concourse. Since my pipelines keep growing I decided to move the tasks to separate files. One of the tasks use a custom Docker image from our own private registry. So, in that task file I have:
image_resource:
type: docker-image
source:
repository: docker.mycomp.com:443/app-builder
tag: latest
username: {{dckr-user}}
password: {{dckr-pass}}
When I do a set-pipeline, I pass the --load-from-vars argument to load credentials etc from a seperate file.
Now here's my problem: I notice that the vars in my pipeline files are replaced with the actual correct values, but once the task runs, the afore mentioned {{dckr-user}} and {{dckr-pass}} are not replaced.
How do I achieve this?
In addition to what was provided in this answer
If specifically you are looking to use private images in a task, you can do the following in your pipeline.yml:
resources:
- name: some-private-image
type: docker
params:
repository: ...
username: {{my-username}}
password: {{my-password}}
jobs:
- name: foo
plan:
- get: some-private-image
- task: some-task
image: some-private-image
Because this is your pipeline, you can use --load-vars-from, which will first get your image as a resource and then use it for the subsequent task.
You can also see this article on pre-fetching ruby gems in test containers on Concourse
The only downside to this is you cannot use this technique when running a fly execute.
As of concourse v3.3.0, you can set up Credential Management in order to use variables from one of the supported credential managers which are currently Vault, Credhub, Amazon SSM, and Amazon Secrets Manager. So you don't have to separate your task files partially in the pipeline.yml anymore. The values you set in the Vault will be also accessible from the task.yml files.
And since v3.2.0 {{foo}} is deprecated in favor of ((foo)).
Using the Credential Manager you can parameterize:
source under resources in a pipeline
source under resource_types in a pipeline
webhook_token under resources in a pipeline
image_resource.source under image_resource in a task config
params in a pipeline
params in a task config
For setting up vault with concourse you can refer to:
https://concourse-ci.org/creds.html
You can always define tasks in a pipeline.yml...
For example:
jobs:
- name: dotpersecond
plan:
- task: dotpersecond
config:
image_resource:
type: docker-image
source:
repository: docker.mycomp.com:443/app-builder
tag: latest
username: {{dckr-user}}
password: {{dckr-pass}}
run:
path: sh
args:
- "-c"
- |
for i in `seq 1000`; do echo hi; sleep 2; done