If I wanted to run a scheduled task once a month for checking for outdated dependencies but I already have a CI pipeline how can I do that? For example I have a pipeline that runs though code sniffs -> checkmarx + twistlock -> deploy to dev -> stage and whatnot. This triggers on master. I want to also include the ability to have a scheduled task of dependabot to occur once every month. How can I mix this scheduled task into an established CI pipeline? This is all contained within Azure Devops as well.
I only want to run the single task of dependabot once a month. I don't want to run the entire pipeline once a month
I suggest creating a second - entirely separate - pipeline to run dependabot once a month.
That way, you can have the appropriate triggers for the CI pipeline, and the appropriate schedule for the dependabot pipeline, with exactly the right tasks in each one and no duplication.
You can run a pipeline using both the trigger and schedules.
For example, to run a stage on the 1st day of every month at 08:00 UTC, you can use:
trigger:
- master #This is the trigger for other stages. It is not needed for the scheduled stage.
schedules:
- cron: '0 8 1 * *'
displayName: 'Deploy every 1st day of every month at 08:00Z'
branches:
include:
- master
always: true
Then, to ensure that a specific stage runs as part of the scheduled run, use the condition expression, for example:
- stage: 'Test'
displayName: 'Deploy to the test environment'
dependsOn: Dev
condition: eq(variables['Build.Reason'], 'Schedule')
Refer to this MSDocs article for more on the syntax of schedules: https://learn.microsoft.com/en-us/azure/devops/pipelines/process/scheduled-triggers?view=azure-devops&tabs=yaml#scheduled-triggers
Related
With the classic Azure DevOps release pipeline our release flow was very easy to setup.
We had a build pipeline running many times during the day. On success it deployed to our development environment. Every night the latest successful deployment to dev was released to our test environment (running automated tests for hours), before it deployed to UAT. But often we also need to deploy to test during the day, if we have a new change which needs to go directly into test or UAT. The classic pipelines allowed us to skip a stage, or deploy if the previous was only partly successful.
1) Development - automatic
2) Test - nightly or manually
3) UAT - nightly or manually
4) Staging - manual approval
5) Production - manual approval
With the multi-stage pipelines the same flow seems to be very difficult to do. At least when it comes to making it as a single deployment pipeline. The first part is fine. We can have our build trigger the development deployment. But how can we hold back the release to the test environment until 0:30am, while still retain the ability to also release it manually? If I created a separate test environment pipeline, then it could work if it had no triggers, but a schedule.
Same with the UAT, as we also need the flexibility to manually run UAT deployments, then it would also need to go into its own pipeline. Releases to our staging and production environment we "gate" with manual approvals, which is fine.
While this technically could work, if we split the deployment pipeline into multiple pipelines it really gets difficult to manage "a release". Not to say that it kind of goes against the whole multi-stage pipeline principle if we create a separate pipeline per stage.
But with this being so easy to setup like this in the classic pipelines, then I cannot really imaging that other companies have not run into the same limitations. Is it just me who cannot see the light, or cannot this really not be done with multi-stage pipelines?
manually run UAT deployments
We could add Azure DevOps Multi-Stage Pipelines Approval Strategies in the yaml build.
Steps:
Open the tab Environments and click the button New environment-> Click the button approvals and checks-> My environment name is TEST.
Then use it in the yaml pipeline(just a sample):
trigger: none
pool:
vmImage: 'ubuntu-latest'
stages:
- stage: A
jobs:
- deployment: 'MyDeployment'
displayName: MyDeployment
environment: 'TEST'
- job: A1
steps:
- script: echo "##vso[task.setvariable variable=skipsubsequent;isOutput=true]false"
name: printvar
- stage: B
condition: and(succeeded(), ne(stageDependencies.A.A1.outputs['printvar.skipsubsequent'], 'true'))
dependsOn: A
jobs:
- job: B1
steps:
- script: echo hello from Stage B
Result:
We could also configure schedule Trigger and use them in the multi-stage pipelines.
Note: The schedule trigger and Approval Strategies are using in the stage level.
For scheduled jobs: you can use something like this in your YAML:
(Copied from Microsoft documentation)
schedules:
- cron: string # cron syntax defining a schedule
displayName: string # friendly name given to a specific schedule
branches:
include: [ string ] # which branches the schedule applies to
exclude: [ string ] # which branches to exclude from the schedule
always: boolean # whether to always run the pipeline or only if there have been source code changes since the last successful scheduled run. The default is false.
For manual jobs, you can use the Create Release button to create and deploy a release manually. Do note that sometimes this can create a conflict with the schedule. Also, to "hold back a release" put an approver on the release, and then when approving, defer the release:
noting that it's in UTC, and it defaults to tomorrow - you can change it to any time after now.
Azure scheduled pipeline runs for every push in master.What am I missing?Here's the yaml code:
schedules:
- cron: "0 23 * * *"
displayName: Nightly build
branches:
include:
- master
Apparently, the previous builds aren't failing.
I've already tried to remove this pipeline and create it again, but it keeps running for every push.
The scheduled runs are correct:
Ok, I think I figured this out.
Just added the following lines to the yml file and it no longer runs the scheduled pipeline on every push.
trigger: none
pr: none
Is there a way to customize the pipeline scheduling options in Azure to have it run only the second week of each month?
I know you can schedule it to run on individual days of the week, but I cannot figure out how I would do this on a monthly scale.
Can I do this if my pipeline was made as a classic/GUI based, and not
as a YAML pipeline?
In the classic pipelines, you can only set scheduled triggers for each week. As far as I know, you can not have it run only the second week of each month in the classic pipelines. However, you can set schedule triggers in yaml pipeline and use it to trigger your classic pipeline.
Here is the sample if you are going to use a YAML pipeline:
schedules:
- cron: "0 0 8-14 * *"
displayName: schedule
branches:
include:
- main
always: true
In this example:
The pipeline will be triggered from the 8th to the 14th of this
month. You need to update the date each month.
always: true means run even when there are no code changes.
Agree with iikkoo that if you want to run your pipeline by only using scheduled triggers, you must disable PR and continuous integration triggers by specifying pr: none and trigger: none in your YAML file.
You can add a build completion trigger in this yaml pipeline to trigger your classic pipeline:
Please find more detailed information about Configure schedules for pipelines in the document.
You can achive this by creating a scheduling trigger in your YAML config. Note tough, you must disabled PR and CI triggers to run your pipeline by using scheduled triggers.
You disable the trigger by setting pr: none and trigger: none. Then you define the schedule using cron syntax.
schedules:
- cron: "0 0 1/14 * *" # At 00:00 on every 14th day-of-month from 1 through 31.
displayName: Second week of each month
branches:
include:
- master
...
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/scheduled-triggers?view=azure-devops&tabs=yaml
https://github.com/atifaziz/NCrontab/wiki/Crontab-Expression
https://crontab.guru/
It doesn't seem to do so in the UI, but you can still trigger the build via an API call on your own schedule.
https://learn.microsoft.com/en-us/rest/api/azure/devops/build/builds/queue?view=azure-devops-rest-6.1
Had a scheduled job running every 30 minutes of Azure devops, it was running fine, last scheduled build I saw was on 2019-10-02ยท14:00, and since then there were no changes made to azure devops or even to the repository for which the pipeline builds.
Devops is not triggering any new builds, not sure about the issue and where should I look for issues.
Tried a manual run, thinking that it could invoke the sleeping process somewhere, but it did not help
trigger:
branches:
include:
- master
schedules:
- cron: "*/30 * * * *"
displayName: Daily half-hourly build
branches:
include:
- master
always: true
....
....
For your issue ,there could be two reasons why the schedule trigger stopped scheduling new builds.
The first one : Set the schedule trigger in UI
Scheduled triggers defined using the pipeline settings UI take precedence over YAML scheduled triggers.
If your YAML pipeline has both YAML scheduled triggers and UI defined scheduled triggers, only the UI defined scheduled triggers are run. To run the YAML defined scheduled triggers in your YAML pipeline, you must remove the scheduled triggers defined in the pipeline setting UI.
The second one: Each cron schedule has a limit
Each cron schedule has a maximum of 100 pipeline runs per week.From your description: running every 30 minutes, it should exceed the limit of 100 pipelines per week. If you need more, you can split your cron schedule into multiple cron schedules that each result in 100 or less pipeline runs per week.
You can check these two points to see if this is the cause of the issue.For details please refer to this official document.
I have a pipeline with 2 stages - a build/test stage, and a Teardown stage that cleans up external resources after the build/test stage. The teardown stage depends on some state information that gets generated in the build/test stage. I'm trying to use Azure hosted agents to do this. The problem is that the way I have it now, each stage deploys a new agent, so I lose the state I need for the teardown stage.
My pipeline looks something like this:
trigger:
- master
stages:
- stage: Build_stage
jobs:
- job: Build_job
pool:
vmImage: 'ubuntu-latest'
steps:
- task: InstallSomeTool#
- script: invoke someTool
- script: run some test
- stage: Teardown_stage
condition: always()
jobs:
- job: Teardown_job
pool:
vmImage: 'ubuntu-latest'
steps:
- script: invoke SomeTool --cleanup
The teardown stage fails because it's a brand new agent that knows nothing about the state created by the previous invoke someTool script.
I'm trying to do it this way because the Build stage creates some resources externally that I want to be cleaned up every time, even if the Build stage fails.
Is it possible to have an Azure hosted build agent persist between
pipeline stages?
No, you can't. The hosted agent are all randomly assigned by server. You could not use any script or command to specify a specific one.
Since you said that the Build_Stage will create some resources externally, so that you want to execute clean up to clean it.
In fact, for this, you can execute this clean up command as the last steps in Build_Stage. If this, whether using hosted or private agent will not affect what you want.