I have 5 repositories :
2 of Java,1 of R, 1 of Python and last of documents
I would like to create one master repository for this combinedly as folders and would like to create jenkins jobs for these folders in a repository.
Please let me know is this possible and if yes how I should approach.
Yes, this is possible.
You can create 5 different jenkins jobs referring to the same git repo. What will differ is the content of execution phase for each of them. Say, if you're using ant, this might look:
Java_1 job: ant -f java_1/build.xml all
Doc job: ant -f doc/build.xml all
Note: this approach implies an overhead in the amount of files: each job will copy and sync-up all the repo while needing only a subset of it.
Use [dsl plugin] (https://wiki.jenkins-ci.org/display/JENKINS/Job+DSL+Plugin) - just generate a job for control new jobs
Related
I am trying to solve a problem where I can't find the right documentation for the problem that I have.
At the moment, in my project, I am using Azure DevOps pipelines to build and deploy a simples code in a function. What I am trying to approach is to have multiple stages doing something concrete.
Example of the pipeline
Stage 1 - Validation of code (checktsyle, guidelines,...)
Stage 2 - Tests
Job1 : Unit tests
Job2 : Integration tests
Stage 3 - Deployment on the cloud
Stage 4 - Function tests against the deployment done on stage 3.
Problem
As you may know, when you do different stages, the pipelines run into different slaves, which means that will apply the git checkout in all of them. What I am trying to make is to avoid this checkouts and only make one single checkout on the first stage and use the checkout of the first stage for the rest (the code is the same..)
Do you have any clue what I am missing here? I know that I can do this process in a single stage with all steps/jobs inside but I want to split this in different stages to make sure that each stage has it own responsability.
Thanks in advance for your time.
It depends on where your agents run, if the agents are self-hosted, you can of course use a common location and avoid checking out the self repo. With hosted agents, I don't think you can do this using the stage concept in azure pipelines, stages have specific semantics which do not map to your desired outcome AFAIK. There are other ways to split the responsibilities without insisting on using azure pipeline stages. It depends on what you want to achieve with this splitting of responsibilities;
If you simply want to logically partition the pipeline there are alternatives e.g. templates which will allow you to separate the partitions into files which can be maintained separately, if that would satisfy your requirement for separation of responsibilities. They can even be separated into different repositories like in the example below, of course they can also reside in the same repository.
An example I use for caching and restoring dependencies for C++ projects using a common repository.
- checkout: DevOpsScripts
- template: up-restore.yml#DevOpsScripts
parameters:
CachePath: $(updepsCache)
CacheKeyPrefix: 'updeps | "$(Agent.OS)"'
DependenciesManifest: $(updepsPrefix)$(osSuffix).json
As stated in order to accomplish that you should have a custom agent on which you can have a folder to store the code for example C:\code. Then you can checkout the repository on this code path and disable checkout on the next stages.
You can disable checkout on the job inside your stage.
- job: DeployCode
displayName: Deploy code
steps:
- checkout: none
- script: echo deploying code
displayName: deploy code
In order to checkout on a specific directory on your self hosted agent you should:
- checkout: self
clean: true
path: C:\code
I am using Azure DevOps Server 2020 and I have a release pipeline which has around 21 copy file tasks in it to copy the output of multiple microservices to different target paths and this takes almost around 23 mins to complete the release pipeline.
I want to optimize the release pipeline and save some time and thus I am thinking of running all the copy task simultaneously.
Under the copy tasks in Control Options section, I see Run this task option is available where we do have the option to define custom conditions but I am not sure which custom conditions do I need to define exactly so that all my copy tasks gets executed parallelly.
Could anyone please let me know what custom conditions will allow all the copy task to get executed in one go?
Currently it is not possible to have tasks run in parallel. It has been raised as a suggestion here but the feature hasn't been implemented
How to run multiple Copy Files task in a Azure DevOps Release pipeline simultaneously with Custom Conditions?
Just as TheWinterCoder pointed, Currently it is not possible to have tasks run in parallel.
But, as a workaround, you could divide the replication task into several different jobs and make the jobs run in parallel:
This requires you to have multiple agents available in the local agent pool:
I'm having a hard time getting a grasp on this to be honest.
Right now my lab project is as follows:
PR to master -> Triggers Pre-Build Pipeline as condition to merge the code ->
On merge Infrastructure pipe runs only if any changes happen in my Infrastructure folder ->
On merge I want to run my deploy pipeline to deploy my web app to Azure.
The pipes in question do the things they ought to, i.e.
Pre build builds, publishes artifact, runs Unit tests, validates ARM templates.
Infra pipe deploys the necessary infra for my web app such as ResourceGroup, App plan, app service, key vault.
Deploy Pipe downloads the artifact produced in pre deploy and deploys to a stage slot and swaps it to production slot.
What I can't seem to get to work is the pipeline chaining through dependencies, if changes happen to both infra and web app code in master I want the infra pipe to run first and the deploy pipe only if it succeeds.
If I merge only app code I want only the deploy pipe to run regardless if the infra pipe ran or not.
If I merge only infra code I want only the infra pipe to run.
If I merge both app and infra code I want both infra and deploy pipe to run in specific order.
I feel this shouldn't be all that hard to accomplish, but I've spent way too much time trying to solve this to no avail, anyone able to help? :)
Edit:
Hey Sorry #HughLin-MSFT Been Trying to work around this a bit since we're trying to avoid running scripts left and right. :)
I saw you have Build Queuing planned in an upcoming release so for now I think we might have to wait for that.
If I were to merge my deploy and infra pipe, can I use:
trigger:
branches:
include:
- master
paths:
include:
- Infrastructure/*
At stage level and somehow skip a stage instead?
Seen multiple articles mention "Continue if skipped" but can't find any information on how to actually skip a stage.
For the first and second cases, you just need to set Path filters in Triggers, the pipeline only triggers when the file at the specified path is changed. Please refer to this.
For the third case, you can try to add two agent jobs in the infra pipe, add Trigger Azure DevOps Pipeline task to the second agent job to trigger the deploy pipe, and then set Only when all previous jobs have succeeded in Run this job drop-down box for job2. In addition, you need to add a powershell task before the Trigger Azure DevOps Pipeline task, and use a script to detect whether there is app code, run job2 if there is, and cancel job2 if not.
Update:
First you can create a new pipeline and create a variable:changedcode
Use Builds - Get rest api to get the commit , then get the changed code folder with Commits - Get Changes rest api.
Assign changed code folder name as value to changedcode variable.
Set custom conditions for the agent job. In the Infra job, if the changedcode variable value is Infra, run the Infra job. In the Infra job, use the Builds-Queue rest api or Trigger Azure DevOps Pipeline task to trigger the Infra pipeline. The same is true for Deploy job, the only difference is the custom condition expression.
Here is a sample structure in yaml:
jobs:
variables:
changedcode: ""
- job:
steps:
- powershell： |
#Get the changed code folder with rest api
- job: Infra
condition: containsValue($(changedcode), "Infra"))
- powershell： |
#queue Infra pipeline with rest api or Trigger Azure DevOps Pipeline task
- job: Deploy
condition: (containsValue($(changedcode), "deploy")),and ....
- powershell： |
#queue Deploy pipeline with rest api or Trigger Azure DevOps Pipeline task
Our current pipeline deploys a new instance of our app for every new branch created on azure repos, just like review apps on Heroku or Gitlab.
The creation part went smooth, but I'm not sure what to do with the orphaned resources and deployment once the branch is deleted (hopefully by an accepted pr).
Deleting them manually is not an option and there I can't find a trigger in the documentation for branch deletes.
The only option I can see right now is to create a scheduled job for the master branch(since it always exists) with a bash script that compares the list of deployed apps and existing branches and cleans up the resources.
Is it my only option, or is there another way without a fairly complex, all-access, destroy machine?
So did a little investigation dumping all enviroment vars to Notepad++ and using the compare plugin i realized that when a PR is accepted 2 env variables are different.
First of the initial variable "BUILD_REASON" during a push is set to "IndividualCI", but with the "BUILD_SOURCEBRANCH" set to "refs/heads/feature/******". When a pull request is initiated the "BUILD_REASON" changes to "pullrequest" and "BUILD_SOURCEBRANCH" to "refs/pull/***".
Finally when a PR is accepted the variables change to "BUILD_REASON" = "IndividualCI" and "BUILD_SOURCEBRANCH" = "refs/heads/master".
Once i figured out this i could create stage that have the following conditions:
- stage: CleanUp
displayName: 'CleanUp'
dependsOn: Test
condition: and(succeeded(), in(variables['Build.Reason'], 'IndividualCI'),in(variables['Build.SourceBranchName'], 'master'))
The above stage will be triggered when PR is accepted so to cleanup resources created during PR :-) havnt tested all the way but seems to do the job.
You can use a webhook in Azure DevOps to watch the pull request for updates. When the pull request status changes to completed, fire a script that deletes the resources used for the PR.
I have a build with multiple jobs, where they depend on each other's output. But I also have multiple agents, which gives me the following issue:
If Agent1 runs Job1, Agent2 runs Job2, and Job3 requires the output from both Job1 and Job2, I can't access the files from just one agent, since they are located on different machines.
How do I make my jobs able to download the output of other agents?
I looked for the workspace on MS Docs, but it doesn't describe how to handle this scenario.
To add more details on top of JukkaK's answer.
I looked for the workspace on MS Docs, but it doesn't describe how to
handle this scenario.
The workspace is something corresponding to agents. No sure which kind of agent do you use, but different agents have different OS instance, so the content under same path(workspace) in one agent should be quite different from that in another agent.
So workspace is not the approach for you needs.
How do I make my jobs able to download the output of other agents?
You can use Publish Artifacts+Download Artifacts combination to do what you need. See this:
You can place Publish build Artifacts task as the last task of agent job1 and job2. Then add a Download buil Artifacts as the first one of agent job3.
And make sure agent job3 depends on agent job1 and job2 like this:
In this way, the output from agent job1 and job2 can be installed in agent job3's machine for further usage. Hope it helps.
Pipeline artifacts in multi-stage pipelines would be a perfect match for this, if the current features available with multi-stage pipelines otherwise satisfy your needs.
https://learn.microsoft.com/en-us/azure/devops/pipelines/artifacts/pipeline-artifacts?view=azure-devops&tabs=yaml
If not, the best I can come up with is directing the the jobs to same agent by adding a capability to agent and adding a demand to the pool-assignment (or by creating your own pool). With Deployment group agents, adding tags is a handy way to direct jobs to a certain agent in deployment group, but haven't found anything similar on build agents.
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/demands?view=azure-devops&tabs=yaml