We have a project written in C that depends on several libraries as git submodules. We built an Azure Pipeline to build it, using multiple containers targeting multiple environments.
The challenge is that the build takes more time than we'd like, partly because of the fact that the submodules are being recompiled every time, even though they do not change.
What I'm looking for is a way to build the submodules only when needed, store them as artifacts, and have the main build consume them.
As far as I understand, I can set up a build for the submodule's repos which will poll for changes, but I want my product to depend on specific commits of the submodules - i.e. I'm not always taking the latest submodule version.
So I'm looking to trigger a submodule build whenever we switch to a new commit. Can this be achieved in Azure Pipelines? What would be the best way to manage the artifacts (e.g. store the commit ID as part of the artifact name)?
Related
Anybody know if it is possible to pass in a repo name / base the build on a dynamic repo name? This would allow us to share the same build definition across different branches, cutting down on definitions when creating a feature branch, etc.
When using a TFVC repo we would store the different releases in the same repo but different paths. We could reuse the same build definition across different releases/FB's by altering the source path such as $/product/$(release)/......
It appears Git likes to have the repo hard-coded into the build (hence the dropdown - no way to plug in a variable.
While the question is targeted to On-prem Azure DevOps, if it is possible in the hosted environment it would be helpful to know.
I recommend using YAML build templates. By default these check out "self" and are stored in the repo. That way they work on forks, branches etc. Each branch can contain tweaks to the build process as well.
With the 'old' UI based builds this isn't possible.
What you are looking for is actually two things:
templates - this allows you reuse definition accross different pipelines
triggers - this allows you to trigger pipeline when commit happens on different branches
Looks like Task Groups solved the need (mostly). I was hoping to have one build definition that could be shared across multiple branches; while this appears to be possible on the hosted model, on prem is different.
I am able to clone a build (or use templates) to have an entry point into the repo/branch to get the sources, then pass off the work to a common task group. If I need to modify the build process for multiple branches, just modify the task group.
TL;DR
How do I avoid rebuilding artifacts on master when a feature is merged without creating multiple pipelines per project? Where do I access the information about which branch was merged?
More Info
I run Jenkins to build many projects stored in two different VCSs (Gitlab, Bitbucket). Auto-discovery for both VCSs work and create multi-branch pipelines for every project/branch/PR containing a Jenkinsfile (Gitlab Branch Source Plugin, Bitbucket Branch Source Plugin).
Build artifacts get produced and stored on every build (e.g. docker images pushed to registry).
As I follow a feature branch workflow, these features get eventually merged into master, master will then be deployed in irregular intervals.
When doing the merge, there is an artifact already built and stored for this code(see appendix:1). It was built for the feature branch the code originated from (e.g. container mysuperapp:feat-add-better-things-3). I would like to take this artifact and promote it as the new master artifact (e.g. mysuperapp:master), avoiding a rebuild (and unit + integration testing everything).
But, merging a feature branch just kicks off a new build pipeline on branch master without any information about the merged branch (see appendix:2). This is correct behavior concerning master (new commit(s) where pushed) but prevents me from reacting to the merged branch (e.g. the aforementioned promoting or even just deleting unused artifacts). Is there any way to get the information, which branch was merged?
I am aware, that I can create a new pipeline listening for PR webhooks from my VCSs, running a pipeline to do the promotion and ignore builds on master completely. But this moves visibility of this process to a different pipeline and requires additional pipelines for projects, e.g. reducing the advantage of auto-discovery to 50% (have to create these merge pipelines for each project).
How can I keep the advantages of auto-discovery and visibility of executed steps while also executing something on a merge?
Ideas: Tag artifacts differently, but how (needs to be able to clean up correctly)? Parameterize pipelines and setup a single merge pipeline which re-triggers the pipeline 'push on master' with parameters of the merged branch. But can this be done without having to setup the webhooks for every project? Ask the VCSs via REST about which branch belonged to a commit?
Greets and thanks for the help you all! This may be a complicated one, but it would be so cool to get this to work. It's the last barrier for me to enable continuos delivery for a lot of projects!
Appendix:
1: I am also aware, that to have consistent builds, I have to enforce --ff-only merges. This question is not about the pitfalls of git but rather about the way to go with Jenkins.
2: Git provides me with the parent commits, I can easily find out, which commit was merged. But, especially using "Delete branch after merge", leaves me without the branch ref in git. Tagging my docker images with commits instead of branches leaves me with backtracking the last commit on each build to delete the old, obsolete build.
I have a TFVC project with about 4 branches. I need to somehow setup build and release pipelines that build an artifact for each branch. Because I will eventually need to repeat this process elsewhere, I would like to prevent having to duplicate the same build pipelines for each branch. I am able to configure a single build pipeline that works on whatever branch I need using a user-defined variable when the pipeline is kicked off, but now I need to enable continuous integration on the build.
My current build pipeline trigger configuration
I need this to work such that whenever someone checks into one of the TFVC branches, the build is kicked off and can correctly identify which branch was updated. From what I have found, this means that my initial idea of a user-defined variable is not going to work any longer. Is there a predefined pipeline variable that I can use to tell which branch was checked into, so that that branch is the one that is checked out and built? If not, is there some other way to do this in one pipeline, or do I ultimately need to duplicate this build pipeline for each branch?
Sorry, it's not available with TFVC in Azure DevOps/TFS build pipeline.
For CI trigger, you could select the version control paths you want to include and exclude. In most cases, you should make sure that these filters are consistent with your TFVC mappings on the Repository tab. It's not able to dynamically set workspace mapping path based on the branch which continuous integration trigger your build pipeline.
You could also take a look at Daniel's explanation in this question: When my TFS build is triggered by a branch-specific check-in, why doesn't it set that branch as its source?
TFVC relies on workspace mappings to know what to download. The workspace mappings can encompass multiple TFVC repos across different
team projects, multiple branches within a single repository......
As a result, there's no way for it to understand how to dynamically change workspace mappings to be for a specific branch.
Conclusion: You may need one build for every branch, duplicate the pipeline simply change the path filters in trigger and workspace mappings.
I am using VSTS build to run a CI build. This build definition is the same for all my git branches (master, develop, features, etc).
I am now trying to implement a deployment pipeline using VSTS release management. I plan to have two distinct release definitions. One for feature branches and one for the more important branches like develop and master. The release definition for feature branches would be more lightweight.
I think this is pretty basic and usual. In fact, that's pretty much what is documented in Microsoft's typical use case for Release Management.
For both of these pipelines, I want to configure them to start automatically using the 'Continuous Deployment' trigger. When I select this trigger, I must select an artifact source.
Unfortunately, an artifact source takes all artifacts (regardless of the branch) coming from a given build definition (my CI build). Since I use the same CI build definition for all my branches, it looks like I can't configure my two release pipelines in 'Continuous Deployment' and still use the same build definition as an artifact source.
Anybody knows how to share the same build definition for multiple release definitions but only start a release for a specific branch? Anybody knows of a way to filter by branch when we define an artifact source?
Configure branch specific release deployment
Go to release management in VSTS
Go to the definition of the release
Go to tab triggers
Add Continuous Deployment trigger
Here you can select a specific branch (for branch)
Availability of feature
This feature is available in VSTS
In the TFS on-premise version it was supposed to be available in version Server 2017.1 but it is still not available in version 2018.1
Currently there is no way in VSTS Release Management to do conditional deployment based on branch.
An alternate would be to create separate BDs for different branches and then configure them as artifact sources for the RDs.
This will also give users a clarity about the artifact from its name itself.
In the spirit of keeping my SVN trunk clean and ready for deployment, I've been utilizing the following source control model. For the impatient, the basic concept is that you create development branches to do actual development, and leave the trunk clean and ready for deployment, at any time (no junk in the trunk).
In addition to this, I am configuring TeamCity for continuous integration. Within TeamCity, I'd like to ensure that all development branches, as well as the deployment-ready branch (the trunk, in my case) build correctly and pass all unit tests.
This might be a stupid question, but not being overly familiar with TeamCity, should I create a new TeamCity project for each branch? The deployment-ready branch, in particular, has a few additional rules than the development branch. For example, releases should be saved in versioned directories on the file system (e.g., C:\Projects\MyProject\1.0.187..., C:\Projects\MyProject\1.0.188...) to enable easy access to the binaries, at any point in time. On the other hands, saving versioned copies of the assemblies in the development branches is not necessary and would waste hard disk space.
Within TeamCity, I'd prefer to see only a single project for each software project. In other words, if my company is working on X number of development projects, I'd prefer to see that project listed only once, not X * 2 (assuming each project has only two branches).
You only need to create a single project, but you will need multiple build configurations - 1 for each branch. As far as I know, you can't customize the artifact folder name on disk (it's an auto-increment number), however you can download all artifacts as a zip file in TeamCity 4.5 from the UI. There is also a scheduler included with TeamCity that lets you cleanup artifacts so they don't consume too much disk space.
TeamCity 2018.1.5
TeamCity doesn't support multi branches for SVN as for GIT - so I solved such problem with Configuration parameter - where I set active branch from which I need to build and after can easily switch to another branch by running a custom build or change that configuration parameter.
After need just configures triggers to start building from a specific branch:
So on project side you can see different branches
And easily switch between branches by running Custom Build and change branch there: